CoE 3SK3: Project 3

Color Demosaicing for Digital Cameras with Linear Regression

Due: April 8, 2023

1 Problem Description

Photo sensors are in general sensitive to a wide range of visible light spectrum, unable to distinguish between colours. To achieve colour imaging, modern digital cameras employ colour filter array (CFA) allowing each pixel to sense only one of the three primary colours. A raw image captured by this type of camera is a mosaic of colour pixels laid out in the so-called Bayer Pattern. Full colour images we commonly use are digitally restored from such raw images using demosaicing algorithms. In this project, your task is to implement a highly effective demosaicing algorithm based on linear regression.

Most natural images are piecewise smooth with high correlations among adjacent pixels and among colour channels. Thus, each missing component in a mosaic image is approximately a linear combination of the surrounding known pixels. For instance, given a mosaic patch X as follows,

AXg

      a1,1 a1,2 a1,3 a1,4 a1,5 a2,1 a2,2 a2,3 a2,4 a2,5 a3,1 a3,2 a3,3 a3,4 a3,5 a4,1 a4,2 a4,3 a4,4 a4,5 , a5,1 a5,2 a5,3 a5,4 a5,5

x1,1 x1,2 x1,3 x1,4 x1,5

x2,1 x2,2 x2,3 x2,4 x2,5

x3,1 x3,2 x3,3 x3,4 x3,5 ≈ x4,1 x4,2 x4,3 x4,4 x4,5

x5,1 x5,2 x5,3 x5,4 x5,5

it is possible to approximate the missing green component g at the centre using the inner product of some coefficient matrix A and X, i.e.,

55

g ≈ g ̃ = ⟨A,X⟩ = XXai,jxi,j. i=1 j=1

The optimal A can be learned from a large number of sample patches similar to X. Suppose that we have n such mosaic patches X1, X2, . . . , Xn and their corresponding ground truth missing centre green component g1, g2, . . . , gn, then the optimal A is a matrix that minimizes the approximation error as follows,

n

minX(⟨A,Xk⟩−gk)2.

A

k=0

1

This is a tractable linear least square problem. Please note that matrix A only predicts the green component; for the missing blue component, a different matrix, say Z, should be used following the same approach,

n

minX(⟨Z,Xk⟩−bk)2.

B

k=0

where b1, b2, . . . , bn are the corresponding ground truth centre blue components. Additionally, A, Z only apply to mosaic patches with the same pattern as X above. For each of the other three different mosaic patterns as follows,

we need to use different coefficient matrices to predict the missing colour components. Therefore, 8 coefficient matrices are required in total.

2 Tasks and Requirements

To implement the linear regression based demosaicing algorithm, you can follow the steps below.

1. Simulate the 4 types of mosaic patches from full-colour patches.

2. Solve the linear least square problem for each case and get the 8 optimal coefficient matrices.

3. Apply the matrices on each patch of a simulated mosaic image to approximate the missing colours.

4. Measure the RMSE between the demosaiced image and the ground truth.

5. Run your program on test raw mosaic data of our choice (to be released prior to deadline), and record the process and output results in video, and submit this demo video.

You need to write a report detailing your implementation and experimental results. You should compare the performance of the algorithm with the builtin demosaic(...) function in Matlab.

Bonus: Up to 25 percent bonus will be given to students whose algorithm can outperform our benchmark algorithm.

2