CoE 3SK3: Project 3
Color Demosaicing for Digital Cameras with Linear Regression
Due: April 8, 2023
1 Problem Description
Photo sensors are in general sensitive to a wide range of visible light spectrum, unable to distinguish
between colours. To achieve colour imaging, modern digital cameras employ colour filter array
(CFA) allowing each pixel to sense only one of the three primary colours. A raw image captured by
this type of camera is a mosaic of colour pixels laid out in the so-called Bayer Pattern. Full colour
images we commonly use are digitally restored from such raw images using demosaicing algorithms.
In this project, your task is to implement a highly effective demosaicing algorithm based on linear
regression.
Most natural images are piecewise smooth with high correlations among adjacent pixels and
among colour channels. Thus, each missing component in a mosaic image is approximately a linear
combination of the surrounding known pixels. For instance, given a mosaic patch X as follows,
x1,1
x2,1
x3,1
x4,1
x5,1
x1,2
x2,2
x3,2
x4,2
x5,2
x1,3
x2,3
x3,3
x4,3
x5,3
x1,4
x2,4
x3,4
x4,4
x5,4
x1,5
x2,5
x3,5
x4,5
x5,5
g
≈
,
a1,1
a2,1
a3,1
a4,1
a5,1
a1,2
a2,2
a3,2
a4,2
a5,2
a1,3
a2,3
a3,3
a4,3
a5,3
a1,4
a2,4
a3,4
a4,4
a5,4
a1,5
a2,5
a3,5
a4,5
a5,5
A X
it is possible to approximate the missing green component g at the centre using the inner product
of some coefficient matrix A and X, i.e.,
g ≈ g˜ = ⟨A,X⟩ =
5∑
i=1
5∑
j=1
ai,jxi,j .
The optimal A can be learned from a large number of sample patches similar to X. Suppose that
we have n such mosaic patches X1, X2, . . . , Xn and their corresponding ground truth missing centre
green component g1, g2, . . . , gn, then the optimal A is a matrix that minimizes the approximation
error as follows,
min
A
n∑
k=0
(⟨A,Xk⟩ − gk)2.
1
This is a tractable linear least square problem. Please note that matrix A only predicts the green
component; for the missing blue component, a different matrix, say Z, should be used following
the same approach,
min
B
n∑
k=0
(⟨Z,Xk⟩ − bk)2.
where b1, b2, . . . , bn are the corresponding ground truth centre blue components. Additionally, A,Z
only apply to mosaic patches with the same pattern as X above. For each of the other three
different mosaic patterns as follows,
we need to use different coefficient matrices to predict the missing colour components. Therefore,
8 coefficient matrices are required in total.
2 Tasks and Requirements
To implement the linear regression based demosaicing algorithm, you can follow the steps below.
1. Simulate the 4 types of mosaic patches from full-colour patches.
2. Solve the linear least square problem for each case and get the 8 optimal coefficient matrices.
3. Apply the matrices on each patch of a simulated mosaic image to approximate the missing
colours.
4. Measure the RMSE between the demosaiced image and the ground truth.
5. Run your program on test raw mosaic data of our choice (to be released prior to deadline),
and record the process and output results in video, and submit this demo video.
You need to write a report detailing your implementation and experimental results. You should
compare the performance of the algorithm with the builtin demosaic(...) function in Matlab.
Bonus: Up to 25 percent bonus will be given to students whose algorithm can outperform our
benchmark algorithm.
2

欢迎咨询51作业君