COMP 776 Spring 2009

Assignment 1: Demosaicing

Due date: February 2, 5PM

The goal of the assignment is to get started with image processing in MATLAB by implementing a very simple demosaicing algorithm. You will be working with the following data:

  • original image, "mosaic" image

    NOTE: Before doing anything else with the image, you should convert it to double-precision format using the im2double function.

    The "mosaic" image was created by taking the original color image and keeping only one color component for each pixel, according to the standard Bayer pattern: 

    
R G . . .
G B
.
.
.
    So, once you read the "mosaic" image into a matrix, entry (1,1) will be the value of the "red" component for pixel (1,1), entry (1,2) will be "green", etc.

    Part 1: Linear Interpolation

    Implement a very simple linear interpolation approach to demosaicing: for each pixel, fill in the two missing channels by averaging either the four or the two neighboring known channel values:

    Avoid using loops! Instead, use the imfilter function (filter2 or conv2 work just as well).

    The above method, being very simple, does not work perfectly. You can see where it makes mistakes by computing a map of squared differences between the original and reconstructed values for each pixel, summed over the three color channels. Compute such a map and display it using the following commands:

    imagesc(I_diff); colormap(jet); colorbar % I_diff is the difference image

    In addition, compute and display the average and maximum of the per-pixel squared differences for the image. Finally, show a close-up of some patch of the reconstructed image where the artifacts are particularly apparent and explain the cause of these artifacts.

    Part 2: The Freeman Method

    Bill Freeman proposed an improvement of the simple bilinear interpolation approach. Since the G channel is sampled at a higher rate than the R and B channels, one might expect interpolation to work better for G values. Then it would make sense to use the interpolated G channel to modify the interpolated R and B channels. The improved algorithm begins with linear interpolation applied separately to each channel, just as you have already done above. The estimated G channel is not changed, but R and B channels are modified as follows. First, compute the difference images R-G and B-G between the respective interpolated channels. Mosaicing artifacts tend to show up as small "splotches" in these images. To eliminate the "splotches", apply median filtering (medfilt2 command in MATLAB) to the R-G and B-G images. Finally, create the modified R and B channels by adding the G channel to the respective difference images.

    Implement the above algorithm and visualize the quality of the results in the same way as for Part 1 by displaying the per-pixel difference image and computing average and maximum differences. Compare the output to that of Part 1. Are there visible improvements (especially in the close-up patch selected in Part 1)?

    Hint: Implementing this method should take you about two lines of code.

    Part 3: MATLAB's Demosaic Function

    Compare the output of the above two methods to the output of MATLAB's demosaic function. To find out how to use this function, consult MATLAB's documentation. Visualize the results as above and compare them to the output of Parts 1 and 2. As in Part 1 and 2, display average and maximum per-pixel differences.

    Part 4: Additional Images

    Take two additional images of your own choosing, create fake "mosaics" by removing the two color channels according to the Bayer pattern, and apply all three demosaicing algorithms to them. Try to find difficult images that would "break" at least the simpler methods. Explain what makes the images difficult and what accounts for the artifacts or errors you observe.

    Hint: You may see some interesting results if you apply demosaicing to a grayscale image (i.e., an image all of whose color channels are the same).

    For Bonus Points

    Propose (or look up online) additional improvements to make demosaicing work better. Implement these improvements and evaluate the results.

    Turning in the Assignment

    Your submission should consist of the following:
    • All your MATLAB code and additional images in a single zip file. The filename should be firstname_lastname.zip.
    • A brief report in a single PDF file showing all your results and answering the above questions. The filename should be firstname_lastname.pdf.
    Email me the above two files as attachments by 5PM, Tuesday, February 2nd. The email subject should be COMP 776 Assignment 1. Points will be taken off for not following directions and for late submission.

    Useful MATLAB Functions

    • imread, imwrite
    • imfilter, filter2, conv2
    • imshow, imagesc
    • medfilt2
    • max, mean