1.1 Description:

The objective is to combine a pair of images, one taken without a flash and one with flash to get a image which represents the ambience the best. This algorithm is described by Petschnigg et al in this paper.
The flash image is visually incorrect in representing the ambient lighting although its SNR is high. The aim of the paper is to improve the SNR of the noflash image to get a good representation of the actual scene.

1.2 Bilateral Filter

The bilateral filter was introduced by Tomasi & Manduchi in this work. Instead of doing a blind gaussian smoothing for high SNR which results into blurring and loss of data at edges, bilateral filter allows us to keep the high gradient information while smoothing out the noise. This is done by considering the color similarity while blurring. But this results into a non-linear filter which when naively implemented is very inefficient. Although constant time bilateral filter algorithm exists, in this assignment I have used the naive implementation because the objective is not the bilateral filter itself but rather its application to do achieve something.. The bilateral filter can be expressed mathematically as follows:

A is the image, (in our case noflash image)
p is is the pixel,
gd and gr are gaussian in spatial and color domain respectively.

1. Bilateral filter formulation

1.2 Joint(Cross)-Bilateral Filter

The bilateral filter was introduced by Tomasi & Manduchi in this work. Instead of doing a blind gaussian smoothing for high SNR which results into blurring and loss of data at edges, bilateral filter allows us to keep the high gradient information while smoothing out the noise. This is done by considering the color similarity while blurring. But this results into a non-linear filter which when naively implemented is very inefficient. Although constant time bilateral filter algorithm exists, in this assignment I have used the naive implementation. The bilateral filter can be expressed mathematically as follows:

A is the image, (in our case noflash image)
F is the guiding image, (in our case flash image)
p is is the pixel,
gd and gr are gaussian in spatial and color domain respectively.

2. Joint bilateral filter formulation

1.3 Detail transfer

The joint bilateral filter scheme can increase SNR but it cannot transfer detail which are present in flash image. for this the following Scheme is used.

3. Detail transfer

1.4 Mask computation
The bilateral filter may under-blur or over-blur depending upon the region of image. to mitigate this a mask is computed.The mask computed has two parts, the flash shadows and specularities in the scene.

1.4.1 Flash shadows

The lash shadows are taken care of by thresholding the difference between the flash no-flash images. When the difference is small than a user assigned constant tau, Mask is 1 else 0. The program lets the user choose a suitable tau by aking for numeric input. If user is satisfied with tau, he should enter 'q' including the single quotes to finalise the tau.

4. Mshadow calculation

1.4.1 Flash specularities

Flash specularities are detected by using heuristic that specular surface should be bright in flash image and hence will saturate it. Hence the region above 95% of the maximum sensor value is assigned as secular region.
Mshadow and Mspec are cleaned by a combination of morphological operations to fill holes and remove isolated regions. Finally, these two mask are combined by taking a union of these to masks to get the final mask M.

1.4 White balancing

I have implemented the white balancing scheme. The function which does this is wbColor.m But there are some problems with scaling and the program results into worsening the white balance.

5.Algorithm overview

1.6 Results

As I have used the naive implementation of bilateral filter, the program is excruciatingly slow. So, it may even take 2 hrs to process a 6MP image. The time taken will strongly depend upon the neighbourhood size selected. All the parameters are set to whatever the authors have recommended in the paper. The images used are provided by the authors at their site. As these images are in .tif format, they are very large in size. The result for .tif images can be found zipped here. All the images on this website are in .jpg format which may result into loss of detail.