COMP 776 Spring 2011
Assignment 3: Image stitching
Due date: March 17 11:59PM
The goal of this assignment is to write a simple panorama
application for stitching together pairs of images. You will be working with
this pair:
Images courtesy of Kristen Grauman
Directions
- Load both images, convert to double and to grayscale.
- Detect feature points in both images. You can use this corner detector code.
- Extract fixed-size patches around every keypoint in both images, and form descriptors simply by
"flattening" the pixel values in each patch to one-dimensional vectors. Experiment with different patch
sizes to see which one works the best.
- Compute distances between every descriptor in one image and every descriptor in the other image.
You can use this code for fast computation of Euclidean distance. Alternatively,
experiment with computing normalized correlation, or Euclidean distance after normalizing all descriptors
to have zero mean and unit standard deviation.
- Select putative matches based on the matrix of pairwise descriptor distances obtained above.
You can select all pairs whose descriptor distances are below a specified threshold,
or select the top few hundred descriptor pairs with the smallest pairwise distances.
- Run RANSAC to estimate (1) an affine transformation and (2) a homography mapping one image onto the other.
For each transformation, report the inlier number and the average residual
(squared distance between the point coordinates in one image and
the transformed coordinates of the matching point in the other image).
Also, display the locations of inlier matches in both images.
- Warp one image onto the other using the estimated transformation. To do this, you will need to learn
about maketform and imtransform functions.
- Create a new image big enough to hold the panorama and composite the two images into it.
You can composite by simply averaging the pixel values where the two images overlap.
Your result should look something like this (but hopefully with a more precise alignment):
You should create color panoramas by applying the same compositing step to each of the color channels separately (for estimating the transformation, it is sufficient to use grayscale images). Show panorama results both for affine transformation and homography.
- Find or take two more pairs of overlapping images and demonstrate the output of your system on them.
Tips and Details
- For RANSAC, a very simple implementation is sufficient. Use 3 and 4 matches,
respectively, to initialize affine transformations and homographies. You should
output a single transformation that gets the most inliers in the course of all
the iterations. For the various RANSAC parameters (number of iterations, inlier
threshold), play around with a few "reasonable" values and pick the ones
that work best. Refer to this lecture for details on RANSAC.
For randomly sampling matches, you can use the randperm or randsample functions.
- For details of affine and homography fitting, you should review
this lecture.
- In MATLAB, the solution to a nonhomogeneous linear least squares system AX=B is given by
X = A\B;
- Homography fitting calls for homogeneous least squares. The solution
to the homogeneous least squares system AX=0 is obtained from the SVD of A
by the singular vector corresponding to the smallest singular value:
[U,S,V]=svd(A); X = V(:,end);
For bonus points
- Experiment with registering very "difficult" image pairs -- for instance, try to
find a modern and a historical view of the same location to mimic the kinds of
composites found here.
Or try to find two views of the same location taken at different times of day, different
times of year, etc. Another idea is to try to register images with a lot of repetition,
or images separated by an extreme transformation (large rotation, scaling, etc.).
To make stitching work for such challenging situations, you may need to experiment with
alternative feature detectors and/or descriptors, as well as feature space outlier
rejection techniques such as Lowe's ratio test.
- Try to stitch a panorama composed of more than a pair of images.
- Learn about and experiment with image blending techniques.