COMP 776 Spring 2010
Assignment 3: Image stitching, epipolar geometry estimation, triangulation
Due date: March 23 5PM
Part 1
The goal of this part of the assignment is to write a simple panorama
application for stitching together pairs of images. You will be working with
this pair:
Images courtesy of Kristen Grauman
Part 1 Directions
- Load both images, convert to double and to grayscale.
- Detect feature points in both images. You can use this corner detector code.
- Extract fixed-size patches around every keypoint in both images, and form descriptors simply by
"flattening" the pixel values in each patch to one-dimensional vectors. Experiment with different patch
sizes to see which one works the best.
- Compute distances between every descriptor in one image and every descriptor in the other image.
You can use this code for fast computation of Euclidean distance. Alternatively,
experiment with computing normalized correlation, or Euclidean distance after normalizing all descriptors
to have zero mean and unit standard deviation.
- Select putative matches based on the matrix of pairwise descriptor distances obtained above.
You can select all pairs whose descriptor distances are below a specified threshold,
or select the top few hundred descriptor pairs with the smallest pairwise distances.
- Run RANSAC to estimate (1) an affine transformation and (2) a homography mapping one image onto the other.
For each transformation, report the inlier number and the average residual
(squared distance between the point coordinates in one image and
the transformed coordinates of the matching point in the other image).
Also, display the locations of inlier matches in both images.
- Warp one image onto the other using the estimated transformation. To do this, you will need to learn
about maketform and imtransform functions.
- Create a new image big enough to hold the panorama and composite the two images into it.
You can composite by simply averaging the pixel values where the two images overlap.
Your result should look something like this:
You should create color panoramas by applying the same compositing step to each of the color channels separately (for estimating the transformation, it is sufficient to use grayscale images). Show panorama results both for affine transformation and homography.
- Find or take another pair of panorama images and demonstrate the output of your system on this pair.
Part 1 Bonus points
- Try to increase your inlier ratio by applying the ratio test from David Lowe's paper or other advanced techniques.
- Experiment with alternative feature detectors and/or descriptors -- for example, the blob detector that you wrote in the previous assignment.
- Experiment with advanced image blending techniques.
Part 2
The goals of this part of the assignment are fundamental matrix estimation and triangulation.
You will be using these two image pairs:
Download full size images, matching points,
camera matrices, and sample code.
Part 2 Directions
- Load the image pair and matching points file into MATLAB (see sample code
in the data file).
- Fit a fundamental matrix to the matching points. There is no need
to use RANSAC here since all the matches are correct. Use the sample code
provided to visualize the results. Try both the normalized and the
unnormalized algorithms. In each case, report your residual, which should be the mean
squared distance (in pixels) between points in the second image and the corresponding epipolar
lines.
- Now fit a fundamental matrix based on putative correspondences obtained
by your code from Part 1. Because the set of putative matches includes outliers,
you will need to use RANSAC. For this part, use only the normalized fitting
approach.
- Load the camera matrices for the two images (they are stored as 3x4 matrices
and can be loaded with the load command, i.e., P1 = load('house1_camera.txt');
Find the centers of the two cameras. Use linear least squares
to triangulate the position of each matching pair of points given the two cameras.
Display the two camera centers and scene matches in 3D. Also compute the residuals
between the observed 2D points and the projected 3D points in the two images.
- Apply triangulation to inlier matches that you obtained in step 3 above.
Tips and Details
- For RANSAC, a very simple implementation is sufficient. Use 3 and 4 matches,
respectively, to initialize affine transformations and homographies. You should
output a single transformation that gets the most inliers in the course of all
the iterations. For the various RANSAC parameters (number of iterations, inlier
threshold), play around with a few "reasonable" values and pick the ones
that work best. Refer to this lecture for details on RANSAC.
For randomly sampling matches, you can use the randperm function.
- For details of the fitting methods, you should review the lectures on
alignment and epipolar geometry.
- In MATLAB, the solution to a linear least squares system AX=B is given by
X = A\B;
- Homography fitting calls for homogeneous least squares. The solution
to the homogeneous least squares system AX=0 is obtained from the SVD of A
by the singular vector corresponding to the smallest singular value:
[U,S,V]=svd(A); X = V(:,end);
- For fundamental matrix estimation, don't forget to enforce the rank-2 constraint.
This can be done by taking the SVD of F, setting the smallest
singular value to zero, and recomputing F.
- Recall that the camera centers are
given by the null spaces of the matrices. They can be found by taking the SVD of the
camera matrix and taking the last column of V.
- The linear least squares method for triangulation is described in
this lecture. To simplify the implementation,
it is not necessary to use data normalization for this part of the assignment
(in my implementation, normalization made very little difference for this part).
- Plotting in 3D can be done using the plot3 command. Use the axis equal option to avoid automatic nonuniform scaling of the 3D space.