Refining structure and motion

Once the structure and motion has been obtained for the whole sequence, it is recommended to refine it through a global minimization step. A maximum likelihood estimation can be obtained through bundle adjustment [158,137]. The goal is to find the parameters of the camera view ${\bf P}_k$ and the 3D points ${\tt M}_i$ for which the mean squared distances between the observed image points ${\tt m}_{ki}$ and the reprojected image points ${\bf P}_k({\tt M}_i)$ is minimized. The camera projection model should also take radial distortion into account. For $m$ views and $n$ points the following criterion should be minimized:

$$\min_{{\bf P}_k,{\tt M}_i} \sum_{k=1}^{m} \sum_{i=1}^{n} D({\tt m}_{ki},{\bf P}_k({\tt M}_i))^2$$ (E8)

If the image error is zero-mean Gaussian then bundle adjustment is the Maximum Likelihood Estimator. Although it can be expressed very simply, this minimization problem is huge. For a typical sequence of 20 views and 2000 points, a minimization problem in more than 6000 variables has to be solved. A straight-forward computation is obviously not feasible. However, the special structure of the problem can be exploited to solve the problem much more efficiently. More details on this approach is given in Appendix A.

To conclude this section an overview of the algorithm to retrieve structure and motion from a sequence of images is given. Two views are selected and a projective frame is initialized. The matched corners are reconstructed to obtain an initial structure. The other views in the sequence are related to the existing structure by matching them with their predecessor. Once this is done the structure is updated. Existing points are refined and new points are initialized. When the camera motion implies that points continuously disappear and reappear it is interesting to relate an image to other close views. Once the structure and motion has been retrieved for the whole sequence, the results can be refined through bundle adjustment. The whole procedure is resumed in Table 5.1.

Table 5.1: Overview of the projective structure and motion algorithm.