An Image-Based Rendering System
Leonard McMillan and Gary Bishop
Image-based rendering systems are a powerful new approach for generating real-time photorealistic computer graphics. They can provide convincing animations without an explicit geometric representation. The "plenoptic function" of Adelson and Bergen is a parameterized function for describing everything that is visible from a given point in space. We use this plenoptic function to provide a concise problem statement for image-based rendering paradigms, such as morphing and view interpolation. We also present our own image-based rendering system based on sampling, reconstructing, and resampling the plenoptic function. In addition, we introduce a novel visible surface algorithm and a geometric invariant for cylindrical projections that is equivalent the epipolar constraint defined for planar projections.
Figure 1. A panoramic image constructed by registering and reprojecting 36 images that were acquired by an uncalibrated video camera as it was panned through 360 degrees.
Figure 2. A second panoramic image generated by registering 31 images acquired from a position approximately 60 inches from the first panoramic image.
Figure 3. A display of the cylindrical epipolar curves relating the two panoramic views. The two convergence points, called epipoles, correspond to the virtual image of the second camera as seen from the first, and the vanishing point along the optical axis of the second camera.
Figure 4.Arbitrary reprojections of the first panoramic image. Each image corresponds to a change in viewing position. These images were generated by an parameterized image-warp that directly maps pixels from the source image to their position in the desired view. These reprojections are accomplished without any explicit underlying geometric scene description. Notice how both occlusion and perspective effects are modeled correctly.
Figure 5. Just for fun here is a short MPEG animation of 16 cylinders (635K). Notice how these cylinders have a considerably larger vertical field-of-view (~110 degrees) than the ones shown previously. That's because these days we're acquiring images using a fisheye lens. This requires adding a lens distortion model to the registration process described in the paper. Surprisingly, a fisheye lens is easier (only a single parameter distortion model is needed) to undistort than alternative wide-angle lenses. More details will be available soon. You might also notice that our motorized translation platform is visible along the bottom of these images. We'll add more about that in the near future, also.
Figure 6. These frames are from a video that was shown at SIGGRAPH'95. If you'd like to actually see the animation for yourself an MPEG version is available. It's a little over 10 Mbytes long, runs about 5 minutes, and was digitized at 5 fps. We owe a lot (of praise and headaches) to two undergrads, Steve Wendel and Arthur Gregory, who spent a large chunk of their summer working on this video.
For more about visibility. For more about warping.