ImageFlow: Real-Time Image-Based Rendering

Principal Investigators: Henry Fuchs and John Poulton
Funding Agencies: National Science Foundation / Defense Advanced Projects Agency
Agency Number: MIP-9612643 / DABT63-96-6-0041

Abstract
We propose to design, build, and demonstrate a high performance graphics engine that uses depth images as the principal rendering primitive. A depth image is a two-dimensional image that includes depth, relative to the viewpoint, in addition to color and other appearance parameters. From a relatively small number of such depth images, each of which represents an arbitrarily complex scene from a single viewpoint, an image of similar complexity can be computed for any viewpoint within the neighborhood of the original viewpoints. This approach, which we call image-based rendering, promises the following dramatic advantages:

• The rendered images can be as realistic as photographs, because the input images can be photographs of real scenes (with depth information); alternatively, they can be synthesized off-line using state-of-the-art photo-realistic rendering techniques. Image-based rendering correctly warps each pixel of each input depth-image to the proper location in the output image when a new view is required, allowing for depth and occlusion as appropriate.
• An image-based rendered can be used as a post-processor for a conventional image generation system that generates the depth-images. By re-projecting the depth images to new viewpoints, the post-processor can provide viewpoint update rates far higher than the image-generation system's update rate. This decoupling between depth image generation and the final viewpoint-specific image generation allows multiple users to time-share the same depth image generator. It also allows users remote from the depth image generator to change viewpoints without a communications lag.

These new capabilities promise richer environments and greater interactivity in a wide range of applications such as flight and vehicle simulation, architectural previewing, and site analysis, particularly for natural outdoor environments. The complex environments required by such applications can be modeled more efficiently with depth images; for example, a single 512x512-sample depth image would require about 500,000 triangles in a triangle-mesh tessellation.

Our preliminary results in developing image-based rendering are very encouraging, but several problems remain: 1) accurately and automatically computing depth information for photographic images, 2) occlusion: the hiding of features behind closer surfaces within one depth image, and 3) managing the large of amount of data in the multiple depth images needed to provide all viewpoints required in a specific application. We propose a two-phase research plan to explore and resolve these problems, and then to build a real-time machine:

1. Construct a hardware platform for algorithm development and experimentation, based on our PixelFlow image generation system (on-line in summer 1996), by adding image capture and storage hardware. We will then develop and refine algorithms which will form the basis for a new image generation system,ImageFlow, specifically designed for image-based rendering.
2. Design, build, and demonstrate ImageFlow. ImageFlow will be based on a new microarchitecture, implemented using custom VLSI, specialized for rendering with depth images. To validate the ImageFlow design, we propose to use the prototype to demonstrate several demanding test applications, including visual simulation (using a combination of depth-image and polygonal models), remote visualization, image fusion of sensor data with geometric data, and medical visualization.

Image-based rendering may well represent the next major step in real-time image generation, providing unprecedented richness of detail in real-time imagery. The approach may also enable low-cost systems capable of rendering extremely realistic scenes.