This is the final report on Andy Wilson's project for COMP290-072: Computational Geometry. It assumes familiarity with the concept of viewpoint cells for massive model rendering. (See the Walkthrough group's paper, "A Framework for the Real-time Walkthrough of Massive Models", available online as a PDF file.)

Sections

Project Goals
Approach
Problems Encountered
Achievements and Contributions
Future Work
Homework 3 (CG Details)

Goals

My original intent was to automatically find paths through a massive model, generate a stream of images from that path, encoding it as an MPEG2 video stream, integrate the decoder into the UNC Walkthrough group's Massive Model Rendering (MMR) system, and then show the system running with video images.

This proved to be a bit too ambitious. In particular, I was unable to successfully adapt the decoder for random and bidirectional access to video streams. See the Achievements section for a discussion of the final product.

Approach Taken

I chose to integrate my work into the MMR system. In particular, I retained the textured depth mesh representation of distant geometry, but replaced the PNG-encoded images with frames taken from an MPEG stream. My goal was to extract and decode these images at runtime.

Step 1: Generate Paths

First, I computed a path through the portion of the model to be visualized. The MMR system uses a cell graph covering the walkways on each floor of the power plant model; I selected portions to treat as a single path by finding connected components of the cell-adjacency graph. An ordering was imposed on these cells, and in each cell six 512x512 images were acquired of distant geometry.

Step 2: MPEG2 Encoding

I used the MPEG Software Simulation Group's freely available MPEG2 codec to turn the captured images into video streams. The best image quality came from an encoding pattern of IBBPBBPBBPBB; I also experimented with exclusively using I-frames to simplify random access. The MPEG streams suffered a considerable loss of visual quality; for comparison, here is the same view in its original (JPEG, 72K) and MPEG-encoded (JPEG, 32K) versions. (The artifacts in the MPEG version are not due to the JPEG encoding.)

Step 3: Runtime Decoding

I attempted to modify the MSSG's MPEG2 decoder to allow decoding of individual frames from an MPEG stream. (I was unsuccessful; see below.) The idea was to decode frames both on demand (when the user moved quickly between cells) and speculatively (when the decoding process could get ahead of the user). The ability to play streams both forward and backward would have been quite useful here.

Setbacks and Problems

I spent a fairly large amount of time trying to cope with three particular problems. Perhaps a different video decoder would ameliorate these.

Path generation is NP-complete

The challenge of path generation is to find an order in which to visit cells so that the images acquired change as little as possible from cell to cell. The path must visit all cells, and a "distance" encoding the difference between images can be computed for any two cells. This, unfortunately, is the Traveling Salesman Problem, which is NP-complete in the strong sense.

In this case, I chose the difference function to be the Euclidean distance between the centers of the various cells. This doesn't directly represent image coherence, but causes the cell graph to satisfy the triangle inequality: for any three cells A, B, and C, the distance from A to C is no greater than the sum of the distances from A to B and B to C. Given this restriction, there is an approximation algorithm for the traveling salesman problem which runs in O(n log n) time and yields a solution with a cost no worse than twice the value of the optimal tour.

Random access to video streams is difficult

The MPEG2 standard is designed to start at the beginning of the stream, decode all the frames, and then stop. Frames are encoded in groups of pictures, only one of which may typically be decoded in isolation. In order to display the last frame in a group, it may be necessary to decode all the others for use as bases for prediction. Again, it may be the case that other video encoding schemes make this easier; I don't expect that fully random access will be at all simple, though. Of greater utility would be the ability to decode either forward or backward in the stream given a starting point.

Software for MPEG2 streams is fragile

The codec written by the MPEG Software and Simulation Group does a good job of encoding and decoding entire streams in one fell swoop. Adapting it to any other purpose proved difficult: aside from the (minor) hassle of converting the code from K&R C to ANSI C++, the decoder has a great deal of state which is initialized and manipulated at different stages of processing. To date, I haven't been able to find all of it. (To be fair, the reference implementation was never really designed for what I was trying to do with it.)

Achievements and Contributions

While some of the implementation details eluded me, I was able to accomplish most of my goals. This project has resulted in the following:

Utility for path generation given a cell graph - it works, although a better approximation algorithm (if one exists) that doesn't rely upon the triangle inequality would be quite helpful.

C++ version of reference MPEG2 decoder - Software written in C++ is much easier to integrate into the Walkthrough group's systems than that written in C. This isn't exactly a research result, but the conversion is sufficiently time-consuming that I'd rather keep the package around than duplicate the effort at a later date.

Framework for integrating video into MMR system - In the process of integrating MPEG video for textured depth meshes, I've found precisely those functions and points of entry which need to be changed to support different far-field representations. Again, this is not particularly revolutionary, but was a lot of work.

More open problems than I know what to do with (see Future Work, below) - I believe there's easily a dissertation's worth of work to be done in this area.

Future Work

Given more time to work, I've got some ideas I think are worth pursuing. Here are a few:

Do away with the textured depth meshes entirely. Why not put a video stream on the wall of a cell instead of a polygonal representation? This stream could capture views of the far field from different points within the cell; given the ability to navigate both forward and backward within a stream, this could be fairly efficient, and would probably be much less expensive than image warping.
Instead of sampling the path only at the centers of viewpoint cells, acquire a continuous stream of images, one per frame. It would be more difficult to prefetch images and geometry for distant cells, given that video decoding would take a significant chunk of bandwidth and CPU time; however, if specialized hardware is available to handle video streams, this becomes much more appealing.
Sample larger portions of the far field than a single 90-degree field of view. Perhaps acquire the entire sphere of view as a single image; perhaps panoramas, plus top and bottom images.
Encode depth along with the images. The Z-buffer is simply a grayscale image. This has a number of drawbacks, including occlusion artifacts and the necessity of simplifying the depth images. I also suspect that lossy video compression would produce worse artifacts when the depth buffer is restored than when flat pixels are displayed.

http://www.mpeg.org/index.html/MSSG/ http://drogo.cselt.stet.it/mpeg/ http://www.mpeg.org/index.html/video.html

Andy Wilson

Last modified: Sat Dec 12 14:05:53 EST 1998