ITR: Collaborative Research: Image-Based Rendering in Forensic Science, Education, and Historical Preservation

Principal Investigator: Anselmo Lastra
Funding Agency: National Science Foundation 
Agency Number: CCF-0205425

Abstract
Valuable information about the world around us, of both current and historical significance, is being lost because it is just too difficult to capture 3D models; today, this is primarily done by hand, making measurements with tape measures and entering the results manually. Furthermore, even when models are constructed, it is not known how to visualize them effectively. We propose to design and develop an end-to-end solution to the problem. To keep us focused, we are targeting two driving problems: forensic science and education. Two of the project collaborators are from the law enforcement community, the Federal Bureau of Investigation and the Armed Forces Institute of Pathology. One of our team members, from the New Orleans Museum of Art, is directing a major exhibition in which we propose to include a virtual environment of Monticello, President Jefferson's home. We will tackle the problems of acquiring, processing, representing, rendering and display of realworld data, focusing on the two target applications. Our goal is to design and build a complete end-to-end prototype system. The work includes: Acquisition. We have an accurate rangefinder built with prior NSF funding. We will mount it on a motorized cart and tackle the problem of capturing as complete a model as possible. This requires accurate tracking and a solution to the "next best view" problem in a practical sense, including resolution, reflectance, and importance of the surfaces. Processing. A very difficult problem is the registration of color with depth data, and of scans from multiple locations. We will develop algorithms to robustly register the data, and filter to reduce outliers and noise. Representation. We will develop algorithms to process the massive amounts of data (100 MB per scan, tens of scans per room) into both efficient geometric and imago-based representations. We will compare and contrast the representations in terms of quality and efficiency. Rendering. We will develop hierarchical and out-of-core algorithms to render the imagebased and geometric models at interactive rates. Display. This is perhaps the most difficult part of the problem because it involves the way humans perceive the environment. We will develop a hybrid projector/head-mounted-display system to provide stereo views to multiple viewers in a life-sized walking environment. The goal is to give the participants a sense of presence, of being in Mr. Jefferson's house. The measure of success of our work is in the benefit to the target applications. We will evaluate the results of our methods against the traditional methods that our collaborators have been using.