The Challenge
We are currently working on what we call the "Office of the Future". This work is based on a unified application of computer vision and computer graphics in a system that combines and builds upon the notions of panoramic image display, tiled display systems, image-based modeling, and immersive environments. Our goals include a better everyday graphical display environment, and 3D tele-immersion capabilities that allow distant people to feel as though they are together in a shared office space.

The Approach
Our basic idea is to use computer vision techniques, in real time, to dynamically extract per-pixel depth and reflectance information for the visible surfaces in the Office, such as walls, furniture, objects and people (see image below). Using this depth information one could then display (project) images on the surfaces, render images of captured models of the surfaces, or interpret changes in the surfaces. We are primarily concentrating on the display and capture aspects. Our approach to the former is to designate everyday, irregular surfaces in a person’s office to be part of a spatially immersive display, and then project high resolution graphics and text onto those surfaces. Our approach to the latter is to capture, reconstruct, and then transmit dynamic image-based models over a network for display at a remote site.

There are several specific sub-projects under the umbrella of the Office of the Future. Our Tele-Immersion work (a collaboration with the University of Pennsylvania) involves data acquisition of a distant office using a miniature “sea” of seven cameras; modeling of that data; tracking of the local user’s eye positions, using UNC-developed HiBallTM Wide-Area tracking technology; rendering the dynamic model of the remote scene based on the tracked user’s positions; and a stereo presentation to display a remote collaborator, all dynamically in real-time. The approach aims to make distant collaborators feel as if they were in an adjoining room, and could see each other through a large “window” in the connecting wall. Our PixelFlex team is developing an automatically reconfigurable, wide-area, high-resolution display wall. Eight projectors, each with computer-controlled pan, tilt, and zoom adjustments, seamlessly display one image that is geometrically and photometrically correct. We will eventually incorporate this into the Office of the Future, to vastly increase the amount of the “adjoining room” that can be seen.

Our "Being There" team is exploring a new projector-based approach to visualizing re-creations of real or imagined places. Rather than using a planar display wall, we use display surfaces that closely approximate the virtual scene geometry in size and shape. The effect is a virtual environment that is both visually and spatially realistic, providing the user with a strong sense of immersion. We envision museum spaces with re-creations of famous places such as Monticello, President Thomas Jefferson’s home. Finally, our "Electronic Books" team (in collaboration with researchers at Brown University) is exploring the use of 3D capture and display to help train trauma surgeons. The idea is to create life-sized, high-fidelity, three-dimensional, dynamic, and annotated graphical reconstructions of surgery related to rare, but often fatal, trauma events. telecollaboration; tele-immersion; spatially-immersive displays; image-based modeling.