Electronic Books

A joint project with Brown University (support from NSF).

Tele-immersion will provide a dramatic new medium for groups of people remote from each other to work and share experiences together in an immersive 3D virtual environment, much as if they were co-located in a shared physical space. Immersive time machines, the subject of this proposal, will add an additional important dimension, that of being able to record experiences in which a viewer, immersed in the 3D reconstruction, can literally walk through the scene or move backward and forward in time. While there are many potential application areas for such novel technologies including maintenance and repair, design and virtual prototyping, paleontological and archaeological reconstruction, etc., we focus here on a socially important and technologically challenging driving application, teaching surgical management of difficult, potentially lethal, injuries.

Today, the pace of surgical innovations has increased dramatically, as have the societal demands for safe and effective practices. The mechanisms for training and retraining suffer from inflexible timing, extended time commitments, and limited content. Traditional videotaped instruction has also long been available to help surgeons learn new procedures, but these videos are universally seen as marginally effective at best for a number of reasons, such as the fixed point of view that is integral to the narration, lack of depth perception and interactivity, and missing information. In short, the experience of watching a video is not sufficiently close to being there and witnessing the actual procedure.

Our goal is to develop a new paradigm for teaching surgical procedures - immersive electronic books that in effect blend a "time machine" with 3D hypermedia. We plan to allow surgeons to witness and explore (in time and space) a past surgical procedure as if they were there, with the added benefit of integrated instruction from the original surgeon or another instructor, as well as incorporated 3D illustrations, annotations, and relevant medical metadata. The trainees should be able to freely and naturally walk around a life-sized, high-fidelity, 3D graphical reconstruction of the original time-varying events, pausing or stepping forward and backward in time to satisfy curiosity or allay confusion. Existing tools for time-varying 3D scene capture are crude at best, and the authoring tools, interaction metaphors, and "cinematic" techniques for this fundamentally new medium are unknown. We are bringing together experts in the respective disciplines and we are leveraging our prior work in tele-immersion to achieve our goals.

Top: Surgeon's view during emergency surgery involving blunt liver trauma. Bottom: Trainee's view within immersive time machine.  Note VCR-like controls and medical annotations.  The trainee's hands are captured in real-time and merged into the reconstructed event as white outlines (artist's impressions).

Recent Results:
3D+time reconstructions