|(a) Original image. (b) Planes found by RANSAC in depthmap. (c) Planar class likelihood. (d) Final plane labeling overlaid on depthmap. Colors = planes, gray = no plane, and black = discarded. (e) Resulting 3D model with planes highlighted.|
In this work, we address the problem of piecewise-planar stereo reconstruction in the presence of non-planar surfaces. Piecewise planar stereo is an effective model for dealing with textureless and specular surfaces commonly found in man-made scenes. However most scenes contain elements which are not planar. Our method segments the scene into planar regions as well as non-planar regions which are modeled with a standard stereo depthmap. The segmentation is guided by multi-view photoconsistency as well as the learned appearance of planar surfaces. Appearance information is necessary to distinguish between truly flat surfaces such as walls and rooftops and nearly flat surfaces such as turf and pruned bushes.
David Gallup, Jan-Michael Frahm, Marc Pollefeys, "Piecewise Planar and Non-Planar Stereo for Urban Scene Reconstruction", CVPR 2010. [Oral] (5% of papers accepted as orals)
We have developed a novel multi-view depthmap fusion algorithm where the surface is represented by a heightmap. Whereas most other systems attempt to produce true 3D surfaces, our simplified model is a 2.5D representation. This model may seem to be a more natural fit to aerial and satellite data, but we have found it to also be a powerful representation for ground-level reconstructions. While it cannot represent overhanging surfaces, it has the advantage of producing purely vertical facades, and it also yields a continuous surface without holes. Compared to more general 3D reconstruction methods, our algorithm is more efficient, uses less memory, and produces more compact models at the expense of losing some detail. Our GPU implementation can compute a 200x200 heightmap from 64 depthmaps in just 92 milliseconds. The resulting 3D models are clean, complete, and compact.
David Gallup, Jan-Michael Frahm, Marc Pollefeys, "A Heightmap Model for Efficient 3D Reconstruction from Street-Level Video", 3DPVT 2010.
During my internship at Microsoft Research I worked with Cha Zhang on 3D face tracking using a depth camera. Hopefully there will be a paper soon!
In conjunction with NVIDIA, we developed a real-time binocular stereo system using CUDA.
|Left: Standard stereo. Note that the distance between depths increases quadratically. Right: Variable Baseline/Resolution Stereo. The distance between depths is held constant by increasing the baseline and selecting the appropriate resolution.|
We present a novel multi-baseline, multi-resolution stereo method, which varies the baseline and resolution proportionally to depth to obtain a reconstruction in which the depth resolution is constant. This is in contrast to traditional stereo, in which the error grows quadratically with depth. Many datasets, such as video captured from a moving camera, allow the baseline to be selected with significant flexibility. By selecting an appropriate baseline and resolution (realized using an image pyramid), our algorithm computes a depthmap which has these properties: 1) the depth accuracy is constant over the reconstructed volume, 2) the computational effort is spread evenly over the volume, 3) the angle of triangulation is held constant w.r.t. depth. Our approach achieves a given target accuracy with minimal computational effort, and is orders of magnitude faster than traditional stereo.
D. Gallup, J.-M. Frahm, P. Mordohai, M. Pollefeys, "Variable Baseline/Resolution Stereo", CVPR 2008. [Oral] (4% of papers accepted as orals)
Red = Wall 1
Top Left: An Input Image. Top Right: Surface segmentation. Bottom Left: Depthmap. Bottom Right: 3D model view.
Urban environments exhibit mostly planar surfaces. These surfaces, which are often imaged at oblique angles, pose a challenge for many window-based stereo matchers which suffer in the presence of slanted surfaces. We have developed a multi-view plane sweep stereo algorithm which correctly handles slanted surfaces and runs in real-time using the GPU. Our algorithm consists of (1) identifying the scene's p rinciple plane orientations, (2) estimating depth by performing a plane sweep for each direction, (3) combining the results of each sweep. Additionally, by incorporating priors on the locations of planes in the scene, we can increase the quality of the reconstruction and reduce computation time.
C. Zach, D. Gallup, J.-M. Frahm and M. Niethammer, "Fast Global Labeling for Real-Time Stereo Using Multiple Plane Sweeps", VMV 2008. [Oral]
D. Gallup, J.-M. Frahm, P. Mordohai, Q. Yang, M. Pollefeys, "Real-time Plane-sweeping Stereo with Multiple Sweeping Directions, CVPR 2007. [Poster]
I was part of the "UrbanScape" project which produced the first real-time system for 3D reconstruction from video.
During my time as an undergrad at the University of Utah, I worked on non-photorealistic rendering with Pete Shirley, embedded systems with John Regehr, and computer vision with Tom Hendersen.
|<< Home||Top||Publications >>|