Marc's picture
Marc Pollefeys
Assistant Professor
Department of Computer Science
University of North Carolina at Chapel Hill

Tel: (919) 962 1845
Fax: (919) 962 1699
E-mail: marc@cs.unc.edu
Home
Research
Teaching
Publications
Contact
Personal


Research projects

Combining photo-consistency and silhouette constraints

optimizing photoconsistency while preserving exact silhouettes
We have developed an approach to compute a watertight 3D surface that trades off photo-consistency and shape priors while exactly enforcing silhouette constraints. A special graph is constructed for which each valid cut separating source from sink yields a watertight surface that exactly satisfies the silhouette constraints.  By translating costs for photo-consistency and shape priors to edge capacities, the minimal cut can be made to correspond to the optimal surface. 

S. Sinha, M. Pollefeys, Multi-view Reconstruction using Photo-consistency and Exact Silhouette Constraints: A Maximum-Flow Formulation, Proc. International Conference on Computer Vision, 2005. [pdf]

Camera network calibration and synchronization

 Camera network
We have developed a practical approach to automatically calibrate a network of cameras without the need for recording specific calibration data.  Our approach is robust and can deal with unsynchronized video streams and widely separated cameras.  The approach relies on the information provided by silhouettes of moving objects. 

S. Sinha, M. Pollefeys. Synchronization and Calibration of Camera Networks from Silhouettes, Proc. International Conference on Pattern Recognition 2004. [pdf]
S. Sinha, M. Pollefeys. Camera Network Calibration from Dynamic Silhouettes, Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, 2004. [pdf]

Calibration of omnidirectional sensors
linear wide-angle lens calibration
We have developed a simple linear approach that can calibrate wide-angle lenses from three views of a plane or when the camera performs pure rotation.  The key concept for this is the radial trifocal tensor.  We are now working on the use a non-parametric model of distortion, to deal with cata-dioptric cameras and some non-central cameras and to use four views in general configuration (using the radial quadrifocal tensor). 

S. Thirthala, M. Pollefeys, Multi-view geometry of 1D radial cameras and its application to omnidirectional camera calibration, Proc. International Conference on Computer Vision 2005. [pdf]
S. Thirthala, M. Pollefeys, The Radial Trifocal Tensor: A Tool for Calibrating Radial Disortion of Wide-Angle Cameras, IEEE Conf. on Computer Vision and Pattern Recognition, 2005. [pdf]
S. Thirthala, M. Pollefeys, Trifocal Tensor for Heterogeneous Cameras,  OMNIVIS Workshop, 2005. [pdf]


Articulated and Non-Rigid Motion Analysis

rotation axisrotational joint
We are working on a factorization-based approach for analysing articulated and non-rigid motion.  An important result that we have obtained is that the linear motion subspaces of articulated bodies intersect and that their intersection corresponds to the motion subspace of the joints. 

J. Yan, M. Pollefeys, Recovering Articulated Non-rigid Shapes, Motions and Kinematic Chains From Video, Proc. IV Conference on Articulated Motion and Deformable Objects, (to appear), 2006.
J. Yan, M. Pollefeys, Automatic Kinematic Chain Building from Feature Trajectories of Articulated Objects, Proc. CVPR’06 (IEEE Conf. on Computer Vision and Pattern Recognition), 2006. [pdf]
J. Yan, M. Pollefeys, A General Framework for Motion Segmentation: Independent, Articulated, Rigid, Non-rigid, Degenerate and Non-degenerate , European Conference on Computer Vision, 2006. [pdf]
J. Yan, M. Pollefeys, A factorization approach to articulated motion recovery, IEEE Conf. on Computer Vision and Pattern Recognition, 2005.[pdf]
J. Yan, M. Pollefeys, Articulated Motion Segmentation Using RANSACWith Priors, ICCV Workshop on Dynamical Vision 2005. [pdf]

This work was supported by the NSF award IIS - 0313047  Converting 2D Video to 3D with Applications to 3D-TV, Video Analysis and Compression

Real-time computer vision on commodity graphics hardware

stereo demo

Todays Graphic Processing Units (GPU) have tremendous processing power and are programmable.  Since the GPU is build to process images it is particularly well suited to perform some computer vision and image processing algorithms very efficiently.  We developed a real-time stereo algorithm that runs on the GPU and is several times faster than most CPU-based implementations. 
[more details]

R. Yang, M. Pollefeys, A Versatile Stereo Implementation on Commodity Graphics Hardware, Journal of Real-Time Imaging, 2005 (to appear). [pdf]
R. Yang, M. Pollefeys, H. Yang, G. Welch, A Unified Approach to Real-Time, Multi-Resolution, Multi-Baseline 2D View Synthesis and 3D Depth Estimation using Commodity Graphics Hardware, International Journal of Image and Graphics (to appear).  [pdf]
R. Yang, M. Pollefeys, and S. Li, Improved Real-Time Stereo on Commodity Graphics Hardware, Workshop on Real Time 3D Sensors and Their Use 2004 (in conjunction with CVPR) (to appear). [pdf]
R. Yang and M. Pollefeys. Multi-Resolution Real-Time Stereo on Commodity Graphics Hardware, Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pp. 211-218, 2003. [pdf]

Besides stereo, we have also implemented other computer vision algorithms on the GPU, such as the KLT feature tracker and the SIFT feature detector.  More details of our KLT implementation, including the code area available through the following link.

Sudipta N Sinha, Jan-Michael Frahm, Marc Pollefeys and Yakup Genc, "GPU-Based Video Feature Tracking and Matching", EDGE 2006, workshop on Edge Computing Using New Commodity Architectures, Chapel Hill, May 2006. [pdf]


3D-from-Video

3D medusa reconstruction    [mpg]

We have developed one of the first systems to automatically extract detailed 3D models from photo or video sequences.  Our first implementation was ready in 1997.  Since then we have continued to improve our image-to-3D pipeline, removing limitations and increasing accuracy and robustness. 

M. Pollefeys, L. Van Gool, M. Vergauwen, F. Verbiest, K. Cornelis, J. Tops, R. Koch, Visual modeling with a hand-held camera, International Journal of Computer Vision 59(3), 207-232, 2004. [IJCV][pdf]
M. Pollefeys and L. Van Gool. Visual modeling: from images to images, The Journal of Visualization and Computer Animation, 13: 199-209, 2002. [pdf]
M. Pollefeys and L. Van Gool. From Images to 3D Models, Communications of the ACM, July 2002/Vol. 45, No. 7, pp.50-55. [pdf]
M. Pollefeys, L. Van Gool, M. Vergauwen, K. Cornelis, F. Verbiest, J. Tops, Video-to-3D, Proc. Photogrammetric Computer Vision 2002 (ISPRS Commission III Symposium), International Archive of Photogrammetry and Remote Sensing. [pdf]
M. Pollefeys, R. Koch, M. Vergauwen, L. Van Gool. Automated reconstruction of 3D scenes from sequences of images, ISPRS Journal Of Photogrammetry And Remote Sensing (55)4 (2000) pp. 251-267. [pdf]
M. Pollefeys, R. Koch, M. Vergauwen and L. Van Gool, Hand-held acquisition of 3D models with a video camera, Proc. 3DIM'99 (Second International Conference on 3-D Digital Imaging and Modeling), IEEE Computer Society Press, pp.14-23, 1999. [pdf]
M. Pollefeys, R. Koch, M. Vergauwen and L. Van Gool, Metric 3D Surface Reconstruction from Uncalibrated Image Sequences,  Proc. SMILE Workshop (post-ECCV'98), LNCS 1506, pp.138-153, Springer-Verlag, 1998. [pdf]

Polar rectification and stereo geometry

begijnhof/beguinage Leuven  example polar rectification

Our polar rectification scheme allows to rectify image pairs for any relative camera motion (including when the epipole is in the image) and guarantees a minimal image size.  The price to pay is that co-linearity is in general not preserved, only epipolar lines are guaranteed to remain straight.  The iso-disparity surfaces corresponding to polar rectification have some interesting properties and reveal more fundamental properties of stereo configurations. 

M. Pollefeys, S. Sinha. Iso-disparity surfaces for general stereo configurations, T. Pajdla and J. Matas (Eds.), Computer Vision - ECCV 2004 (European Conference on Computer Vision), LNCS, Vol. 3023, pp. 509-520, Springer-Verlag, 2004. [pdf]
M. Pollefeys, R. Koch and L. Van Gool, A simple and efficient rectification method for general motion, Proc. International Conference on Computer Vision, pp.496-501, Corfu (Greece), 1999. [pdf]

Pan-Tilt-Zoom cameras and active camera networks

PTZ cube map  [camera]

Pan-tilt-zoom cameras allow to construct far more flexible vision systems than static cameras.  We are exploring the possibility to build active networks of pan-tit-zoom cameras to observe, track and reconstruct events in 3D.  As a first step we have developed an approach to automatically calibrate the pan-tilt-zoom camera over its full zoom range and to build very high-resolution panoramas. 

S. Sinha, M. Pollefeys, S.J. Kim, High-Resolution Multiscale Panoramic Mosaics from Pan-Tilt-Zoom Cameras, 4th Indian Conference on Computer Vision, Graphics and Image Processing (accepted).
S. Sinha, M. Pollefeys. Towards Calibrating a Pan-Tilt-Zoom Cameras Network, OMNIVIS 2004, ECCV Conference Workshop CD-rom proceedings, 2004. [pdf]

Multi-projector displays

multi-projector auto-calibration

Very high-resolution displays can be constructed by combining multiple projectors.  We have developed a fully automated method to calibrate a system consisting of casually alligned projectors.  Compared to previous approaches, we have removed the need for 4 or more fiducial points to establish a metric reference frame on the display. 

A. Raij, M. Pollefeys. Auto-Calibration of Multi-Projector Display Walls, International Conference on Pattern Recogniton 2004. [pdf]

[Wide Area Visuals (WAV) project]
[PixelFlex project]

Multi-view reconstruction from specular surfaces

    3D reconstruction of specular surfaces

By extending the photo-consistency criterion to include specular effects we are able to compute the 3D geometry of specular surfaces.  Our approach does not require to know the light geometry and can deal with multiple lightsources, including area lightsources. 

R. Yang, M. Pollefeys, and G. Welch. Dealing with Textureless Regions and Specular Highlight: A Progressive Space Carving Scheme Using a Novel Photo-consistency Measure, Proc. of the International Conference on Computer Vision, pp. 576-584, 2003. [pdf]

Photometric calibration

photometric aligment example

We have extended previous work on radiometric/photometric calibratioin and high-dynamic range (HDR) imaging to a moving camera.  This mainly consisted of developing a robust approach to estimate the brightness transfer function in the presence of occlusion (due to wrong correspondences and view-dependent effects).  One of the applications consist of dealing with auto-exposure while constructing a textured 3D model from a video sequence.   In the image shown above the image on the left was not radiometrically aligned, while the image on the right was.  We are now working on also dealing with vignetting. 

S.J. Kim, M. Pollefeys. Radiometric Self-Alignment of Image Sequences, Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, 2004. [pdf]

Self-calibration

We have developed algorithms to automatically calibrate the camera from video sequences.  This allows to obtain metric measurements with an uncalibrated camera.  Among other contributions we have proposed the first practical algorithm to deal with a varying focal length. 

M. Pollefeys, F. Verbiest, L. Van Gool, Surviving dominant planes in uncalibrated structure and motion recovery, A. Heyden, G. Sparr, M. Nielsen, P. Johansen (Eds.) Computer Vision - ECCV 2002, 7th European Conference on Computer Vision, Lecture Notes in Computer Science, Vol.2351, pp. 837-851. [pdf]
M. Pollefeys, L. Van Gool. Do ambiguous reconstructions always give ambiguous images?, Proc. International Conference on Computer Vision, 2001, pp.187-192. [pdf]
M. Pollefeys and L. Van Gool. Some Geometric Insight in Self-Calibration and Critical Motion Sequences, Technical Report Nr. KUL/ESAT/PSI/0001, PSI-ESAT, K.U.Leuven, Belgium, 2000. [pdf]
M. Pollefeys, R. Koch and L. Van Gool. Self-Calibration and Metric Reconstruction in spite of Varying and Unknown Internal Camera Parameters, International Journal of Computer Vision, 32(1), 7-25, 1999. [pdf]
M. Pollefeys and L. Van Gool, Stratified Self-Calibration with the Modulus Constraint, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol 21, No.8, pp.707-724, 1999. [pdf]
M. Pollefeys, R. Koch and L. Van Gool. Self-Calibration and Metric Reconstruction in spite of Varying and Unknown Internal Camera Parameters, Proc.ICCV'98 (international Conference on Computer Vision), pp.90-95, Bombay, 1998. joint winner of the David Marr prize (best paper). [pdf]

Image-based rendering

image-based rendering    [avi]

We have proposed the first unstructured lightfield rendering algorithm that can render directly from original images acquired with a hand-held camera.  Our approach combines view-dependent geometry and textures.  This allow to circumvent the difficult problem of having to reconstruct a consistent 3D representation of a recorded scene. 

M. Pollefeys and L. Van Gool. Visual modeling: from images to images, The Journal of Visualization and Computer Animation, 13: 199-209, 2002. [pdf]
R. Koch, B. Heigl, and M. Pollefeys, Image-Based Rendering from Uncalibrated Lightfields with Scalable Geometry, In R. Klette, T. Huang, G. Gimel'farb (Eds.), Multi-Image Analysis, Lecture Notes in Computer Science, Vol. 2032, pp.51-66, Springer-Verlag, 2001. [pdf]
B. Heigl, R. Koch, M. Pollefeys, J. Denzler and L. Van Gool, Plenoptic Modeling and Rendering from Image Sequences taken by Hand-held Camera, Proc. DAGM'99, pp.94-101, joint best paper. [pdf]
R. Koch, M. Pollefeys, B. Heigl, L. Van Gool and H. Niemann. Calibration of Hand-held Camera Sequences for Plenoptic Modeling, Proc. International Conference on Computer Vision,  pp.585-591, Corfu (Greece), 1999. [pdf]
R. Koch, B. Heigl, M. Pollefeys, L. Van Gool and H. Niemann, A Geometric Approach to Lightfield Calibration, Proc. CAIP99, LNCS 1689, Springer-Verlag, pp.596-603, 1999. [pdf]


Applications

Archaeology and cultural heritage

Archaeology has an important need for flexible 3D measurement techniques.  One of the reasons is that the process of excavation is destructive and it is therefore important to record as much information as possible.  More in general cultural heritage is often irreplacable and can be endangered.  Detailed 3D recording and archiving is thus very important. 

3D medusa
Medusa head decorating a fountain in the ancient town of Sagalassos, reconstructed from video.

       3D sagalassos
Landscape of Sagalassos combined with hypothetical reconstructions of monuments and buildings.

3D stratigraphy
Two different excavation layers recorded in 3D using a hand-held photo camera.

  3D pillar fit
Three fracture surfaces of pillars recorded to form a 3D puzzle.

(All the examples above were recorded in Sagalassos, Turkey.  We are grateful to Marc Waelkens for his collaboration, for providing access to the site and for helpful discussions.)

M. Pollefeys, L. Van Gool, M. Vergauwen, K. Cornelis, F. verbiest, J. Tops, Image-based 3D Recording for Archaeological Field Work, Computer Graphics and Applications (CGA), vol. 23, no. 3., May/June 2003, pp. 20-27. [pdf]
M. Pollefeys, L. Van Gool, M. Vergauwen, K. Cornelis, F. Verbiest, J. Tops, 3D Capture of Archaeology and Architecture with a Hand-Held Camera, Proc. of the ISPRS workshop on Vision Techniques for Digital Architectural and Archaeological Archives, July 1-3 2003, Ancona, Italy, The International Archive of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. XXXIV, Part 5/W12, pp. 262--267, 2003. [pdf]
M. Pollefeys, L. Van Gool, M. Vergauwen, K. Cornelis, F. Verbiest, J. Tops. Image-based 3D Acquisition of Archaeological Heritage and Applications, Proc. VAST2001 (Virtual Reality, Archaeology, and Cultural Heritage), ACM Press, pp.255-262. [pdf]
M. Pollefeys, L. Van Gool, I. Akkermans, D. De Becker, A Guided Tour to Virtual Sagalassos, Proc. VAST2001 (Virtual Reality, Archaeology, and Cultural Heritage), ACM press, pp.213-218. [pdf]
M. Pollefeys, M. Vergauwen, K. Cornelis, L. Van Gool, 3D Acquisition of Archaeological Heritage from Images, Proc. XVIII International Symposium of CIPA,    ISPRS International Archive of Photogrammetry, Remote Sensing and Spatial Information Sciences Vol. XXXIV-5/C7 /CIPA International Archive for Documentation of Cultural Heritage Vol. XVIII-2001,pp. 381--388, 2001. [pdf]
J. Schouteden, M. Pollefeys, M. Vergauwen, L. Van Gool, Image-based 3D acquisition tool for architectural conservation, Proc. XVIII International Symposium of CIPA, ISPRS International Archive of Photogrammetry, Remote Sensing and Spatial Information Sciences Vol. XXXIV-5/C7 /CIPA International Archive for Documentation of Cultural Heritage Vol. XVIII-2001, pp. 301--305, 2001. [pdf]
K. Nuyts, J.-P. Kruth, B. Lauwers, H. Neuckermans, M. Pollefeys, L. Qiongyan, J. Schouteden, P. Smars, K. Van Balen, L. Van Gool and M. Vergauwen. From a conservationist's point of view, Proc. Conference on Optical 3-D Measurement Techniques V, Gruen, Kahmen (Eds.), Vienna, October, 2001, pp. 179-186. [pdf]
M. Pollefeys, M. Proesmans, R. Koch, M. Vergauwen and L. Van Gool, Detailed model acquisition for virtual reality,  in J. Barcelo, M. Forte and D. Sanders,  "Virtual Reality in Archaeology", ArcheoPress (BAR International Series S 843) ISBN 1-84171-047-4, Oxford, pp.71-77, 2000. [pdf]
M. Pollefeys, R. Koch, M. Vergauwen and L. Van Gool, Virtualizing Archaeological Sites, Proceedings Virtual Systems and MultiMedia (VSMM'98).  [pdf]

Medecine

3D recorded surgery demo

Mock-up surgical operation recorded in 3D using multiple cameras.
[more info]

Urban 3D modeling

video frame      urban 3D model from video    [wmv]

Reconstruction of urban scene from drive-by video.  More details of the UrbanScape project can be found here

A. Akbarzadeh, J.-M. Frahm, P. Mordohai, B. Clipp, C. Engels, D. Gallup, P. Merrell, M. Phelps, S. Sinha, B. Talton, L. Wang, Q. Yang, H. Stewenius, R. Yang, G. Welch, H. Towles, D. Nister and M. Pollefeys, Towards Urban 3D Reconstruction From Video, Proc. 3DPVT'06 (Int. Symp. on 3D Data, Processing, Visualization and Transmission), 2006. [pdf]

Robot navigation

ROBUST mars rover prototype

The ROBUST system was a prototype for controling a micro-rover on Mars.  Our contribution consisted of designing and calibrating a stereo vision system and reconstructing the Martian surface from images. 

K. Landzettel, K. Arbter, B. Brunner, B.-M. Steinmetz, M. Pollefeys, M. Vergauwen, R. Moreas, F. Xu, L. Steinicke, B. Fontaine, A Micro-Rover Navigation and Control System for Autonomous Planetary Exploration, Advanced Robotics, Vol. 18, No.3, pp.285-314, 2004. [pdf]
M. Vergauwen, M. Pollefeys, L. Van Gool. A stereo vision system for support of planetary surface exploration, special issue Machine Vision and Applications, Vol. 14(1), 2003, pp.5-14. [pdf]
M. Vergauwen, M. Pollefeys, L. Van Gool, Calibration and 3D measurements from Martian Terrain Images, Proceedings of the 2001 IEEE International Conference on Robotics and Automation, pp 2153-2158, Seoul, Korea, IEEE Computer Society Press, 2001. [pdf]
B. Fontaine, D. Termont, L. Steinicke, M. Pollefeys, M. Vergauwen, R. Moreas, F. Xu, K. Landzettel, M. Steinmetz, B. Brunner, H. Michaelis, T. Behnke, R. Dequeker, P. Degezelle, R. Bertrand, G. Visentin, Autonomous Operations of a Micro-Rover for Geo-Science on Mars, Proc. 6th ESA workshop on Advanced Space Technologies for Robotics and Automation (ASTRA 2000), paper 3.6-3, 2000. [pdf]

3D-TV

C. Fehn, P. Kauff, M. Op de Beeck, F. Ernst, W. IJsselsteijn, M. Pollefeys, L. Van Gool, E. Ofek and I. Sexton, An Evolutionary and Optimised Approach on 3D-TV, In Proc. IBC '02, pages 357-365, Amsterdam, Netherlands, September 2002. [pdf]
M. op de Beeck, P. Wilinski, C. Fehn, P. Kauff, W. IJsselsteijn, M. Pollefeys, L. Van Gool, E. Ofek and I. Sexton, Towards an Optimized 3D Broadcast Chain, In Proc. SPIE ITCom '02, Boston, MA, USA, July - August 2002. [pdf]
A. Redert, M. Op de Beeck, C. Fehn, W. IJsselsteijn, M. Pollefeys, L. van Gool, E. Ofek, I. Sexton, P. Surman, ATTEST, Advanced Three-dimensional Television System Technologies, Proc. of First International Symposium on 3D Data Processing Visualization and Transmission. [pdf]

Entertainment

augmented sagalassos nymphaeum    [avi]

Once the motion (and calibration) of a camera has been estimated with respect to the background it becomes possible to augment a video sequence with virtual objects rigidly attached to the background.  The image above is a frame from a video sequence in which a virtual reconstruction of an ancient monumental fountain has been inserted.  This video was recorded by Axell Communications in Sagalassos, Turkey, for their film project

K. Cornelis, M. Pollefeys, M. Vergauwen, F. Verbiest, L. Van Gool, Tracking based structure and motion recovery for augmented video productions, Proceedings A.C.M. symposium on virtual reality software and technology - VRST 2001, pp. 17-24, November 14-17, 2001,  Banff, Canada. [pdf]
K. Cornelis, M. Pollefeys, M. Vergauwen and L. Van Gool, Augmented Reality from Uncalibrated Video Sequences, In M. Pollefeys, L. Van Gool, A. Zisserman, A. Fitzgibbon (Eds.), 3D Structure from Images - SMILE 2000, Lecture Notes in Computer Science, Vol. 2018, pp.144-160, Springer-Verlag, 2001. [pdf]



Current research sponsors

NSF  DARPA  Packard Foundation
Past sponsors
Siemens


Part of this research was carried out at the Visics/PSI/ESAT research group of the K.U.Leuven