Soumyadip (Roni) Sengupta

I am an Assistant Professor of Computer Science at the University of North Carolina at Chapel Hill.

Previously I was a Postdoctoral Research Associate in Computer Science & Engineering at University of Washington, working with Prof. Steve Seitz, Prof. Brian Curless and Prof. Ira Kemelmacher-Shlizerman in the UW Reality Lab and GRAIL (2019-2022). I completed my Ph.D. (2013 - 2019) from University of Maryland - College Park (UMD), advised by Prof. David Jacobs and my undergraduate degree (2009-2013) in Electronics and Tele-Communication Engineering from Jadavpur University, Kolkata, India. I also had the pleasure to spend time and work with many amazing researchers from NVIDIA Research, Snapchat Research, The Weizmann Institute of Science (Israel), and TU Dortmund (Germany).

Email: ronisen at

Website: Google Scholar
Resume/CV: CV
Office: Sitterson Hall 255, University of North Carolina at Chapel Hill

Research Interest

My research lies at the intersection of Computer Vision and Computer Graphics, mainly centered around 3D Vision and Computational Photography. I am particularly interested in solving Inverse Graphics problems where the goal is to decompose images into its' intrinsic components (e.g. geometry, material reflectance, lighting, alpha matte etc.). I solve Inverse Graphics problems to create next-generation video communication and content creation by democratizing high-quality video production and 3D capture.

Research Group | Teaching | Publications

Research Group

Graduate Advisees

Undergraduate Advisees

If you are a Junior or a Senior UG student at UNC interested in pursuing research in my group, reach out to me via email.

Past Advisees


Selected Publications

MVPSNet: Fast Generalizable Multi-view Photometric Stereo
Dongxu Zhao, Daniel Lichy, Pierre-Nicolas Perrin, Jan-Michael Frahm, Soumyadip Sengupta
ICCV 2023
[Paper][Project Page][Code (Coming Soon)]

We propose generalized approach to multi-view photometric stereo that is significantly better than only multi-view stereo. It produces same reconstruction quality while being 400x faster than per-scene optimization techniques.

Measured Albedo in the Wild: Filling the Gap in Intrinsics Evaluation
Jiaye Wu, Sanjoy Chowdhury, Hariharmano Shanmugaraja, David Jacobs, Soumyadip Sengupta
ICCP (International Conference on Computational Photography) 2023
[Paper][Project Page][Code & Dataset(Coming Soon)]

Existing benchmark (WHDR metric on IIW) for evaluating Intrinsic Image decomposition in the wild are often incomplete as it relies on pair-wise relative human judgements. In order to comprehensively evaluate albedo, we collect a new dataset, Measured Albedo in the Wild (MAW), and propose three new metrics that complement WHDR: intensity, chromaticity and texture metrics. We show that SOTA inverse rendering and intrinsic image decomposition algorithms overfit on WHDR metric and our proposed MAW benchmark can properly evaluate these algorithms that match their visual quality.

My3DGen: Building Lightweight Personalized 3D Generative Model
Luchao Qi, Jiaye Wu, Shengze Wang, Soumyadip Sengupta
arXiv 2023
[Paper][Project Page][Code (Coming Soon)]

We propose a parameter efficient approach for building personalized 3D generative priors by updating only 0.6 million parameters compared to a full finetuning of 31 million parameters. Personalized 3D generative priors can reconstruct any test image and synthesize novel 3D images of an individual without any test-time optimization or finetuning.

Bringing Telepresence to Every Desk
Shengze Wang, Ziheng Wang, Ryan Schmelzle, Liujie Zheng, Youngjoong Kwon, Soumyadip Sengupta, Henry Fuchs
arXiv 2023
[Paper][Project Page]

We introduce a novel system that can render high-quality novel views from 4 RGBD camera focused on a tele-conferencing setup. We introduce a novel multiview point cloud rendering algorithm.

Motion Matters: Neural Motion Transfer for Better Camera Physiological Sensing
Akshay Paruchuri, Xin Liu, Yulu Pan, Shwetak Patel, Daniel McDuff*, Soumyadip Sengupta*
arXiv 2023
[Paper][Project Page][Code]

Neural Motion Transfer serves as an effective data augmentation technique for PPG signal estimation from facial videos. We devise the best strategy to augment publicly available datasets with motion augmentation, improving up to 75% over SOTA techniques on five benchmark datasets.

Universal Guidance for Diffusion Models
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, Tom Goldstein
arXiv 2023
[Paper][Project Page + Code]

Enables controlling diffusion models by arbitrary guidance modalities without the need to retrain any use-specific components.

A Surface-normal Based Neural Framework for Colonoscopy Reconstruction
Shuxian Wang, Yubo Zhang, Sarah K McGill, Julian G Rosenman, Jan-Michael Frahm, Soumyadip Sengupta, Stephen M Pizer
International Conference on Image Processing and Machine Intelligence (IPMI 2023)

Using SLAM + near-field Photometric Stereo for 3D colon reconstruction from colonoscopy videos.

Towards Unified Keyframe Propagation Models
Patrick Esser, Peter Michael, Soumyadip Sengupta
(CVPRW 2022 - AI for Content Creation Workshop)
[Paper][Project Page][Code]

We present a two-stream approach for video in-painting, where high-frequency features interact locally and low-frequency features interact globally via attention mechanism.

Real-Time Light-Weight Near-Field Photometric Stereo
Daniel Lichy, Soumyadip Sengupta, David Jacobs
(CVPR 2022)
[Paper][Project Page][Code]

Near-field Photometric Stereo technique is useful for 3D imaging of large objects. We capture multiple images of an object by moving a flashlight and reconstruct the 3D mesh. Our method is significnatly faster and memory-efficient while producing better quality than SOTA methods.

Robust High-Resolution Video Matting with Temporal Guidance
Peter Lin, Linjie Yang, Imran Saleemi, Soumyadip Sengupta
(WACV 2022)
[Paper][Project Page][Code]

Background Removal a.k.a Alpha matting on videos by exploiting temporal information with a recurrent architecture. Does not require capturing background image or manual annotations.

A Light Stage on Every Desk
Soumyadip Sengupta, Brian Curless, Ira Kemelmacher-Shlizerman, Steve Seitz
(ICCV 2021)
[Paper][Project Page]

We learn a personalized relighting model by capturing a person watching YouTube videos. Potential application includes relighting during a zoom call.

Shape and Material Capture at Home
Daniel Lichy, Jiaye Wu, Soumyadip Sengupta, David Jacobs
(CVPR 2021)
[Paper][Project Page][Code]

High-quality Photometric Stereo can be achieved with a simple flashlight. Recovers hi-res geometry and reflectance by progressively refining the predictions at each scale, conditioned on the prediction at previous scale.

Real-Time High Resolution Background Matting
Peter Lin*, Andrey Ryabtsev*, Soumyadip Sengupta, Brian Curless, Steve Seitz, Ira Kemelmacher-Shlizerman
(CVPR 2021 ORAL)(Best Paper Candidate 32/1600+ accepted papers)(Best Student Paper Honorable Mentions)
[Paper][Project Page][Code]

Background replacement at 30fps on 4K and 60fps on HD. Alpha matte is first extracted at low-res and then selectively refined with patches.

Lifespan Age Transformation Synthesis
Roy Or-El, Soumyadip Sengupta, Ohad Fried, Eli Shechtman, Ira Kemelmacher-Shlizerman
(ECCV 2020)
[Paper][Project Page][Code]

Age transformation from 0-70. Continuous aging is modeled by assuming 10 anchor age classes with interpolation in the latent space between them.

Background Matting: The World is Your Green Screen
Soumyadip Sengupta, Vivek Jayaram, Brian Curless, Steve Seitz, Ira Kemelmacher-Shlizerman
(CVPR 2020)
[Paper][Project Page][Code][Two Minute Papers Video][Microsoft AI using our code][CEO of Microsoft Satya Nadella talks about our work]

By simply capturing an additional image of the background, alpha matte can be extracted easily without requiring extensive human annotation in form of trimap.

Neural Inverse Rendering of an Indoor Scene from a Single Image
Soumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu, David Jacobs, Jan Kautz
(ICCV 2019)
[Paper][Project Page]

Self-supervision on real data is achieved with a Residual Appearnace Renderer network. It can cast shadows, add inter-reflections and near-field lighting, given the normal and albedo of the scene.

SfSNet : Learning Shape, Reflectance and Illuminance of Faces in the Wild
Soumyadip Sengupta, Angjoo Kanazawa, Carlos D. Castillo, David Jacobs.
CVPR 2018 [Spotlight].
[Paper] [Project Page / Code][Download PyTorch Code]

Decomposes an unconstrained human face into surface normal, albedo and spherical harmonics lighting. Learns from synthetic 3DMM followed by self-supervised finetuning on unlabelled real images.

Soumyadip Sengupta, Daniel Lichy, Angjoo Kanazawa, Carlos D. Castillo, David Jacobs.
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2020.

Introduces SfSMesh that utilizes the surface normal predicted by SfSNet to reconstruct a 3D face mesh.

A New Rank Constraint on Multi-view Fundamental Matrices, and its Application to Camera Location Recovery
Soumyadip Sengupta, Tal Amir, Meirav Galun, Amit Singer, Tom Goldstein, David Jacobs, Ronen Basri.
CVPR 2017 [Spotlight].
[Paper] [Code]

We prove that a matrix formed by stacking fundamental matrices between pairs of images has rank 6. We then introduce a non-linear optimization algorithm based on ADMM, that can better estimate the camera parameters using this rank constraint. This improves Structure-from-Motion algorithms which require initial camera estimation (bundle adjustment).

Solving Uncalibrated Photometric Stereo Using Fewer Images by Jointly Optimizing Low-rank Matrix Completion and Integrability
Soumyadip Sengupta, Hao Zhou, Walter Forkel, Ronen Basri, Tom Goldstein, David Jacobs.
Journal of Mathematical Imaging and Vision (JMIV), 2018.

We solve uncalibrated Photometric Stereo using as few as 4-6 images as a rank-constrained non-linear optimization with ADMM.

Frontal to Profile Face Verification in the Wild
Soumyadip Sengupta, Jun-Cheng Chen, Carlos D. Castillo, Vishal M. Patel, Rama Chellappa, David Jacobs.
WACV 2016.
[Project Page] , [Paper]

We introduce a dataset of frontal vs profile face verfication in the wild -- CFP. We show that SOTA face verification algorithms degrade about 10% on frontal-profile verification compared to frontal-frontal. Our dataset has been widely used to improve face verification across poses, but also for face warping and pose synthesis with GAN.

A Frequency Domain Approach to Silhouette Based Gait Recognition
Soumyadip Sengupta, Udit Halder, Rameshwar Panda, Ananda S Chowdhury.


Constraints and Priors for Inverse Rendering from Limited Observations
Soumyadip Sengupta
Doctoral Thesis, University of Maryland, January 2019