COMP 590/790-177 Special Topics: Neural Rendering

Fall 2022

Instructor: Soumyadip (Roni) Sengupta
Tuesday and Thursday, 2:00-3:15pm, Sitterson - Room 0014

[ Overview | Course Details | Schedule | Pre-Requisite ]

Course Description

This is an advanced seminar course focusing on recent developements in the field of Neural Rendering. Neural Rendering can be loosely defined as a technique where classical rendering pipelines (ray-tracing, volumetric rendering, image-based rendering etc.) are augmented with neural networks. We will first review basics of computer graphics and deep neural networks. This course is then divided into 2 sections. (a) In the first part, we will focus on 2D image-based neural rendering. (b) In the second part, we will discuss advances in 3D volumetric rendering via Neural Radiance Fields (NeRFs).

Survey papers & tutorials: State of the Art on Neural Rendering - Eurographics 2020 (SIGGRAPH 2021 course) & Advances in Neural Rendering - Eurographics 2022.

Office Hours: Thrs 3.30-4.30pm (SN 255). Canvas Site: 590 | 790.

Target Audience

590: Junior/Senior/MS students interested in learning the latest research trends in Computer Vision & Graphics

790: PhD students focusing on latest research trends in Computer Vision & Graphics

Goal/Student Learning Outcomes

590: (a) Understand an overview of 2D & 3D Neural Rendering (b) Learn how to code and synthesize 2D and 3D content in Pytorch

790: (a) Understand details of different recent techniques in Neural Rendering (b) Learn how to code and synthesize 2D and 3D content in Pytorch (c) Formulate novel project ideas and explore new solutions in the field of image/video synthesis and 3D reconstruction.

Grading: for 590

• 2 Paper Summary: 20 points
• Assignment 1: Train neural network for image classification: 10 points
• Assignment 2: Train DCGAN: 20 points
• Assignment 3: Test StyleGAN Inversion and editing: 20 points
• Assignment 4: Generate new images from text and sketch using Stability Diffusion: 10 points
• Assignment 5: Train Tiny-NeRF (Neural Radiance Field): 20 points

Students who have registered for 590 but want to engage in research projects can opt for 790 grading system. You have to declare your intention by Aug 25 by sending me an email.

Grading: for 790

• 2 Paper Presentation/ Review: 20 points (Some students will do 2 presentations, some will do 1 presentation + 1 review)
• Assignment 1: Generate new images from text and sketch using Stability Diffusion: 10 points
• Assignment 2: Train NeRF on pre-captured and self-captured imagery: 20 points
• Course Project: 50 points

Special Competition

Each student will use Stability Diffusion code (similar to DALL-E) to create image from text prompt only. Each student will be entitled to 1 submission. The whole CS department can vote on these creations. Top 3 winners will receive a small prize at the end of the semester. This does not impact your grade! Bring out your creativity and have fun!

Course Projects for 790

You can propose any project that involves neural rendering, i.e. using a neural network for any image or 3D synthesis tasks. Projects should be completed in groups of two or three. If you want to pursue a project individually, please come talk to me before you do so.

The goal of the course project is to find an interesting problem, propose a baseline and show that there is a research potential to introduce a better solution to the problem. I am looking for interesting problems/applications and not looking for neural network based solutions. Feel free to use exisiting neural architectures to come up with a reasonable baseline

Proposal: Project proposal will be in form of a 5mins presentation + 2mins Q&A (exact timing may vary). You should clearly present: (a)The problem that you are trying to solve. (b) The motivation behind the problem. (c) Your proposed approach, and how it relates to prior work. (c) The experiments that you plan to conduct. (d) Your plan for evaluation. You will need to submit a google slide presentation only.

Mid-term: Mid-term evaluation will also consist of a 5mins presentation + 2mins Q&A (exact timing may vary). You should extend your project proposal to include the following: (a) A preliminary set of results. (b) Your analysis of those results. (c) Additional experiments that you plan to run. You will need to submit a google slide presentation only.

Final: Final evaluation will consist of 5mins presentation + 2mins Q&A + 4 page project report in CVPR format (double column). You will also need to submit a 2-4mins teaser video describing your project and showcasing the results. This will be published in the course webpage for future references.

Paper Presentation

Each paper will be given a 20 minutes presentation slot, followed by 10 mins of Q&A. Present the paper as if you are the lead author. Please follow Prof. Gedas Bertasius's instructions on paper presentation. Your grades will depend on how well you present the paper and how well you answer the questions and clarify doubts from audience.

Intro lectures are specifically setup to give you good coverage of foundation topics, e.g. generative models, computer graphics and neural radiance fields. Understanding the lecture materials will help you better prepare paper presentation and review.

Note: You do not have to submit any written report. However, you have to submit the presentation by 11.59pm ET on the day of your presentation. This gives you extra 9 hours after your presentation to improve it based on the feedback during the class.

Paper Summary/Review for 590/790

You will be also required to review or summarize a paper. You have to submit a written review in CVPR reviewing format by 11.59pm on the day BEFORE the presentation of the paper. Your review will be public, open to all the students in the class. Each paper will have 2 reviewers. Both reviewers will be required to sit in the front of the class and ask clarifying questions to the presenter and carry out discussions.

Misc.

Class Participation: Please come to class prepared for a paper discussion with your peers. Furthermore, please do not discuss the papers with your peers before the class. I'm interested in hearing your own opinion about the papers.

Late Submissions: The class is structured around a tight paper presentation schedule. Therefore, late assignments will not be accepted. You lose 1 points for each late day.

Academic Integrity: For your presentations and projects, you are allowed to use materials from external sources. However, you must clearly acknowledge those sources.

Schedule

The paper presentation and reviewing schedule is published in this excel sheet.

Instructions for paper presentation and reviewing

Date	Topic	Details	Presenter	Special Dates
WEEK 1
Tue Aug 16	Welcome + Intro to Deep Learning	Lecture Slide | Deep Learning Crash Course for begineers	Roni	590: Image Classification Assignment Announced
Thrs Aug 18	Introduction to Computer Graphics (Geometry, Camera, Reflectance)	Lecture Slide | Watch 5 mins YouTube introduction on each topics you do not know. Mesh, Affine Transformation, Perspective Projection, Perspective Projection cont., Diffuse Reflection, Specular Reflection.	Roni
WEEK 2
Tue Aug 23	Introduction to Computer Graphics (Lighting & Rendering)	Lecture Slide	Roni
Thrs Aug 25	Generative Models (PixelCNN, VAE)	Lecture Slides: 1, 2	Roni	590: Image Classification Assignment Due
WEEK 3
Tue Aug 30	Generative Adversarial Networks (GANs)	Lecture Slide	Roni	590: DCGAN Assignment Announced
Thrs Sept 1	Applications of GANs	Lecture Slide	Roni
WEEK 4
Tue Sept 6	No Class (well-being day)
Thrs Sept 8	Semantic Manipulation with GANs	[1]Multimodal Conditional Image Synthesis with Product-of-Experts GANs. [2] Local Relighting of Real Scenes.	[1] Akshay Paruchuri [2] Max Lennon	590: DCGAN Assignment Due
WEEK 5
Tue Sept 13	StyleGAN Inversion	[1]Pivotal Tuning for Latent-based Editing of Real Images. [2]Third Time's the Charm? Image and Video Editing with StyleGAN3.	[1] William Stanford [2] Nurislam Tursynbek	590: StyleGAN Assignment Announced
Thrs Sept 15	StyleGAN editing	[1]CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directions. [2] DyStyle: Dynamic Neural Network for Multi-Attribute-Conditioned Style Editing.	[1] Sam Ehrenstein [2] Qiwei Zhao
WEEK 6
Tue Sept 20	790: Project Proposal Presentation		790 Students	790: Project Proposal Slide Submission
Thrs Sept 23	View Synthesis + Relighting	[1]Dynamic View Synthesis from Dynamic Monocular Video. [2]Neural Light Transport for Relighting and View Synthesis	[1] Jade Kandel [2] Jun Myeong Choi
WEEK 7
Tue Sept 27	Diffusion models - Part 1	Lecture Slide	Roni	590: StyleGAN Assignment Due
Thrs Sept 29	Diffusion models - Part 2	Lecture Slide	Roni
WEEK 8
Tue Oct 4	Diffusion models	[1]High-Resolution Image Synthesis with Latent Diffusion Models [2] Elucidating the Design Space of Diffusion-Based Generative Models	[1] Amartya Banerjee [2] Nurislam Tursynbek
Thrs Oct 6	Diffusion models	[1]ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models [2] Diffusion Autoencoders: Toward a Meaningful and Decodable Representation	[1] Luchao Qi [2] Max Lennon
WEEK 9
Tue Oct 11	Diffusion models	[1]DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation [2] Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise	[1] Kenya Lucero Vazquez Martinez [2] Amartya Banerjee	590/790: Diffusion Model Assignment Announced
Thrs Oct 13	Neural Fields - 1	Lecture Slide	Roni
WEEK 10
Tue Oct 18	Neural Fields - 2	Lecture Slide	Roni
Thrs Oct 20	Fall Break (no class)
WEEK 11
Tue Oct 25	Neural Fields - 3	Lecture Slide NeRF Volume Rendering Derivation	Roni
Thrs Oct 27	790: Mid-term Project Presentation		790 Students	790: Mid-term Project Slide Submission
WEEK 12
Tue Nov 1	Faster Inference	[1]Plenoxels: Radiance Fields without Neural Networks. [2]Instant Neural Graphics Primitives with a Multiresolution Hash Encoding.	[1] Raul Chun-Hung Chao [2] Ian Thomas	590/790: Diffusion Model Assignment Due on Oct 31st
Thrs Nov 3	Generalization	[1]GRF: Learning a General Radiance Field for 3D Scene Representation and Rendering. [2]pixelNeRF: Neural Radiance Fields from One or Few Images.	[1] Chin Tseng [2] Austin Hale
WEEK 13
Tue Nov 8	Multi-View	[1]IBRNet: Learning Multi-View Image-Based Rendering. [2]MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo.	[1] Jun Myeong Choi [2] Pierre-Nicolas Perrin
Thrs Nov 10	Inverse Rendering	[1]NeROIC: Neural Object Capture and Rendering from Online Image Collections . [2]SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections.	[1] Basar Demir [2] Chin Tseng
WEEK 14
Tue Nov 15	Misc.	[1]SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse views. [2]Dex-NeRF: Using a Neural Radiance field to Grasp Transparent Objects.	[1] Austin Hale [2] Ian Thomas
Thrs Nov 17	Misc.	[1]Nerfies: Deformable Neural Radiance Fields. [2]GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields.	[1] Basar Demir [2] Pierre-Nicolas Perrin	590/790: NeRF Assignment Announced
WEEK 15
Tue Nov 22	Guest Lecture: Ethics & Bias in Computer Graphics Research	Online via Zoom (details to be announced later)	Silvia Sellan (U. Toronto) & Ana Dodik (MIT)
Thrs Nov 24	Thanksgiving (no class)
WEEK 16
Tue Nov 29	790: Final Project Presentation		790 Students
Tue Dec 6	No Class			790: Final Slide Submission; 590/790: NeRF Assignment Due

 

Pre-Requisite

Basic knowledge of Computer Graphics and/or Deep Neural Networks.

Q) I am an undergrad, I don't have much background, but I am interested. Should I take this class?
Answer: You can consider taking the class if you are (a) junior/senior, (b) interested in Computer Vision, Graphics, and Machine Learning, (c) interested in being involved in undergraduate research, (d) interested or exploring grad school (MS/PhD) (e) willing to put extra hours to learn new concepts encountered while reading papers or working on projects. The purpose of the course for UGs is to be introduced to research, learn how to read papers, write reviews and work on research projects. This gives you some insight into how a Ph.D. life would look like ;)

Q) I do not know Deep Learning, what can I do to learn the basics?
Answer: I created a short tutorial for UGs when I was a postdoc at Univ. of Washington, introducing what Computer Vision and Deep Learning is. Here are the slides. The slides end (Slide #73) with a Google Colab tutorial on training a convolutional network for classifying hand-written numerical digits. Here are the following concepts or topics that you want to know: (a) What is a neural network? (b) How to train a neural network for classification? (c) What is backpropagation for training neural networks? (d) What is a convolutional neural network (CNN)? (e) What are some of the popular CNN architectures - LeNet, AlexNet, VGG, ResNet? (f) How to train a neural network in PyTorch? (best to use Google Colab for simple experiments)
Here are some crash course tutorial on Deep Learning, feel free to watch them at your convenience: SIGGRAPH 2018 tutorial (3hours 30mins), freeCodeCamp.org (1hours 30mins)

Q) I do not know Computer Graphics, what should I do?
Answer: Lecture 2 and 3 will introduce most relevant concepts in Computer Graphics for this course. They will be a bit fast paced. But re-reading the materials after class should help. Knowing any of the following Computer Graphics engine: OpenGL, MeshLab, Blender, Mitsuba, Unity, Unreal Engine, can be helpful for course projects. But you can always learn them on the fly.