1) Dolly zoom The dolly zoom is an optical effect used by cinematographers. The effect consists in adjusting the distance of the camera to a foreground object in the scene, while simultaneously controlling the camera’s field of view (a function of the focal length), in order for the foreground object to retain a constant size in the image throughout the entire capture sequence.

In order to approximate a photorealistic effect, you are provided a dense point cloud augmented with RGB color information. To obtain a rendered image you can use the provided rendering function PointCloud2Image, which takes as input a projection matrix and transforms the 3D point cloud into a 2D image (see below for details). Your task will be to:

1) Design a sequence of projection matrices corresponding to each frame of a ”dolly zoom” capture sequence and effect.

2) Use the provided code to render each of the individual images (capture frames).

Setup: Start the sequence using the cameras original internal calibration matrix K (provided in the data.mat file) and position the camera in such a way that the foreground object occupies in the initial image a bounding box of approx 400 by 640 pixels (width and height) respectively. (Per reference, positioning the camera at the origin renders the foreground object within a bounding box of size 250 by 400 pixels).

Solution:

Code: dollyzoom.m

Generate the video of the dollyzoom effect “dollyzoom.avi“ and the video “dollyzoom_distorted.avi“ with radio distortion. The output videos are converted to wmv format by “ffmpeg“

Explanation: According to the camera model, projection matrix is P = K[R^T   - R^T C]. In our cases, there is no rotation and only movement in z axis. So, the rotation matrix R is identity matrix and the elements in C are 0 except elements of index for z axis. We can obtain the modified sensor calibration K by geometric calculation:

2) Radial Distortion Radial distortion is the most significant type of geometric distortion in todays cameras. It is most evident in images produced with low-cost, wide-angle lenses. Radial distortion bends straight lines into circular arcs, violating the main invariance preserved in the pinhole camera model that straight lines in the world map to straight lines in the image plane. Radial distortion may appear as barrel distortion, usually arising at short focal lengths, or pincushion distortion, usually arising at longer focal lengths.

Distortion can be compensated mathematically, first by applying a parametric distortion model, estimating the distortion coefficients and then correcting the distortion. The Brown distortion model is an example of a polynomial approximation model:

Where (u,v)^T and (u^′,v^′)^T are normalized camera coordinates (not image coordinates!), r = |(u,v)^T |₂ and K_i denotes a real valued distortion coefficient. Note that the Brown model serves to describe the pixel displacement field between a pair of images, where one of them is denoted (by convention) as ”distorted”, while the other as ”undistorted”.

Your task is to apply a ”radial distortion” procedure to each of the synthetic frames generated for the dolly zoom sequence. Consider a single radial distortion coefficient K₁. After converting the pixel coordinates of the synthetically generated image into normalized camera coordinates, they will correspond to the (u^′,v^′)^T coordinates. Hence, your distortion procedure needs to estimate the coordinates (u,v)^T = g(K₁,u^′,v^′). In this way, you will be solving for a backwards radial distortion model. Once the distorted image coordinates have been computed, generate a new image. In you report explain the methods used to solve the problem and discuss their possible extension to a full radial distortion model.

Setup: Let the distortion parameter K₁ be defined in the range [-1.0,1.0] and interpolate linearly between these two values along the entire dolly zoom sequence.

Solution:

Code: distort.m (distort_oval.m)

The matlab function to generate the radio distortion effect. (distort_oval.m generate a ‘oval‘ radial distortion effect due to the different normalization method)

Explanation:

Consider a single radial distortion coefficient K₁. According to the Brown distortion model, we can get the model only considered K₁:

(u,v are normalized camera coordinates. In function distort.m, the coordinates are normalized by max(height,width), while the coordinated are normalized by height and width respectively in function distort_oval.m)