This project is motivated by the latency requirements for Augmented Reality(AR) or Virtual Reality(VR) systems. IN AR/VR, there are three subsystems:
In this project we target the tracking sub-system. Our goal is to create a high frequency camera-based tracker.
To maintain a reliable registration of the virtual world with the real world, augmented reality (AR) applications require highly accurate, low-latency tracking of the device. In this paper, we propose a novel method for performing this fast 6-DOF head pose tracking using a cluster of rolling shutter cameras. The key idea is that a rolling shutter camera works by capturing the rows of an image in rapid succession, essentially acting as a high-frequency 1D image sensor. By integrating multiple rolling shutter cameras on the AR device, our tracker is able to perform 6-DOF markerless tracking in a static indoor environment with minimal latency. Compared to state-of-the-art tracking systems, this tracking approach performs at significantly higher frequency, and it works in generalized environments. To demonstrate the feasibility of our system, we present thorough evaluations on synthetically generated data with tracking frequencies reaching 56.7 kHz. We further validate the method's accuracy on real-world images collected from a prototype of our tracking system against ground truth data using standard commodity GoPro cameras capturing at 120 Hz frame rate tracking at 80.4kHz.
Most of the cameras in your smartphones and GoPros, employ what is called a rolling shutter(RS). In rolling shutter, EACH row of the images is captured at a slightly different time. Hence each row is actually a snapshot of the scene at different time instances. We treat these captured rows as an individual sample rather than treating the 2-D image as a single frame. The frequency of such row-images(row-samples) is very high. Even for a camera in your iphone capturing an image of height 720 rows at a fps of 120 is more than 80kHz. In our system, we process each row-image to produce a tracking estimate.
There are two important disadvantages of using a rolling shutter camera.
To tackle these two challenges we use a set of redundant cameras.
As we are tracking at very high frequency, we can assume that the motion between two timwstamps is small. This lets us linearize the motion model making the motion model simple and efficient to compute.
Direct pixel-wise comparison instead of feature-extracton-and-matching pipeline keeps our system efficient. We use our binary decriptors to match pixels and hamming distanc as a measure of cost. This makes our system even more efficient.
PDF version of my talk at ISMAR 2016