Real-Time Computer Vision in Autonomous Vehicles: Real Fast Isn't Good Enough

Funded by NSF Cyber-Physical Systems Program.

PI: Jim Anderson. Co-PIs: Don Smith, Jan-Michael Frahm, and Shige Wang (General Motors Research).

The Challenge.

The push towards deploying autonomous-driving capabilities in vehicles is happening at breakneck speed. Semi-autonomous features are becoming increasingly common, and fully autonomous vehicles at mass-market scales are on the horizon. Cameras are cost-effective sensors, so computer-vision (CV) techniques have loomed large in implementing autonomous features and will continue to do so. In a vehicle, these techniques must function "in real time." Unfortunately, this requirement lies at the heart of a significant disconnect: when CV researchers refer to "real time," they usually mean "real fast"; in contrast, certifiable automotive systems must be "real time" in the sense of being predictable.

This disconnect between real-fast CV and real-time safety is already having unfortunate consequences. Several highly publicized recent accidents involving semi-autonomous and fully autonomous cars on open roads have resulted in fatalities. The post-crash analyses of these incidents have revealed a critical trade-off between time and accuracy at the nexus between CV and real time. Today, CV-based features are largely provided in settings where the driver is assumed to be able to retake full control of the vehicle. As such, these features are not yet subject to strict certification requirements. Moving forward, however, we will see greater and greater autonomy in mass-market vehicles, with the line between semi-autonomy and full autonomy being crossed eventually. When this happens, strict certification will be enormously important. Certifying CV applications that are merely "real fast" will be difficult, if not impossible.

The Approach.

Motivated by this disconnect, this project is directed at enabling CV programmers to produce code that is amenable to real-time certification. Specifically, a real-time CV API will be produced by extending OpenVX, which is a recently ratified standard intended for embedded systems. OpenVX utilizes a graph-based formalism for specifying CV workloads that presents several challenges when validating real-time constraints. These challenges will be addressed. A library of new CV algorithms that exploit the features of the API will be created and methods developed to transform existing algorithms. Also, an experimental evaluation of "real-fast" vs. "real-time" CV will be conducted using driving simulators, sub-scale autonomous vehicles, and advanced testing infrastructure at General Motors.


While industry is pushing hard in the area of autonomous driving, autonomous vehicles will never become a common mode of transportation unless methods for certifying real-time safety are produced. This project will focus on a key aspect of certification: validating the real-time correctness of CV applications. The results that are produced will be made available to the world at large through open-source software. This software will include the new OpenVX API to be produced as well as tools for validating the real-time correctness of applications developed using this API.


T. Amert, S. Voronov, and J. Anderson " OpenVX and Real-Time Certification: The Troublesome History", Proceedings of the 40th IEEE Real-Time Systems Symposium, December 2019, to appear. PDF .

M. Yang, S. Wang, J. Bakita, T. Vu, F.D. Smith, J. Anderson, and J.-M. Frahm, " Re-thinking CNN Frameworks for Time-Sensitive Autonomous-Driving Applications: Addressing an Industrial Challenge", Proceedings of the 25th IEEE Real-Time and Embedded Technology and Applications Symposium, pp. 305-317, April 2019. PDF .

Other papers that acknowledge this grant can be found on the PI's Publications Page .

Last modified 25 October 2019