Real-Time Computer Vision in Autonomous Vehicles: Real Fast Isn't Good Enough

Funded by NSF Cyber-Physical Systems Program.

PI: Jim Anderson. Co-PIs: Don Smith, Jan-Michael Frahm, and Shige Wang (General Motors Research).

The Challenge.

The push towards deploying autonomous-driving capabilities in vehicles is happening at breakneck speed. Semi-autonomous features are becoming increasingly common, and fully autonomous vehicles at mass-market scales are on the horizon. Cameras are cost-effective sensors, so computer-vision (CV) techniques have loomed large in implementing autonomous features and will continue to do so. In a vehicle, these techniques must function "in real time." Unfortunately, this requirement lies at the heart of a significant disconnect: when CV researchers refer to "real time," they usually mean "real fast"; in contrast, certifiable automotive systems must be "real time" in the sense of being predictable.

This disconnect between real-fast CV and real-time safety is already having unfortunate consequences. Several highly publicized recent accidents involving semi-autonomous and fully autonomous cars on open roads have resulted in fatalities. The post-crash analyses of these incidents have revealed a critical trade-off between time and accuracy at the nexus between CV and real time. Today, CV-based features are largely provided in settings where the driver is assumed to be able to retake full control of the vehicle. As such, these features are not yet subject to strict certification requirements. Moving forward, however, we will see greater and greater autonomy in mass-market vehicles, with the line between semi-autonomy and full autonomy being crossed eventually. When this happens, strict certification will be enormously important. Certifying CV applications that are merely "real fast" will be difficult, if not impossible.

The Approach.

Motivated by this disconnect, this project is directed at enabling CV programmers to produce code that is amenable to real-time certification. Specifically, a real-time CV API will be produced by extending OpenVX, which is a recently ratified standard intended for embedded systems. OpenVX utilizes a graph-based formalism for specifying CV workloads that presents several challenges when validating real-time constraints. These challenges will be addressed. A library of new CV algorithms that exploit the features of the API will be created and methods developed to transform existing algorithms. Also, an experimental evaluation of "real-fast" vs. "real-time" CV will be conducted using driving simulators, sub-scale autonomous vehicles, and advanced testing infrastructure at General Motors.

Significance.

While industry is pushing hard in the area of autonomous driving, autonomous vehicles will never become a common mode of transportation unless methods for certifying real-time safety are produced. This project will focus on a key aspect of certification: validating the real-time correctness of CV applications. The results that are produced will be made available to the world at large through open-source software. This software will include the new OpenVX API to be produced as well as tools for validating the real-time correctness of applications developed using this API.



Publications


M. Yang, Sharing GPUs for Real-Time Autonomous-Driving Systems, Ph.D. Dissertation, Department of Computer Science, The University of North Carolina at Chapel Hill, December 2020. PDF .


T. Amert, Enabling Real-Time Certification of Autonomous Driving Applications, Ph.D. Dissertation, Department of Computer Science, The University of North Carolina at Chapel Hill, August 2021. PDF . Winner, 2022 ACM SIGBED Paul Caspi Memorial Dissertation Award.


N. Otterness, Developing Real-Time GPU-Sharing Platforms for Artificial-Intelligence Applications, Ph.D. Dissertation, Department of Computer Science, The University of North Carolina at Chapel Hill, August 2022. PDF .


S. Voronov, Scheduling Real-Time Graph-Based Workloads, Ph.D. Dissertation, Department of Computer Science, The University of North Carolina at Chapel Hill, August 2023. PDF .


N. Otterness and J. Anderson, " Exploring AMD GPU Scheduling Details by Experimenting With `Worst Practices'", Real-Time Systems, special issue of outstanding papers from the 29th International Conference on Real-Time Networks and Systems (RTNS 2021), Volume 58, pp. 105–133, March 2022. PDF .


T. Amert, M. Yang, S. Nandi, T. Vu, J. Anderson, and F.D. Smith, " The Price of Schedulability in Cyclic Workloads: The History-vs.-Response-Time-vs.-Accuracy Trade-Off", Journal of Systems Architecture, special issue of outstanding papers from the 23rd IEEE International Symposium on Real-Time Distributed Computing (ISORC 2020), Volume 120, November 2021. PDF .


R. Wagle, Z. Tong, R. Sites, and J. Anderson, " Want Predictable GPU Execution? Beware SMIs!", Proceedings of the 29th IEEE International Conference on Parallel and Distributed Systems, December 2023. PDF .


Z. Tong, S. Ahmed, and J. Anderson, " Holistically Budgeting Processing Graphs", Proceedings of the 44th IEEE Real-Time Systems Symposium, pp. 27–39, December 2023. PDF .


J. Bakita and J. Anderson, " Hardware Compute Partitioning on NVIDIA GPUs", Proceedings of the 29th IEEE Real-Time and Embedded Technology and Applications Symposium, pp. 54–66, May 2023. Winner, outstanding paper award. PDF .


T. Yandrofski, J. Chen, N. Otterness, J. Anderson, and F.D. Smith, " Making Powerful Enemies on NVIDIA GPUs", Proceedings of the 43rd IEEE Real-Time Systems Symposium, pp. 383–395, December 2022. PDF .


J. Bakita and J. Anderson, " Enabling GPU Memory Oversubscription via Transparent Paging to an NVMe SSD", Proceedings of the 43rd IEEE Real-Time Systems Symposium, pp. 370–382, December 2022. PDF .


T. Amert, Z. Tong, S. Voronov, J. Bakita, F.D. Smith, and J. Anderson, " TimeWall: Enabling Time Partitioning for Real-Time Multicore+Accelerator Platforms", Proceedings of the 42nd IEEE Real-Time Systems Symposium, pp. 455-468, December 2021. PDF .


S. Voronov, S. Tang, T. Amert, and J. Anderson, " AI Meets Real-Time: Addressing Real-World Complexities in Graph Response-Time Analysis", Proceedings of the 42nd IEEE Real-Time Systems Symposium, pp. 82–96, December 2021. PDF .


T. Amert and J. Anderson, " CUPiDRT: Detecting Improper GPU Usage in Real-Time Applications", Proceedings of the 24th IEEE International Symposium on Real-Time Distributed Computing, pp. 86-95, June 2021. PDF .


N. Otterness and J. Anderson, " Exploring AMD GPU Scheduling Details by Experimenting With `Worst Practices'", Proceedings of the 29th International Conference on Real-Time Networks and Systems , pp. 24-34, April 2021. PDF . Winner, best paper award.


N. Otterness and J. Anderson, " AMD GPUs as an Alternative to NVIDIA for Supporting Real-Time Workloads", Proceedings of the 32nd Euromicro Conference on Real-Time Systems, pp. 10:1-10:23, July 2020. PDF .


T. Amert, M. Yang, S. Nandi, T. Vu, J. Anderson, and F.D. Smith, " The Price of Schedulability in Multi-Object Tracking: The History-vs.-Accuracy Trade-Off", Proceedings of the 23rd International Symposium on Real-Time Distributed Computing, pp. 124-133, May 2020. PDF .


T. Vu, M. Eder, T. Price, and J.-M. Frahm " Any-Width Networks", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020. PDF .


T. Amert, S. Voronov, and J. Anderson " OpenVX and Real-Time Certification: The Troublesome History", Proceedings of the 40th IEEE Real-Time Systems Symposium, pp. 312 -325, December 2019. PDF .


M. Yang, S. Wang, J. Bakita, T. Vu, F.D. Smith, J. Anderson, and J.-M. Frahm, " Re-thinking CNN Frameworks for Time-Sensitive Autonomous-Driving Applications: Addressing an Industrial Challenge", Proceedings of the 25th IEEE Real-Time and Embedded Technology and Applications Symposium, pp. 305-317, April 2019. PDF .


Other papers that acknowledge this grant can be found on the PI's Publications Page .



Last modified 12 March 2024