Automated Driving System Testing and Evaluation

Mcity Augmented Reality

This technology develops an augmented reality environment for connected and automated vehicle (CAV) testing and evaluation, in which background traffic is generated in microscopic simulation and provided to testing CAVs. The augmented reality combines the real-world testing facility (Mcity) and a simulation platform, in which movements of testing CAVs and traffic signals in the real-world can be synchronized in simulation, while simulated traffic information can be provided to testing CAVs. Testing CAVs “think” they are surrounded by other vehicles and adjust behaviors accordingly. At the same time, behaviors of simulated vehicles are also influenced by the testing CAVs. Information between real world and simulation is transmitted through DSRC.

Safety Assessment of Highly Automated Driving Systems: A New Framework

Safety assessment is critical in the development and deployment of highly automated driving systems (ADS). A new framework is proposed for closed-facility testing, which can quantitatively, accurately, and efficiently assess the safety of highly ADS in a cost-effective fashion. To this end, two major problems of closed-facility testing approach are resolved by two pillars of the framework. First, an augmented reality (AR) testing platform is constructed to augment the real ADS interacting with simulated background traffic. Second, a testing scenario library generation (TSLG) method is designed to systematically generate a set of critical scenarios for each operational design domain (ODD). By the important sampling theory, generating a library is transformed as constructing an importance function. A new definition of criticality and critical scenario searching methods are proposed. The framework is implemented in the Mcity test facility at the University of Michigan. Field test results validate the accuracy and efficiency of the proposed framework. In the cut-in case study, the proposed framework can accelerate the safety assessment process by times faster than the public-road testing approach.

Intelligent Driving Intelligence Test for Autonomous Vehicles with Naturalistic and Adversarial Environment

Driving intelligence tests are critical to the development and deployment of autonomous vehicles. The prevailing approach tests autonomous vehicles in life-like simulations of the naturalistic driving environment. However, due to the high dimensionality of the environment and the rareness of safety-critical events, hundreds of millions of miles would be required to demonstrate the safety performance of autonomous vehicles, which is severely inefficient. We discover that sparse but adversarial adjustments to the naturalistic driving environment, resulting in the naturalistic and adversarial driving environment, can significantly reduce the required test miles without loss of evaluation unbiasedness. By training the background vehicles to learn when to execute what adversarial maneuver, the proposed environment becomes an intelligent environment for driving intelligence testing. We demonstrate the effectiveness of the proposed environment in a highway-driving simulation. Comparing with the naturalistic driving environment, the proposed environment can accelerate the evaluation process by multiple orders of magnitude.

Publications

1. Feng Y., Yu C., Xu S., Liu H. X. and Peng H. (2018). An Augmented Reality Environment for Connected and Automated Vehicle Testing and Evaluation, 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, , pp. 1549-1554. [PDF]

2. Feng Y. and Liu H.X. (2019). Self-Driving Cars Learn About Road Hazards Through Augmented Reality. IEEE Spectrum. [Link]

3. Feng Y. and Liu H.X. (2019). Real World Meets Virtual World: Augmented Reality Makes Driverless Vehicle Testing Faster, Safer, and Cheaper. Mcity white paper. [PDF]

4. Feng S., Feng Y., Yu, C., Zhang Y., and Liu H.X. (2020). Testing Scenario Library Generation for Connected and Automated Vehicles, Part I: Methodology. IEEE Transactions on Intelligent Transportation Systems. [PDF]

5. Feng S., Feng Y., Sun H., Bao S., Zhang Y., and Liu H.X. (2020). Testing Scenario Library Generation for Connected and Automated Vehicles, Part II: Case Studies. IEEE Transactions on Intelligent Transportation Systems. [PDF]

6. Feng S., Feng Y., Sun H., Zhang Y., and Liu H.X. (2020). Testing Scenario Library Generation for Connected and Automated Vehicles: An Adaptive Framework. IEEE Transactions on Intelligent Transportation Systems. [PDF]

7. Feng S., Feng Y., Yan X., Shen S., Xu S., and Liu H.X. (2020). Safety Assessment of Highly Automated Driving Systems in Test Tracks: A New Framework. Accident Analysis and Prevention Volume 144, 105664. [PDF]

8. Feng, S., Yan, X., Sun, H., Feng Y., and Liu H.X. (2021) Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment. Nat Commun 12, 748. https://doi.org/10.1038/s41467-021-21007-8 [PDF] [Link]

Patents

Liu, H. and Feng, Y. (2017) Simulated Vehicle Traffic for Autonomous Vehicles, US Provisional Patent, Application No. 62/500,299, Filed on May 2, 2017. [PDF]

Related News:

02/02/2021

A research paper from CCAT Director, Prof. Henry Liu, has been published and chosen as a featured article in the scientific journal, Nature Communications. It proposes a naturalistic and adversarial driving environment (NADE) that will drastically improve the efficiency of autonomous vehicle testing and evaluation.

Media Coverage

News from Center for Connected and Automated Transportation

News from Tech Xplore

12/28/2020

Professor Henry Liu gave a keynote lecture presentation titled "Testing Autonomous Vehicles in a Naturalistic and Adversarial Environment" at the Purdue Next-Generation Transportation System (NGTS) 2020 conference.

12/12/2019

Another news of our Augments Reality Environment for Mcity was posted in engineering.com: Augmented Reality Is Used to Improve Autonomous Vehicle Testing

11/22/2019

Our Paper titled "Self-Driving Cars Learn About Road Hazards Through Augmented Reality" written by Yiheng Feng and Henry X. Liu was published on IEEE SPECTRUM.

11/19/2018

A white Paper that Prof. Henry Liu and Dr. Yiheng Feng outlined the unique augmented reality technology used to test connected and automated vehicles at Mcity was published in today's University Record.