Automated Driving System Testing and Evaluation
Safe AI Framework for Trustworthy Edge Scenario Test (SAFE-TEST)
It is well-known that scenario generation is essential for testing and evaluation of automated vehicles (AVs), but how to generate realistic and trustworthy scenarios efficiently remains an open question. To address this challenge, in the past few years, University of Michigan’s Center for Connected and Automated Transportation (CCAT) has developed a scenario generation toolbox that can accelerate the development and validation of automated vehicles. This toolbox, which has been implemented at the American Center for Mobility (ACM), includes the augmented reality (AR) testing platform and the naturalistic and adversarial driving environment (NADE). With AR, a real AV can be tested at a test track with interaction from virtual traffic flow. With NADE, the maneuvers of virtual background vehicles will be controlled intelligently, in that most of scenarios are generated from naturalistic driving data, and only at selected moments, adversarial scenarios are generated to challenge the AV under test. The theory behind NADE ensures both the unbiasedness and the efficiency of the testing scenario generation. With this toolbox, every testing mile at ACM can be converted into thousands of equivalent miles on public roads, which can significantly reduce the development costs and shorten the development cycle. In this work, we demonstrate its effectiveness by testing a L4 experiment vehicle at ACM.
Intelligent Driving Intelligence Test for Autonomous Vehicles with Naturalistic and Adversarial Environment
Driving intelligence tests are critical to the development and deployment of autonomous vehicles. The prevailing approach tests autonomous vehicles in life-like simulations of the naturalistic driving environment. However, due to the high dimensionality of the environment and the rareness of safety-critical events, hundreds of millions of miles would be required to demonstrate the safety performance of autonomous vehicles, which is severely inefficient. We discover that sparse but adversarial adjustments to the naturalistic driving environment, resulting in the naturalistic and adversarial driving environment, can significantly reduce the required test miles without loss of evaluation unbiasedness. By training the background vehicles to learn when to execute what adversarial maneuver, the proposed environment becomes an intelligent environment for driving intelligence testing. We demonstrate the effectiveness of the proposed environment in a highway-driving simulation. Comparing with the naturalistic driving environment, the proposed environment can accelerate the evaluation process by multiple orders of magnitude.
Safety Assessment of Highly Automated Driving Systems: A New Framework
Safety assessment is critical in the development and deployment of highly automated driving systems (ADS). A new framework is proposed for closed-facility testing, which can quantitatively, accurately, and efficiently assess the safety of highly ADS in a cost-effective fashion. To this end, two major problems of closed-facility testing approach are resolved by two pillars of the framework. First, an augmented reality (AR) testing platform is constructed to augment the real ADS interacting with simulated background traffic. Second, a testing scenario library generation (TSLG) method is designed to systematically generate a set of critical scenarios for each operational design domain (ODD). By the important sampling theory, generating a library is transformed as constructing an importance function. A new definition of criticality and critical scenario searching methods are proposed. The framework is implemented in the Mcity test facility at the University of Michigan. Field test results validate the accuracy and efficiency of the proposed framework. In the cut-in case study, the proposed framework can accelerate the safety assessment process by times faster than the public-road testing approach.
Mcity Augmented Reality
This technology develops an augmented reality environment for connected and automated vehicle (CAV) testing and evaluation, in which background traffic is generated in microscopic simulation and provided to testing CAVs. The augmented reality combines the real-world testing facility (Mcity) and a simulation platform, in which movements of testing CAVs and traffic signals in the real-world can be synchronized in simulation, while simulated traffic information can be provided to testing CAVs. Testing CAVs “think” they are surrounded by other vehicles and adjust behaviors accordingly. At the same time, behaviors of simulated vehicles are also influenced by the testing CAVs. Information between real world and simulation is transmitted through DSRC.
1. Feng Y., Yu C., Xu S., Liu H. X. and Peng H. (2018). An Augmented Reality Environment for Connected and Automated Vehicle Testing and Evaluation, 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, , pp. 1549-1554. [PDF]
2. Feng Y. and Liu H.X. (2019). Self-Driving Cars Learn About Road Hazards Through Augmented Reality. IEEE Spectrum. [Link]
3. Feng Y. and Liu H.X. (2019). Real World Meets Virtual World: Augmented Reality Makes Driverless Vehicle Testing Faster, Safer, and Cheaper. Mcity white paper. [PDF]
4. Feng S., Feng Y., Yu, C., Zhang Y., and Liu H.X. (2020). Testing Scenario Library Generation for Connected and Automated Vehicles, Part I: Methodology. IEEE Transactions on Intelligent Transportation Systems. [PDF]
5. Feng S., Feng Y., Sun H., Bao S., Zhang Y., and Liu H.X. (2020). Testing Scenario Library Generation for Connected and Automated Vehicles, Part II: Case Studies. IEEE Transactions on Intelligent Transportation Systems. [PDF]
6. Feng S., Feng Y., Sun H., Zhang Y., and Liu H.X. (2020). Testing Scenario Library Generation for Connected and Automated Vehicles: An Adaptive Framework. IEEE Transactions on Intelligent Transportation Systems. [PDF]
7. Feng S., Feng Y., Yan X., Shen S., Xu S., and Liu H.X. (2020). Safety Assessment of Highly Automated Driving Systems in Test Tracks: A New Framework. Accident Analysis and Prevention Volume 144, 105664. [PDF]
8. Feng, S., Yan, X., Sun, H., Feng Y., and Liu H.X. (2021) Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment. Nat Commun 12, 748. https://doi.org/10.1038/s41467-021-21007-8 [PDF] [Link]
9. Liu L., Feng S., Feng Y., Zhu X., Liu H.X. (2022). Learning-Based Stochastic Driving Model for Autonomous Vehicle Testing. Transportation Research Record. 2022; 2676(1):54-64. doi:10.1177/03611981211035756 [PDF]
A research paper "Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment" published in the scientific journal, Nature Communications received the SIG ITS Outstanding Paper Award at the INFORMS Transportation Science and Logistics Society Meeting.
A webinar of Scenario Generation Toolbox for Accelerating AV Development and Validation at the American Center for Mobility (ACM) was hosted by ACM.
A research paper from CCAT Director, Prof. Henry Liu, has been published and chosen as a featured article in the scientific journal, Nature Communications. It proposes a naturalistic and adversarial driving environment (NADE) that will drastically improve the efficiency of autonomous vehicle testing and evaluation.
A white Paper that Prof. Henry Liu and Dr. Yiheng Feng outlined the unique augmented reality technology used to test connected and automated vehicles at Mcity was published in today's University Record.