Timezone: »

 
Poster
SafeBench: A Benchmarking Platform for Safety Evaluation of Autonomous Vehicles
Chejian Xu · Wenhao Ding · Weijie Lyu · ZUXIN LIU · Shuai Wang · Yihan He · Hanjiang Hu · DING ZHAO · Bo Li

Tue Nov 29 02:00 PM -- 04:00 PM (PST) @ Hall J #1032
in Poster Session 2 »

As shown by recent studies, machine intelligence-enabled systems are vulnerable to test cases resulting from either adversarial manipulation or natural distribution shifts. This has raised great concerns about deploying machine learning algorithms for real-world applications, especially in safety-critical domains such as autonomous driving (AD). On the other hand, traditional AD testing on naturalistic scenarios requires hundreds of millions of driving miles due to the high dimensionality and rareness of the safety-critical scenarios in the real world. As a result, several approaches for autonomous driving evaluation have been explored, which are usually, however, based on different simulation platforms, types of safety-critical scenarios, scenario generation algorithms, and driving route variations. Thus, despite a large amount of effort in autonomous driving testing, it is still challenging to compare and understand the effectiveness and efficiency of different testing scenario generation algorithms and testing mechanisms under similar conditions. In this paper, we aim to provide the first unified platform SafeBench to integrate different types of safety-critical testing scenarios, scenario generation algorithms, and other variations such as driving routes and environments. In particular, we consider 8 safety-critical testing scenarios following National Highway Traffic Safety Administration (NHTSA) and develop 4 scenario generation algorithms considering 10 variations for each scenario. Meanwhile, we implement 4 deep reinforcement learning-based AD algorithms with 4 types of input (e.g., bird’s-eye view, camera) to perform fair comparisons on SafeBench. We find our generated testing scenarios are indeed more challenging and observe the trade-off between the performance of AD agents under benign and safety-critical testing scenarios. We believe our unified platform SafeBench for large-scale and effective autonomous driving testing will motivate the development of new testing scenario generation and safe AD algorithms. SafeBench is available at https://safebench.github.io.

Keywords: autonomous driving · safety-critical scenarios · safety benchmarking platform
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Chejian Xu (University of Illinois at Urbana-Champaign)
Wenhao Ding (Carnegie Mellon University)
Weijie Lyu (University of Illinois at Urbana-Champaign)
ZUXIN LIU (Carnegie Mellon University)
Shuai Wang (CMU, Carnegie Mellon University)
Yihan He (CMU, Carnegie Mellon University)
Hanjiang Hu (Carnegie Mellon University)

I am a second-year PhD student at Safe AI Lab at CMU advised by Prof. Ding Zhao. I work closely with Secure Learning Lab at UIUC advised by Prof. Bo Li. Before that, I worked at IRMV Lab advised by Prof. Hesheng Wang and got my bachelor and master degree from Shanghai Jiao Tong University in 2018 and 2021. I also worked with MSC Lab at UC Berkeley in 2020 advised by Prof. Masayoshi Tomizuka. My main research focuses on robotic perception and robust machine learning for safe autonomous driving and robotics. Specifically, I am working on the robustness and certification with heterogeneous data in robotic and vision applications. I am also interested in building new datasets and benchmarks for robust machine learning algorithms.

DING ZHAO (Carnegie Mellon University)
Bo Li (UIUC)

More from the Same Authors