Scenario Generation for Testing of Autonomous Driving Systems Using Real-World Failure Records
Quick Answer
This study introduces a novel scenario generation pipeline for Autonomous Driving Systems (ADS) testing, leveraging historical failure records in natural language.
Quick Take
This study introduces a novel scenario generation pipeline for Autonomous Driving Systems (ADS) testing, leveraging historical failure records in natural language. By utilizing modular LLM-based synthetic scenario generation, the method produces diverse scenarios compatible with testing constraints, successfully applying it to generate 20 scenarios for the Metadrive simulator using NHTSA ADS crash data.
Key Points
- Proposes a scenario generation pipeline using historical ADS failure records.
- Utilizes modular LLM-based generation for diverse scenario creation.
- Successfully generates 20 scenarios for Metadrive simulator testing.
- Combines 4 road types and 3 non-ego vehicle movements in scenarios.
- Code available on GitHub for further use and exploration.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 31131v1 Announce Type: new Abstract: To ensure safe on-road behavior, pre-deployment testing and failure discovery of Autonomous Driving Systems (ADS) is crucial. Present day simulation based testing methods focus largely on mathematical models for efficient search of optimal scenarios, assuming a fixed scenario representation. On the other hand, real-world testing involves substantial manual effort to design scenario templates for testing.
These templates represent distinct failure scenarios consisting of pre-deployment vehicle movements, map types, etc. Historical failure records for ADS are a reliable source of real-world failure conditions, which can be used for scenario generation. In this work, we propose a scenario generation pipeline using categorical and contextual information available from historical records in natural language format.
Our approach consists of modular LLM based synthetic scenario generation, compatible with the testing constraints of a given system. We successfully apply our method to generate a diverse set of scenarios for testing autonomous navigation on Metadrive simulator using the NHTSA ADS crash records. Our approach results in accurate and diverse scenario generation with a combination of 4 road types, 3 non ego vehicle movement types, including on road anomalies in the form of working zones.
Generated scenarios align with the provided testing conditions, and reveals interesting failures of the system within a limited testing budget of 20 scenarios. Code is available at https://github. com/anjaliParashar/crash2scenario.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →The Verification Horizon: No Silver Bullet for Coding Agent Rewards
As coding agents evolve, verifying solutions becomes more challenging than generating them, necessitating a focus on scalable, faithful, and robust verification methods. The study reveals that no fixed reward function can sustain effectiveness as model capabilities advance, emphasizing the need for verification to evolve alongside solution generation.