Can World Foundation Models Generate Realistic Driving Videos? A Case Study on Pedestrian Crossing Scenarios

Engineering notes

Key topics: autonomous driving, autonomous vehicle, foundation model, real-world driving. See the paper for implementation details and experimental results.

Chinese explanation / 中文解读

中文解读待补充：本站会优先为端到端自动驾驶、BEV感知、3D目标检测、轨迹预测、路径规划、LiDAR感知等高价值论文补充中文说明。

Original abstract

Autonomous vehicle (AV) technologies have advanced rapidly in recent years, driving an increasing demand for large-scale, high-quality annotated data. However, collecting and annotating real-world pedestrian video datasets is time-consuming, costly, and often insufficient to cover rare and safety-critical scenarios. Recent world foundation models have demonstrated impressive capabilities in generating realistic videos, yet their suitability for safety-critical autonomous driving applications remains largely unexplored. In this work, we investigate whether current world foundation models can generate driving scenarios that are sufficiently realistic and behaviourally consistent for autonomous driving research. We conduct a case study centred on pedestrian–vehicle interactions captured from ego-vehicle dashcam viewpoints, where subtle behavioural and geometric errors can have significant safety implications. To support this investigation, we develop SynPeDAS, an open research framework comprising a collection of synthetic pedestrian-interaction videos, a reusable generation pipeline for transforming real-world driving footage into synthetic scenarios, an automated evaluation suite, and downstream demonstration code. Through quantitative evaluation and structured human assessment, we identify several recurring failure modes, including dynamic misalignment, depth drift, and object persistence inconsistencies. More importantly, we find that commonly used evaluation metrics frequently exhibit ceiling effects and weak alignment with human judgement, limiting their ability to detect safety-critical behavioural errors. These findings indicate that, despite high perceptual realism at the frame level, current generative world models and existing evaluation methodologies remain insufficient for capturing physically grounded motion and task-critical semantics. Consequently, significant challenges remain before world model-generated videos can be considered reliable for safety-critical autonomous driving applications. SynPeDAS provides an open platform for systematically studying these challenges and developing improved generation and evaluation methods.

Links and sources

• PDF from original source

Comments