Discrimination Is Not Timeliness: Benchmarking Surrogate Safety Metrics Against Ground-Truth Collisions in Monte Carlo Microsimulation

Engineering notes

I then score PORA, inverse time-to-collision, and a time-to-stop margin on a common benchmark: 60 seeded SUMO runs of a deliberately crash-prone priority intersection, where 1,134 logged collisions provide ground-truth labels for 258,770 vehicle-time samples. Implementation: github.com/chenggma/pora-replication.

Chinese explanation / 中文解读

中文解读待补充：本站会优先为端到端自动驾驶、BEV感知、3D目标检测、轨迹预测、路径规划、LiDAR感知等高价值论文补充中文说明。

Original abstract

Surrogate safety metrics for autonomous driving are usually compared on their ability to separate dangerous situations from safe ones. Whether the best-separating metric also warns early enough to act on is a different question, and this note measures the gap between the two. I first re-implement the recently proposed PORA collision-risk metric independently from its arXiv text (arXiv:2501.16480), documenting five quantities the paper leaves unspecified. I then score PORA, inverse time-to-collision, and a time-to-stop margin on a common benchmark: 60 seeded SUMO runs of a deliberately crash-prone priority intersection, where 1,134 logged collisions provide ground-truth labels for 258,770 vehicle-time samples. All three metrics see identical kinematic information. PORA separates best (AUROC 0.739 against 0.711 and 0.706) and, at a fixed 5% false-alarm rate, warns the most collision-bound vehicles (1,659). The same threshold, however, fires latest: median warning lead of 1.0 s, against 4.0 s for the time-to-stop margin, which warns roughly half as many vehicles (902). Discrimination and timeliness rank the metrics in opposite orders. AUROC alone is therefore not a sufficient statistic for choosing a safety metric; the choice depends on whether a deployment values coverage or lead time. Code, seeds, and data are public and the full experiment is regenerable from a pinned simulator version. Implementation: github.com/chenggma/pora-replication. Benchmark: github.com/chenggma/risk-metric-bench.

Links and sources

Comments