A Multimodal Trustworthy Joint Perception Prediction Model for Autonomous Driving

Engineering notes

Key topics: autonomous driving, bev, nuscenes, kitti, perception, prediction. See the paper for implementation details and experimental results.

Chinese explanation / 中文解读

中文解读待补充：本站会优先为端到端自动驾驶、BEV感知、3D目标检测、轨迹预测、路径规划、LiDAR感知等高价值论文补充中文说明。

Original abstract

The emergence of vision-focused joint perception and prediction (PnP) marks a novel trend in the field of autonomous driving research. It predicts the future states of traffic participants within the surrounding environment from perceptual data. However, perception from a single vehicle is insufficient to obtain more accurate environmental information; therefore, the fusion of perception data from different sources and modalities becomes increasingly crucial for processing and predicting environmental data. To this end, this paper proposes a novel multimodal trustworthiness fusion and prediction model. First, we introduce a Bird's-Eye View (BEV) encoder that is synchronized with poses and based on multimodal data. This encoder is capable of projecting raw image inputs from any modality camera, captured at any pose and time, into a shared, synchronized BEV space, thereby enhancing spatiotemporal synchronization. Second, we present a trustworthy Spatial-Temporal Pyramid Transform (TSTPT), which is designed to comprehensively extract multiscale features from BEV and forecast future BEV states, leveraging spatial priors. A comprehensive series of experiments conducted on the KITTI and nuScenes datasets demonstrate that the proposed model is overall feasible more reliable and safe compared to existing vision-based prediction methods.

Links and sources

• Official / arXiv page

Comments