GPFusion: enhancing multi-modal 3D object detection via geometric pseudo-image fusion and adaptive feature fusion

Engineering notes

Key topics: autonomous driving, bev, 3d object detection, depth estimation, object detection, lidar, point cloud, nuscenes. See the paper for implementation details and experimental results.

Chinese explanation / 中文解读

中文解读待补充：本站会优先为端到端自动驾驶、BEV感知、3D目标检测、轨迹预测、路径规划、LiDAR感知等高价值论文补充中文说明。

Original abstract

With the rapid progress of autonomous driving technologies, camera–LiDAR-based multi-modal 3D object detection has attracted increasing interest. BEV-based approaches have become a prevalent paradigm, in which image features are lifted into Bird’s Eye View representations via depth estimation and subsequently fused with LiDAR BEV features. Nevertheless, errors in depth estimation and inefficient fusion mechanisms remain major bottlenecks for detection accuracy. To address these limitations, we introduce a novel multi-modal fusion network for 3D object detection, consisting of two principal components: Geometric Pseudo-Image Feature Fusion and the Feature Fusion Encoder. In particular, the Geometric Pseudo-Image Fusion module embeds LiDAR-derived 3D structural information into image features, enabling camera representations to retain reliable geometric cues (derived from the pixel-wise alignment between LiDAR point clouds and images) even in the presence of depth estimation inaccuracies. Furthermore, during the transformation of image features into BEV space, sparse point cloud depth is incorporated as a prior to enhance depth estimation quality. The Feature Fusion Encoder dynamically regulates the contributions of both modalities, alleviates cross-modal feature misalignment, and adopts adaptive convolutional kernels to better handle objects of varying scales. Additionally, temporal information is leveraged to improve the detection of occluded objects. Extensive experiments conducted on the nuScenes dataset validate the effectiveness and robustness of the proposed method.

Links and sources

Comments