FSNet: Redesign Self-Supervised MonoDepth for Full-Scale Depth Prediction for Autonomous Driving

Engineering notes

Key topics: autonomous driving, monocular depth, nuscenes, kitti, deployment, prediction. See the paper for implementation details and experimental results.

Chinese explanation / 中文解读

中文解读待补充：本站会优先为端到端自动驾驶、BEV感知、3D目标检测、轨迹预测、路径规划、LiDAR感知等高价值论文补充中文说明。

Original abstract

Predicting accurate depth with monocular images is important for low-cost robotic applications and autonomous driving. This study proposes a comprehensive self-supervised framework for accurate scale-aware depth prediction on autonomous driving scenes utilizing inter-frame poses obtained from inertial measurements. In particular, we introduce a Full-Scale depth prediction network named FSNet. FSNet contains four important improvements over existing self-supervised models: (1) a multichannel output representation for stable training of depth prediction in driving scenarios, (2) an optical-flow-based mask designed for dynamic object removal, (3) a self-distillation training strategy to augment the training process, and (4) an optimization-based post-processing algorithm in test time, fusing the results from visual odometry. With this framework, robots and vehicles with only one well-calibrated camera can collect sequences of training image frames and camera poses, and infer accurate 3D depths of the environment without extra labeling work or 3D data. Extensive experiments on the KITTI dataset, KITTI-360 dataset and the nuScenes dataset demonstrate the potential of FSNet. More visualizations are presented in https://sites.google.com/view/fsnet/home Note to Practitioners—This paper was motivated by the problem of unsupervised monocular depth for robotic deployment. We notice that PoseNet is not generalizable and by nature monodepth2 only predict depths up to a scale. We believe that we should not expect PoseNet, a ResNet on a concatenation of two images, to produce more reliable poses than the localization module in a robot. So we try our best to completely avoid using PoseNet. This creates much unstability in training, but we managed to fix it in FSNet with multichannel output and self-distillation. We also believe the network should try to directly predict accurate depth with a correct scale at any cases. So our method could produce meaningful results on static frames or scenes with little/no VO points (same as the network’s direct prediction). There are images without VO points in our multi-frame experiment, but our method is robust enough to fix this problem. In future research, we will include multi-frame depth predictions for more accurate depth prediction.

Links and sources

• Official / arXiv page
• PDF from original source

Comments