Enhancing Photorealism in Carla Autonomous Driving Simulator

Engineering notes

In this work, we extend the applicability of CARLA2Real 11https://github.com/stefanos50/CARLA2Real-a previously developed publicly available tool for enhancing CARLA's visual realism-to the latest Unreal Engine 5 version of the simulator, which already features improved rendering capabilities. Our approach utilizes a state-of-the-art image-to-image translation method to generate photorealism-enhanced autonomous driving-related visual data that target the characteristics of the real-world datasets, Cityscapes and KITTI.

Chinese explanation / 中文解读

中文解读待补充：本站会优先为端到端自动驾驶、BEV感知、3D目标检测、轨迹预测、路径规划、LiDAR感知等高价值论文补充中文说明。

Original abstract

Simulators are essential in autonomous systems research, providing a controlled test environment for self-driving vehicles, autonomous robots, and unmanned aerial vehicles. Despite recent significant improvements in simulation realism, the noticeable gap between the simulation and the real-world complexities persists, hindering the direct applicability of algorithms trained on simulated data to real-world scenarios. In this work, we extend the applicability of CARLA2Real 11https://github.com/stefanos50/CARLA2Real-a previously developed publicly available tool for enhancing CARLA's visual realism-to the latest Unreal Engine 5 version of the simulator, which already features improved rendering capabilities. Our approach utilizes a state-of-the-art image-to-image translation method to generate photorealism-enhanced autonomous driving-related visual data that target the characteristics of the real-world datasets, Cityscapes and KITTI. Based on this, we generated synthetic datasets from both the simulator and the photorealism enhancement model outputs, including their corresponding ground truth annotations for semantic segmentation. Subsequently, by employing the photorealism-improved synthetic data as training data, we conducted experiments to assess how the proposed approach affected the accuracy of a semantic segmentation approach. The findings illustrated that, although Unreal Engine 5 improves the baseline realism of CARLA, a sim2real gap still persists in various aspects of the scenes. However, our method significantly reduces this gap, leading to improved segmentation performance on real-world data.

Links and sources

• Official / arXiv page

Comments