Intelligent Autopilot Agent based on Neural Network Technologies in an Autonomous Driving Simulation Environment

Engineering notes

Key topics: autonomous driving system, autonomous driving, autonomous vehicle, end-to-end, panoptic segmentation, carla, control. See the paper for implementation details and experimental results.

Chinese explanation / 中文解读

中文解读待补充：本站会优先为端到端自动驾驶、BEV感知、3D目标检测、轨迹预测、路径规划、LiDAR感知等高价值论文补充中文说明。

Original abstract

The paper presents the architecture of an intelligent agent for end-to-end autonomous vehicle control based on the M3Drive neural network and multimodal data fusion in the CARLA simulation environment. The proposed approach combines visual features (depth maps and panoptic segmentation for reconstructing 3D geometry and scene semantics), navigation commands and spatial coordinates of route control points, as well as time series of agent states (history of control actions and critical event indicators). The model architecture includes specialized encoders for each data type: EfficientNet-lite0 for visual modalities followed by cross-attention fusion, bidirectional LSTM with an attention mechanism for processing temporal context, and a gating module for tabular features. The combined representations are fed into a multilayer perceptron with residual blocks and multi-task output "heads" that predict continuous control actions: steering angle, acceleration, and braking. Training was conducted in two stages with a weighted loss function. In the second stage, the model was fine-tuned using scenarios involving turns and reactions to traffic lights, which reduced steering errors by more than half. Experimental results at the independent test site confirmed the effectiveness of the system: a route completion rate (RC) of 89 % and an integral driving score (DS) of 67.6 were achieved at a high performance of 58 FPS. The results confirm the applicability of the proposed architecture for creating lightweight real-time autonomous driving systems and lay the foundation for transfer to the Mitsubishi i-MiEV physical platform.

Links and sources

• Official / arXiv page

Comments