RM-MLLM: Robust Scene Understanding in Adverse Weather with RGB-Multispectral Fusion

Engineering notes

First, we introduce DENSE-QA, a question-answering benchmark derived from the DENSE adverse-weather dataset, comprising 12,997 scenes and over 77,000 instruction-response pairs, built on synchronized RGB and multispectral imagery. Across the evaluated models, RM-MLLM achieves the highest observed average accuracy on DENSE-QA (72.18%), compared with 71.82% for the strongest fine-tuned RGB baseline.

Chinese explanation / 中文解读

中文解读待补充：本站会优先为端到端自动驾驶、BEV感知、3D目标检测、轨迹预测、路径规划、LiDAR感知等高价值论文补充中文说明。

Original abstract

Multimodal large language models (MLLMs) for autonomous driving often become less reliable in adverse weather, partly because training data is dominated by clear, well-lit scenes, and most models primarily rely on RGB imagery. We address these issues with two contributions. First, we introduce DENSE-QA, a question-answering benchmark derived from the DENSE adverse-weather dataset, comprising 12,997 scenes and over 77,000 instruction-response pairs, built on synchronized RGB and multispectral imagery. Second, we propose RM-MLLM (RGB–Multispectral Multimodal Large Language Model), an MLLM architecture with modality-specific encoders and an adaptive fusion module, trained with a three-stage progressive strategy to better integrate complementary sensor cues. Across multiple LLM backbones, RM-MLLM improves QA performance under adverse conditions. Across the evaluated models, RM-MLLM achieves the highest observed average accuracy on DENSE-QA (72.18%), compared with 71.82% for the strongest fine-tuned RGB baseline. The RGB-only variant exceeds the best zero-shot or prompt-augmented baseline by 10.94 percentage points. In the controlled within-architecture ablation, adding the gated near-infrared (NIR) stream improves the six-task average from 68.58% to 72.18%, with the largest gain on weather classification (+7.23 points), followed by infrastructure classification (+4.23 points) and road-surface classification (+4.08 points). Across the five classification tasks, the mean macro-F1 increases by 10.37 points relative to the RGB-only variant. We release the benchmark and models to support future all-weather autonomous driving research https://yonsei-stl.github.io/RM-MLLM/ .

Links and sources

• PDF from original source

Comments