More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models
2510.04532v1
cs.AI, cs.CL, cs.RO
2025-10-08
Авторы:
Xurui Song, Shuo Huai, JingJing Jiang, Jiayi Kong, Jun Luo
Abstract
Vision-Language Model (VLM) driving agents promise explainable end-to-end
autonomy by first producing natural-language reasoning and then predicting
trajectory planning. However, whether planning is causally driven by this
reasoning remains a critical but unverified assumption. To investigate this, we
build DriveMind, a large-scale driving Visual Question Answering (VQA) corpus
with plan-aligned Chain-of-Thought (CoT), automatically generated from nuPlan.
Our data generation process converts sensors and annotations into structured
inputs and, crucially, separates priors from to-be-reasoned signals, enabling
clean information ablations. Using DriveMind, we train representative VLM
agents with Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization
(GRPO) and evaluate them with nuPlan's metrics. Our results, unfortunately,
indicate a consistent causal disconnect in reasoning-planning: removing
ego/navigation priors causes large drops in planning scores, whereas removing
CoT produces only minor changes. Attention analysis further shows that planning
primarily focuses on priors rather than the CoT. Based on this evidence, we
propose the Reasoning-Planning Decoupling Hypothesis, positing that the
training-yielded reasoning is an ancillary byproduct rather than a causal
mediator. To enable efficient diagnosis, we also introduce a novel,
training-free probe that measures an agent's reliance on priors by evaluating
its planning robustness against minor input perturbations. In summary, we
provide the community with a new dataset and a diagnostic tool to evaluate the
causal fidelity of future models.
Ссылки и действия
Дополнительные ресурсы: