From Spatial to Actions: Grounding Vision-Language-Action Model in Spatial Foundation Priors
2510.17439v1
cs.RO, cs.AI, cs.CV, cs.LG
2025-10-22
Авторы:
Zhengshen Zhang, Hao Li, Yalun Dai, Zhengbang Zhu, Lei Zhou, Chenchen Liu, Dong Wang, Francis E. H. Tay, Sijin Chen, Ziwei Liu, Yuxiao Liu, Xinghang Li, Pan Zhou
Abstract
Existing vision-language-action (VLA) models act in 3D real-world but are
typically built on 2D encoders, leaving a spatial reasoning gap that limits
generalization and adaptability. Recent 3D integration techniques for VLAs
either require specialized sensors and transfer poorly across modalities, or
inject weak cues that lack geometry and degrade vision-language alignment. In
this work, we introduce FALCON (From Spatial to Action), a novel paradigm that
injects rich 3D spatial tokens into the action head. FALCON leverages spatial
foundation models to deliver strong geometric priors from RGB alone, and
includes an Embodied Spatial Model that can optionally fuse depth, or pose for
higher fidelity when available, without retraining or architectural changes. To
preserve language reasoning, spatial tokens are consumed by a Spatial-Enhanced
Action Head rather than being concatenated into the vision-language backbone.
These designs enable FALCON to address limitations in spatial representation,
modality transferability, and alignment. In comprehensive evaluations across
three simulation benchmarks and eleven real-world tasks, our proposed FALCON
achieves state-of-the-art performance, consistently surpasses competitive
baselines, and remains robust under clutter, spatial-prompt conditioning, and
variations in object scale and height.