Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views
2510.18632v1
cs.CV, cs.AI, I.2.10
2025-10-23
Авторы:
Zhangquan Chen, Manyuan Zhang, Xinlei Yu, Xufang Luo, Mingze Sun, Zihao Pan, Yan Feng, Peng Pei, Xunliang Cai, Ruqi Huang
Abstract
Though recent advances in vision-language models (VLMs) have achieved
remarkable progress across a wide range of multimodal tasks, understanding 3D
spatial relationships from limited views remains a significant challenge.
Previous reasoning methods typically rely on pure text (e.g., topological
cognitive maps) or on 2D visual cues. However, their limited representational
capacity hinders performance in specific tasks that require 3D spatial
imagination. To address this limitation, we propose 3DThinker, a framework that
can effectively exploits the rich geometric information embedded within images
while reasoning, like humans do. Our framework is the first to enable 3D
mentaling during reasoning without any 3D prior input, and it does not rely on
explicitly labeled 3D data for training. Specifically, our training consists of
two stages. First, we perform supervised training to align the 3D latent
generated by VLM while reasoning with that of a 3D foundation model (e.g.,
VGGT). Then, we optimize the entire reasoning trajectory solely based on
outcome signals, thereby refining the underlying 3D mentaling. Extensive
experiments across multiple benchmarks show that 3DThinker consistently
outperforms strong baselines and offers a new perspective toward unifying 3D
representations into multimodal reasoning. Our code will be available at
https://github.com/zhangquanchen/3DThinker.
Ссылки и действия
Дополнительные ресурсы: