Terra: Explorable Native 3D World Model with Point Latents
2510.14977v1
cs.CV, cs.AI, cs.LG
2025-10-18
Авторы:
Yuanhui Huang, Weiliang Chen, Wenzhao Zheng, Xin Tao, Pengfei Wan, Jie Zhou, Jiwen Lu
Abstract
World models have garnered increasing attention for comprehensive modeling of
the real world. However, most existing methods still rely on pixel-aligned
representations as the basis for world evolution, neglecting the inherent 3D
nature of the physical world. This could undermine the 3D consistency and
diminish the modeling efficiency of world models. In this paper, we present
Terra, a native 3D world model that represents and generates explorable
environments in an intrinsic 3D latent space. Specifically, we propose a novel
point-to-Gaussian variational autoencoder (P2G-VAE) that encodes 3D inputs into
a latent point representation, which is subsequently decoded as 3D Gaussian
primitives to jointly model geometry and appearance. We then introduce a sparse
point flow matching network (SPFlow) for generating the latent point
representation, which simultaneously denoises the positions and features of the
point latents. Our Terra enables exact multi-view consistency with native 3D
representation and architecture, and supports flexible rendering from any
viewpoint with only a single generation process. Furthermore, Terra achieves
explorable world modeling through progressive generation in the point latent
space. We conduct extensive experiments on the challenging indoor scenes from
ScanNet v2. Terra achieves state-of-the-art performance in both reconstruction
and generation with high 3D consistency.
Ссылки и действия
Дополнительные ресурсы: