Agent-Omni: Test-Time Multimodal Reasoning via Model Coordination for Understanding Anything
2511.02834v2
cs.AI, cs.CL, cs.LG
2025-11-06
Авторы:
Huawei Lin, Yunzhi Shi, Tong Geng, Weijie Zhao, Wei Wang, Ravender Pal Singh
Abstract
Multimodal large language models (MLLMs) have shown strong capabilities but
remain limited to fixed modality pairs and require costly fine-tuning with
large aligned datasets. Building fully omni-capable models that can integrate
text, images, audio, and video remains impractical and lacks robust reasoning
support. In this paper, we propose an Agent-Omni framework that coordinates
existing foundation models through a master-agent system, enabling flexible
multimodal reasoning without retraining. The master agent interprets user
intent, delegates subtasks to modality-specific agents, and integrates their
outputs into coherent responses. Extensive experiments across text, image,
audio, video, and omni benchmarks show that Agent-Omni consistently achieves
state-of-the-art performance, particularly on tasks requiring complex
cross-modal reasoning. Its agent-based design enables seamless integration of
specialized foundation models, ensuring adaptability to diverse inputs while
maintaining transparency and interpretability. In addition, the framework is
modular and easily extensible, allowing future improvements as stronger models
become available.
Ссылки и действия
Дополнительные ресурсы: