When One Modality Sabotages the Others: A Diagnostic Lens on Multimodal Reasoning
2511.02794v1
cs.AI, cs.MA
2025-11-06
Авторы:
Chenyu Zhang, Minsol Kim, Shohreh Ghorbani, Jingyao Wu, Rosalind Picard, Patricia Maes, Paul Pu Liang
Abstract
Despite rapid growth in multimodal large language models (MLLMs), their
reasoning traces remain opaque: it is often unclear which modality drives a
prediction, how conflicts are resolved, or when one stream dominates. In this
paper, we introduce modality sabotage, a diagnostic failure mode in which a
high-confidence unimodal error overrides other evidence and misleads the fused
result. To analyze such dynamics, we propose a lightweight, model-agnostic
evaluation layer that treats each modality as an agent, producing candidate
labels and a brief self-assessment used for auditing. A simple fusion mechanism
aggregates these outputs, exposing contributors (modalities supporting correct
outcomes) and saboteurs (modalities that mislead). Applying our diagnostic
layer in a case study on multimodal emotion recognition benchmarks with
foundation models revealed systematic reliability profiles, providing insight
into whether failures may arise from dataset artifacts or model limitations.
More broadly, our framework offers a diagnostic scaffold for multimodal
reasoning, supporting principled auditing of fusion dynamics and informing
possible interventions.
Ссылки и действия
Дополнительные ресурсы: