Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2512.16250 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908719962390528 |
|---|---|
| author | Chowdhury, Sanjoy Yang, Karren D. Liu, Xudong Faghri, Fartash Vasu, Pavan Kumar Anasosalu Tuzel, Oncel Manocha, Dinesh Li, Chun-Liang Vemulapalli, Raviteja |
| author_facet | Chowdhury, Sanjoy Yang, Karren D. Liu, Xudong Faghri, Fartash Vasu, Pavan Kumar Anasosalu Tuzel, Oncel Manocha, Dinesh Li, Chun-Liang Vemulapalli, Raviteja |
| contents | Recent multimodal large language models (MLLMs) such as GPT-4o and Qwen3-Omni show strong perception but struggle in multi-speaker, dialogue-centric settings that demand agentic reasoning tracking who speaks, maintaining roles, and grounding events across time. These scenarios are central to multimodal audio-video understanding, where models must jointly reason over audio and visual streams in applications such as conversational video assistants and meeting analytics. We introduce AMUSE, a benchmark designed around tasks that are inherently agentic, requiring models to decompose complex audio-visual interactions into planning, grounding, and reflection steps. It evaluates MLLMs across three modes zero-shot, guided, and agentic and six task families, including spatio-temporal speaker grounding and multimodal dialogue summarization. Across all modes, current models exhibit weak multi-speaker reasoning and inconsistent behavior under both non-agentic and agentic evaluation. Motivated by the inherently agentic nature of these tasks and recent advances in LLM agents, we propose RAFT, a data-efficient agentic alignment framework that integrates reward optimization with intrinsic multimodal self-evaluation as reward and selective parameter adaptation for data and parameter efficient updates. Using RAFT, we achieve up to 39.52\% relative improvement in accuracy on our benchmark. Together, AMUSE and RAFT provide a practical platform for examining agentic reasoning in multimodal models and improving their capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_16250 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understanding Chowdhury, Sanjoy Yang, Karren D. Liu, Xudong Faghri, Fartash Vasu, Pavan Kumar Anasosalu Tuzel, Oncel Manocha, Dinesh Li, Chun-Liang Vemulapalli, Raviteja Artificial Intelligence Multiagent Systems Recent multimodal large language models (MLLMs) such as GPT-4o and Qwen3-Omni show strong perception but struggle in multi-speaker, dialogue-centric settings that demand agentic reasoning tracking who speaks, maintaining roles, and grounding events across time. These scenarios are central to multimodal audio-video understanding, where models must jointly reason over audio and visual streams in applications such as conversational video assistants and meeting analytics. We introduce AMUSE, a benchmark designed around tasks that are inherently agentic, requiring models to decompose complex audio-visual interactions into planning, grounding, and reflection steps. It evaluates MLLMs across three modes zero-shot, guided, and agentic and six task families, including spatio-temporal speaker grounding and multimodal dialogue summarization. Across all modes, current models exhibit weak multi-speaker reasoning and inconsistent behavior under both non-agentic and agentic evaluation. Motivated by the inherently agentic nature of these tasks and recent advances in LLM agents, we propose RAFT, a data-efficient agentic alignment framework that integrates reward optimization with intrinsic multimodal self-evaluation as reward and selective parameter adaptation for data and parameter efficient updates. Using RAFT, we achieve up to 39.52\% relative improvement in accuracy on our benchmark. Together, AMUSE and RAFT provide a practical platform for examining agentic reasoning in multimodal models and improving their capabilities. |
| title | AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understanding |
| topic | Artificial Intelligence Multiagent Systems |
| url | https://arxiv.org/abs/2512.16250 |