Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910899035439104 |
|---|---|
| author | Chowdhury, Sanjoy Gani, Hanan Anand, Nishit Nag, Sayan Gao, Ruohan Elhoseiny, Mohamed Khan, Salman Manocha, Dinesh |
| author_facet | Chowdhury, Sanjoy Gani, Hanan Anand, Nishit Nag, Sayan Gao, Ruohan Elhoseiny, Mohamed Khan, Salman Manocha, Dinesh |
| contents | Recent advancements in reasoning optimization have greatly enhanced the performance of large language models (LLMs). However, existing work fails to address the complexities of audio-visual scenarios, underscoring the need for further research. In this paper, we introduce AURELIA, a novel actor-critic based audio-visual (AV) reasoning framework that distills structured, step-by-step reasoning into AVLLMs at test time, improving their ability to process complex multi-modal inputs without additional training or fine-tuning. To further advance AVLLM reasoning skills, we present AVReasonBench, a challenging benchmark comprising 4500 audio-visual questions, each paired with detailed step-by-step reasoning. Our benchmark spans six distinct tasks, including AV-GeoIQ, which evaluates AV reasoning combined with geographical and cultural knowledge. Evaluating 18 AVLLMs on AVReasonBench reveals significant limitations in their multi-modal reasoning capabilities. Using AURELIA, we achieve up to a 100% relative improvement, demonstrating its effectiveness. This performance gain highlights the potential of reasoning-enhanced data generation for advancing AVLLMs in real-world applications. Our code and data will be publicly released at: https: //github.com/schowdhury671/aurelia. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_23219 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMs Chowdhury, Sanjoy Gani, Hanan Anand, Nishit Nag, Sayan Gao, Ruohan Elhoseiny, Mohamed Khan, Salman Manocha, Dinesh Audio and Speech Processing Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Recent advancements in reasoning optimization have greatly enhanced the performance of large language models (LLMs). However, existing work fails to address the complexities of audio-visual scenarios, underscoring the need for further research. In this paper, we introduce AURELIA, a novel actor-critic based audio-visual (AV) reasoning framework that distills structured, step-by-step reasoning into AVLLMs at test time, improving their ability to process complex multi-modal inputs without additional training or fine-tuning. To further advance AVLLM reasoning skills, we present AVReasonBench, a challenging benchmark comprising 4500 audio-visual questions, each paired with detailed step-by-step reasoning. Our benchmark spans six distinct tasks, including AV-GeoIQ, which evaluates AV reasoning combined with geographical and cultural knowledge. Evaluating 18 AVLLMs on AVReasonBench reveals significant limitations in their multi-modal reasoning capabilities. Using AURELIA, we achieve up to a 100% relative improvement, demonstrating its effectiveness. This performance gain highlights the potential of reasoning-enhanced data generation for advancing AVLLMs in real-world applications. Our code and data will be publicly released at: https: //github.com/schowdhury671/aurelia. |
| title | Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMs |
| topic | Audio and Speech Processing Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2503.23219 |