Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915295988285440 |
|---|---|
| author | Cappellazzo, Umberto Kim, Minsu Petridis, Stavros Falavigna, Daniele Brutti, Alessio |
| author_facet | Cappellazzo, Umberto Kim, Minsu Petridis, Stavros Falavigna, Daniele Brutti, Alessio |
| contents | Audio-Visual Speech Recognition (AVSR) enhances robustness in noisy environments by integrating visual cues. While recent advances integrate Large Language Models (LLMs) into AVSR, their high computational cost hinders deployment in resource-constrained settings. To address this, we propose Llama-SMoP, an efficient Multimodal LLM that employs a Sparse Mixture of Projectors (SMoP) module to scale model capacity without increasing inference costs. By incorporating sparsely-gated mixture-of-experts (MoE) projectors, Llama-SMoP enables the use of smaller LLMs while maintaining strong performance. We explore three SMoP configurations and show that Llama-SMoP DEDR (Disjoint-Experts, Disjoint-Routers), which uses modality-specific routers and experts, achieves superior performance on ASR, VSR, and AVSR tasks. Ablation studies confirm its effectiveness in expert activation, scalability, and noise robustness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_14336 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach Cappellazzo, Umberto Kim, Minsu Petridis, Stavros Falavigna, Daniele Brutti, Alessio Audio and Speech Processing Computer Vision and Pattern Recognition Multimedia Sound Audio-Visual Speech Recognition (AVSR) enhances robustness in noisy environments by integrating visual cues. While recent advances integrate Large Language Models (LLMs) into AVSR, their high computational cost hinders deployment in resource-constrained settings. To address this, we propose Llama-SMoP, an efficient Multimodal LLM that employs a Sparse Mixture of Projectors (SMoP) module to scale model capacity without increasing inference costs. By incorporating sparsely-gated mixture-of-experts (MoE) projectors, Llama-SMoP enables the use of smaller LLMs while maintaining strong performance. We explore three SMoP configurations and show that Llama-SMoP DEDR (Disjoint-Experts, Disjoint-Routers), which uses modality-specific routers and experts, achieves superior performance on ASR, VSR, and AVSR tasks. Ablation studies confirm its effectiveness in expert activation, scalability, and noise robustness. |
| title | Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach |
| topic | Audio and Speech Processing Computer Vision and Pattern Recognition Multimedia Sound |
| url | https://arxiv.org/abs/2505.14336 |