ALAS: Measuring Latent Speech-Text Alignment For Spoken Language Understanding In Multimodal LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911037856415744 |
|---|---|
| author | Mousavi, Pooneh Wang, Yingzhi Ravanelli, Mirco Subakan, Cem |
| author_facet | Mousavi, Pooneh Wang, Yingzhi Ravanelli, Mirco Subakan, Cem |
| contents | Large Language Models (LLMs) are increasingly used in Spoken Language Understanding (SLU), where effective multimodal learning depends on the alignment between audio and text. Despite various fusion methods, no standard metric exists to assess this alignment. This work introduces ALAS (Automatic Latent Alignment Score), a metric that evaluates alignment by measuring correlations between audio and text representations across transformer layers. Experiments on Spoken Question Answering and Emotion Recognition show that ALAS captures meaningful patterns across tasks and layers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_19937 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ALAS: Measuring Latent Speech-Text Alignment For Spoken Language Understanding In Multimodal LLMs Mousavi, Pooneh Wang, Yingzhi Ravanelli, Mirco Subakan, Cem Computation and Language Sound Audio and Speech Processing Large Language Models (LLMs) are increasingly used in Spoken Language Understanding (SLU), where effective multimodal learning depends on the alignment between audio and text. Despite various fusion methods, no standard metric exists to assess this alignment. This work introduces ALAS (Automatic Latent Alignment Score), a metric that evaluates alignment by measuring correlations between audio and text representations across transformer layers. Experiments on Spoken Question Answering and Emotion Recognition show that ALAS captures meaningful patterns across tasks and layers. |
| title | ALAS: Measuring Latent Speech-Text Alignment For Spoken Language Understanding In Multimodal LLMs |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2505.19937 |