Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2604.21928 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916063443156992 |
|---|---|
| author | Bañeras-Roux, Thibault Kumar, Shashi Khalil, Driss Burdisso, Sergio Motlicek, Petr Liu, Shiran Rouvier, Mickael Wottawa, Jane Dufour, Richard |
| author_facet | Bañeras-Roux, Thibault Kumar, Shashi Khalil, Driss Burdisso, Sergio Motlicek, Petr Liu, Shiran Rouvier, Mickael Wottawa, Jane Dufour, Richard |
| contents | Automatic Speech Recognition (ASR) is traditionally evaluated using Word Error Rate (WER), a metric that is insensitive to meaning. Embedding-based semantic metrics are better correlated with human perception, but decoder-based Large Language Models (LLMs) remain underexplored for this task. This paper evaluates their relevance through three approaches: (1) selecting the best hypothesis between two candidates, (2) computing semantic distance using generative embeddings, and (3) qualitative classification of errors. On the HATS dataset, the best LLMs achieve 92--94\% agreement with human annotators for hypothesis selection, compared to 63\% for WER, also outperforming semantic metrics. Embeddings from decoder-based LLMs show performance comparable to encoder models. Finally, LLMs offer a promising direction for interpretable and semantic ASR evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_21928 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Evaluation of Automatic Speech Recognition Using Generative Large Language Models Bañeras-Roux, Thibault Kumar, Shashi Khalil, Driss Burdisso, Sergio Motlicek, Petr Liu, Shiran Rouvier, Mickael Wottawa, Jane Dufour, Richard Computation and Language Automatic Speech Recognition (ASR) is traditionally evaluated using Word Error Rate (WER), a metric that is insensitive to meaning. Embedding-based semantic metrics are better correlated with human perception, but decoder-based Large Language Models (LLMs) remain underexplored for this task. This paper evaluates their relevance through three approaches: (1) selecting the best hypothesis between two candidates, (2) computing semantic distance using generative embeddings, and (3) qualitative classification of errors. On the HATS dataset, the best LLMs achieve 92--94\% agreement with human annotators for hypothesis selection, compared to 63\% for WER, also outperforming semantic metrics. Embeddings from decoder-based LLMs show performance comparable to encoder models. Finally, LLMs offer a promising direction for interpretable and semantic ASR evaluation. |
| title | Evaluation of Automatic Speech Recognition Using Generative Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.21928 |