Multi-Modal Retrieval For Large Language Model Based Speech Recognition
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913390811676672 |
|---|---|
| author | Kolehmainen, Jari Gourav, Aditya Shivakumar, Prashanth Gurunath Gu, Yile Gandhe, Ankur Rastrow, Ariya Strimel, Grant Bulyko, Ivan |
| author_facet | Kolehmainen, Jari Gourav, Aditya Shivakumar, Prashanth Gurunath Gu, Yile Gandhe, Ankur Rastrow, Ariya Strimel, Grant Bulyko, Ivan |
| contents | Retrieval is a widely adopted approach for improving language models leveraging external information. As the field moves towards multi-modal large language models, it is important to extend the pure text based methods to incorporate other modalities in retrieval as well for applications across the wide spectrum of machine learning tasks and data types. In this work, we propose multi-modal retrieval with two approaches: kNN-LM and cross-attention techniques. We demonstrate the effectiveness of our retrieval approaches empirically by applying them to automatic speech recognition tasks with access to external information. Under this setting, we show that speech-based multi-modal retrieval outperforms text based retrieval, and yields up to 50 % improvement in word error rate over the multi-modal language model baseline. Furthermore, we achieve state-of-the-art recognition results on the Spoken-Squad question answering dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_09618 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multi-Modal Retrieval For Large Language Model Based Speech Recognition Kolehmainen, Jari Gourav, Aditya Shivakumar, Prashanth Gurunath Gu, Yile Gandhe, Ankur Rastrow, Ariya Strimel, Grant Bulyko, Ivan Computation and Language Artificial Intelligence Information Retrieval Sound Audio and Speech Processing Retrieval is a widely adopted approach for improving language models leveraging external information. As the field moves towards multi-modal large language models, it is important to extend the pure text based methods to incorporate other modalities in retrieval as well for applications across the wide spectrum of machine learning tasks and data types. In this work, we propose multi-modal retrieval with two approaches: kNN-LM and cross-attention techniques. We demonstrate the effectiveness of our retrieval approaches empirically by applying them to automatic speech recognition tasks with access to external information. Under this setting, we show that speech-based multi-modal retrieval outperforms text based retrieval, and yields up to 50 % improvement in word error rate over the multi-modal language model baseline. Furthermore, we achieve state-of-the-art recognition results on the Spoken-Squad question answering dataset. |
| title | Multi-Modal Retrieval For Large Language Model Based Speech Recognition |
| topic | Computation and Language Artificial Intelligence Information Retrieval Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.09618 |