Speech Recognition Rescoring with Large Speech-Text Foundation Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909325353549824 |
|---|---|
| author | Shivakumar, Prashanth Gurunath Kolehmainen, Jari Gourav, Aditya Gu, Yi Gandhe, Ankur Rastrow, Ariya Bulyko, Ivan |
| author_facet | Shivakumar, Prashanth Gurunath Kolehmainen, Jari Gourav, Aditya Gu, Yi Gandhe, Ankur Rastrow, Ariya Bulyko, Ivan |
| contents | Large language models (LLM) have demonstrated the ability to understand human language by leveraging large amount of text data. Automatic speech recognition (ASR) systems are often limited by available transcribed speech data and benefit from a second pass rescoring using LLM. Recently multi-modal large language models, particularly speech and text foundational models have demonstrated strong spoken language understanding. Speech-Text foundational models leverage large amounts of unlabelled and labelled data both in speech and text modalities to model human language. In this work, we propose novel techniques to use multi-modal LLM for ASR rescoring. We also explore discriminative training to further improve the foundational model rescoring performance. We demonstrate cross-modal knowledge transfer in speech-text LLM can benefit rescoring. Our experiments demonstrate up-to 20% relative improvements over Whisper large ASR and up-to 15% relative improvements over text-only LLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_16654 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Speech Recognition Rescoring with Large Speech-Text Foundation Models Shivakumar, Prashanth Gurunath Kolehmainen, Jari Gourav, Aditya Gu, Yi Gandhe, Ankur Rastrow, Ariya Bulyko, Ivan Audio and Speech Processing Computation and Language Sound Large language models (LLM) have demonstrated the ability to understand human language by leveraging large amount of text data. Automatic speech recognition (ASR) systems are often limited by available transcribed speech data and benefit from a second pass rescoring using LLM. Recently multi-modal large language models, particularly speech and text foundational models have demonstrated strong spoken language understanding. Speech-Text foundational models leverage large amounts of unlabelled and labelled data both in speech and text modalities to model human language. In this work, we propose novel techniques to use multi-modal LLM for ASR rescoring. We also explore discriminative training to further improve the foundational model rescoring performance. We demonstrate cross-modal knowledge transfer in speech-text LLM can benefit rescoring. Our experiments demonstrate up-to 20% relative improvements over Whisper large ASR and up-to 15% relative improvements over text-only LLM. |
| title | Speech Recognition Rescoring with Large Speech-Text Foundation Models |
| topic | Audio and Speech Processing Computation and Language Sound |
| url | https://arxiv.org/abs/2409.16654 |