Enhancing Large Language Model-based Speech Recognition by Contextualization for Rare and Ambiguous Words
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910645087109120 |
|---|---|
| author | Nozawa, Kento Masuko, Takashi Taniguchi, Toru |
| author_facet | Nozawa, Kento Masuko, Takashi Taniguchi, Toru |
| contents | We develop a large language model (LLM) based automatic speech recognition (ASR) system that can be contextualized by providing keywords as prior information in text prompts. We adopt decoder-only architecture and use our in-house LLM, PLaMo-100B, pre-trained from scratch using datasets dominated by Japanese and English texts as the decoder. We adopt a pre-trained Whisper encoder as an audio encoder, and the audio embeddings from the audio encoder are projected to the text embedding space by an adapter layer and concatenated with text embeddings converted from text prompts to form inputs to the decoder. By providing keywords as prior information in the text prompts, we can contextualize our LLM-based ASR system without modifying the model architecture to transcribe ambiguous words in the input audio accurately. Experimental results demonstrate that providing keywords to the decoder can significantly improve the recognition performance of rare and ambiguous words. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_08027 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Large Language Model-based Speech Recognition by Contextualization for Rare and Ambiguous Words Nozawa, Kento Masuko, Takashi Taniguchi, Toru Audio and Speech Processing Computation and Language Sound We develop a large language model (LLM) based automatic speech recognition (ASR) system that can be contextualized by providing keywords as prior information in text prompts. We adopt decoder-only architecture and use our in-house LLM, PLaMo-100B, pre-trained from scratch using datasets dominated by Japanese and English texts as the decoder. We adopt a pre-trained Whisper encoder as an audio encoder, and the audio embeddings from the audio encoder are projected to the text embedding space by an adapter layer and concatenated with text embeddings converted from text prompts to form inputs to the decoder. By providing keywords as prior information in the text prompts, we can contextualize our LLM-based ASR system without modifying the model architecture to transcribe ambiguous words in the input audio accurately. Experimental results demonstrate that providing keywords to the decoder can significantly improve the recognition performance of rare and ambiguous words. |
| title | Enhancing Large Language Model-based Speech Recognition by Contextualization for Rare and Ambiguous Words |
| topic | Audio and Speech Processing Computation and Language Sound |
| url | https://arxiv.org/abs/2408.08027 |