Contextualization of ASR with LLM using phonetic retrieval-based augmentation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929512188477440 |
|---|---|
| author | Lei, Zhihong Na, Xingyu Xu, Mingbin Pusateri, Ernest Van Gysel, Christophe Zhang, Yuanyuan Han, Shiyi Huang, Zhen |
| author_facet | Lei, Zhihong Na, Xingyu Xu, Mingbin Pusateri, Ernest Van Gysel, Christophe Zhang, Yuanyuan Han, Shiyi Huang, Zhen |
| contents | Large language models (LLMs) have shown superb capability of modeling multimodal signals including audio and text, allowing the model to generate spoken or textual response given a speech input. However, it remains a challenge for the model to recognize personal named entities, such as contacts in a phone book, when the input modality is speech. In this work, we start with a speech recognition task and propose a retrieval-based solution to contextualize the LLM: we first let the LLM detect named entities in speech without any context, then use this named entity as a query to retrieve phonetically similar named entities from a personal database and feed them to the LLM, and finally run context-aware LLM decoding. In a voice assistant task, our solution achieved up to 30.2% relative word error rate reduction and 73.6% relative named entity error rate reduction compared to a baseline system without contextualization. Notably, our solution by design avoids prompting the LLM with the full named entity database, making it highly efficient and applicable to large named entity databases. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_15353 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Contextualization of ASR with LLM using phonetic retrieval-based augmentation Lei, Zhihong Na, Xingyu Xu, Mingbin Pusateri, Ernest Van Gysel, Christophe Zhang, Yuanyuan Han, Shiyi Huang, Zhen Audio and Speech Processing Computation and Language Machine Learning Sound Large language models (LLMs) have shown superb capability of modeling multimodal signals including audio and text, allowing the model to generate spoken or textual response given a speech input. However, it remains a challenge for the model to recognize personal named entities, such as contacts in a phone book, when the input modality is speech. In this work, we start with a speech recognition task and propose a retrieval-based solution to contextualize the LLM: we first let the LLM detect named entities in speech without any context, then use this named entity as a query to retrieve phonetically similar named entities from a personal database and feed them to the LLM, and finally run context-aware LLM decoding. In a voice assistant task, our solution achieved up to 30.2% relative word error rate reduction and 73.6% relative named entity error rate reduction compared to a baseline system without contextualization. Notably, our solution by design avoids prompting the LLM with the full named entity database, making it highly efficient and applicable to large named entity databases. |
| title | Contextualization of ASR with LLM using phonetic retrieval-based augmentation |
| topic | Audio and Speech Processing Computation and Language Machine Learning Sound |
| url | https://arxiv.org/abs/2409.15353 |