PromptASR for contextualized ASR with controllable style
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929220773478400 |
|---|---|
| author | Yang, Xiaoyu Kang, Wei Yao, Zengwei Yang, Yifan Guo, Liyong Kuang, Fangjun Lin, Long Povey, Daniel |
| author_facet | Yang, Xiaoyu Kang, Wei Yao, Zengwei Yang, Yifan Guo, Liyong Kuang, Fangjun Lin, Long Povey, Daniel |
| contents | Prompts are crucial to large language models as they provide context information such as topic or logical relationships. Inspired by this, we propose PromptASR, a framework that integrates prompts in end-to-end automatic speech recognition (E2E ASR) systems to achieve contextualized ASR with controllable style of transcriptions. Specifically, a dedicated text encoder encodes the text prompts and the encodings are injected into the speech encoder by cross-attending the features from two modalities. When using the ground truth text from preceding utterances as content prompt, the proposed system achieves 21.9% and 6.8% relative word error rate reductions on a book reading dataset and an in-house dataset compared to a baseline ASR system. The system can also take word-level biasing lists as prompt to improve recognition accuracy on rare words. An additional style prompt can be given to the text encoder and guide the ASR system to output different styles of transcriptions. The code is available at icefall. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_07414 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | PromptASR for contextualized ASR with controllable style Yang, Xiaoyu Kang, Wei Yao, Zengwei Yang, Yifan Guo, Liyong Kuang, Fangjun Lin, Long Povey, Daniel Audio and Speech Processing Computation and Language Sound Prompts are crucial to large language models as they provide context information such as topic or logical relationships. Inspired by this, we propose PromptASR, a framework that integrates prompts in end-to-end automatic speech recognition (E2E ASR) systems to achieve contextualized ASR with controllable style of transcriptions. Specifically, a dedicated text encoder encodes the text prompts and the encodings are injected into the speech encoder by cross-attending the features from two modalities. When using the ground truth text from preceding utterances as content prompt, the proposed system achieves 21.9% and 6.8% relative word error rate reductions on a book reading dataset and an in-house dataset compared to a baseline ASR system. The system can also take word-level biasing lists as prompt to improve recognition accuracy on rare words. An additional style prompt can be given to the text encoder and guide the ASR system to output different styles of transcriptions. The code is available at icefall. |
| title | PromptASR for contextualized ASR with controllable style |
| topic | Audio and Speech Processing Computation and Language Sound |
| url | https://arxiv.org/abs/2309.07414 |