Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.07607 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913631216599040 |
|---|---|
| author | Zhou, Wei Jia, Junteng Sari, Leda Mahadeokar, Jay Kalinli, Ozlem |
| author_facet | Zhou, Wei Jia, Junteng Sari, Leda Mahadeokar, Jay Kalinli, Ozlem |
| contents | CTC compressor can be an effective approach to integrate audio encoders to decoder-only models, which has gained growing interest for different speech applications. In this work, we propose a novel CTC compressor based joint speech and text training (CJST) framework for decoder-only ASR. CJST matches speech and text modalities from both directions by exploring a simple modality adaptor and several features of the CTC compressor, including sequence compression, on-the-fly forced peaky alignment and CTC class embeddings. Experimental results on the Librispeech and TED-LIUM2 corpora show that the proposed CJST achieves an effective text injection without the need of duration handling, leading to the best performance for both in-domain and cross-domain scenarios. We also provide a comprehensive study on CTC compressor, covering various compression modes, edge case handling and behavior under both clean and noisy data conditions, which reveals the most robust setting to use CTC compressor for decoder-only models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_07607 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CJST: CTC Compressor based Joint Speech and Text Training for Decoder-Only ASR Zhou, Wei Jia, Junteng Sari, Leda Mahadeokar, Jay Kalinli, Ozlem Audio and Speech Processing Machine Learning Sound CTC compressor can be an effective approach to integrate audio encoders to decoder-only models, which has gained growing interest for different speech applications. In this work, we propose a novel CTC compressor based joint speech and text training (CJST) framework for decoder-only ASR. CJST matches speech and text modalities from both directions by exploring a simple modality adaptor and several features of the CTC compressor, including sequence compression, on-the-fly forced peaky alignment and CTC class embeddings. Experimental results on the Librispeech and TED-LIUM2 corpora show that the proposed CJST achieves an effective text injection without the need of duration handling, leading to the best performance for both in-domain and cross-domain scenarios. We also provide a comprehensive study on CTC compressor, covering various compression modes, edge case handling and behavior under both clean and noisy data conditions, which reveals the most robust setting to use CTC compressor for decoder-only models. |
| title | CJST: CTC Compressor based Joint Speech and Text Training for Decoder-Only ASR |
| topic | Audio and Speech Processing Machine Learning Sound |
| url | https://arxiv.org/abs/2411.07607 |