Improving Practical Aspects of End-to-End Multi-Talker Speech Recognition for Online and Offline Scenarios
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909651360022528 |
|---|---|
| author | Subramanian, Aswin Shanmugam Das, Amit Kanda, Naoyuki Li, Jinyu Wang, Xiaofei Gong, Yifan |
| author_facet | Subramanian, Aswin Shanmugam Das, Amit Kanda, Naoyuki Li, Jinyu Wang, Xiaofei Gong, Yifan |
| contents | We extend the frameworks of Serialized Output Training (SOT) to address practical needs of both streaming and offline automatic speech recognition (ASR) applications. Our approach focuses on balancing latency and accuracy, catering to real-time captioning and summarization requirements. We propose several key improvements: (1) Leveraging Continuous Speech Separation (CSS) single-channel front-end with end-to-end (E2E) systems for highly overlapping scenarios, challenging the conventional wisdom of E2E versus cascaded setups. The CSS framework improves the accuracy of the ASR system by separating overlapped speech from multiple speakers. (2) Implementing dual models -- Conformer Transducer for streaming and Sequence-to-Sequence for offline -- or alternatively, a two-pass model based on cascaded encoders. (3) Exploring segment-based SOT (segSOT) which is better suited for offline scenarios while also enhancing readability of multi-talker transcriptions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_14204 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Improving Practical Aspects of End-to-End Multi-Talker Speech Recognition for Online and Offline Scenarios Subramanian, Aswin Shanmugam Das, Amit Kanda, Naoyuki Li, Jinyu Wang, Xiaofei Gong, Yifan Audio and Speech Processing Computation and Language Sound We extend the frameworks of Serialized Output Training (SOT) to address practical needs of both streaming and offline automatic speech recognition (ASR) applications. Our approach focuses on balancing latency and accuracy, catering to real-time captioning and summarization requirements. We propose several key improvements: (1) Leveraging Continuous Speech Separation (CSS) single-channel front-end with end-to-end (E2E) systems for highly overlapping scenarios, challenging the conventional wisdom of E2E versus cascaded setups. The CSS framework improves the accuracy of the ASR system by separating overlapped speech from multiple speakers. (2) Implementing dual models -- Conformer Transducer for streaming and Sequence-to-Sequence for offline -- or alternatively, a two-pass model based on cascaded encoders. (3) Exploring segment-based SOT (segSOT) which is better suited for offline scenarios while also enhancing readability of multi-talker transcriptions. |
| title | Improving Practical Aspects of End-to-End Multi-Talker Speech Recognition for Online and Offline Scenarios |
| topic | Audio and Speech Processing Computation and Language Sound |
| url | https://arxiv.org/abs/2506.14204 |