Streaming Bilingual End-to-End ASR model using Attention over Multiple Softmax
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910304580927488 |
|---|---|
| author | Patil, Aditya Joshi, Vikas Agrawal, Purvi Mehta, Rupesh |
| author_facet | Patil, Aditya Joshi, Vikas Agrawal, Purvi Mehta, Rupesh |
| contents | Even with several advancements in multilingual modeling, it is challenging to recognize multiple languages using a single neural model, without knowing the input language and most multilingual models assume the availability of the input language. In this work, we propose a novel bilingual end-to-end (E2E) modeling approach, where a single neural model can recognize both languages and also support switching between the languages, without any language input from the user. The proposed model has shared encoder and prediction networks, with language-specific joint networks that are combined via a self-attention mechanism. As the language-specific posteriors are combined, it produces a single posterior probability over all the output symbols, enabling a single beam search decoding and also allowing dynamic switching between the languages. The proposed approach outperforms the conventional bilingual baseline with 13.3%, 8.23% and 1.3% word error rate relative reduction on Hindi, English and code-mixed test sets, respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_11645 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Streaming Bilingual End-to-End ASR model using Attention over Multiple Softmax Patil, Aditya Joshi, Vikas Agrawal, Purvi Mehta, Rupesh Audio and Speech Processing Computation and Language Sound Even with several advancements in multilingual modeling, it is challenging to recognize multiple languages using a single neural model, without knowing the input language and most multilingual models assume the availability of the input language. In this work, we propose a novel bilingual end-to-end (E2E) modeling approach, where a single neural model can recognize both languages and also support switching between the languages, without any language input from the user. The proposed model has shared encoder and prediction networks, with language-specific joint networks that are combined via a self-attention mechanism. As the language-specific posteriors are combined, it produces a single posterior probability over all the output symbols, enabling a single beam search decoding and also allowing dynamic switching between the languages. The proposed approach outperforms the conventional bilingual baseline with 13.3%, 8.23% and 1.3% word error rate relative reduction on Hindi, English and code-mixed test sets, respectively. |
| title | Streaming Bilingual End-to-End ASR model using Attention over Multiple Softmax |
| topic | Audio and Speech Processing Computation and Language Sound |
| url | https://arxiv.org/abs/2401.11645 |