Online Register for Dual-Mode Self-Supervised Speech Models: Mitigating The Lack of Future Context
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918359450255360 |
|---|---|
| author | Goto, Keita Maekaku, Takashi Sakuma, Jin Tian, Jinchuan Shinohara, Yusuke Watanabe, Shinji |
| author_facet | Goto, Keita Maekaku, Takashi Sakuma, Jin Tian, Jinchuan Shinohara, Yusuke Watanabe, Shinji |
| contents | Dual-mode self-supervised speech models (S3Ms), which jointly pre-trained in the offline and online mode, suffer from attention mismatch in streaming scenarios due to missing future context. To address this challenge, we proposed online registers, learnable tokens appended to each chunk in online mode. These tokens act as virtual placeholders for unseen future frames, enabling the model to compensate for missing context without introducing additional latency. Furthermore, we introduce a future prediction loss that explicitly guides the registers to capture predictive cues, thereby enriching their ability to retain future information. Experiments on LibriSpeech, and out-of-domain benchmarks demonstrate that online registers consistently reduce the performance gap between offline and online modes, achieving a 3.4% relative improvement on LibriSpeech with 160 ms chunks, especially in low-latency settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_23702 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Online Register for Dual-Mode Self-Supervised Speech Models: Mitigating The Lack of Future Context Goto, Keita Maekaku, Takashi Sakuma, Jin Tian, Jinchuan Shinohara, Yusuke Watanabe, Shinji Sound Dual-mode self-supervised speech models (S3Ms), which jointly pre-trained in the offline and online mode, suffer from attention mismatch in streaming scenarios due to missing future context. To address this challenge, we proposed online registers, learnable tokens appended to each chunk in online mode. These tokens act as virtual placeholders for unseen future frames, enabling the model to compensate for missing context without introducing additional latency. Furthermore, we introduce a future prediction loss that explicitly guides the registers to capture predictive cues, thereby enriching their ability to retain future information. Experiments on LibriSpeech, and out-of-domain benchmarks demonstrate that online registers consistently reduce the performance gap between offline and online modes, achieving a 3.4% relative improvement on LibriSpeech with 160 ms chunks, especially in low-latency settings. |
| title | Online Register for Dual-Mode Self-Supervised Speech Models: Mitigating The Lack of Future Context |
| topic | Sound |
| url | https://arxiv.org/abs/2602.23702 |