Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.20300 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908793279873024 |
|---|---|
| author | Xu, Jing Wu, Minglin Chen, Xueyuan Wu, Xixin Meng, Helen |
| author_facet | Xu, Jing Wu, Minglin Chen, Xueyuan Wu, Xixin Meng, Helen |
| contents | Self-supervised learning (SSL) has greatly advanced speech representation learning, but multilingual SSL models remain constrained to languages encountered during pretraining. Retraining from scratch to incorporate new languages is computationally expensive, while sequential training without migitation strategies often leads to catastrophic forgetting. To address this, we propose MiLorE-SSL, a lightweight framework that combines LoRA modules with a soft mixture-of-experts (MoE) mechanism for efficient continual multilingual training. LoRA provides efficient low-rank adaptation, while soft MoE promotes flexible expert sharing across languages, reducing cross-lingual interference. To further mitigate forgetting, we introduce limited replay data from existing languages, avoiding reliance on large historical corpora. Experiments on ML-SUPERB demonstrate that MiLorE-SSL achieves strong performance in new languages and improves the ability in existing ones with only 2.14% trainable parameters. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_20300 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MiLorE-SSL: Scaling Multilingual Capabilities in Self-Supervised Models without Forgetting Xu, Jing Wu, Minglin Chen, Xueyuan Wu, Xixin Meng, Helen Computation and Language Audio and Speech Processing Self-supervised learning (SSL) has greatly advanced speech representation learning, but multilingual SSL models remain constrained to languages encountered during pretraining. Retraining from scratch to incorporate new languages is computationally expensive, while sequential training without migitation strategies often leads to catastrophic forgetting. To address this, we propose MiLorE-SSL, a lightweight framework that combines LoRA modules with a soft mixture-of-experts (MoE) mechanism for efficient continual multilingual training. LoRA provides efficient low-rank adaptation, while soft MoE promotes flexible expert sharing across languages, reducing cross-lingual interference. To further mitigate forgetting, we introduce limited replay data from existing languages, avoiding reliance on large historical corpora. Experiments on ML-SUPERB demonstrate that MiLorE-SSL achieves strong performance in new languages and improves the ability in existing ones with only 2.14% trainable parameters. |
| title | MiLorE-SSL: Scaling Multilingual Capabilities in Self-Supervised Models without Forgetting |
| topic | Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2601.20300 |