SONAR: Self-Distilled Continual Pre-training for Domain Adaptive Audio Representation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915748856725504 |
|---|---|
| author | Zhang, Yizhou Gao, Yuan Zhou, Wangjin Yuan, Zicheng Imoto, Keisuke Kawahara, Tatsuya |
| author_facet | Zhang, Yizhou Gao, Yuan Zhou, Wangjin Yuan, Zicheng Imoto, Keisuke Kawahara, Tatsuya |
| contents | Self-supervised learning (SSL) on large-scale datasets like AudioSet has become the dominant paradigm for audio representation learning. While the continuous influx of new, unlabeled audio presents an opportunity to enrich these static representations, a naive approach is to retrain the model from scratch using all available data. However, this method is computationally prohibitive and discards the valuable knowledge embedded in the previously trained model weights. To address this inefficiency, we propose SONAR (Self-distilled cONtinual pre-training for domain adaptive Audio Representation), a continual pre-training framework built upon BEATs. SONAR effectively adapts to new domains while mitigating catastrophic forgetting by tackling three key challenges: implementing a joint sampling strategy for new and prior data, applying regularization to balance specificity and generality, and dynamically expanding the tokenizer codebook for novel acoustic patterns. Experiments across four distinct domains demonstrate that our method achieves both high adaptability and robust resistance to forgetting. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_15703 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SONAR: Self-Distilled Continual Pre-training for Domain Adaptive Audio Representation Zhang, Yizhou Gao, Yuan Zhou, Wangjin Yuan, Zicheng Imoto, Keisuke Kawahara, Tatsuya Sound Audio and Speech Processing Self-supervised learning (SSL) on large-scale datasets like AudioSet has become the dominant paradigm for audio representation learning. While the continuous influx of new, unlabeled audio presents an opportunity to enrich these static representations, a naive approach is to retrain the model from scratch using all available data. However, this method is computationally prohibitive and discards the valuable knowledge embedded in the previously trained model weights. To address this inefficiency, we propose SONAR (Self-distilled cONtinual pre-training for domain adaptive Audio Representation), a continual pre-training framework built upon BEATs. SONAR effectively adapts to new domains while mitigating catastrophic forgetting by tackling three key challenges: implementing a joint sampling strategy for new and prior data, applying regularization to balance specificity and generality, and dynamically expanding the tokenizer codebook for novel acoustic patterns. Experiments across four distinct domains demonstrate that our method achieves both high adaptability and robust resistance to forgetting. |
| title | SONAR: Self-Distilled Continual Pre-training for Domain Adaptive Audio Representation |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2509.15703 |