Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.21772 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908872897200128 |
|---|---|
| author | Chen, Yuxuan He, Peize Yu, Haoyuan Zhang, Junzi |
| author_facet | Chen, Yuxuan He, Peize Yu, Haoyuan Zhang, Junzi |
| contents | A universal audio representation should capture fine-grained speech cues and high-level semantics for environmental sounds and music in a single encoder. Existing encoders often excel in one domain but degrade in others. We propose UniWhisper, an efficient continual multi-task training framework that casts heterogeneous audio tasks into a unified instruction and answer format. This enables standard next-token training without task-specific heads and losses. We train it on 38k hours of public audio and assess the encoder using shallow MLP probes and k-nearest neighbors (kNN) on 20 tasks spanning speech, environmental sound, and music. UniWhisper reaches normalized weighted averages of 0.81 with MLP probes and 0.61 with kNN, compared to 0.64 and 0.46 for Whisper, while retaining strong speech performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_21772 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation Chen, Yuxuan He, Peize Yu, Haoyuan Zhang, Junzi Sound Artificial Intelligence A universal audio representation should capture fine-grained speech cues and high-level semantics for environmental sounds and music in a single encoder. Existing encoders often excel in one domain but degrade in others. We propose UniWhisper, an efficient continual multi-task training framework that casts heterogeneous audio tasks into a unified instruction and answer format. This enables standard next-token training without task-specific heads and losses. We train it on 38k hours of public audio and assess the encoder using shallow MLP probes and k-nearest neighbors (kNN) on 20 tasks spanning speech, environmental sound, and music. UniWhisper reaches normalized weighted averages of 0.81 with MLP probes and 0.61 with kNN, compared to 0.64 and 0.46 for Whisper, while retaining strong speech performance. |
| title | UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation |
| topic | Sound Artificial Intelligence |
| url | https://arxiv.org/abs/2602.21772 |