Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classification
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916065758412800 |
|---|---|
| author | Rauch, Lukas Heinrich, René Ghaffari, Houtan Miklautz, Lukas Moummad, Ilyass Sick, Bernhard Scholz, Christoph |
| author_facet | Rauch, Lukas Heinrich, René Ghaffari, Houtan Miklautz, Lukas Moummad, Ilyass Sick, Bernhard Scholz, Christoph |
| contents | Although probing frozen models has become a standard evaluation paradigm, self-supervised learning in audio defaults to fine-tuning when pursuing state-of-the-art on AudioSet. A key reason is that global pooling creates an information bottleneck causing linear probes to misrepresent the embedding quality: The $\texttt{cls}$-token discards crucial token information about dispersed, localized events in audio. This weakness is rooted in the mismatch between the pretraining objective (globally) and the downstream task (localized). Across a comprehensive benchmark of 13 datasets and 6 spectrogram-based encoders, we investigate the global pooling bottleneck. We introduce binarized prototypical probes: a lightweight and simple pooling method that learns prototypes to perform class-wise information aggregation. Despite its simplicity, our method notably outperforms linear and attentive probing. Our work establishes probing as a competitive and efficient paradigm for evaluating audio SSL models, challenging the reliance on costly fine-tuning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_24901 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classification Rauch, Lukas Heinrich, René Ghaffari, Houtan Miklautz, Lukas Moummad, Ilyass Sick, Bernhard Scholz, Christoph Sound Machine Learning Although probing frozen models has become a standard evaluation paradigm, self-supervised learning in audio defaults to fine-tuning when pursuing state-of-the-art on AudioSet. A key reason is that global pooling creates an information bottleneck causing linear probes to misrepresent the embedding quality: The $\texttt{cls}$-token discards crucial token information about dispersed, localized events in audio. This weakness is rooted in the mismatch between the pretraining objective (globally) and the downstream task (localized). Across a comprehensive benchmark of 13 datasets and 6 spectrogram-based encoders, we investigate the global pooling bottleneck. We introduce binarized prototypical probes: a lightweight and simple pooling method that learns prototypes to perform class-wise information aggregation. Despite its simplicity, our method notably outperforms linear and attentive probing. Our work establishes probing as a competitive and efficient paradigm for evaluating audio SSL models, challenging the reliance on costly fine-tuning. |
| title | Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classification |
| topic | Sound Machine Learning |
| url | https://arxiv.org/abs/2509.24901 |