Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2506.23271 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915364223320064 |
|---|---|
| author | Zhou, Jinxing Li, Zhihui Yu, Yongqiang Zhou, Yanghao Guo, Ruohao Li, Guangyao Mao, Yuxin Han, Mingfei Chang, Xiaojun Wang, Meng |
| author_facet | Zhou, Jinxing Li, Zhihui Yu, Yongqiang Zhou, Yanghao Guo, Ruohao Li, Guangyao Mao, Yuxin Han, Mingfei Chang, Xiaojun Wang, Meng |
| contents | We present \textbf{Met}a-\textbf{T}oken \textbf{Le}arning (Mettle), a simple and memory-efficient method for adapting large-scale pretrained transformer models to downstream audio-visual tasks. Instead of sequentially modifying the output feature distribution of the transformer backbone, Mettle utilizes a lightweight \textit{Layer-Centric Distillation (LCD)} module to distill in parallel the intact audio or visual features embedded by each transformer layer into compact meta-tokens. This distillation process considers both pretrained knowledge preservation and task-specific adaptation. The obtained meta-tokens can be directly applied to classification tasks, such as audio-visual event localization and audio-visual video parsing. To further support fine-grained segmentation tasks, such as audio-visual segmentation, we introduce a \textit{Meta-Token Injection (MTI)} module, which utilizes the audio and visual meta-tokens distilled from the top transformer layer to guide feature adaptation in earlier layers. Extensive experiments on multiple audiovisual benchmarks demonstrate that our method significantly reduces memory usage and training time while maintaining parameter efficiency and competitive accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_23271 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptation Zhou, Jinxing Li, Zhihui Yu, Yongqiang Zhou, Yanghao Guo, Ruohao Li, Guangyao Mao, Yuxin Han, Mingfei Chang, Xiaojun Wang, Meng Computer Vision and Pattern Recognition We present \textbf{Met}a-\textbf{T}oken \textbf{Le}arning (Mettle), a simple and memory-efficient method for adapting large-scale pretrained transformer models to downstream audio-visual tasks. Instead of sequentially modifying the output feature distribution of the transformer backbone, Mettle utilizes a lightweight \textit{Layer-Centric Distillation (LCD)} module to distill in parallel the intact audio or visual features embedded by each transformer layer into compact meta-tokens. This distillation process considers both pretrained knowledge preservation and task-specific adaptation. The obtained meta-tokens can be directly applied to classification tasks, such as audio-visual event localization and audio-visual video parsing. To further support fine-grained segmentation tasks, such as audio-visual segmentation, we introduce a \textit{Meta-Token Injection (MTI)} module, which utilizes the audio and visual meta-tokens distilled from the top transformer layer to guide feature adaptation in earlier layers. Extensive experiments on multiple audiovisual benchmarks demonstrate that our method significantly reduces memory usage and training time while maintaining parameter efficiency and competitive accuracy. |
| title | Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2506.23271 |