When Vision Models Meet Parameter Efficient Look-Aside Adapters Without Large-Scale Audio Pretraining
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929619964264448 |
|---|---|
| author | Yeo, Juan Jang, Jinkwan Chae, Kyubyung Mun, Seongkyu Kim, Taesup |
| author_facet | Yeo, Juan Jang, Jinkwan Chae, Kyubyung Mun, Seongkyu Kim, Taesup |
| contents | Recent studies show that pretrained vision models can boost performance in audio downstream tasks. To enhance the performance further, an additional pretraining stage with large scale audio data is typically required to infuse audio specific knowledge into the vision model. However, such approaches require extensive audio data and a carefully designed objective function. In this work, we propose bypassing the pretraining stage by directly fine-tuning the vision model with our Look Aside Adapter (LoAA) designed for efficient audio understanding. Audio spectrum data is represented across two heterogeneous dimensions time and frequency and we refine adapters to facilitate interactions between tokens across these dimensions. Our experiments demonstrate that our adapters allow vision models to reach or surpass the performance of pretrained audio models in various audio and speech tasks, offering a resource efficient and effective solution for leveraging vision models in audio applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_05951 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | When Vision Models Meet Parameter Efficient Look-Aside Adapters Without Large-Scale Audio Pretraining Yeo, Juan Jang, Jinkwan Chae, Kyubyung Mun, Seongkyu Kim, Taesup Sound Computer Vision and Pattern Recognition Machine Learning Audio and Speech Processing Recent studies show that pretrained vision models can boost performance in audio downstream tasks. To enhance the performance further, an additional pretraining stage with large scale audio data is typically required to infuse audio specific knowledge into the vision model. However, such approaches require extensive audio data and a carefully designed objective function. In this work, we propose bypassing the pretraining stage by directly fine-tuning the vision model with our Look Aside Adapter (LoAA) designed for efficient audio understanding. Audio spectrum data is represented across two heterogeneous dimensions time and frequency and we refine adapters to facilitate interactions between tokens across these dimensions. Our experiments demonstrate that our adapters allow vision models to reach or surpass the performance of pretrained audio models in various audio and speech tasks, offering a resource efficient and effective solution for leveraging vision models in audio applications. |
| title | When Vision Models Meet Parameter Efficient Look-Aside Adapters Without Large-Scale Audio Pretraining |
| topic | Sound Computer Vision and Pattern Recognition Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2412.05951 |