Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.12591 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914039165091840 |
|---|---|
| author | Govindarajan, Vijay Patel, Pratik Tripathi, Sahil Hoque, Md Azizul Kashyap, Gautam Siddharth |
| author_facet | Govindarajan, Vijay Patel, Pratik Tripathi, Sahil Hoque, Md Azizul Kashyap, Gautam Siddharth |
| contents | Automated Audio Captioning (AAC) generates captions for audio clips but faces challenges due to limited datasets compared to image captioning. To overcome this, we propose the zero-shot AAC system that leverages pre-trained models, eliminating the need for extensive training. Our approach uses a pre-trained audio CLIP model to extract auditory features and generate a structured prompt, which guides a Large Language Model (LLM) in caption generation. Unlike traditional greedy decoding, our method refines token selection through the audio CLIP model, ensuring alignment with the audio content. Experimental results demonstrate a 35% improvement in NLG mean score (from 4.7 to 7.3) using MAGIC search with the WavCaps model. The performance is heavily influenced by the audio-text matching model and keyword selection, with optimal results achieved using a single keyword prompt, and a 50% performance drop when no keyword list is used. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_12591 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MAGIC-Enhanced Keyword Prompting for Zero-Shot Audio Captioning with CLIP Models Govindarajan, Vijay Patel, Pratik Tripathi, Sahil Hoque, Md Azizul Kashyap, Gautam Siddharth Computation and Language Automated Audio Captioning (AAC) generates captions for audio clips but faces challenges due to limited datasets compared to image captioning. To overcome this, we propose the zero-shot AAC system that leverages pre-trained models, eliminating the need for extensive training. Our approach uses a pre-trained audio CLIP model to extract auditory features and generate a structured prompt, which guides a Large Language Model (LLM) in caption generation. Unlike traditional greedy decoding, our method refines token selection through the audio CLIP model, ensuring alignment with the audio content. Experimental results demonstrate a 35% improvement in NLG mean score (from 4.7 to 7.3) using MAGIC search with the WavCaps model. The performance is heavily influenced by the audio-text matching model and keyword selection, with optimal results achieved using a single keyword prompt, and a 50% performance drop when no keyword list is used. |
| title | MAGIC-Enhanced Keyword Prompting for Zero-Shot Audio Captioning with CLIP Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2509.12591 |