Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.23323 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911623078215680 |
|---|---|
| author | Liu, Meizhu Rowe, Matthew Agarwal, Amit Avendi, Michael Abbasi, Yassi Patel, Hitesh Laxmichand Li, Paul Han, Kyu J. Sheng, Tao Ravi, Sujith Roth, Dan |
| author_facet | Liu, Meizhu Rowe, Matthew Agarwal, Amit Avendi, Michael Abbasi, Yassi Patel, Hitesh Laxmichand Li, Paul Han, Kyu J. Sheng, Tao Ravi, Sujith Roth, Dan |
| contents | Audio-text retrieval enables semantic alignment between audio content and natural language queries, supporting applications in multimedia search, accessibility, and surveillance. However, current state-of-the-art approaches struggle with long, noisy, and weakly labeled audio due to their reliance on contrastive learning and large-batch training. We propose a novel multimodal retrieval framework that refines audio and text embeddings using a cross-modal embedding refinement module combining transformer-based projection, linear mapping, and bidirectional attention. To further improve robustness, we introduce a hybrid loss function blending cosine similarity, $\mathcal{L}_{1}$, and contrastive objectives, enabling stable training even under small-batch constraints. Our approach efficiently handles long-form and noisy audio (SNR 5 to 15) via silence-aware chunking and attention-based pooling. Experiments on benchmark datasets demonstrate improvements over prior methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_23323 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Robust Audio-Text Retrieval via Cross-Modal Attention and Hybrid Loss Liu, Meizhu Rowe, Matthew Agarwal, Amit Avendi, Michael Abbasi, Yassi Patel, Hitesh Laxmichand Li, Paul Han, Kyu J. Sheng, Tao Ravi, Sujith Roth, Dan Computation and Language Sound Audio-text retrieval enables semantic alignment between audio content and natural language queries, supporting applications in multimedia search, accessibility, and surveillance. However, current state-of-the-art approaches struggle with long, noisy, and weakly labeled audio due to their reliance on contrastive learning and large-batch training. We propose a novel multimodal retrieval framework that refines audio and text embeddings using a cross-modal embedding refinement module combining transformer-based projection, linear mapping, and bidirectional attention. To further improve robustness, we introduce a hybrid loss function blending cosine similarity, $\mathcal{L}_{1}$, and contrastive objectives, enabling stable training even under small-batch constraints. Our approach efficiently handles long-form and noisy audio (SNR 5 to 15) via silence-aware chunking and attention-based pooling. Experiments on benchmark datasets demonstrate improvements over prior methods. |
| title | Robust Audio-Text Retrieval via Cross-Modal Attention and Hybrid Loss |
| topic | Computation and Language Sound |
| url | https://arxiv.org/abs/2604.23323 |