Vela: Scalable Embeddings with Voice Large Language Models for Multimodal Retrieval
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918061642088448 |
|---|---|
| author | Hu, Ruofan Xia, Yan Hong, Minjie Zhu, Jieming Chen, Bo Yang, Xiaoda Fang, Minghui Jin, Tao |
| author_facet | Hu, Ruofan Xia, Yan Hong, Minjie Zhu, Jieming Chen, Bo Yang, Xiaoda Fang, Minghui Jin, Tao |
| contents | Multimodal large language models (MLLMs) have seen substantial progress in recent years. However, their ability to represent multimodal information in the acoustic domain remains underexplored. In this work, we introduce Vela, a novel framework designed to adapt MLLMs for the generation of universal multimodal embeddings. By leveraging MLLMs with specially crafted prompts and selected in-context learning examples, Vela effectively bridges the modality gap across various modalities. We then propose a single-modality training approach, where the model is trained exclusively on text pairs. Our experiments show that Vela outperforms traditional CLAP models in standard text-audio retrieval tasks. Furthermore, we introduce new benchmarks that expose CLAP models' limitations in handling long texts and complex retrieval tasks. In contrast, Vela, by harnessing the capabilities of MLLMs, demonstrates robust performance in these scenarios. Our code will soon be available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_14445 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Vela: Scalable Embeddings with Voice Large Language Models for Multimodal Retrieval Hu, Ruofan Xia, Yan Hong, Minjie Zhu, Jieming Chen, Bo Yang, Xiaoda Fang, Minghui Jin, Tao Information Retrieval Multimodal large language models (MLLMs) have seen substantial progress in recent years. However, their ability to represent multimodal information in the acoustic domain remains underexplored. In this work, we introduce Vela, a novel framework designed to adapt MLLMs for the generation of universal multimodal embeddings. By leveraging MLLMs with specially crafted prompts and selected in-context learning examples, Vela effectively bridges the modality gap across various modalities. We then propose a single-modality training approach, where the model is trained exclusively on text pairs. Our experiments show that Vela outperforms traditional CLAP models in standard text-audio retrieval tasks. Furthermore, we introduce new benchmarks that expose CLAP models' limitations in handling long texts and complex retrieval tasks. In contrast, Vela, by harnessing the capabilities of MLLMs, demonstrates robust performance in these scenarios. Our code will soon be available. |
| title | Vela: Scalable Embeddings with Voice Large Language Models for Multimodal Retrieval |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2506.14445 |