CART: A Generative Cross-Modal Retrieval Framework with Coarse-To-Fine Semantic Modeling
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915644181577728 |
|---|---|
| author | Fang, Minghui Ji, Shengpeng Zuo, Jialong Huang, Hai Xia, Yan Zhu, Jieming Cheng, Xize Yang, Xiaoda Liu, Wenrui Wang, Gang Dong, Zhenhua Zhao, Zhou |
| author_facet | Fang, Minghui Ji, Shengpeng Zuo, Jialong Huang, Hai Xia, Yan Zhu, Jieming Cheng, Xize Yang, Xiaoda Liu, Wenrui Wang, Gang Dong, Zhenhua Zhao, Zhou |
| contents | Cross-modal retrieval aims to search for instances, which are semantically related to the query through the interaction of different modal data. Traditional solutions utilize a single-tower or dual-tower framework to explicitly compute the score between queries and candidates, which is challenged by training cost and inference latency with large-scale data. Inspired by the remarkable performance and efficiency of generative models, we propose a generative cross-modal retrieval framework (CART) based on coarse-to-fine semantic modeling, which assigns identifiers to each candidate and treats the generating identifier as the retrieval target. Specifically, we explore an effective coarse-to-fine scheme, combining K-Means and RQ-VAE to discretize multimodal data into token sequences that support autoregressive generation. Further, considering the lack of explicit interaction between queries and candidates, we propose a feature fusion strategy to align their semantics. Extensive experiments demonstrate the effectiveness of the strategies in the CART, achieving excellent results in both retrieval performance and efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_17507 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CART: A Generative Cross-Modal Retrieval Framework with Coarse-To-Fine Semantic Modeling Fang, Minghui Ji, Shengpeng Zuo, Jialong Huang, Hai Xia, Yan Zhu, Jieming Cheng, Xize Yang, Xiaoda Liu, Wenrui Wang, Gang Dong, Zhenhua Zhao, Zhou Information Retrieval Cross-modal retrieval aims to search for instances, which are semantically related to the query through the interaction of different modal data. Traditional solutions utilize a single-tower or dual-tower framework to explicitly compute the score between queries and candidates, which is challenged by training cost and inference latency with large-scale data. Inspired by the remarkable performance and efficiency of generative models, we propose a generative cross-modal retrieval framework (CART) based on coarse-to-fine semantic modeling, which assigns identifiers to each candidate and treats the generating identifier as the retrieval target. Specifically, we explore an effective coarse-to-fine scheme, combining K-Means and RQ-VAE to discretize multimodal data into token sequences that support autoregressive generation. Further, considering the lack of explicit interaction between queries and candidates, we propose a feature fusion strategy to align their semantics. Extensive experiments demonstrate the effectiveness of the strategies in the CART, achieving excellent results in both retrieval performance and efficiency. |
| title | CART: A Generative Cross-Modal Retrieval Framework with Coarse-To-Fine Semantic Modeling |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2406.17507 |