Smart Multi-Modal Search: Contextual Sparse and Dense Embedding Integration in Adobe Express
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910581800304640 |
|---|---|
| author | Aroraa, Cherag King, Tracy Holloway Kumar, Jayant Lu, Yi Sharma, Sanat Srikantan, Arvind Uvalle, David Valls-Vargas, Josep Vardhan, Harsha |
| author_facet | Aroraa, Cherag King, Tracy Holloway Kumar, Jayant Lu, Yi Sharma, Sanat Srikantan, Arvind Uvalle, David Valls-Vargas, Josep Vardhan, Harsha |
| contents | As user content and queries become increasingly multi-modal, the need for effective multi-modal search systems has grown. Traditional search systems often rely on textual and metadata annotations for indexed images, while multi-modal embeddings like CLIP enable direct search using text and image embeddings. However, embedding-based approaches face challenges in integrating contextual features such as user locale and recency. Building a scalable multi-modal search system requires fine-tuning several components. This paper presents a multi-modal search architecture and a series of AB tests that optimize embeddings and multi-modal technologies in Adobe Express template search. We address considerations such as embedding model selection, the roles of embeddings in matching and ranking, and the balance between dense and sparse embeddings. Our iterative approach demonstrates how utilizing sparse, dense, and contextual features enhances short and long query search, significantly reduces null rates (over 70\%), and increases click-through rates (CTR). Our findings provide insights into developing robust multi-modal search systems, thereby enhancing relevance for complex queries. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_14698 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Smart Multi-Modal Search: Contextual Sparse and Dense Embedding Integration in Adobe Express Aroraa, Cherag King, Tracy Holloway Kumar, Jayant Lu, Yi Sharma, Sanat Srikantan, Arvind Uvalle, David Valls-Vargas, Josep Vardhan, Harsha Information Retrieval Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition As user content and queries become increasingly multi-modal, the need for effective multi-modal search systems has grown. Traditional search systems often rely on textual and metadata annotations for indexed images, while multi-modal embeddings like CLIP enable direct search using text and image embeddings. However, embedding-based approaches face challenges in integrating contextual features such as user locale and recency. Building a scalable multi-modal search system requires fine-tuning several components. This paper presents a multi-modal search architecture and a series of AB tests that optimize embeddings and multi-modal technologies in Adobe Express template search. We address considerations such as embedding model selection, the roles of embeddings in matching and ranking, and the balance between dense and sparse embeddings. Our iterative approach demonstrates how utilizing sparse, dense, and contextual features enhances short and long query search, significantly reduces null rates (over 70\%), and increases click-through rates (CTR). Our findings provide insights into developing robust multi-modal search systems, thereby enhancing relevance for complex queries. |
| title | Smart Multi-Modal Search: Contextual Sparse and Dense Embedding Integration in Adobe Express |
| topic | Information Retrieval Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.14698 |