Gecko: Versatile Text Embeddings Distilled from Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914734428651520 |
|---|---|
| author | Lee, Jinhyuk Dai, Zhuyun Ren, Xiaoqi Chen, Blair Cer, Daniel Cole, Jeremy R. Hui, Kai Boratko, Michael Kapadia, Rajvi Ding, Wen Luan, Yi Duddu, Sai Meher Karthik Abrego, Gustavo Hernandez Shi, Weiqiang Gupta, Nithi Kusupati, Aditya Jain, Prateek Jonnalagadda, Siddhartha Reddy Chang, Ming-Wei Naim, Iftekhar |
| author_facet | Lee, Jinhyuk Dai, Zhuyun Ren, Xiaoqi Chen, Blair Cer, Daniel Cole, Jeremy R. Hui, Kai Boratko, Michael Kapadia, Rajvi Ding, Wen Luan, Yi Duddu, Sai Meher Karthik Abrego, Gustavo Hernandez Shi, Weiqiang Gupta, Nithi Kusupati, Aditya Jain, Prateek Jonnalagadda, Siddhartha Reddy Chang, Ming-Wei Naim, Iftekhar |
| contents | We present Gecko, a compact and versatile text embedding model. Gecko achieves strong retrieval performance by leveraging a key idea: distilling knowledge from large language models (LLMs) into a retriever. Our two-step distillation process begins with generating diverse, synthetic paired data using an LLM. Next, we further refine the data quality by retrieving a set of candidate passages for each query, and relabeling the positive and hard negative passages using the same LLM. The effectiveness of our approach is demonstrated by the compactness of the Gecko. On the Massive Text Embedding Benchmark (MTEB), Gecko with 256 embedding dimensions outperforms all existing entries with 768 embedding size. Gecko with 768 embedding dimensions achieves an average score of 66.31, competing with 7x larger models and 5x higher dimensional embeddings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_20327 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Gecko: Versatile Text Embeddings Distilled from Large Language Models Lee, Jinhyuk Dai, Zhuyun Ren, Xiaoqi Chen, Blair Cer, Daniel Cole, Jeremy R. Hui, Kai Boratko, Michael Kapadia, Rajvi Ding, Wen Luan, Yi Duddu, Sai Meher Karthik Abrego, Gustavo Hernandez Shi, Weiqiang Gupta, Nithi Kusupati, Aditya Jain, Prateek Jonnalagadda, Siddhartha Reddy Chang, Ming-Wei Naim, Iftekhar Computation and Language Artificial Intelligence We present Gecko, a compact and versatile text embedding model. Gecko achieves strong retrieval performance by leveraging a key idea: distilling knowledge from large language models (LLMs) into a retriever. Our two-step distillation process begins with generating diverse, synthetic paired data using an LLM. Next, we further refine the data quality by retrieving a set of candidate passages for each query, and relabeling the positive and hard negative passages using the same LLM. The effectiveness of our approach is demonstrated by the compactness of the Gecko. On the Massive Text Embedding Benchmark (MTEB), Gecko with 256 embedding dimensions outperforms all existing entries with 768 embedding size. Gecko with 768 embedding dimensions achieves an average score of 66.31, competing with 7x larger models and 5x higher dimensional embeddings. |
| title | Gecko: Versatile Text Embeddings Distilled from Large Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2403.20327 |