Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913582226079744 |
|---|---|
| author | Ziarko, Alicja Jiang, Albert Q. Piotrowski, Bartosz Li, Wenda Jamnik, Mateja Miłoś, Piotr |
| author_facet | Ziarko, Alicja Jiang, Albert Q. Piotrowski, Bartosz Li, Wenda Jamnik, Mateja Miłoś, Piotr |
| contents | Text embeddings are essential for many tasks, such as document retrieval, clustering, and semantic similarity assessment. In this paper, we study how to contrastively train text embedding models in a compute-optimal fashion, given a suite of pre-trained decoder-only language models. Our innovation is an algorithm that produces optimal configurations of model sizes, data quantities, and fine-tuning methods for text-embedding models at different computational budget levels. The resulting recipe, which we obtain through extensive experiments, can be used by practitioners to make informed design choices for their embedding models. Specifically, our findings suggest that full fine-tuning and low-rank adaptation fine-tuning produce optimal models at lower and higher computational budgets respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_04165 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe Ziarko, Alicja Jiang, Albert Q. Piotrowski, Bartosz Li, Wenda Jamnik, Mateja Miłoś, Piotr Machine Learning Text embeddings are essential for many tasks, such as document retrieval, clustering, and semantic similarity assessment. In this paper, we study how to contrastively train text embedding models in a compute-optimal fashion, given a suite of pre-trained decoder-only language models. Our innovation is an algorithm that produces optimal configurations of model sizes, data quantities, and fine-tuning methods for text-embedding models at different computational budget levels. The resulting recipe, which we obtain through extensive experiments, can be used by practitioners to make informed design choices for their embedding models. Specifically, our findings suggest that full fine-tuning and low-rank adaptation fine-tuning produce optimal models at lower and higher computational budgets respectively. |
| title | Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2406.04165 |