Language Models are Universal Embedders
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916750140899328 |
|---|---|
| author | Zhang, Xin Li, Zehan Zhang, Yanzhao Long, Dingkun Xie, Pengjun Zhang, Meishan Zhang, Min |
| author_facet | Zhang, Xin Li, Zehan Zhang, Yanzhao Long, Dingkun Xie, Pengjun Zhang, Meishan Zhang, Min |
| contents | In the large language model (LLM) revolution, embedding is a key component of various systems, such as retrieving knowledge or memories for LLMs or building content moderation filters. As such cases span from English to other natural or programming languages, from retrieval to classification and beyond, it is advantageous to build a unified embedding model rather than dedicated ones for each scenario. In this context, the pre-trained multilingual decoder-only large language models, e.g., BLOOM, emerge as a viable backbone option. To assess their potential, we propose straightforward strategies for constructing embedders and introduce a universal evaluation benchmark. Experimental results show that our trained model is proficient at generating good embeddings across languages and tasks, even extending to languages and tasks for which no finetuning/pretraining data is available. We also present detailed analyses and additional evaluations. We hope that this work could encourage the development of more robust open-source universal embedders. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_08232 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Language Models are Universal Embedders Zhang, Xin Li, Zehan Zhang, Yanzhao Long, Dingkun Xie, Pengjun Zhang, Meishan Zhang, Min Computation and Language In the large language model (LLM) revolution, embedding is a key component of various systems, such as retrieving knowledge or memories for LLMs or building content moderation filters. As such cases span from English to other natural or programming languages, from retrieval to classification and beyond, it is advantageous to build a unified embedding model rather than dedicated ones for each scenario. In this context, the pre-trained multilingual decoder-only large language models, e.g., BLOOM, emerge as a viable backbone option. To assess their potential, we propose straightforward strategies for constructing embedders and introduce a universal evaluation benchmark. Experimental results show that our trained model is proficient at generating good embeddings across languages and tasks, even extending to languages and tasks for which no finetuning/pretraining data is available. We also present detailed analyses and additional evaluations. We hope that this work could encourage the development of more robust open-source universal embedders. |
| title | Language Models are Universal Embedders |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2310.08232 |