Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.01028 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912189041868800 |
|---|---|
| author | Hu, Xinshuo Shan, Zifei Zhao, Xinping Sun, Zetian Liu, Zhenyu Li, Dongfang Ye, Shaolin Wei, Xinyuan Chen, Qian Hu, Baotian Wang, Haofen Yu, Jun Zhang, Min |
| author_facet | Hu, Xinshuo Shan, Zifei Zhao, Xinping Sun, Zetian Liu, Zhenyu Li, Dongfang Ye, Shaolin Wei, Xinyuan Chen, Qian Hu, Baotian Wang, Haofen Yu, Jun Zhang, Min |
| contents | As retrieval-augmented generation prevails in large language models, embedding models are becoming increasingly crucial. Despite the growing number of general embedding models, prior work often overlooks the critical role of training data quality. In this work, we introduce KaLM-Embedding, a general multilingual embedding model that leverages a large quantity of cleaner, more diverse, and domain-specific training data. Our model has been trained with key techniques proven to enhance performance: (1) persona-based synthetic data to create diversified examples distilled from LLMs, (2) ranking consistency filtering to remove less informative samples, and (3) semi-homogeneous task batch sampling to improve training efficacy. Departing from traditional BERT-like architectures, we adopt Qwen2-0.5B as the pre-trained model, facilitating the adaptation of auto-regressive language models for general embedding tasks. Extensive evaluations of the MTEB benchmark across multiple languages show that our model outperforms others of comparable size, setting a new standard for multilingual embedding models with <1B parameters. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_01028 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model Hu, Xinshuo Shan, Zifei Zhao, Xinping Sun, Zetian Liu, Zhenyu Li, Dongfang Ye, Shaolin Wei, Xinyuan Chen, Qian Hu, Baotian Wang, Haofen Yu, Jun Zhang, Min Computation and Language As retrieval-augmented generation prevails in large language models, embedding models are becoming increasingly crucial. Despite the growing number of general embedding models, prior work often overlooks the critical role of training data quality. In this work, we introduce KaLM-Embedding, a general multilingual embedding model that leverages a large quantity of cleaner, more diverse, and domain-specific training data. Our model has been trained with key techniques proven to enhance performance: (1) persona-based synthetic data to create diversified examples distilled from LLMs, (2) ranking consistency filtering to remove less informative samples, and (3) semi-homogeneous task batch sampling to improve training efficacy. Departing from traditional BERT-like architectures, we adopt Qwen2-0.5B as the pre-trained model, facilitating the adaptation of auto-regressive language models for general embedding tasks. Extensive evaluations of the MTEB benchmark across multiple languages show that our model outperforms others of comparable size, setting a new standard for multilingual embedding models with <1B parameters. |
| title | KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2501.01028 |