KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909842997772288 |
|---|---|
| author | Zhao, Xinping Hu, Xinshuo Shan, Zifei Huang, Shouzheng Zhou, Yao Zhang, Xin Sun, Zetian Liu, Zhenyu Li, Dongfang Wei, Xinyuan Pan, Youcheng Xiang, Yang Zhang, Meishan Wang, Haofen Yu, Jun Hu, Baotian Zhang, Min |
| author_facet | Zhao, Xinping Hu, Xinshuo Shan, Zifei Huang, Shouzheng Zhou, Yao Zhang, Xin Sun, Zetian Liu, Zhenyu Li, Dongfang Wei, Xinyuan Pan, Youcheng Xiang, Yang Zhang, Meishan Wang, Haofen Yu, Jun Hu, Baotian Zhang, Min |
| contents | Recent advancements in Large Language Models (LLMs)-based text embedding models primarily focus on data scaling or synthesis, yet limited exploration of training techniques and data quality, thereby constraining performance. In this work, we propose KaLM-Embedding-V2, a series of versatile and compact embedding models, systematically incentivizing advanced embedding capability in LLMs by superior training techniques and high-quality data. For model architecture, we implement the models on a 0.5B compact size with simple mean-pooling to produce fixed-length embeddings and remove the causal attention mask to enable fully bidirectional representation learning. For training techniques, we propose a progressive multi-stage training pipeline: pre-training on weakly supervised large-scale datasets, fine-tuning with supervised high-quality datasets, and contrastive distillation with fine-grained soft signals, integrated with focal-style reweighting and online hard-negative mixing to emphasize difficult samples and enrich hard negatives, respectively. For training data, we curate over 20 categories for pre-training and 100 categories for fine-tuning and contrastive distillation, to improve both performance and generalization, leveraging task-specific instructions, hard-negative mining, and example-based multi-class labeling to ensure high quality. Combining these techniques, our KaLM-Embedding-V2 series achieves state-of-the-art performance on the Massive Text Embedding Benchmark, outperforming models of comparable size and rivaling models 3-26x larger, setting a new standard for versatile and compact embedding models under 1B parameters. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_20923 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model Zhao, Xinping Hu, Xinshuo Shan, Zifei Huang, Shouzheng Zhou, Yao Zhang, Xin Sun, Zetian Liu, Zhenyu Li, Dongfang Wei, Xinyuan Pan, Youcheng Xiang, Yang Zhang, Meishan Wang, Haofen Yu, Jun Hu, Baotian Zhang, Min Computation and Language Recent advancements in Large Language Models (LLMs)-based text embedding models primarily focus on data scaling or synthesis, yet limited exploration of training techniques and data quality, thereby constraining performance. In this work, we propose KaLM-Embedding-V2, a series of versatile and compact embedding models, systematically incentivizing advanced embedding capability in LLMs by superior training techniques and high-quality data. For model architecture, we implement the models on a 0.5B compact size with simple mean-pooling to produce fixed-length embeddings and remove the causal attention mask to enable fully bidirectional representation learning. For training techniques, we propose a progressive multi-stage training pipeline: pre-training on weakly supervised large-scale datasets, fine-tuning with supervised high-quality datasets, and contrastive distillation with fine-grained soft signals, integrated with focal-style reweighting and online hard-negative mixing to emphasize difficult samples and enrich hard negatives, respectively. For training data, we curate over 20 categories for pre-training and 100 categories for fine-tuning and contrastive distillation, to improve both performance and generalization, leveraging task-specific instructions, hard-negative mining, and example-based multi-class labeling to ensure high quality. Combining these techniques, our KaLM-Embedding-V2 series achieves state-of-the-art performance on the Massive Text Embedding Benchmark, outperforming models of comparable size and rivaling models 3-26x larger, setting a new standard for versatile and compact embedding models under 1B parameters. |
| title | KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.20923 |