Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.03020 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914082137833472 |
|---|---|
| author | Su, Chang Shi, Dengliang Huang, Siyuan Du, Jintao Meng, Changhua Cheng, Yu Wang, Weiqiang Lin, Zhouhan |
| author_facet | Su, Chang Shi, Dengliang Huang, Siyuan Du, Jintao Meng, Changhua Cheng, Yu Wang, Weiqiang Lin, Zhouhan |
| contents | Large language models (LLMs) have increasingly been explored as powerful text embedders. Existing LLM-based text embedding approaches often leverage the embedding of the final token, typically a reserved special token such as [EOS]. However, these tokens have not been intentionally trained to capture the semantics of the whole context, limiting their capacity as text embeddings, especially for retrieval and re-ranking tasks. We propose to add a new training stage before contrastive learning to enrich the semantics of the final token embedding. This stage employs bidirectional generative reconstruction tasks, namely EBQ2D (Embedding-Based Query-to-Document) and EBD2Q (Embedding-Based Document-to-Query), which interleave to anchor the [EOS] embedding and reconstruct either side of Query-Document pairs. Experimental results demonstrate that our additional training stage significantly improves LLM performance on the Massive Text Embedding Benchmark (MTEB), achieving new state-of-the-art results across different LLM base models and scales. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_03020 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Training LLMs to be Better Text Embedders through Bidirectional Reconstruction Su, Chang Shi, Dengliang Huang, Siyuan Du, Jintao Meng, Changhua Cheng, Yu Wang, Weiqiang Lin, Zhouhan Computation and Language Information Retrieval Large language models (LLMs) have increasingly been explored as powerful text embedders. Existing LLM-based text embedding approaches often leverage the embedding of the final token, typically a reserved special token such as [EOS]. However, these tokens have not been intentionally trained to capture the semantics of the whole context, limiting their capacity as text embeddings, especially for retrieval and re-ranking tasks. We propose to add a new training stage before contrastive learning to enrich the semantics of the final token embedding. This stage employs bidirectional generative reconstruction tasks, namely EBQ2D (Embedding-Based Query-to-Document) and EBD2Q (Embedding-Based Document-to-Query), which interleave to anchor the [EOS] embedding and reconstruct either side of Query-Document pairs. Experimental results demonstrate that our additional training stage significantly improves LLM performance on the Massive Text Embedding Benchmark (MTEB), achieving new state-of-the-art results across different LLM base models and scales. |
| title | Training LLMs to be Better Text Embedders through Bidirectional Reconstruction |
| topic | Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2509.03020 |