Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913880577409024 |
|---|---|
| author | Conti, Max Faysse, Manuel Viaud, Gautier Bosselut, Antoine Hudelot, Céline Colombo, Pierre |
| author_facet | Conti, Max Faysse, Manuel Viaud, Gautier Bosselut, Antoine Hudelot, Céline Colombo, Pierre |
| contents | A limitation of modern document retrieval embedding methods is that they typically encode passages (chunks) from the same documents independently, often overlooking crucial contextual information from the rest of the document that could greatly improve individual chunk representations.
In this work, we introduce ConTEB (Context-aware Text Embedding Benchmark), a benchmark designed to evaluate retrieval models on their ability to leverage document-wide context. Our results show that state-of-the-art embedding models struggle in retrieval scenarios where context is required. To address this limitation, we propose InSeNT (In-sequence Negative Training), a novel contrastive post-training approach which combined with late chunking pooling enhances contextual representation learning while preserving computational efficiency. Our method significantly improves retrieval quality on ConTEB without sacrificing base model performance. We further find chunks embedded with our method are more robust to suboptimal chunking strategies and larger retrieval corpus sizes. We open-source all artifacts at https://github.com/illuin-tech/contextual-embeddings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_24782 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings Conti, Max Faysse, Manuel Viaud, Gautier Bosselut, Antoine Hudelot, Céline Colombo, Pierre Information Retrieval A limitation of modern document retrieval embedding methods is that they typically encode passages (chunks) from the same documents independently, often overlooking crucial contextual information from the rest of the document that could greatly improve individual chunk representations. In this work, we introduce ConTEB (Context-aware Text Embedding Benchmark), a benchmark designed to evaluate retrieval models on their ability to leverage document-wide context. Our results show that state-of-the-art embedding models struggle in retrieval scenarios where context is required. To address this limitation, we propose InSeNT (In-sequence Negative Training), a novel contrastive post-training approach which combined with late chunking pooling enhances contextual representation learning while preserving computational efficiency. Our method significantly improves retrieval quality on ConTEB without sacrificing base model performance. We further find chunks embedded with our method are more robust to suboptimal chunking strategies and larger retrieval corpus sizes. We open-source all artifacts at https://github.com/illuin-tech/contextual-embeddings. |
| title | Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2505.24782 |