Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912469146927104 |
|---|---|
| author | Günther, Michael Mohr, Isabelle Williams, Daniel James Wang, Bo Xiao, Han |
| author_facet | Günther, Michael Mohr, Isabelle Williams, Daniel James Wang, Bo Xiao, Han |
| contents | Many use cases require retrieving smaller portions of text, and dense vector-based retrieval systems often perform better with shorter text segments, as the semantics are less likely to be over-compressed in the embeddings. Consequently, practitioners often split text documents into smaller chunks and encode them separately. However, chunk embeddings created in this way can lose contextual information from surrounding chunks, resulting in sub-optimal representations. In this paper, we introduce a novel method called late chunking, which leverages long context embedding models to first embed all tokens of the long text, with chunking applied after the transformer model and just before mean pooling - hence the term late in its naming. The resulting chunk embeddings capture the full contextual information, leading to superior results across various retrieval tasks. The method is generic enough to be applied to a wide range of long-context embedding models and works without additional training. To further increase the effectiveness of late chunking, we propose a dedicated fine-tuning approach for embedding models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_04701 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models Günther, Michael Mohr, Isabelle Williams, Daniel James Wang, Bo Xiao, Han Computation and Language Information Retrieval 68T50 I.2.7 Many use cases require retrieving smaller portions of text, and dense vector-based retrieval systems often perform better with shorter text segments, as the semantics are less likely to be over-compressed in the embeddings. Consequently, practitioners often split text documents into smaller chunks and encode them separately. However, chunk embeddings created in this way can lose contextual information from surrounding chunks, resulting in sub-optimal representations. In this paper, we introduce a novel method called late chunking, which leverages long context embedding models to first embed all tokens of the long text, with chunking applied after the transformer model and just before mean pooling - hence the term late in its naming. The resulting chunk embeddings capture the full contextual information, leading to superior results across various retrieval tasks. The method is generic enough to be applied to a wide range of long-context embedding models and works without additional training. To further increase the effectiveness of late chunking, we propose a dedicated fine-tuning approach for embedding models. |
| title | Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models |
| topic | Computation and Language Information Retrieval 68T50 I.2.7 |
| url | https://arxiv.org/abs/2409.04701 |