Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914991395831808 |
|---|---|
| author | Coelho, João Martins, Bruno Magalhães, João Callan, Jamie Xiong, Chenyan |
| author_facet | Coelho, João Martins, Bruno Magalhães, João Callan, Jamie Xiong, Chenyan |
| contents | This study investigates the existence of positional biases in Transformer-based models for text representation learning, particularly in the context of web document retrieval. We build on previous research that demonstrated loss of information in the middle of input sequences for causal language models, extending it to the domain of representation learning. We examine positional biases at various stages of training for an encoder-decoder model, including language model pre-training, contrastive pre-training, and contrastive fine-tuning. Experiments with the MS-MARCO document collection reveal that after contrastive pre-training the model already generates embeddings that better capture early contents of the input, with fine-tuning further aggravating this effect. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_04163 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval Coelho, João Martins, Bruno Magalhães, João Callan, Jamie Xiong, Chenyan Information Retrieval Computation and Language This study investigates the existence of positional biases in Transformer-based models for text representation learning, particularly in the context of web document retrieval. We build on previous research that demonstrated loss of information in the middle of input sequences for causal language models, extending it to the domain of representation learning. We examine positional biases at various stages of training for an encoder-decoder model, including language model pre-training, contrastive pre-training, and contrastive fine-tuning. Experiments with the MS-MARCO document collection reveal that after contrastive pre-training the model already generates embeddings that better capture early contents of the input, with fine-tuning further aggravating this effect. |
| title | Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval |
| topic | Information Retrieval Computation and Language |
| url | https://arxiv.org/abs/2404.04163 |