Information Representation Fairness in Long-Document Embeddings: The Peculiar Interaction of Positional and Language Bias
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908979430424576 |
|---|---|
| author | Schuhmacher, Elias Michail, Andrianos Opitz, Juri Sennrich, Rico Clematide, Simon |
| author_facet | Schuhmacher, Elias Michail, Andrianos Opitz, Juri Sennrich, Rico Clematide, Simon |
| contents | To be discoverable in an embedding-based search process, each part of a document should be reflected in its embedding representation. To quantify any potential reflection biases, we introduce a permutation-based evaluation framework. With this, we observe that state-of-the-art embedding models exhibit systematic positional and language biases when documents are longer and consist of multiple segments. Specifically, early segments and segments in higher-resource languages like English are over-represented, while later segments and segments in lower-resource languages are marginalized. In our further analysis, we find that the positional bias stems from front-loaded attention distributions in pooling-token embeddings, where early tokens receive more attention. To mitigate this issue, we introduce an inference-time attention calibration method that redistributes attention more evenly across document positions, increasing discoverabiltiy of later segments. Our evaluation framework and attention calibration is available at https://github.com/impresso/fair-sentence-transformers |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_16934 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Information Representation Fairness in Long-Document Embeddings: The Peculiar Interaction of Positional and Language Bias Schuhmacher, Elias Michail, Andrianos Opitz, Juri Sennrich, Rico Clematide, Simon Computation and Language Artificial Intelligence To be discoverable in an embedding-based search process, each part of a document should be reflected in its embedding representation. To quantify any potential reflection biases, we introduce a permutation-based evaluation framework. With this, we observe that state-of-the-art embedding models exhibit systematic positional and language biases when documents are longer and consist of multiple segments. Specifically, early segments and segments in higher-resource languages like English are over-represented, while later segments and segments in lower-resource languages are marginalized. In our further analysis, we find that the positional bias stems from front-loaded attention distributions in pooling-token embeddings, where early tokens receive more attention. To mitigate this issue, we introduce an inference-time attention calibration method that redistributes attention more evenly across document positions, increasing discoverabiltiy of later segments. Our evaluation framework and attention calibration is available at https://github.com/impresso/fair-sentence-transformers |
| title | Information Representation Fairness in Long-Document Embeddings: The Peculiar Interaction of Positional and Language Bias |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2601.16934 |