Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914196293156864 |
|---|---|
| author | Menschikov, Mikhail Kharitonov, Alexander Kotyga, Maiia Porvatov, Vadim Zhukovskaya, Anna Kagramanyan, David Shvetsov, Egor Burnaev, Evgeny |
| author_facet | Menschikov, Mikhail Kharitonov, Alexander Kotyga, Maiia Porvatov, Vadim Zhukovskaya, Anna Kagramanyan, David Shvetsov, Egor Burnaev, Evgeny |
| contents | Large Language Models (LLMs) exhibit position bias systematically underweighting information based on its location in the context but how this bias varies across languages and models remains unclear. We conduct a multilingual study across five typologically diverse languages (English, Russian, German, Hindi, Vietnamese) and five model architectures, analyzing how position bias interacts with prompting strategies and affects output entropy. Our key findings are: (1) Position bias is primarily model-driven but shows language-specific nuances. Notably, Qwen2.5-7B-Instruct, DeepSeek 7B Chat and Mistral 7B consistently favor late positions challenging the common assumption of universal early-token preference. (2) Explicitly instructing the model, in the presence of irrelevant distractors, that "the most relevant context to the query is marked as 1" unexpectedly reduces accuracy across all languages, questioning standard prompt-engineering practices. (3) Accuracy consistently drops most when relevant information appears in the middle of the context, yet this is not reflected in a corresponding increase in output entropy, suggesting the model remains confident even when it fails to use mid-context cues. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_16134 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs Menschikov, Mikhail Kharitonov, Alexander Kotyga, Maiia Porvatov, Vadim Zhukovskaya, Anna Kagramanyan, David Shvetsov, Egor Burnaev, Evgeny Computation and Language Machine Learning Large Language Models (LLMs) exhibit position bias systematically underweighting information based on its location in the context but how this bias varies across languages and models remains unclear. We conduct a multilingual study across five typologically diverse languages (English, Russian, German, Hindi, Vietnamese) and five model architectures, analyzing how position bias interacts with prompting strategies and affects output entropy. Our key findings are: (1) Position bias is primarily model-driven but shows language-specific nuances. Notably, Qwen2.5-7B-Instruct, DeepSeek 7B Chat and Mistral 7B consistently favor late positions challenging the common assumption of universal early-token preference. (2) Explicitly instructing the model, in the presence of irrelevant distractors, that "the most relevant context to the query is marked as 1" unexpectedly reduces accuracy across all languages, questioning standard prompt-engineering practices. (3) Accuracy consistently drops most when relevant information appears in the middle of the context, yet this is not reflected in a corresponding increase in output entropy, suggesting the model remains confident even when it fails to use mid-context cues. |
| title | Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2505.16134 |