Beg to Differ: Understanding Reasoning-Answer Misalignment Across Languages
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917367858069504 |
|---|---|
| author | Ovalle, Anaelia Ross, Candace Ruder, Sebastian Williams, Adina Ullrich, Karen Ibrahim, Mark Sagun, Levent |
| author_facet | Ovalle, Anaelia Ross, Candace Ruder, Sebastian Williams, Adina Ullrich, Karen Ibrahim, Mark Sagun, Levent |
| contents | Large language models demonstrate strong reasoning capabilities through chain-of-thought prompting, but whether this reasoning quality transfers across languages remains underexplored. We introduce a human-validated framework to evaluate whether model-generated reasoning traces logically support their conclusions across languages. Analyzing 65k reasoning traces from GlobalMMLU questions across 6 languages and 6 frontier models, we uncover a critical blind spot: while models achieve high task accuracy, their reasoning can fail to support their conclusions. Reasoning traces in non-Latin scripts show at least twice as much misalignment between their reasoning and conclusions than those in Latin scripts. We develop an error taxonomy through human annotation to characterize these failures, finding they stem primarily from evidential errors (unsupported claims, ambiguous facts) followed by illogical reasoning steps. Our findings demonstrate that current multilingual evaluation practices provide an incomplete picture of model reasoning capabilities and highlight the need for reasoning-aware evaluation frameworks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_22712 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beg to Differ: Understanding Reasoning-Answer Misalignment Across Languages Ovalle, Anaelia Ross, Candace Ruder, Sebastian Williams, Adina Ullrich, Karen Ibrahim, Mark Sagun, Levent Computation and Language Large language models demonstrate strong reasoning capabilities through chain-of-thought prompting, but whether this reasoning quality transfers across languages remains underexplored. We introduce a human-validated framework to evaluate whether model-generated reasoning traces logically support their conclusions across languages. Analyzing 65k reasoning traces from GlobalMMLU questions across 6 languages and 6 frontier models, we uncover a critical blind spot: while models achieve high task accuracy, their reasoning can fail to support their conclusions. Reasoning traces in non-Latin scripts show at least twice as much misalignment between their reasoning and conclusions than those in Latin scripts. We develop an error taxonomy through human annotation to characterize these failures, finding they stem primarily from evidential errors (unsupported claims, ambiguous facts) followed by illogical reasoning steps. Our findings demonstrate that current multilingual evaluation practices provide an incomplete picture of model reasoning capabilities and highlight the need for reasoning-aware evaluation frameworks. |
| title | Beg to Differ: Understanding Reasoning-Answer Misalignment Across Languages |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2512.22712 |