Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915229762322432 |
|---|---|
| author | Wang, Mingyang Adel, Heike Lange, Lukas Liu, Yihong Nie, Ercong Strötgen, Jannik Schütze, Hinrich |
| author_facet | Wang, Mingyang Adel, Heike Lange, Lukas Liu, Yihong Nie, Ercong Strötgen, Jannik Schütze, Hinrich |
| contents | Multilingual language models (MLMs) store factual knowledge across languages but often struggle to provide consistent responses to semantically equivalent prompts in different languages. While previous studies point out this cross-lingual inconsistency issue, the underlying causes remain unexplored. In this work, we use mechanistic interpretability methods to investigate cross-lingual inconsistencies in MLMs. We find that MLMs encode knowledge in a language-independent concept space through most layers, and only transition to language-specific spaces in the final layers. Failures during the language transition often result in incorrect predictions in the target language, even when the answers are correct in other languages. To mitigate this inconsistency issue, we propose a linear shortcut method that bypasses computations in the final layers, enhancing both prediction accuracy and cross-lingual consistency. Our findings shed light on the internal mechanisms of MLMs and provide a lightweight, effective strategy for producing more consistent factual outputs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_04264 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models Wang, Mingyang Adel, Heike Lange, Lukas Liu, Yihong Nie, Ercong Strötgen, Jannik Schütze, Hinrich Computation and Language Multilingual language models (MLMs) store factual knowledge across languages but often struggle to provide consistent responses to semantically equivalent prompts in different languages. While previous studies point out this cross-lingual inconsistency issue, the underlying causes remain unexplored. In this work, we use mechanistic interpretability methods to investigate cross-lingual inconsistencies in MLMs. We find that MLMs encode knowledge in a language-independent concept space through most layers, and only transition to language-specific spaces in the final layers. Failures during the language transition often result in incorrect predictions in the target language, even when the answers are correct in other languages. To mitigate this inconsistency issue, we propose a linear shortcut method that bypasses computations in the final layers, enhancing both prediction accuracy and cross-lingual consistency. Our findings shed light on the internal mechanisms of MLMs and provide a lightweight, effective strategy for producing more consistent factual outputs. |
| title | Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.04264 |