Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2406.16135 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917944224645120 |
|---|---|
| author | Chua, Lynn Ghazi, Badih Huang, Yangsibo Kamath, Pritish Kumar, Ravi Manurangsi, Pasin Sinha, Amer Xie, Chulin Zhang, Chiyuan |
| author_facet | Chua, Lynn Ghazi, Badih Huang, Yangsibo Kamath, Pritish Kumar, Ravi Manurangsi, Pasin Sinha, Amer Xie, Chulin Zhang, Chiyuan |
| contents | Large language models (LLMs) are typically multilingual due to pretraining on diverse multilingual corpora. But can these models relate corresponding concepts across languages, i.e., be crosslingual? This study evaluates state-of-the-art LLMs on inherently crosslingual tasks. We observe that while these models show promising surface-level crosslingual abilities on machine translation and embedding space analyses, they struggle with deeper crosslingual knowledge transfer, revealing a crosslingual knowledge barrier in both general (MMLU benchmark) and domain-specific (Harry Potter quiz and TOFU benchmark) contexts. Since simple inference-time mitigation methods offer only limited improvement, we propose fine-tuning of LLMs on mixed-language data, which effectively reduces these gaps, even when using out-of-domain datasets like WikiText. Our findings suggest the need for explicit optimization to unlock the full crosslingual potential of LLMs. Our code is publicly available at https://github.com/google-research/crosslingual-knowledge-barriers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_16135 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models Chua, Lynn Ghazi, Badih Huang, Yangsibo Kamath, Pritish Kumar, Ravi Manurangsi, Pasin Sinha, Amer Xie, Chulin Zhang, Chiyuan Computation and Language Machine Learning Large language models (LLMs) are typically multilingual due to pretraining on diverse multilingual corpora. But can these models relate corresponding concepts across languages, i.e., be crosslingual? This study evaluates state-of-the-art LLMs on inherently crosslingual tasks. We observe that while these models show promising surface-level crosslingual abilities on machine translation and embedding space analyses, they struggle with deeper crosslingual knowledge transfer, revealing a crosslingual knowledge barrier in both general (MMLU benchmark) and domain-specific (Harry Potter quiz and TOFU benchmark) contexts. Since simple inference-time mitigation methods offer only limited improvement, we propose fine-tuning of LLMs on mixed-language data, which effectively reduces these gaps, even when using out-of-domain datasets like WikiText. Our findings suggest the need for explicit optimization to unlock the full crosslingual potential of LLMs. Our code is publicly available at https://github.com/google-research/crosslingual-knowledge-barriers. |
| title | Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2406.16135 |