Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.20052 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916673415544832 |
|---|---|
| author | Marchisio, Kelly Ko, Wei-Yin Bérard, Alexandre Dehaze, Théo Ruder, Sebastian |
| author_facet | Marchisio, Kelly Ko, Wei-Yin Bérard, Alexandre Dehaze, Théo Ruder, Sebastian |
| contents | We investigate a surprising limitation of LLMs: their inability to consistently generate text in a user's desired language. We create the Language Confusion Benchmark (LCB) to evaluate such failures, covering 15 typologically diverse languages with existing and newly-created English and multilingual prompts. We evaluate a range of LLMs on monolingual and cross-lingual generation reflecting practical use cases, finding that Llama Instruct and Mistral models exhibit high degrees of language confusion and even the strongest models fail to consistently respond in the correct language. We observe that base and English-centric instruct models are more prone to language confusion, which is aggravated by complex prompts and high sampling temperatures. We find that language confusion can be partially mitigated via few-shot prompting, multilingual SFT and preference tuning. We release our language confusion benchmark, which serves as a first layer of efficient, scalable multilingual evaluation at https://github.com/for-ai/language-confusion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_20052 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Understanding and Mitigating Language Confusion in LLMs Marchisio, Kelly Ko, Wei-Yin Bérard, Alexandre Dehaze, Théo Ruder, Sebastian Computation and Language We investigate a surprising limitation of LLMs: their inability to consistently generate text in a user's desired language. We create the Language Confusion Benchmark (LCB) to evaluate such failures, covering 15 typologically diverse languages with existing and newly-created English and multilingual prompts. We evaluate a range of LLMs on monolingual and cross-lingual generation reflecting practical use cases, finding that Llama Instruct and Mistral models exhibit high degrees of language confusion and even the strongest models fail to consistently respond in the correct language. We observe that base and English-centric instruct models are more prone to language confusion, which is aggravated by complex prompts and high sampling temperatures. We find that language confusion can be partially mitigated via few-shot prompting, multilingual SFT and preference tuning. We release our language confusion benchmark, which serves as a first layer of efficient, scalable multilingual evaluation at https://github.com/for-ai/language-confusion. |
| title | Understanding and Mitigating Language Confusion in LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.20052 |