Can Large Language Models Understand, Reason About, and Generate Code-Switched Text?
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911368585674752 |
|---|---|
| author | Winata, Genta Indra Anugraha, David Irawan, Patrick Amadeus Das, Anirban Yoo, Haneul Dashore, Paresh Kulkarni, Shreyas Zhang, Ruochen Sakajo, Haruki Hudi, Frederikus Ovalle, Anaelia Montariol, Syrielle Gaschi, Felix Anugraha, Michael Puranik, Rutuj Ravindra Ahmed, Zawad Hayat Merin, Adril Putra Chersoni, Emmanuele |
| author_facet | Winata, Genta Indra Anugraha, David Irawan, Patrick Amadeus Das, Anirban Yoo, Haneul Dashore, Paresh Kulkarni, Shreyas Zhang, Ruochen Sakajo, Haruki Hudi, Frederikus Ovalle, Anaelia Montariol, Syrielle Gaschi, Felix Anugraha, Michael Puranik, Rutuj Ravindra Ahmed, Zawad Hayat Merin, Adril Putra Chersoni, Emmanuele |
| contents | Code-switching is a pervasive phenomenon in multilingual communication, yet the robustness of large language models (LLMs) in mixed-language settings remains insufficiently understood. In this work, we present a comprehensive evaluation of LLM capabilities in understanding, reasoning over, and generating code-switched text. We introduce CodeMixQA a novel benchmark with high-quality human annotations, comprising 16 diverse parallel code-switched language-pair variants that span multiple geographic regions and code-switching patterns, and include both original scripts and their transliterated forms. Using this benchmark, we analyze the reasoning behavior of LLMs on code-switched question-answering tasks, shedding light on how models process and reason over mixed-language inputs. We further conduct a systematic evaluation of LLM-generated synthetic code-switched text, focusing on both naturalness and semantic fidelity, and uncover key limitations in current generation capabilities. Our findings reveal persistent challenges in both reasoning and generation under code-switching conditions and provide actionable insights for building more robust multilingual LLMs. We release the dataset and code as open source. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_07153 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Can Large Language Models Understand, Reason About, and Generate Code-Switched Text? Winata, Genta Indra Anugraha, David Irawan, Patrick Amadeus Das, Anirban Yoo, Haneul Dashore, Paresh Kulkarni, Shreyas Zhang, Ruochen Sakajo, Haruki Hudi, Frederikus Ovalle, Anaelia Montariol, Syrielle Gaschi, Felix Anugraha, Michael Puranik, Rutuj Ravindra Ahmed, Zawad Hayat Merin, Adril Putra Chersoni, Emmanuele Computation and Language Artificial Intelligence Code-switching is a pervasive phenomenon in multilingual communication, yet the robustness of large language models (LLMs) in mixed-language settings remains insufficiently understood. In this work, we present a comprehensive evaluation of LLM capabilities in understanding, reasoning over, and generating code-switched text. We introduce CodeMixQA a novel benchmark with high-quality human annotations, comprising 16 diverse parallel code-switched language-pair variants that span multiple geographic regions and code-switching patterns, and include both original scripts and their transliterated forms. Using this benchmark, we analyze the reasoning behavior of LLMs on code-switched question-answering tasks, shedding light on how models process and reason over mixed-language inputs. We further conduct a systematic evaluation of LLM-generated synthetic code-switched text, focusing on both naturalness and semantic fidelity, and uncover key limitations in current generation capabilities. Our findings reveal persistent challenges in both reasoning and generation under code-switching conditions and provide actionable insights for building more robust multilingual LLMs. We release the dataset and code as open source. |
| title | Can Large Language Models Understand, Reason About, and Generate Code-Switched Text? |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2601.07153 |