Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.27649 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911723051548672 |
|---|---|
| author | Zhan, Qishi Hu, Minxuan Jang, Seoyeon Zhao, Lei Chen, Ziheng Liang, Man Xiang, Xinyue Liu, Jiaxin Wang, Guansu He, Liang |
| author_facet | Zhan, Qishi Hu, Minxuan Jang, Seoyeon Zhao, Lei Chen, Ziheng Liang, Man Xiang, Xinyue Liu, Jiaxin Wang, Guansu He, Liang |
| contents | Multilingual LLMs are increasingly used when instruction, source content, and required response languages do not coincide. Existing benchmarks have expanded multilingual instruction-following evaluation, but they rarely isolate these three roles within a fully crossed design. We introduce MTM-Bench, a controlled benchmark for language-conditioned task execution in which each instance is defined by a triplet \((L_{\text{instr}}, L_{\text{content}}, L_{\text{resp}})\). Across English, Spanish, and Chinese, MTM-Bench enumerates all 27 triplets and contains 2{,}430 instances per model across semantic reversal, final-state extraction, and language purity with update realization. We evaluate 20 frontier and open-weight LLMs using decomposed metrics for semantic correctness, target-language adherence, constraint satisfaction, contamination ratio, and joint success, with scoring validated by a targeted human audit. The fully crossed design reveals that degradation is organized by the role a language occupies in the task structure, not merely by mismatch count. The response-language role is the dominant axis of variation, and a single response-slot mismatch accounts for most degradation. The response-only and full-mismatch comparison suggests that mismatch count is not a monotonic predictor of difficulty, with model-level ordering varying across systems. Task families fail through distinct channels, showing that semantic correctness alone does not capture reliable multilingual task execution. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_27649 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Disentangling Language Roles in Multilingual LLM Task Execution Zhan, Qishi Hu, Minxuan Jang, Seoyeon Zhao, Lei Chen, Ziheng Liang, Man Xiang, Xinyue Liu, Jiaxin Wang, Guansu He, Liang Computation and Language Machine Learning Multilingual LLMs are increasingly used when instruction, source content, and required response languages do not coincide. Existing benchmarks have expanded multilingual instruction-following evaluation, but they rarely isolate these three roles within a fully crossed design. We introduce MTM-Bench, a controlled benchmark for language-conditioned task execution in which each instance is defined by a triplet \((L_{\text{instr}}, L_{\text{content}}, L_{\text{resp}})\). Across English, Spanish, and Chinese, MTM-Bench enumerates all 27 triplets and contains 2{,}430 instances per model across semantic reversal, final-state extraction, and language purity with update realization. We evaluate 20 frontier and open-weight LLMs using decomposed metrics for semantic correctness, target-language adherence, constraint satisfaction, contamination ratio, and joint success, with scoring validated by a targeted human audit. The fully crossed design reveals that degradation is organized by the role a language occupies in the task structure, not merely by mismatch count. The response-language role is the dominant axis of variation, and a single response-slot mismatch accounts for most degradation. The response-only and full-mismatch comparison suggests that mismatch count is not a monotonic predictor of difficulty, with model-level ordering varying across systems. Task families fail through distinct channels, showing that semantic correctness alone does not capture reliable multilingual task execution. |
| title | Disentangling Language Roles in Multilingual LLM Task Execution |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2605.27649 |