Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918469969117184 |
|---|---|
| author | Yuan, Zekun Ye, Yangfan Feng, Xiaocheng Li, Baohang Hong, Qichen Lu, Yunfei Tu, Dandan Qin, Bing |
| author_facet | Yuan, Zekun Ye, Yangfan Feng, Xiaocheng Li, Baohang Hong, Qichen Lu, Yunfei Tu, Dandan Qin, Bing |
| contents | Large language models (LLMs) have achieved strong performance in general machine translation, yet their ability in culture-aware scenarios remains poorly understood. To bridge this gap, we introduce CanMT, a Culture-Aware Novel-Driven Parallel Dataset for Machine Translation, together with a theoretically grounded, multi-dimensional evaluation framework for assessing cultural translation quality. Leveraging CanMT, we systematically evaluate a wide range of LLMs and translation systems under different translation strategy constraints. Our findings reveal substantial performance disparities across models and demonstrate that translation strategies exert a systematic influence on model behavior. Further analysis shows that translation difficulty varies across types of culture-specific items, and that a persistent gap remains between models' recognition of culture-specific knowledge and their ability to correctly operationalize it in translation outputs. In addition, incorporating reference translations is shown to substantially improve evaluation reliability in LLM-as-a-judge, underscoring their essential role in assessing culture-aware translation quality. The corpus and code are available at CanMT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_24361 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation Yuan, Zekun Ye, Yangfan Feng, Xiaocheng Li, Baohang Hong, Qichen Lu, Yunfei Tu, Dandan Qin, Bing Computation and Language Large language models (LLMs) have achieved strong performance in general machine translation, yet their ability in culture-aware scenarios remains poorly understood. To bridge this gap, we introduce CanMT, a Culture-Aware Novel-Driven Parallel Dataset for Machine Translation, together with a theoretically grounded, multi-dimensional evaluation framework for assessing cultural translation quality. Leveraging CanMT, we systematically evaluate a wide range of LLMs and translation systems under different translation strategy constraints. Our findings reveal substantial performance disparities across models and demonstrate that translation strategies exert a systematic influence on model behavior. Further analysis shows that translation difficulty varies across types of culture-specific items, and that a persistent gap remains between models' recognition of culture-specific knowledge and their ability to correctly operationalize it in translation outputs. In addition, incorporating reference translations is shown to substantially improve evaluation reliability in LLM-as-a-judge, underscoring their essential role in assessing culture-aware translation quality. The corpus and code are available at CanMT. |
| title | Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.24361 |