Exploring the Capability Boundaries of LLMs in Mastering of Chinese Chouxiang Language
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915945448996864 |
|---|---|
| author | Lin, Dianqing Lan, Tian Zhu, Jiali Li, Jiang Chen, Wei Liu, Xu Aruukhan Su, Xiangdong Hou, Hongxu Gao, Guanglai |
| author_facet | Lin, Dianqing Lan, Tian Zhu, Jiali Li, Jiang Chen, Wei Liu, Xu Aruukhan Su, Xiangdong Hou, Hongxu Gao, Guanglai |
| contents | While large language models (LLMs) have achieved remarkable success in general language tasks, their performance on Chouxiang Language, a representative subcultural language in the Chinese internet context, remains largely unexplored. In this paper, we introduce Mouse, a specialized benchmark designed to evaluate the capabilities of LLMs on NLP tasks involving Chouxiang Language across six tasks. Experimental results show that, current state-of-the-art (SOTA) LLMs exhibit clear limitations on multiple tasks, while performing well on tasks that involve contextual semantic understanding. In addition, we further discuss the reasons behind the generally low performance of SOTA LLMs on Chouxiang Language, examine whether the LLM-as-a-judge approach adopted for translation tasks aligns with human judgments and values, and analyze the key factors that influence Chouxiang translation. Our study aims to promote further research in the NLP community on multicultural integration and the dynamics of evolving internet languages. Our code and data are publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_15841 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Exploring the Capability Boundaries of LLMs in Mastering of Chinese Chouxiang Language Lin, Dianqing Lan, Tian Zhu, Jiali Li, Jiang Chen, Wei Liu, Xu Aruukhan Su, Xiangdong Hou, Hongxu Gao, Guanglai Computation and Language While large language models (LLMs) have achieved remarkable success in general language tasks, their performance on Chouxiang Language, a representative subcultural language in the Chinese internet context, remains largely unexplored. In this paper, we introduce Mouse, a specialized benchmark designed to evaluate the capabilities of LLMs on NLP tasks involving Chouxiang Language across six tasks. Experimental results show that, current state-of-the-art (SOTA) LLMs exhibit clear limitations on multiple tasks, while performing well on tasks that involve contextual semantic understanding. In addition, we further discuss the reasons behind the generally low performance of SOTA LLMs on Chouxiang Language, examine whether the LLM-as-a-judge approach adopted for translation tasks aligns with human judgments and values, and analyze the key factors that influence Chouxiang translation. Our study aims to promote further research in the NLP community on multicultural integration and the dynamics of evolving internet languages. Our code and data are publicly available. |
| title | Exploring the Capability Boundaries of LLMs in Mastering of Chinese Chouxiang Language |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.15841 |