Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.17784 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916754781896704 |
|---|---|
| author | Edman, Lukas Schmid, Helmut Fraser, Alexander |
| author_facet | Edman, Lukas Schmid, Helmut Fraser, Alexander |
| contents | The CUTE benchmark showed that LLMs struggle with character understanding in English. We extend it to more languages with diverse scripts and writing systems, introducing EXECUTE. Our simplified framework allows easy expansion to any language. Tests across multiple LLMs reveal that challenges in other languages are not always on the character level as in English. Some languages show word-level processing issues, some show no issues at all. We also examine sub-character tasks in Chinese, Japanese, and Korean to assess LLMs' understanding of character components. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_17784 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EXECUTE: A Multilingual Benchmark for LLM Token Understanding Edman, Lukas Schmid, Helmut Fraser, Alexander Computation and Language The CUTE benchmark showed that LLMs struggle with character understanding in English. We extend it to more languages with diverse scripts and writing systems, introducing EXECUTE. Our simplified framework allows easy expansion to any language. Tests across multiple LLMs reveal that challenges in other languages are not always on the character level as in English. Some languages show word-level processing issues, some show no issues at all. We also examine sub-character tasks in Chinese, Japanese, and Korean to assess LLMs' understanding of character components. |
| title | EXECUTE: A Multilingual Benchmark for LLM Token Understanding |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.17784 |