UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911106118713344 |
|---|---|
| author | Choudhary, Mukund Srivatsa, KV Aditya Aeron, Gaurja Bhattacharya, Antara Raaghavi Dinh, Dang Khoa Dang Hanif, Ikhlasul Akmal Kotova, Daria Kochmar, Ekaterina Choudhury, Monojit |
| author_facet | Choudhary, Mukund Srivatsa, KV Aditya Aeron, Gaurja Bhattacharya, Antara Raaghavi Dinh, Dang Khoa Dang Hanif, Ikhlasul Akmal Kotova, Daria Kochmar, Ekaterina Choudhury, Monojit |
| contents | Large language models (LLMs) have demonstrated potential in reasoning tasks, but their performance on linguistics puzzles remains consistently poor. These puzzles, often derived from Linguistics Olympiad (LO) contests, provide a minimal contamination environment to assess LLMs' linguistic reasoning abilities across low-resource languages. This work analyses LLMs' performance on 629 problems across 41 low-resource languages by labelling each with linguistically informed features to unveil weaknesses. Our analyses show that LLMs struggle with puzzles involving higher morphological complexity and perform better on puzzles involving linguistic features that are also found in English. We also show that splitting words into morphemes as a pre-processing step improves solvability, indicating a need for more informed and language-specific tokenisers. These findings thus offer insights into some challenges in linguistic reasoning and modelling of low-resource languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_11260 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs? Choudhary, Mukund Srivatsa, KV Aditya Aeron, Gaurja Bhattacharya, Antara Raaghavi Dinh, Dang Khoa Dang Hanif, Ikhlasul Akmal Kotova, Daria Kochmar, Ekaterina Choudhury, Monojit Computation and Language Large language models (LLMs) have demonstrated potential in reasoning tasks, but their performance on linguistics puzzles remains consistently poor. These puzzles, often derived from Linguistics Olympiad (LO) contests, provide a minimal contamination environment to assess LLMs' linguistic reasoning abilities across low-resource languages. This work analyses LLMs' performance on 629 problems across 41 low-resource languages by labelling each with linguistically informed features to unveil weaknesses. Our analyses show that LLMs struggle with puzzles involving higher morphological complexity and perform better on puzzles involving linguistic features that are also found in English. We also show that splitting words into morphemes as a pre-processing step improves solvability, indicating a need for more informed and language-specific tokenisers. These findings thus offer insights into some challenges in linguistic reasoning and modelling of low-resource languages. |
| title | UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2508.11260 |