LINGOLY-TOO: Disentangling Reasoning from Knowledge with Templatised Orthographic Obfuscation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910206225547264 |
|---|---|
| author | Khouja, Jude Yang, Lingyi Korgul, Karolina Hellsten, Simeon Neacsu, Vlad A. Mayne, Harry Kearns, Ryan Othniel Bean, Andrew M. Mahdi, Adam |
| author_facet | Khouja, Jude Yang, Lingyi Korgul, Karolina Hellsten, Simeon Neacsu, Vlad A. Mayne, Harry Kearns, Ryan Othniel Bean, Andrew M. Mahdi, Adam |
| contents | Frontier language models demonstrate increasing ability at solving reasoning problems, but their performance is often inflated by circumventing reasoning and instead relying on their expanding knowledge and memorisation capacity. We introduce LINGOLY-TOO, a challenging reasoning benchmark of 1,203 questions and a total of 6,995 sub-questions that counters these shortcuts by applying expert-designed obfuscations to Linguistics Olympiad problems. These obfuscations preserve the underlying solution logic while reducing the likelihood problems are solvable with via knowledge or memorisation. Our experiments show that models exploit shortcuts on the original question as performance markedly drop upon obfuscation. Even the best reasoning models remain highly sensitive, with scores dropping from around 0.59 on original problems to 0.48 after obfuscation. LINGOLY-TOO disentangles reasoning from knowledge, offering a clearer measure of true reasoning capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_02972 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LINGOLY-TOO: Disentangling Reasoning from Knowledge with Templatised Orthographic Obfuscation Khouja, Jude Yang, Lingyi Korgul, Karolina Hellsten, Simeon Neacsu, Vlad A. Mayne, Harry Kearns, Ryan Othniel Bean, Andrew M. Mahdi, Adam Computation and Language Artificial Intelligence Frontier language models demonstrate increasing ability at solving reasoning problems, but their performance is often inflated by circumventing reasoning and instead relying on their expanding knowledge and memorisation capacity. We introduce LINGOLY-TOO, a challenging reasoning benchmark of 1,203 questions and a total of 6,995 sub-questions that counters these shortcuts by applying expert-designed obfuscations to Linguistics Olympiad problems. These obfuscations preserve the underlying solution logic while reducing the likelihood problems are solvable with via knowledge or memorisation. Our experiments show that models exploit shortcuts on the original question as performance markedly drop upon obfuscation. Even the best reasoning models remain highly sensitive, with scores dropping from around 0.59 on original problems to 0.48 after obfuscation. LINGOLY-TOO disentangles reasoning from knowledge, offering a clearer measure of true reasoning capabilities. |
| title | LINGOLY-TOO: Disentangling Reasoning from Knowledge with Templatised Orthographic Obfuscation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.02972 |