EnigmaEval: A Benchmark of Long Multimodal Reasoning Challenges
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915151881437184 |
|---|---|
| author | Wang, Clinton J. Lee, Dean Menghini, Cristina Mols, Johannes Doughty, Jack Khoja, Adam Lynch, Jayson Hendryx, Sean Yue, Summer Hendrycks, Dan |
| author_facet | Wang, Clinton J. Lee, Dean Menghini, Cristina Mols, Johannes Doughty, Jack Khoja, Adam Lynch, Jayson Hendryx, Sean Yue, Summer Hendrycks, Dan |
| contents | As language models master existing reasoning benchmarks, we need new challenges to evaluate their cognitive frontiers. Puzzle-solving events are rich repositories of challenging multimodal problems that test a wide range of advanced reasoning and knowledge capabilities, making them a unique testbed for evaluating frontier language models. We introduce EnigmaEval, a dataset of problems and solutions derived from puzzle competitions and events that probes models' ability to perform implicit knowledge synthesis and multi-step deductive reasoning. Unlike existing reasoning and knowledge benchmarks, puzzle solving challenges models to discover hidden connections between seemingly unrelated pieces of information to uncover solution paths. The benchmark comprises 1184 puzzles of varying complexity -- each typically requiring teams of skilled solvers hours to days to complete -- with unambiguous, verifiable solutions that enable efficient evaluation. State-of-the-art language models achieve extremely low accuracy on these puzzles, even lower than other difficult benchmarks such as Humanity's Last Exam, unveiling models' shortcomings when challenged with problems requiring unstructured and lateral reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_08859 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EnigmaEval: A Benchmark of Long Multimodal Reasoning Challenges Wang, Clinton J. Lee, Dean Menghini, Cristina Mols, Johannes Doughty, Jack Khoja, Adam Lynch, Jayson Hendryx, Sean Yue, Summer Hendrycks, Dan Artificial Intelligence Computation and Language As language models master existing reasoning benchmarks, we need new challenges to evaluate their cognitive frontiers. Puzzle-solving events are rich repositories of challenging multimodal problems that test a wide range of advanced reasoning and knowledge capabilities, making them a unique testbed for evaluating frontier language models. We introduce EnigmaEval, a dataset of problems and solutions derived from puzzle competitions and events that probes models' ability to perform implicit knowledge synthesis and multi-step deductive reasoning. Unlike existing reasoning and knowledge benchmarks, puzzle solving challenges models to discover hidden connections between seemingly unrelated pieces of information to uncover solution paths. The benchmark comprises 1184 puzzles of varying complexity -- each typically requiring teams of skilled solvers hours to days to complete -- with unambiguous, verifiable solutions that enable efficient evaluation. State-of-the-art language models achieve extremely low accuracy on these puzzles, even lower than other difficult benchmarks such as Humanity's Last Exam, unveiling models' shortcomings when challenged with problems requiring unstructured and lateral reasoning. |
| title | EnigmaEval: A Benchmark of Long Multimodal Reasoning Challenges |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2502.08859 |