Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913146036289536 |
|---|---|
| author | Son, Guijin Kim, Seungone Arnett, Catherine Ko, Hyunwoo Lee, Hyein Kang, Hyeonah Longxi, Jiang Yun, Jin Lee, JungYup Lee, Kyungmin Kim, Sam Yoosuk Park, Sang Hong, Seunghyeok Lee, SeungJae Yi, Seungyeop Shin, Shinae Bok, SunHye Shin, Sunyoung Ji, Yonghoon Kim, Youngtaek Jung, Hanearl Asai, Akari Neubig, Graham Welleck, Sean Yu, Youngjae R, Akshelin Ivanov, Alexander B. Muhammadjon, Boboev Han, Chae Young Stump, Christian Anderson, Cooper R. Karp, Dmitrii Kwon, Dohyun Yi, Dongryung Kwon, DoYong Oh, Duk-Soon Choi, Eunho Resta, Giovanni Panova, Greta Noh, Huiyun Baik, Hyungryul Bae, Hyungsun Mashrafdzhon, Inomov Kim, Jeewon Kim, Jeong-Rae Lee, Ji Eun Liu, Jiaqi Kang, Jieui Kim, Jimin Kim, Jon-Lark Won, Joonyeong Yoon, Junseo Jo, Junwoo Kim, Kibeom Kwon, Kiwoon Kummer, Mario Mercer, Max Kim, Min Hoon Kim, Minjun Lee, Nahyun Ze-An, Ng Libedinsky, Nicolas Łochowski, Rafał Marcin Lachièze-Rey, Raphaël Auffarth, Robert Zhang, Ruichen Park, Sejin Seo, Seonguk Jaehoon, Shin Sunatullo Eom, Taewoong Park, Yeachan Jang, Yongseok Oh, Youchan Wang, Zhaoyang Kovács, Zoltán |
| author_facet | Son, Guijin Kim, Seungone Arnett, Catherine Ko, Hyunwoo Lee, Hyein Kang, Hyeonah Longxi, Jiang Yun, Jin Lee, JungYup Lee, Kyungmin Kim, Sam Yoosuk Park, Sang Hong, Seunghyeok Lee, SeungJae Yi, Seungyeop Shin, Shinae Bok, SunHye Shin, Sunyoung Ji, Yonghoon Kim, Youngtaek Jung, Hanearl Asai, Akari Neubig, Graham Welleck, Sean Yu, Youngjae R, Akshelin Ivanov, Alexander B. Muhammadjon, Boboev Han, Chae Young Stump, Christian Anderson, Cooper R. Karp, Dmitrii Kwon, Dohyun Yi, Dongryung Kwon, DoYong Oh, Duk-Soon Choi, Eunho Resta, Giovanni Panova, Greta Noh, Huiyun Baik, Hyungryul Bae, Hyungsun Mashrafdzhon, Inomov Kim, Jeewon Kim, Jeong-Rae Lee, Ji Eun Liu, Jiaqi Kang, Jieui Kim, Jimin Kim, Jon-Lark Won, Joonyeong Yoon, Junseo Jo, Junwoo Kim, Kibeom Kwon, Kiwoon Kummer, Mario Mercer, Max Kim, Min Hoon Kim, Minjun Lee, Nahyun Ze-An, Ng Libedinsky, Nicolas Łochowski, Rafał Marcin Lachièze-Rey, Raphaël Auffarth, Robert Zhang, Ruichen Park, Sejin Seo, Seonguk Jaehoon, Shin Sunatullo Eom, Taewoong Park, Yeachan Jang, Yongseok Oh, Youchan Wang, Zhaoyang Kovács, Zoltán |
| contents | Following the recent achievement of gold-medal performance on the IMO by frontier LLMs, the community is searching for the next meaningful and challenging target for measuring LLM reasoning. Whereas olympiad-style problems measure step-by-step reasoning alone, research-level problems use such reasoning to advance the frontier of mathematical knowledge itself, emerging as a compelling alternative. Yet research-level math benchmarks remain scarce because such problems are difficult to source (e.g., Riemann Bench and FrontierMath-Tier 4 contain 25 and 50 problems, respectively). To support reliable evaluation of next-generation frontier models, we introduce Soohak, a 439-problem benchmark newly authored from scratch by 64 mathematicians. Soohak comprises two subsets. On the Challenge subset, frontier models including Gemini-3-Pro, GPT-5, and Claude-Opus-4.5 reach 30.4%, 26.4%, and 10.4% respectively, leaving substantial headroom, while leading open-weight models such as Qwen3-235B, GPT-OSS-120B, and Kimi-2.5 remain below 15%. Notably, beyond standard problem solving, Soohak introduces a refusal subset that probes a capability intrinsic to research mathematics: recognizing ill-posed problems and pausing rather than producing confident but unjustified answers. On this subset, no model exceeds 50%, identifying refusal as a new optimization target that current models do not directly address. To prevent contamination, the dataset will be publicly released in late 2026, with model evaluations available upon request in the interim. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_09063 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs Son, Guijin Kim, Seungone Arnett, Catherine Ko, Hyunwoo Lee, Hyein Kang, Hyeonah Longxi, Jiang Yun, Jin Lee, JungYup Lee, Kyungmin Kim, Sam Yoosuk Park, Sang Hong, Seunghyeok Lee, SeungJae Yi, Seungyeop Shin, Shinae Bok, SunHye Shin, Sunyoung Ji, Yonghoon Kim, Youngtaek Jung, Hanearl Asai, Akari Neubig, Graham Welleck, Sean Yu, Youngjae R, Akshelin Ivanov, Alexander B. Muhammadjon, Boboev Han, Chae Young Stump, Christian Anderson, Cooper R. Karp, Dmitrii Kwon, Dohyun Yi, Dongryung Kwon, DoYong Oh, Duk-Soon Choi, Eunho Resta, Giovanni Panova, Greta Noh, Huiyun Baik, Hyungryul Bae, Hyungsun Mashrafdzhon, Inomov Kim, Jeewon Kim, Jeong-Rae Lee, Ji Eun Liu, Jiaqi Kang, Jieui Kim, Jimin Kim, Jon-Lark Won, Joonyeong Yoon, Junseo Jo, Junwoo Kim, Kibeom Kwon, Kiwoon Kummer, Mario Mercer, Max Kim, Min Hoon Kim, Minjun Lee, Nahyun Ze-An, Ng Libedinsky, Nicolas Łochowski, Rafał Marcin Lachièze-Rey, Raphaël Auffarth, Robert Zhang, Ruichen Park, Sejin Seo, Seonguk Jaehoon, Shin Sunatullo Eom, Taewoong Park, Yeachan Jang, Yongseok Oh, Youchan Wang, Zhaoyang Kovács, Zoltán Computation and Language Following the recent achievement of gold-medal performance on the IMO by frontier LLMs, the community is searching for the next meaningful and challenging target for measuring LLM reasoning. Whereas olympiad-style problems measure step-by-step reasoning alone, research-level problems use such reasoning to advance the frontier of mathematical knowledge itself, emerging as a compelling alternative. Yet research-level math benchmarks remain scarce because such problems are difficult to source (e.g., Riemann Bench and FrontierMath-Tier 4 contain 25 and 50 problems, respectively). To support reliable evaluation of next-generation frontier models, we introduce Soohak, a 439-problem benchmark newly authored from scratch by 64 mathematicians. Soohak comprises two subsets. On the Challenge subset, frontier models including Gemini-3-Pro, GPT-5, and Claude-Opus-4.5 reach 30.4%, 26.4%, and 10.4% respectively, leaving substantial headroom, while leading open-weight models such as Qwen3-235B, GPT-OSS-120B, and Kimi-2.5 remain below 15%. Notably, beyond standard problem solving, Soohak introduces a refusal subset that probes a capability intrinsic to research mathematics: recognizing ill-posed problems and pausing rather than producing confident but unjustified answers. On this subset, no model exceeds 50%, identifying refusal as a new optimization target that current models do not directly address. To prevent contamination, the dataset will be publicly released in late 2026, with model evaluations available upon request in the interim. |
| title | Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.09063 |