Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.01531 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908390847938560 |
|---|---|
| author | Liu, Wenhao Lu, Zhenyi Hu, Xinyu Zhang, Jierui Li, Dailin Cen, Jiacheng Cao, Huilin Wang, Haiteng Li, Yuhan Xie, Kun Li, Dandan Zhang, Pei Zhang, Chengbo Ren, Yuxiang Huang, Xiaohong Ma, Yan |
| author_facet | Liu, Wenhao Lu, Zhenyi Hu, Xinyu Zhang, Jierui Li, Dailin Cen, Jiacheng Cao, Huilin Wang, Haiteng Li, Yuhan Xie, Kun Li, Dandan Zhang, Pei Zhang, Chengbo Ren, Yuxiang Huang, Xiaohong Ma, Yan |
| contents | High-quality math datasets are crucial for advancing the reasoning abilities of large language models (LLMs). However, existing datasets often suffer from three key issues: outdated and insufficient challenging content, neglecting human-like reasoning, and limited reliability due to single-LLM generation. To address these, we introduce STORM-BORN, an ultra-challenging dataset of mathematical derivations sourced from cutting-edge academic papers, which includes dense human-like approximations and heuristic cues. To ensure the reliability and quality, we propose a novel human-in-the-loop, multi-agent data generation framework, integrating reasoning-dense filters, multi-agent collaboration, and human mathematicians' evaluations. We curated a set of 2,000 synthetic samples and deliberately selected the 100 most difficult problems. Even most advanced models like GPT-o1 solved fewer than 5% of them. Fine-tuning on STORM-BORN boosts accuracy by 7.84% (LLaMA3-8B) and 9.12% (Qwen2.5-7B). As AI approaches mathematician-level reasoning, STORM-BORN provides both a high-difficulty benchmark and a human-like reasoning training resource. Our code and dataset are publicly available at https://github.com/lwhere/STORM-BORN. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_01531 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | STORM-BORN: A Challenging Mathematical Derivations Dataset Curated via a Human-in-the-Loop Multi-Agent Framework Liu, Wenhao Lu, Zhenyi Hu, Xinyu Zhang, Jierui Li, Dailin Cen, Jiacheng Cao, Huilin Wang, Haiteng Li, Yuhan Xie, Kun Li, Dandan Zhang, Pei Zhang, Chengbo Ren, Yuxiang Huang, Xiaohong Ma, Yan Computation and Language High-quality math datasets are crucial for advancing the reasoning abilities of large language models (LLMs). However, existing datasets often suffer from three key issues: outdated and insufficient challenging content, neglecting human-like reasoning, and limited reliability due to single-LLM generation. To address these, we introduce STORM-BORN, an ultra-challenging dataset of mathematical derivations sourced from cutting-edge academic papers, which includes dense human-like approximations and heuristic cues. To ensure the reliability and quality, we propose a novel human-in-the-loop, multi-agent data generation framework, integrating reasoning-dense filters, multi-agent collaboration, and human mathematicians' evaluations. We curated a set of 2,000 synthetic samples and deliberately selected the 100 most difficult problems. Even most advanced models like GPT-o1 solved fewer than 5% of them. Fine-tuning on STORM-BORN boosts accuracy by 7.84% (LLaMA3-8B) and 9.12% (Qwen2.5-7B). As AI approaches mathematician-level reasoning, STORM-BORN provides both a high-difficulty benchmark and a human-like reasoning training resource. Our code and dataset are publicly available at https://github.com/lwhere/STORM-BORN. |
| title | STORM-BORN: A Challenging Mathematical Derivations Dataset Curated via a Human-in-the-Loop Multi-Agent Framework |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.01531 |