InnoGym: Benchmarking the Innovation Potential of AI Agents
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917301547171840 |
|---|---|
| author | Zhang, Jintian Xu, Kewei Zheng, Jingsheng Yu, Zhuoyun Zhu, Yuqi Luo, Yujie Wei, Lanning Qiao, Shuofei Du, Lun Zheng, Da Deng, Shumin Chen, Huajun Zhang, Ningyu |
| author_facet | Zhang, Jintian Xu, Kewei Zheng, Jingsheng Yu, Zhuoyun Zhu, Yuqi Luo, Yujie Wei, Lanning Qiao, Shuofei Du, Lun Zheng, Da Deng, Shumin Chen, Huajun Zhang, Ningyu |
| contents | LLMs and Agents have achieved impressive progress in code generation, mathematical reasoning, and scientific discovery. However, existing benchmarks primarily measure correctness, overlooking the diversity of methods behind solutions. True innovation depends not only on producing correct answers but also on the originality of the approach. We present InnoGym, the first benchmark and framework designed to systematically evaluate the innovation potential of AI agents. InnoGym introduces two complementary metrics: performance gain, which measures improvement over the best-known solutions, and novelty, which captures methodological differences from prior approaches. The benchmark includes 18 carefully curated tasks from real-world engineering and scientific domains, each standardized through resource filtering, evaluator validation, and solution collection. In addition, we provide iGym, a unified execution environment for reproducible and long-horizon evaluations. Extensive experiments show that while some agents produce novel approaches, their lack of robustness limits performance gains. These results highlight a key gap between creativity and effectiveness, underscoring the need for benchmarks that evaluate both. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_01822 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | InnoGym: Benchmarking the Innovation Potential of AI Agents Zhang, Jintian Xu, Kewei Zheng, Jingsheng Yu, Zhuoyun Zhu, Yuqi Luo, Yujie Wei, Lanning Qiao, Shuofei Du, Lun Zheng, Da Deng, Shumin Chen, Huajun Zhang, Ningyu Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Multiagent Systems LLMs and Agents have achieved impressive progress in code generation, mathematical reasoning, and scientific discovery. However, existing benchmarks primarily measure correctness, overlooking the diversity of methods behind solutions. True innovation depends not only on producing correct answers but also on the originality of the approach. We present InnoGym, the first benchmark and framework designed to systematically evaluate the innovation potential of AI agents. InnoGym introduces two complementary metrics: performance gain, which measures improvement over the best-known solutions, and novelty, which captures methodological differences from prior approaches. The benchmark includes 18 carefully curated tasks from real-world engineering and scientific domains, each standardized through resource filtering, evaluator validation, and solution collection. In addition, we provide iGym, a unified execution environment for reproducible and long-horizon evaluations. Extensive experiments show that while some agents produce novel approaches, their lack of robustness limits performance gains. These results highlight a key gap between creativity and effectiveness, underscoring the need for benchmarks that evaluate both. |
| title | InnoGym: Benchmarking the Innovation Potential of AI Agents |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Multiagent Systems |
| url | https://arxiv.org/abs/2512.01822 |