EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solving
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918477668810752 |
|---|---|
| author | Zhou, Xiyuan Wang, Xinlei He, Yirui Wu, Yang Zou, Ruixi Cheng, Yuheng Xie, Yulu Liu, Wenxuan Zhao, Huan Xu, Yan Gu, Jinjin Zhao, Junhua |
| author_facet | Zhou, Xiyuan Wang, Xinlei He, Yirui Wu, Yang Zou, Ruixi Cheng, Yuheng Xie, Yulu Liu, Wenxuan Zhao, Huan Xu, Yan Gu, Jinjin Zhao, Junhua |
| contents | Large language models (LLMs) have shown strong performance on mathematical reasoning under well-defined conditions. However, real-world engineering problems involve uncertainty, context, and open-ended settings that extend beyond symbolic computation. Existing benchmarks largely focus on well-defined or abstract reasoning and therefore fail to capture these complexities. We introduce EngiBench, a hierarchical benchmark designed to evaluate LLMs on solving engineering problems. It spans three levels of increasing difficulty (foundational knowledge retrieval, contextual reasoning, and open-ended modeling) and covers diverse engineering subfields. To facilitate a deeper understanding of model performance, we systematically rewrite each problem into three controlled variants (perturbed, knowledge-enhanced, and math abstraction), enabling us to separately evaluate the model's robustness, domain-specific knowledge, and mathematical reasoning abilities. Experimental results show clear performance stratification across difficulty levels: model accuracy declines with task complexity, degrades under minor perturbations, and remains substantially below human performance on high-level engineering tasks. These findings reveal that current LLMs still lack the high-level reasoning needed for real-world engineering, highlighting the need for future models with deeper and more reliable problem-solving capabilities. Our source code and data are available at https://github.com/AI4Engi/EngiBench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_17677 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solving Zhou, Xiyuan Wang, Xinlei He, Yirui Wu, Yang Zou, Ruixi Cheng, Yuheng Xie, Yulu Liu, Wenxuan Zhao, Huan Xu, Yan Gu, Jinjin Zhao, Junhua Artificial Intelligence Large language models (LLMs) have shown strong performance on mathematical reasoning under well-defined conditions. However, real-world engineering problems involve uncertainty, context, and open-ended settings that extend beyond symbolic computation. Existing benchmarks largely focus on well-defined or abstract reasoning and therefore fail to capture these complexities. We introduce EngiBench, a hierarchical benchmark designed to evaluate LLMs on solving engineering problems. It spans three levels of increasing difficulty (foundational knowledge retrieval, contextual reasoning, and open-ended modeling) and covers diverse engineering subfields. To facilitate a deeper understanding of model performance, we systematically rewrite each problem into three controlled variants (perturbed, knowledge-enhanced, and math abstraction), enabling us to separately evaluate the model's robustness, domain-specific knowledge, and mathematical reasoning abilities. Experimental results show clear performance stratification across difficulty levels: model accuracy declines with task complexity, degrades under minor perturbations, and remains substantially below human performance on high-level engineering tasks. These findings reveal that current LLMs still lack the high-level reasoning needed for real-world engineering, highlighting the need for future models with deeper and more reliable problem-solving capabilities. Our source code and data are available at https://github.com/AI4Engi/EngiBench. |
| title | EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solving |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2509.17677 |