Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.12290 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908994893774848 |
|---|---|
| author | Chi, Yizhe Hong, Deyao Jiang, Dapeng Luo, Tianwei Yang, Kaisen Zhang, Boshi Cao, Zhe Fan, Xiaoyan He, Bingxiang Hao, Han Jin, Weiyang Lei, Dianqiao Liu, Qingle Qian, Houde Wang, Bowen Wang, Situ Zheng, Youjie Zhou, Yifan Xiao, Calvin Cai, Eren Na, Qinhuai |
| author_facet | Chi, Yizhe Hong, Deyao Jiang, Dapeng Luo, Tianwei Yang, Kaisen Zhang, Boshi Cao, Zhe Fan, Xiaoyan He, Bingxiang Hao, Han Jin, Weiyang Lei, Dianqiao Liu, Qingle Qian, Houde Wang, Bowen Wang, Situ Zheng, Youjie Zhou, Yifan Xiao, Calvin Cai, Eren Na, Qinhuai |
| contents | Current LLM agent benchmarks, which predominantly focus on binary pass/fail tasks such as code generation or search-based question answering, often neglect the value of real-world engineering that is often captured through the iterative optimization of feasible designs. To this end, we introduce Frontier-Eng, a human-verified benchmark for generative optimization -- an iterative propose-execute-evaluate loop in which an agent generates candidate artifacts, receives executable verifier feedback, and revises them under a fixed interaction budget -- spanning $47$ tasks across five broad engineering categories. Unlike previous suites, Frontier-Eng tasks are grounded in industrial-grade simulators and verifiers that provide continuous reward signals and enforce hard feasibility constraints under constrained budgets. We evaluate eight frontier language models using representative search frameworks, finding that while GPT 5.4 achieves the most robust performance, the benchmark remains challenging for all models. Our analysis suggests a dual power-law decay in improvement frequency ($\sim$ 1/iteration) and magnitude ($\sim$ 1/improvement count). We further show that although width improves parallelism and diversity, depth remains crucial for hard-won improvements under a fixed budget. Frontier-Eng establishes a new standard for assessing the capacity of AI agents to integrate domain knowledge with executable feedback to solve complex, open-ended engineering problems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_12290 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization Chi, Yizhe Hong, Deyao Jiang, Dapeng Luo, Tianwei Yang, Kaisen Zhang, Boshi Cao, Zhe Fan, Xiaoyan He, Bingxiang Hao, Han Jin, Weiyang Lei, Dianqiao Liu, Qingle Qian, Houde Wang, Bowen Wang, Situ Zheng, Youjie Zhou, Yifan Xiao, Calvin Cai, Eren Na, Qinhuai Artificial Intelligence Computation and Language Current LLM agent benchmarks, which predominantly focus on binary pass/fail tasks such as code generation or search-based question answering, often neglect the value of real-world engineering that is often captured through the iterative optimization of feasible designs. To this end, we introduce Frontier-Eng, a human-verified benchmark for generative optimization -- an iterative propose-execute-evaluate loop in which an agent generates candidate artifacts, receives executable verifier feedback, and revises them under a fixed interaction budget -- spanning $47$ tasks across five broad engineering categories. Unlike previous suites, Frontier-Eng tasks are grounded in industrial-grade simulators and verifiers that provide continuous reward signals and enforce hard feasibility constraints under constrained budgets. We evaluate eight frontier language models using representative search frameworks, finding that while GPT 5.4 achieves the most robust performance, the benchmark remains challenging for all models. Our analysis suggests a dual power-law decay in improvement frequency ($\sim$ 1/iteration) and magnitude ($\sim$ 1/improvement count). We further show that although width improves parallelism and diversity, depth remains crucial for hard-won improvements under a fixed budget. Frontier-Eng establishes a new standard for assessing the capacity of AI agents to integrate domain knowledge with executable feedback to solve complex, open-ended engineering problems. |
| title | Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2604.12290 |