Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.09108 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912756163149824 |
|---|---|
| author | Brookes, Paul Voskanyan, Vardan Giavrimis, Rafail Truscott, Matthew Ilieva, Mina Pavlou, Chrystalla Staicu, Alexandru Adham, Manal Hood, Will Evers- Gong, Jingzhi Zhang, Kejia Fedoseev, Matvey Sharma, Vishal Bauer, Roman Wang, Zheng Nair, Hema Jie, Wei Xu, Tianhua Constantin, Aurora Kanthan, Leslie Basios, Michail |
| author_facet | Brookes, Paul Voskanyan, Vardan Giavrimis, Rafail Truscott, Matthew Ilieva, Mina Pavlou, Chrystalla Staicu, Alexandru Adham, Manal Hood, Will Evers- Gong, Jingzhi Zhang, Kejia Fedoseev, Matvey Sharma, Vishal Bauer, Roman Wang, Zheng Nair, Hema Jie, Wei Xu, Tianhua Constantin, Aurora Kanthan, Leslie Basios, Michail |
| contents | Agentic AI systems built on large language models (LLMs) offer significant potential for automating complex workflows, from software development to customer support. However, LLM agents often underperform due to suboptimal configurations; poorly tuned prompts, tool descriptions, and parameters that typically require weeks of manual refinement. Existing optimization methods either are too complex for general use or treat components in isolation, missing critical interdependencies.
We present ARTEMIS, a no-code evolutionary optimization platform that jointly optimizes agent configurations through semantically-aware genetic operators. Given only a benchmark script and natural language goals, ARTEMIS automatically discovers configurable components, extracts performance signals from execution logs, and evolves configurations without requiring architectural modifications.
We evaluate ARTEMIS on four representative agent systems: the \emph{ALE Agent} for competitive programming on AtCoder Heuristic Contest, achieving a \textbf{$13.6\%$ improvement} in acceptance rate; the \emph{Mini-SWE Agent} for code optimization on SWE-Perf, with a statistically significant \textbf{10.1\% performance gain}; and the \emph{CrewAI Agent} for cost and mathematical reasoning on Math Odyssey, achieving a statistically significant \textbf{$36.9\%$ reduction} in the number of tokens required for evaluation. We also evaluate the \emph{MathTales-Teacher Agent} powered by a smaller open-source model (Qwen2.5-7B) on GSM8K primary-level mathematics problems, achieving a \textbf{22\% accuracy improvement} and demonstrating that ARTEMIS can optimize agents based on both commercial and local models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_09108 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evolving Excellence: Automated Optimization of LLM-based Agents Brookes, Paul Voskanyan, Vardan Giavrimis, Rafail Truscott, Matthew Ilieva, Mina Pavlou, Chrystalla Staicu, Alexandru Adham, Manal Hood, Will Evers- Gong, Jingzhi Zhang, Kejia Fedoseev, Matvey Sharma, Vishal Bauer, Roman Wang, Zheng Nair, Hema Jie, Wei Xu, Tianhua Constantin, Aurora Kanthan, Leslie Basios, Michail Software Engineering Artificial Intelligence Agentic AI systems built on large language models (LLMs) offer significant potential for automating complex workflows, from software development to customer support. However, LLM agents often underperform due to suboptimal configurations; poorly tuned prompts, tool descriptions, and parameters that typically require weeks of manual refinement. Existing optimization methods either are too complex for general use or treat components in isolation, missing critical interdependencies. We present ARTEMIS, a no-code evolutionary optimization platform that jointly optimizes agent configurations through semantically-aware genetic operators. Given only a benchmark script and natural language goals, ARTEMIS automatically discovers configurable components, extracts performance signals from execution logs, and evolves configurations without requiring architectural modifications. We evaluate ARTEMIS on four representative agent systems: the \emph{ALE Agent} for competitive programming on AtCoder Heuristic Contest, achieving a \textbf{$13.6\%$ improvement} in acceptance rate; the \emph{Mini-SWE Agent} for code optimization on SWE-Perf, with a statistically significant \textbf{10.1\% performance gain}; and the \emph{CrewAI Agent} for cost and mathematical reasoning on Math Odyssey, achieving a statistically significant \textbf{$36.9\%$ reduction} in the number of tokens required for evaluation. We also evaluate the \emph{MathTales-Teacher Agent} powered by a smaller open-source model (Qwen2.5-7B) on GSM8K primary-level mathematics problems, achieving a \textbf{22\% accuracy improvement} and demonstrating that ARTEMIS can optimize agents based on both commercial and local models. |
| title | Evolving Excellence: Automated Optimization of LLM-based Agents |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2512.09108 |