EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economies
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914547141443584 |
|---|---|
| author | Hu, Xavier Xia, Jinxiang Xu, Shengze Song, Kangqi Yuan, Yishuo Zhang, Guibin Ren, JinCheng Feng, Boyu Lu, Li Zeng, Tieyong Liu, Jiaheng Liu, Minghao Zhu, He Jiang, Yuchen Eleanor Wang, Wei Zhou, Wangchunshu |
| author_facet | Hu, Xavier Xia, Jinxiang Xu, Shengze Song, Kangqi Yuan, Yishuo Zhang, Guibin Ren, JinCheng Feng, Boyu Lu, Li Zeng, Tieyong Liu, Jiaheng Liu, Minghao Zhu, He Jiang, Yuchen Eleanor Wang, Wei Zhou, Wangchunshu |
| contents | Long-horizon planning is widely recognized as a core capability of autonomous LLM-based agents; however, current evaluation frameworks suffer from being largely episodic, domain-specific, or insufficiently grounded in persistent economic dynamics. We introduce EcoGym, a generalizable benchmark for continuous plan-and-execute decision making in interactive economies. EcoGym comprises three diverse environments: Vending (adapted from the closed-source Vending-Bench, with full open-source release), Freelance (new), and Operation (new), implemented in a unified decision-making process with standardized interfaces, and budgeted actions over an effectively unbounded horizon (1000+ steps if 365 day-loops for evaluation). The evaluation of EcoGym is based on business-relevant outcomes (e.g., net worth, income, and DAU), targeting long-term strategic coherence and robustness under partial observability and stochasticity. Experiments across eleven leading LLMs expose a systematic tension: no single model dominates across all three scenarios. Critically, we find that models exhibit significant suboptimality in either high-level strategies or efficient actions executions. EcoGym is released as an open, extensible testbed for transparent long-horizon agent evaluation and for studying controllability utility trade-offs in economic settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_09514 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economies Hu, Xavier Xia, Jinxiang Xu, Shengze Song, Kangqi Yuan, Yishuo Zhang, Guibin Ren, JinCheng Feng, Boyu Lu, Li Zeng, Tieyong Liu, Jiaheng Liu, Minghao Zhu, He Jiang, Yuchen Eleanor Wang, Wei Zhou, Wangchunshu Computation and Language Artificial Intelligence Long-horizon planning is widely recognized as a core capability of autonomous LLM-based agents; however, current evaluation frameworks suffer from being largely episodic, domain-specific, or insufficiently grounded in persistent economic dynamics. We introduce EcoGym, a generalizable benchmark for continuous plan-and-execute decision making in interactive economies. EcoGym comprises three diverse environments: Vending (adapted from the closed-source Vending-Bench, with full open-source release), Freelance (new), and Operation (new), implemented in a unified decision-making process with standardized interfaces, and budgeted actions over an effectively unbounded horizon (1000+ steps if 365 day-loops for evaluation). The evaluation of EcoGym is based on business-relevant outcomes (e.g., net worth, income, and DAU), targeting long-term strategic coherence and robustness under partial observability and stochasticity. Experiments across eleven leading LLMs expose a systematic tension: no single model dominates across all three scenarios. Critically, we find that models exhibit significant suboptimality in either high-level strategies or efficient actions executions. EcoGym is released as an open, extensible testbed for transparent long-horizon agent evaluation and for studying controllability utility trade-offs in economic settings. |
| title | EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economies |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2602.09514 |