REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910913399881728 |
|---|---|
| author | Garg, Divyansh VanWeelden, Shaun Caples, Diego Draguns, Andis Ravi, Nikil Putta, Pranav Garg, Naman Abraham, Tomas Lara, Michael Lopez, Federico Liu, James Gundawar, Atharva Hebbar, Prannay Joo, Youngchul Gu, Jindong London, Charles de Witt, Christian Schroeder Motwani, Sumeet |
| author_facet | Garg, Divyansh VanWeelden, Shaun Caples, Diego Draguns, Andis Ravi, Nikil Putta, Pranav Garg, Naman Abraham, Tomas Lara, Michael Lopez, Federico Liu, James Gundawar, Atharva Hebbar, Prannay Joo, Youngchul Gu, Jindong London, Charles de Witt, Christian Schroeder Motwani, Sumeet |
| contents | We introduce REAL, a benchmark and framework for multi-turn agent evaluations on deterministic simulations of real-world websites. REAL comprises high-fidelity, deterministic replicas of 11 widely-used websites across domains such as e-commerce, travel, communication, and professional networking. We also release a benchmark consisting of 112 practical tasks that mirror everyday complex user interactions requiring both accurate information retrieval and state-changing actions. All interactions occur within this fully controlled setting, eliminating safety risks and enabling robust, reproducible evaluation of agent capability and reliability. Our novel evaluation framework combines programmatic checks of website state for action-based tasks with rubric-guided LLM-based judgments for information retrieval. The framework supports both open-source and proprietary agent systems through a flexible evaluation harness that accommodates black-box commands within browser environments, allowing research labs to test agentic systems without modification. Our empirical results show that frontier language models achieve at most a 41% success rate on REAL, highlighting critical gaps in autonomous web navigation and task completion capabilities. Our framework supports easy integration of new tasks, reproducible evaluation, and scalable post-training data generation, marking a significant step forward in evaluating and advancing agent capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_11543 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites Garg, Divyansh VanWeelden, Shaun Caples, Diego Draguns, Andis Ravi, Nikil Putta, Pranav Garg, Naman Abraham, Tomas Lara, Michael Lopez, Federico Liu, James Gundawar, Atharva Hebbar, Prannay Joo, Youngchul Gu, Jindong London, Charles de Witt, Christian Schroeder Motwani, Sumeet Artificial Intelligence We introduce REAL, a benchmark and framework for multi-turn agent evaluations on deterministic simulations of real-world websites. REAL comprises high-fidelity, deterministic replicas of 11 widely-used websites across domains such as e-commerce, travel, communication, and professional networking. We also release a benchmark consisting of 112 practical tasks that mirror everyday complex user interactions requiring both accurate information retrieval and state-changing actions. All interactions occur within this fully controlled setting, eliminating safety risks and enabling robust, reproducible evaluation of agent capability and reliability. Our novel evaluation framework combines programmatic checks of website state for action-based tasks with rubric-guided LLM-based judgments for information retrieval. The framework supports both open-source and proprietary agent systems through a flexible evaluation harness that accommodates black-box commands within browser environments, allowing research labs to test agentic systems without modification. Our empirical results show that frontier language models achieve at most a 41% success rate on REAL, highlighting critical gaps in autonomous web navigation and task completion capabilities. Our framework supports easy integration of new tasks, reproducible evaluation, and scalable post-training data generation, marking a significant step forward in evaluating and advancing agent capabilities. |
| title | REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2504.11543 |