FutureSim: Replaying World Events to Evaluate Adaptive Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911685946638336 |
|---|---|
| author | Goel, Shashwat Chandak, Nikhil Arun, Arvindh Prabhu, Ameya Staab, Steffen Hardt, Moritz Andriushchenko, Maksym Geiping, Jonas |
| author_facet | Goel, Shashwat Chandak, Nikhil Arun, Arvindh Prabhu, Ameya Staab, Steffen Hardt, Moritz Andriushchenko, Maksym Geiping, Jonas |
| contents | AI agents are being increasingly deployed in dynamic, open-ended environments that require adapting to new information as it arrives. To efficiently measure this capability for realistic use-cases, we propose building grounded simulations that replay real-world events in the order they occurred. We build FutureSim, where agents forecast world events beyond their knowledge cutoff while interacting with a chronological replay of the world: real news articles arriving and questions resolving over the simulated period. We evaluate frontier agents in their native harness, testing their ability to predict world events over a three-month period from January to March 2026. FutureSim reveals a clear separation in their capabilities, with the best agent's accuracy being 25%, and many having worse Brier skill score than making no prediction at all. Through careful ablations, we show how FutureSim offers a realistic setting to study emerging research directions like long-horizon test-time adaptation, search, memory, and reasoning about uncertainty. Overall, we hope our benchmark design paves the way to measure AI progress on open-ended adaptation spanning long time-horizons in the real world. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_15188 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | FutureSim: Replaying World Events to Evaluate Adaptive Agents Goel, Shashwat Chandak, Nikhil Arun, Arvindh Prabhu, Ameya Staab, Steffen Hardt, Moritz Andriushchenko, Maksym Geiping, Jonas Machine Learning Artificial Intelligence Computation and Language AI agents are being increasingly deployed in dynamic, open-ended environments that require adapting to new information as it arrives. To efficiently measure this capability for realistic use-cases, we propose building grounded simulations that replay real-world events in the order they occurred. We build FutureSim, where agents forecast world events beyond their knowledge cutoff while interacting with a chronological replay of the world: real news articles arriving and questions resolving over the simulated period. We evaluate frontier agents in their native harness, testing their ability to predict world events over a three-month period from January to March 2026. FutureSim reveals a clear separation in their capabilities, with the best agent's accuracy being 25%, and many having worse Brier skill score than making no prediction at all. Through careful ablations, we show how FutureSim offers a realistic setting to study emerging research directions like long-horizon test-time adaptation, search, memory, and reasoning about uncertainty. Overall, we hope our benchmark design paves the way to measure AI progress on open-ended adaptation spanning long time-horizons in the real world. |
| title | FutureSim: Replaying World Events to Evaluate Adaptive Agents |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2605.15188 |