FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916935346683904 |
|---|---|
| author | Zeng, Zhiyuan Liu, Jiashuo Chen, Siyuan He, Tianci Liao, Yali Tian, Yixiao Wang, Jinpeng Wang, Zaiyuan Yang, Yang Yin, Lingyue Yin, Mingren Zhu, Zhenwei Cai, Tianle Chen, Zehui Chen, Jiecao Du, Yantao Gao, Xiang Guo, Jiacheng Hu, Liang Jiao, Jianpeng Li, Xiangsheng Liu, Jingkai Ni, Shuang Wen, Zhoufutu Zhang, Ge Zhang, Kaiyuan Zhou, Xin Blanchet, Jose Qiu, Xipeng Wang, Mengdi Huang, Wenhao |
| author_facet | Zeng, Zhiyuan Liu, Jiashuo Chen, Siyuan He, Tianci Liao, Yali Tian, Yixiao Wang, Jinpeng Wang, Zaiyuan Yang, Yang Yin, Lingyue Yin, Mingren Zhu, Zhenwei Cai, Tianle Chen, Zehui Chen, Jiecao Du, Yantao Gao, Xiang Guo, Jiacheng Hu, Liang Jiao, Jianpeng Li, Xiangsheng Liu, Jingkai Ni, Shuang Wen, Zhoufutu Zhang, Ge Zhang, Kaiyuan Zhou, Xin Blanchet, Jose Qiu, Xipeng Wang, Mengdi Huang, Wenhao |
| contents | Future prediction is a complex task for LLM agents, requiring a high level of analytical thinking, information gathering, contextual understanding, and decision-making under uncertainty. Agents must not only gather and interpret vast amounts of dynamic information but also integrate diverse data sources, weigh uncertainties, and adapt predictions based on emerging trends, just as human experts do in fields like politics, economics, and finance. Despite its importance, no large-scale benchmark exists for evaluating agents on future prediction, largely due to challenges in handling real-time updates and retrieving timely, accurate answers. To address this, we introduce $\textbf{FutureX}$, a dynamic and live evaluation benchmark specifically designed for LLM agents performing future prediction tasks. FutureX is the largest and most diverse live benchmark for future prediction, supporting real-time daily updates and eliminating data contamination through an automated pipeline for question gathering and answer collection. We evaluate 25 LLM/agent models, including those with reasoning, search capabilities, and integration of external tools such as the open-source Deep Research Agent and closed-source Deep Research models. This comprehensive evaluation assesses agents' adaptive reasoning and performance in dynamic environments. Additionally, we provide in-depth analyses of agents' failure modes and performance pitfalls in future-oriented tasks, including the vulnerability to fake web pages and the temporal validity. Our goal is to establish a dynamic, contamination-free evaluation standard that drives the development of LLM agents capable of performing at the level of professional human analysts in complex reasoning and predictive thinking. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_11987 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction Zeng, Zhiyuan Liu, Jiashuo Chen, Siyuan He, Tianci Liao, Yali Tian, Yixiao Wang, Jinpeng Wang, Zaiyuan Yang, Yang Yin, Lingyue Yin, Mingren Zhu, Zhenwei Cai, Tianle Chen, Zehui Chen, Jiecao Du, Yantao Gao, Xiang Guo, Jiacheng Hu, Liang Jiao, Jianpeng Li, Xiangsheng Liu, Jingkai Ni, Shuang Wen, Zhoufutu Zhang, Ge Zhang, Kaiyuan Zhou, Xin Blanchet, Jose Qiu, Xipeng Wang, Mengdi Huang, Wenhao Artificial Intelligence Machine Learning Future prediction is a complex task for LLM agents, requiring a high level of analytical thinking, information gathering, contextual understanding, and decision-making under uncertainty. Agents must not only gather and interpret vast amounts of dynamic information but also integrate diverse data sources, weigh uncertainties, and adapt predictions based on emerging trends, just as human experts do in fields like politics, economics, and finance. Despite its importance, no large-scale benchmark exists for evaluating agents on future prediction, largely due to challenges in handling real-time updates and retrieving timely, accurate answers. To address this, we introduce $\textbf{FutureX}$, a dynamic and live evaluation benchmark specifically designed for LLM agents performing future prediction tasks. FutureX is the largest and most diverse live benchmark for future prediction, supporting real-time daily updates and eliminating data contamination through an automated pipeline for question gathering and answer collection. We evaluate 25 LLM/agent models, including those with reasoning, search capabilities, and integration of external tools such as the open-source Deep Research Agent and closed-source Deep Research models. This comprehensive evaluation assesses agents' adaptive reasoning and performance in dynamic environments. Additionally, we provide in-depth analyses of agents' failure modes and performance pitfalls in future-oriented tasks, including the vulnerability to fake web pages and the temporal validity. Our goal is to establish a dynamic, contamination-free evaluation standard that drives the development of LLM agents capable of performing at the level of professional human analysts in complex reasoning and predictive thinking. |
| title | FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2508.11987 |