Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.08868 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908705251917824 |
|---|---|
| author | Min, Rui Qiao, Zile Xu, Ze Zhai, Jiawen Gao, Wenyu Chen, Xuanzhong Sun, Haozhen Zhang, Zhen Wang, Xinyu Zhou, Hong Yin, Wenbiao Zhang, Bo Zhou, Xuan Yan, Ming Jiang, Yong Liu, Haicheng Ding, Liang Zou, Ling Fung, Yi R. Li, Yalong Xie, Pengjun |
| author_facet | Min, Rui Qiao, Zile Xu, Ze Zhai, Jiawen Gao, Wenyu Chen, Xuanzhong Sun, Haozhen Zhang, Zhen Wang, Xinyu Zhou, Hong Yin, Wenbiao Zhang, Bo Zhou, Xuan Yan, Ming Jiang, Yong Liu, Haicheng Ding, Liang Zou, Ling Fung, Yi R. Li, Yalong Xie, Pengjun |
| contents | Foundation agents have rapidly advanced in their ability to reason and interact with real environments, making the evaluation of their core capabilities increasingly important. While many benchmarks have been developed to assess agent performance, most concentrate on academic settings or artificially designed scenarios while overlooking the challenges that arise in real applications. To address this issue, we focus on a highly practical real-world setting, the e-commerce domain, which involves a large volume of diverse user interactions, dynamic market conditions, and tasks directly tied to real decision-making processes. To this end, we introduce EcomBench, a holistic E-commerce Benchmark designed to evaluate agent performance in realistic e-commerce environments. EcomBench is built from genuine user demands embedded in leading global e-commerce ecosystems and is carefully curated and annotated through human experts to ensure clarity, accuracy, and domain relevance. It covers multiple task categories within e-commerce scenarios and defines three difficulty levels that evaluate agents on key capabilities such as deep information retrieval, multi-step reasoning, and cross-source knowledge integration. By grounding evaluation in real e-commerce contexts, EcomBench provides a rigorous and dynamic testbed for measuring the practical capabilities of agents in modern e-commerce. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_08868 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EcomBench: Towards Holistic Evaluation of Foundation Agents in E-commerce Min, Rui Qiao, Zile Xu, Ze Zhai, Jiawen Gao, Wenyu Chen, Xuanzhong Sun, Haozhen Zhang, Zhen Wang, Xinyu Zhou, Hong Yin, Wenbiao Zhang, Bo Zhou, Xuan Yan, Ming Jiang, Yong Liu, Haicheng Ding, Liang Zou, Ling Fung, Yi R. Li, Yalong Xie, Pengjun Artificial Intelligence Foundation agents have rapidly advanced in their ability to reason and interact with real environments, making the evaluation of their core capabilities increasingly important. While many benchmarks have been developed to assess agent performance, most concentrate on academic settings or artificially designed scenarios while overlooking the challenges that arise in real applications. To address this issue, we focus on a highly practical real-world setting, the e-commerce domain, which involves a large volume of diverse user interactions, dynamic market conditions, and tasks directly tied to real decision-making processes. To this end, we introduce EcomBench, a holistic E-commerce Benchmark designed to evaluate agent performance in realistic e-commerce environments. EcomBench is built from genuine user demands embedded in leading global e-commerce ecosystems and is carefully curated and annotated through human experts to ensure clarity, accuracy, and domain relevance. It covers multiple task categories within e-commerce scenarios and defines three difficulty levels that evaluate agents on key capabilities such as deep information retrieval, multi-step reasoning, and cross-source knowledge integration. By grounding evaluation in real e-commerce contexts, EcomBench provides a rigorous and dynamic testbed for measuring the practical capabilities of agents in modern e-commerce. |
| title | EcomBench: Towards Holistic Evaluation of Foundation Agents in E-commerce |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2512.08868 |