ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917136327245824 |
|---|---|
| author | Wang, Jiangyuan Xiao, Kejun Sun, Qi Zhao, Huaipeng Luo, Tao Zhang, Jian Dong Zeng, Xiaoyi |
| author_facet | Wang, Jiangyuan Xiao, Kejun Sun, Qi Zhao, Huaipeng Luo, Tao Zhang, Jian Dong Zeng, Xiaoyi |
| contents | Existing benchmarks in e-commerce primarily focus on basic user intents, such as finding or purchasing products. However, real-world users often pursue more complex goals, such as applying vouchers, managing budgets, and finding multi-products seller. To bridge this gap, we propose ShoppingBench, a novel end-to-end shopping benchmark designed to encompass increasingly challenging levels of grounded intent. Specifically, we propose a scalable framework to simulate user instructions based on various intents derived from sampled real-world products. To facilitate consistent and reliable evaluations, we provide a large-scale shopping sandbox that serves as an interactive simulated environment, incorporating over 2.5 million real-world products. Experimental results demonstrate that even state-of-the-art language agents (such as GPT-4.1) achieve absolute success rates under 50% on our benchmark tasks, highlighting the significant challenges posed by our ShoppingBench. In addition, we propose a trajectory distillation strategy and leverage supervised fine-tuning, along with reinforcement learning on synthetic trajectories, to distill the capabilities of a large language agent into a smaller one. As a result, our trained agent achieves competitive performance compared to GPT-4.1. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_04266 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents Wang, Jiangyuan Xiao, Kejun Sun, Qi Zhao, Huaipeng Luo, Tao Zhang, Jian Dong Zeng, Xiaoyi Computation and Language Existing benchmarks in e-commerce primarily focus on basic user intents, such as finding or purchasing products. However, real-world users often pursue more complex goals, such as applying vouchers, managing budgets, and finding multi-products seller. To bridge this gap, we propose ShoppingBench, a novel end-to-end shopping benchmark designed to encompass increasingly challenging levels of grounded intent. Specifically, we propose a scalable framework to simulate user instructions based on various intents derived from sampled real-world products. To facilitate consistent and reliable evaluations, we provide a large-scale shopping sandbox that serves as an interactive simulated environment, incorporating over 2.5 million real-world products. Experimental results demonstrate that even state-of-the-art language agents (such as GPT-4.1) achieve absolute success rates under 50% on our benchmark tasks, highlighting the significant challenges posed by our ShoppingBench. In addition, we propose a trajectory distillation strategy and leverage supervised fine-tuning, along with reinforcement learning on synthetic trajectories, to distill the capabilities of a large language agent into a smaller one. As a result, our trained agent achieves competitive performance compared to GPT-4.1. |
| title | ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2508.04266 |