Beyond Offline A/B Testing: Context-Aware Agent Simulation for Recommender System Evaluation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911737834373120 |
|---|---|
| author | Bougie, Nicolas Marconi, Gian Maria Ye, Xiaotong Watanabe, Narimasa |
| author_facet | Bougie, Nicolas Marconi, Gian Maria Ye, Xiaotong Watanabe, Narimasa |
| contents | Recommender systems are central to online services, enabling users to navigate through massive amounts of content across various domains. However, their evaluation remains challenging due to the disconnect between offline metrics and online performance. The emergence of Large Language Model-powered agents offers a promising solution, yet existing studies model users in isolation, neglecting the contextual factors such as time, location, and needs, which fundamentally shape human decision-making. In this paper, we introduce ContextSim, an LLM agent framework that simulates believable user proxies by anchoring interactions in daily life activities. Namely, a life simulation module generates scenarios specifying when, where, and why users engage with recommendations. To align preferences with genuine humans, we model agents' internal thoughts and enforce consistency at both the action and trajectory levels. Experiments across domains show our method generates interactions more closely aligned with human behavior than prior work. We further validate our approach through offline A/B testing correlation and show that RS parameters optimized using ContextSim yield improved real-world engagement. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_09549 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Beyond Offline A/B Testing: Context-Aware Agent Simulation for Recommender System Evaluation Bougie, Nicolas Marconi, Gian Maria Ye, Xiaotong Watanabe, Narimasa Information Retrieval Artificial Intelligence Recommender systems are central to online services, enabling users to navigate through massive amounts of content across various domains. However, their evaluation remains challenging due to the disconnect between offline metrics and online performance. The emergence of Large Language Model-powered agents offers a promising solution, yet existing studies model users in isolation, neglecting the contextual factors such as time, location, and needs, which fundamentally shape human decision-making. In this paper, we introduce ContextSim, an LLM agent framework that simulates believable user proxies by anchoring interactions in daily life activities. Namely, a life simulation module generates scenarios specifying when, where, and why users engage with recommendations. To align preferences with genuine humans, we model agents' internal thoughts and enforce consistency at both the action and trajectory levels. Experiments across domains show our method generates interactions more closely aligned with human behavior than prior work. We further validate our approach through offline A/B testing correlation and show that RS parameters optimized using ContextSim yield improved real-world engagement. |
| title | Beyond Offline A/B Testing: Context-Aware Agent Simulation for Recommender System Evaluation |
| topic | Information Retrieval Artificial Intelligence |
| url | https://arxiv.org/abs/2604.09549 |