Exploring Recommender System Evaluation: A Multi-Modal User Agent Framework for A/B Testing
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918277830148096 |
|---|---|
| author | Zhang, Wenlin Li, Xiangyang Ge, Qiyuan Dong, Kuicai Jia, Pengyue Li, Xiaopeng Zhang, Zijian Wang, Maolin Wang, Yichao Guo, Huifeng Tang, Ruiming Zhao, Xiangyu |
| author_facet | Zhang, Wenlin Li, Xiangyang Ge, Qiyuan Dong, Kuicai Jia, Pengyue Li, Xiaopeng Zhang, Zijian Wang, Maolin Wang, Yichao Guo, Huifeng Tang, Ruiming Zhao, Xiangyu |
| contents | In recommender systems, online A/B testing is a crucial method for evaluating the performance of different models. However, conducting online A/B testing often presents significant challenges, including substantial economic costs, user experience degradation, and considerable time requirements. With the Large Language Models' powerful capacity, LLM-based agent shows great potential to replace traditional online A/B testing. Nonetheless, current agents fail to simulate the perception process and interaction patterns, due to the lack of real environments and visual perception capability. To address these challenges, we introduce a multi-modal user agent for A/B testing (A/B Agent). Specifically, we construct a recommendation sandbox environment for A/B testing, enabling multimodal and multi-page interactions that align with real user behavior on online platforms. The designed agent leverages multimodal information perception, fine-grained user preferences, and integrates profiles, action memory retrieval, and a fatigue system to simulate complex human decision-making. We validated the potential of the agent as an alternative to traditional A/B testing from three perspectives: model, data, and features. Furthermore, we found that the data generated by A/B Agent can effectively enhance the capabilities of recommendation models. Our code is publicly available at https://github.com/Applied-Machine-Learning-Lab/ABAgent. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_04554 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Exploring Recommender System Evaluation: A Multi-Modal User Agent Framework for A/B Testing Zhang, Wenlin Li, Xiangyang Ge, Qiyuan Dong, Kuicai Jia, Pengyue Li, Xiaopeng Zhang, Zijian Wang, Maolin Wang, Yichao Guo, Huifeng Tang, Ruiming Zhao, Xiangyu Information Retrieval In recommender systems, online A/B testing is a crucial method for evaluating the performance of different models. However, conducting online A/B testing often presents significant challenges, including substantial economic costs, user experience degradation, and considerable time requirements. With the Large Language Models' powerful capacity, LLM-based agent shows great potential to replace traditional online A/B testing. Nonetheless, current agents fail to simulate the perception process and interaction patterns, due to the lack of real environments and visual perception capability. To address these challenges, we introduce a multi-modal user agent for A/B testing (A/B Agent). Specifically, we construct a recommendation sandbox environment for A/B testing, enabling multimodal and multi-page interactions that align with real user behavior on online platforms. The designed agent leverages multimodal information perception, fine-grained user preferences, and integrates profiles, action memory retrieval, and a fatigue system to simulate complex human decision-making. We validated the potential of the agent as an alternative to traditional A/B testing from three perspectives: model, data, and features. Furthermore, we found that the data generated by A/B Agent can effectively enhance the capabilities of recommendation models. Our code is publicly available at https://github.com/Applied-Machine-Learning-Lab/ABAgent. |
| title | Exploring Recommender System Evaluation: A Multi-Modal User Agent Framework for A/B Testing |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2601.04554 |