Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.19219 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910234907246592 |
|---|---|
| author | Li, Han Malik, Vibhor Foumani, Zahra Zanjani Castelo, Alberto Xie, Shuang Fan, Ailin Koay, Keat Yang Zhu, Yuanzheng Feghhi, Meysam Uliana, Ronie Zhang, Zhaoyu Martins, Angelo Ocana Zhao, Mingyu Pelland, Francis Faerman, Jonathan LeBlanc, Nikolas Glazer, Aaron McNamara, Andrew Wu, Zhong Wang, Lingyun |
| author_facet | Li, Han Malik, Vibhor Foumani, Zahra Zanjani Castelo, Alberto Xie, Shuang Fan, Ailin Koay, Keat Yang Zhu, Yuanzheng Feghhi, Meysam Uliana, Ronie Zhang, Zhaoyu Martins, Angelo Ocana Zhao, Mingyu Pelland, Francis Faerman, Jonathan LeBlanc, Nikolas Glazer, Aaron McNamara, Andrew Wu, Zhong Wang, Lingyun |
| contents | A/B testing remains the gold standard for evaluating modifications to e-commerce storefronts, yet it diverts traffic, requires weeks to reach statistical significance, and risks degrading user experience. We present SimGym, a framework for simulating A/B tests on e-commerce storefronts using vision-language model (VLM) agents operating in a live browser. The framework comprises three key components: (a) a traffic-grounded persona generation pipeline that derives per-shop buyer archetypes and intents from production clickstream data; (b) a live-browser agent architecture that combines multimodal perception over visual and browser-structured observations with episodic memory and guardrails to conduct coherent shopping sessions across control and treatment storefronts; and (c) an evaluation protocol that compares simulated outcome shifts with observed shifts in real buyer behavior. We validate SimGym on A/B tests of visually driven UI theme changes from a major e-commerce platform across diverse storefronts and product categories. Empirical results show that SimGym agents achieve strong agreement with observed outcome shifts, attaining 77% directional alignment with add-to-cart shifts observed across interface variants in real-buyer traffic. It reduces experimental cycles from weeks to under an hour, enabling rapid experimentation without exposing real buyers to candidate variants. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_19219 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agents Li, Han Malik, Vibhor Foumani, Zahra Zanjani Castelo, Alberto Xie, Shuang Fan, Ailin Koay, Keat Yang Zhu, Yuanzheng Feghhi, Meysam Uliana, Ronie Zhang, Zhaoyu Martins, Angelo Ocana Zhao, Mingyu Pelland, Francis Faerman, Jonathan LeBlanc, Nikolas Glazer, Aaron McNamara, Andrew Wu, Zhong Wang, Lingyun Artificial Intelligence A/B testing remains the gold standard for evaluating modifications to e-commerce storefronts, yet it diverts traffic, requires weeks to reach statistical significance, and risks degrading user experience. We present SimGym, a framework for simulating A/B tests on e-commerce storefronts using vision-language model (VLM) agents operating in a live browser. The framework comprises three key components: (a) a traffic-grounded persona generation pipeline that derives per-shop buyer archetypes and intents from production clickstream data; (b) a live-browser agent architecture that combines multimodal perception over visual and browser-structured observations with episodic memory and guardrails to conduct coherent shopping sessions across control and treatment storefronts; and (c) an evaluation protocol that compares simulated outcome shifts with observed shifts in real buyer behavior. We validate SimGym on A/B tests of visually driven UI theme changes from a major e-commerce platform across diverse storefronts and product categories. Empirical results show that SimGym agents achieve strong agreement with observed outcome shifts, attaining 77% directional alignment with add-to-cart shifts observed across interface variants in real-buyer traffic. It reduces experimental cycles from weeks to under an hour, enabling rapid experimentation without exposing real buyers to candidate variants. |
| title | SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agents |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2605.19219 |