SimAB: Simulating A/B Tests with Persona-Conditioned AI Agents for Rapid Design Evaluation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917302945972224 |
|---|---|
| author | Rieder, Tim Schneider, Marian Truss, Mario Tsaplin, Vitaly Rublea, Alina Dere, Sinem Sanz, Francisco Chicharro Reiss, Tobias Dogan, Mustafa Doga |
| author_facet | Rieder, Tim Schneider, Marian Truss, Mario Tsaplin, Vitaly Rublea, Alina Dere, Sinem Sanz, Francisco Chicharro Reiss, Tobias Dogan, Mustafa Doga |
| contents | A/B testing is a standard method for validating design decisions, yet its reliance on real user traffic limits iteration speed and makes certain experiments impractical. We present SimAB, a system that reframes A/B testing as a fast, privacy-preserving simulation using persona-conditioned AI agents. Given design screenshots and a conversion goal, SimAB generates user personas, deploys them as agents that state their preference, aggregates results, and synthesizes rationales. Through a formative study with experimentation practitioners, we identified scenarios where traffic constraints hinder testing, including low-traffic pages, multi-variant comparisons, micro-optimizations, and privacy-sensitive contexts. Our design emphasizes speed, early feedback, actionable rationales, and audience specification. We evaluate SimAB against 47 historical A/B tests with known outcomes, achieving 67% overall accuracy, increasing to 83% for high-confidence cases. Additional experiments show robustness to naming and positional bias and demonstrate accuracy gains from personas. Practitioner feedback suggests that SimAB supports faster evaluation cycles and rapid screening of designs difficult to assess with traditional A/B tests. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_01024 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SimAB: Simulating A/B Tests with Persona-Conditioned AI Agents for Rapid Design Evaluation Rieder, Tim Schneider, Marian Truss, Mario Tsaplin, Vitaly Rublea, Alina Dere, Sinem Sanz, Francisco Chicharro Reiss, Tobias Dogan, Mustafa Doga Human-Computer Interaction Artificial Intelligence Multiagent Systems I.2.11; H.5.2 A/B testing is a standard method for validating design decisions, yet its reliance on real user traffic limits iteration speed and makes certain experiments impractical. We present SimAB, a system that reframes A/B testing as a fast, privacy-preserving simulation using persona-conditioned AI agents. Given design screenshots and a conversion goal, SimAB generates user personas, deploys them as agents that state their preference, aggregates results, and synthesizes rationales. Through a formative study with experimentation practitioners, we identified scenarios where traffic constraints hinder testing, including low-traffic pages, multi-variant comparisons, micro-optimizations, and privacy-sensitive contexts. Our design emphasizes speed, early feedback, actionable rationales, and audience specification. We evaluate SimAB against 47 historical A/B tests with known outcomes, achieving 67% overall accuracy, increasing to 83% for high-confidence cases. Additional experiments show robustness to naming and positional bias and demonstrate accuracy gains from personas. Practitioner feedback suggests that SimAB supports faster evaluation cycles and rapid screening of designs difficult to assess with traditional A/B tests. |
| title | SimAB: Simulating A/B Tests with Persona-Conditioned AI Agents for Rapid Design Evaluation |
| topic | Human-Computer Interaction Artificial Intelligence Multiagent Systems I.2.11; H.5.2 |
| url | https://arxiv.org/abs/2603.01024 |