Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.08077 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915439998664704 |
|---|---|
| author | Hayden, Dustin Armitage, Thomas |
| author_facet | Hayden, Dustin Armitage, Thomas |
| contents | Bayesian A/B testing investigates metric changes using the joint posterior distribution of two (or more) experimentally-derived datasets. The construction of said joint posterior is often a time-consuming process requiring specialized knowledge and domain expertise. In businesses that perform tens to hundreds of A/B tests per month it is important to have a robust analysis pipeline that can handle the variety of experiments performed on a modern web platform; requiring a domain expert to select appropriate prior and likelihood distributions for each experiment simply does not scale. In this work, we highlight a solution to this problem using a generalized approximation of the true joint posterior using a Dirichlet-Categorical model. While a manually-constructed, expert-tuned model for every dataset is preferable, the Dirichlet-Categorical approximation performs sufficiently well in both simulations and real-world scenarios to be internally used as the standard analysis method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_08077 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Straightforward Bayesian A/B testing with Dirichlet posteriors Hayden, Dustin Armitage, Thomas Methodology Bayesian A/B testing investigates metric changes using the joint posterior distribution of two (or more) experimentally-derived datasets. The construction of said joint posterior is often a time-consuming process requiring specialized knowledge and domain expertise. In businesses that perform tens to hundreds of A/B tests per month it is important to have a robust analysis pipeline that can handle the variety of experiments performed on a modern web platform; requiring a domain expert to select appropriate prior and likelihood distributions for each experiment simply does not scale. In this work, we highlight a solution to this problem using a generalized approximation of the true joint posterior using a Dirichlet-Categorical model. While a manually-constructed, expert-tuned model for every dataset is preferable, the Dirichlet-Categorical approximation performs sufficiently well in both simulations and real-world scenarios to be internally used as the standard analysis method. |
| title | Straightforward Bayesian A/B testing with Dirichlet posteriors |
| topic | Methodology |
| url | https://arxiv.org/abs/2508.08077 |