Beyond Basic A/B testing: Improving Statistical Efficiency for Business Growth
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914221525041152 |
|---|---|
| author | Wei, Changshuai Nguyen, Phuc Zelditch, Benjamin Chen, Joyce |
| author_facet | Wei, Changshuai Nguyen, Phuc Zelditch, Benjamin Chen, Joyce |
| contents | The standard A/B testing approaches are mostly based on t-test in large scale industry applications. These standard approaches however suffers from low statistical power in business settings, due to nature of small sample-size or non-Gaussian distribution or return-on-investment (ROI) consideration. In this paper, we (i) show the statistical efficiency of using estimating equation and U statistics, which can address these issues separately; and (ii) propose a novel doubly robust generalized U that allows flexible definition of treatment effect, and can handles small samples, distribution robustness, ROI and confounding consideration in one framework. We provide theoretical results on asymptotics and efficiency bounds, together with insights on the efficiency gain from theoretical analysis. We further conduct comprehensive simulation studies, apply the methods to multiple real A/B tests at LinkedIn, and share results and learnings that are broadly useful. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_08128 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond Basic A/B testing: Improving Statistical Efficiency for Business Growth Wei, Changshuai Nguyen, Phuc Zelditch, Benjamin Chen, Joyce Methodology Machine Learning Statistics Theory Computation The standard A/B testing approaches are mostly based on t-test in large scale industry applications. These standard approaches however suffers from low statistical power in business settings, due to nature of small sample-size or non-Gaussian distribution or return-on-investment (ROI) consideration. In this paper, we (i) show the statistical efficiency of using estimating equation and U statistics, which can address these issues separately; and (ii) propose a novel doubly robust generalized U that allows flexible definition of treatment effect, and can handles small samples, distribution robustness, ROI and confounding consideration in one framework. We provide theoretical results on asymptotics and efficiency bounds, together with insights on the efficiency gain from theoretical analysis. We further conduct comprehensive simulation studies, apply the methods to multiple real A/B tests at LinkedIn, and share results and learnings that are broadly useful. |
| title | Beyond Basic A/B testing: Improving Statistical Efficiency for Business Growth |
| topic | Methodology Machine Learning Statistics Theory Computation |
| url | https://arxiv.org/abs/2505.08128 |