Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.03366 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915877884002304 |
|---|---|
| author | Kessler, Ryan McQueen, James Rokkanen, Miikka |
| author_facet | Kessler, Ryan McQueen, James Rokkanen, Miikka |
| contents | We develop a theoretical framework for sample splitting in A/B testing environments, where data for each test are partitioned into two splits to measure methodological performance when the true impacts of tests are unobserved. We show that sample-split estimators are generally biased for full-sample performance but consistently estimate sample-split analogues of it. We derive their asymptotic distributions, construct valid confidence intervals, and characterize the bias-variance trade-offs underlying sample-split design choices. We validate our theoretical results through simulations and provide implementation guidance for A/B testing products seeking to evaluate new estimators and decision rules. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_03366 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evaluating A/B Testing Methodologies via Sample Splitting: Theory and Practice Kessler, Ryan McQueen, James Rokkanen, Miikka Econometrics We develop a theoretical framework for sample splitting in A/B testing environments, where data for each test are partitioned into two splits to measure methodological performance when the true impacts of tests are unobserved. We show that sample-split estimators are generally biased for full-sample performance but consistently estimate sample-split analogues of it. We derive their asymptotic distributions, construct valid confidence intervals, and characterize the bias-variance trade-offs underlying sample-split design choices. We validate our theoretical results through simulations and provide implementation guidance for A/B testing products seeking to evaluate new estimators and decision rules. |
| title | Evaluating A/B Testing Methodologies via Sample Splitting: Theory and Practice |
| topic | Econometrics |
| url | https://arxiv.org/abs/2512.03366 |