It's About Time: What A/B Test Metrics Estimate
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916476186787840 |
|---|---|
| author | Ankargren, Sebastian Frånberg, Mattias Schultzberg, Mårten |
| author_facet | Ankargren, Sebastian Frånberg, Mattias Schultzberg, Mårten |
| contents | Online controlled experiments, or A/B tests, are large-scale randomized trials in digital environments. This paper investigates the estimands of the difference-in-means estimator in these experiments, focusing on scenarios with repeated measurements on users. We compare cumulative metrics that use all post-exposure data for each user to windowed metrics that measure each user over a fixed time window. We analyze the estimands and highlight trade-offs between the two types of metrics. Our findings reveal that while cumulative metrics eliminate the need for pre-defined measurement windows, they result in estimands that are more intricately tied to the experiment intake and runtime. This complexity can lead to counter-intuitive practical consequences, such as decreased statistical power with more observations. However, cumulative metrics offer earlier results and can quickly detect strong initial signals. We conclude that neither metric type is universally superior. The optimal choice depends on the temporal profile of the treatment effect, the distribution of exposure, and the stopping time of the experiment. This research provides insights for experimenters to make informed decisions about how to define metrics based on their specific experimental contexts and objectives. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_06150 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | It's About Time: What A/B Test Metrics Estimate Ankargren, Sebastian Frånberg, Mattias Schultzberg, Mårten Methodology Online controlled experiments, or A/B tests, are large-scale randomized trials in digital environments. This paper investigates the estimands of the difference-in-means estimator in these experiments, focusing on scenarios with repeated measurements on users. We compare cumulative metrics that use all post-exposure data for each user to windowed metrics that measure each user over a fixed time window. We analyze the estimands and highlight trade-offs between the two types of metrics. Our findings reveal that while cumulative metrics eliminate the need for pre-defined measurement windows, they result in estimands that are more intricately tied to the experiment intake and runtime. This complexity can lead to counter-intuitive practical consequences, such as decreased statistical power with more observations. However, cumulative metrics offer earlier results and can quickly detect strong initial signals. We conclude that neither metric type is universally superior. The optimal choice depends on the temporal profile of the treatment effect, the distribution of exposure, and the stopping time of the experiment. This research provides insights for experimenters to make informed decisions about how to define metrics based on their specific experimental contexts and objectives. |
| title | It's About Time: What A/B Test Metrics Estimate |
| topic | Methodology |
| url | https://arxiv.org/abs/2411.06150 |