Evaluating the Surrogate Index as a Decision-Making Tool Using 200 A/B Tests at Netflix
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914659779477504 |
|---|---|
| author | Zhang, Vickie Zhao, Michael Dimakopoulou, and Maria Le, Anh Kallus, Nathan |
| author_facet | Zhang, Vickie Zhao, Michael Dimakopoulou, and Maria Le, Anh Kallus, Nathan |
| contents | Surrogate index approaches have recently become a popular method of estimating longer-term impact from shorter-term outcomes. In this paper, we leverage 1098 test arms from 200 A/B tests at Netflix to empirically investigate to what degree would decisions made using a surrogate index utilizing 14 days of data would align with those made using direct measurement of day 63 treatment effects. Focusing specifically on linear "auto-surrogate" models that utilize the shorter-term observations of the long-term outcome of interest, we find that the statistical inferences that we would draw from using the surrogate index are ~95% consistent with those from directly measuring the long-term treatment effect. Moreover, when we restrict ourselves to the set of tests that would be "launched" (i.e. positive and statistically significant) based on the 63-day directly measured treatment effects, we find that relying instead on the surrogate index achieves 79% and 65% recall. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_11922 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Evaluating the Surrogate Index as a Decision-Making Tool Using 200 A/B Tests at Netflix Zhang, Vickie Zhao, Michael Dimakopoulou, and Maria Le, Anh Kallus, Nathan Applications Methodology Surrogate index approaches have recently become a popular method of estimating longer-term impact from shorter-term outcomes. In this paper, we leverage 1098 test arms from 200 A/B tests at Netflix to empirically investigate to what degree would decisions made using a surrogate index utilizing 14 days of data would align with those made using direct measurement of day 63 treatment effects. Focusing specifically on linear "auto-surrogate" models that utilize the shorter-term observations of the long-term outcome of interest, we find that the statistical inferences that we would draw from using the surrogate index are ~95% consistent with those from directly measuring the long-term treatment effect. Moreover, when we restrict ourselves to the set of tests that would be "launched" (i.e. positive and statistically significant) based on the 63-day directly measured treatment effects, we find that relying instead on the surrogate index achieves 79% and 65% recall. |
| title | Evaluating the Surrogate Index as a Decision-Making Tool Using 200 A/B Tests at Netflix |
| topic | Applications Methodology |
| url | https://arxiv.org/abs/2311.11922 |