Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2401.15811 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929238709370880 |
|---|---|
| author | Zhu, Zhihua Cai, Zheng Zheng, Liang Si, Nian |
| author_facet | Zhu, Zhihua Cai, Zheng Zheng, Liang Si, Nian |
| contents | Two-sided platforms are central to modern commerce and content sharing and often utilize A/B testing for developing new features. While user-side experiments are common, seller-side experiments become crucial for specific interventions and metrics. This paper investigates the effects of interference caused by feedback loops on seller-side experiments in two-sided platforms, with a particular focus on the counterfactual interleaving design, proposed in \citet{ha2020counterfactual,nandy2021b}. These feedback loops, often generated by pacing algorithms, cause outcomes from earlier sessions to influence subsequent ones. This paper contributes by creating a mathematical framework to analyze this interference, theoretically estimating its impact, and conducting empirical evaluations of the counterfactual interleaving design in real-world scenarios. Our research shows that feedback loops can result in misleading conclusions about the treatment effects. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_15811 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Seller-Side Experiments under Interference Induced by Feedback Loops in Two-Sided Platforms Zhu, Zhihua Cai, Zheng Zheng, Liang Si, Nian Methodology Information Retrieval Two-sided platforms are central to modern commerce and content sharing and often utilize A/B testing for developing new features. While user-side experiments are common, seller-side experiments become crucial for specific interventions and metrics. This paper investigates the effects of interference caused by feedback loops on seller-side experiments in two-sided platforms, with a particular focus on the counterfactual interleaving design, proposed in \citet{ha2020counterfactual,nandy2021b}. These feedback loops, often generated by pacing algorithms, cause outcomes from earlier sessions to influence subsequent ones. This paper contributes by creating a mathematical framework to analyze this interference, theoretically estimating its impact, and conducting empirical evaluations of the counterfactual interleaving design in real-world scenarios. Our research shows that feedback loops can result in misleading conclusions about the treatment effects. |
| title | Seller-Side Experiments under Interference Induced by Feedback Loops in Two-Sided Platforms |
| topic | Methodology Information Retrieval |
| url | https://arxiv.org/abs/2401.15811 |