Minimizing Live Experiments in Recommender Systems: User Simulation to Evaluate Preference Elicitation Policies
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913519243362304 |
|---|---|
| author | Hsu, Chih-Wei Mladenov, Martin Meshi, Ofer Pine, James Pham, Hubert Li, Shane Liang, Xujian Polishko, Anton Yang, Li Scheetz, Ben Boutilier, Craig |
| author_facet | Hsu, Chih-Wei Mladenov, Martin Meshi, Ofer Pine, James Pham, Hubert Li, Shane Liang, Xujian Polishko, Anton Yang, Li Scheetz, Ben Boutilier, Craig |
| contents | Evaluation of policies in recommender systems typically involves A/B testing using live experiments on real users to assess a new policy's impact on relevant metrics. This ``gold standard'' comes at a high cost, however, in terms of cycle time, user cost, and potential user retention. In developing policies for ``onboarding'' new users, these costs can be especially problematic, since on-boarding occurs only once. In this work, we describe a simulation methodology used to augment (and reduce) the use of live experiments. We illustrate its deployment for the evaluation of ``preference elicitation'' algorithms used to onboard new users of the YouTube Music platform. By developing counterfactually robust user behavior models, and a simulation service that couples such models with production infrastructure, we are able to test new algorithms in a way that reliably predicts their performance on key metrics when deployed live. We describe our domain, our simulation models and platform, results of experiments and deployment, and suggest future steps needed to further realistic simulation as a powerful complement to live experiments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_17436 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Minimizing Live Experiments in Recommender Systems: User Simulation to Evaluate Preference Elicitation Policies Hsu, Chih-Wei Mladenov, Martin Meshi, Ofer Pine, James Pham, Hubert Li, Shane Liang, Xujian Polishko, Anton Yang, Li Scheetz, Ben Boutilier, Craig Information Retrieval Machine Learning Evaluation of policies in recommender systems typically involves A/B testing using live experiments on real users to assess a new policy's impact on relevant metrics. This ``gold standard'' comes at a high cost, however, in terms of cycle time, user cost, and potential user retention. In developing policies for ``onboarding'' new users, these costs can be especially problematic, since on-boarding occurs only once. In this work, we describe a simulation methodology used to augment (and reduce) the use of live experiments. We illustrate its deployment for the evaluation of ``preference elicitation'' algorithms used to onboard new users of the YouTube Music platform. By developing counterfactually robust user behavior models, and a simulation service that couples such models with production infrastructure, we are able to test new algorithms in a way that reliably predicts their performance on key metrics when deployed live. We describe our domain, our simulation models and platform, results of experiments and deployment, and suggest future steps needed to further realistic simulation as a powerful complement to live experiments. |
| title | Minimizing Live Experiments in Recommender Systems: User Simulation to Evaluate Preference Elicitation Policies |
| topic | Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2409.17436 |