Two-Stage Constrained Actor-Critic for Short Video Recommendation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913189191483392 |
|---|---|
| author | Cai, Qingpeng Xue, Zhenghai Zhang, Chi Xue, Wanqi Liu, Shuchang Zhan, Ruohan Wang, Xueliang Zuo, Tianyou Xie, Wentao Zheng, Dong Jiang, Peng Gai, Kun |
| author_facet | Cai, Qingpeng Xue, Zhenghai Zhang, Chi Xue, Wanqi Liu, Shuchang Zhan, Ruohan Wang, Xueliang Zuo, Tianyou Xie, Wentao Zheng, Dong Jiang, Peng Gai, Kun |
| contents | The wide popularity of short videos on social media poses new opportunities and challenges to optimize recommender systems on the video-sharing platforms. Users sequentially interact with the system and provide complex and multi-faceted responses, including watch time and various types of interactions with multiple videos. One the one hand, the platforms aims at optimizing the users' cumulative watch time (main goal) in long term, which can be effectively optimized by Reinforcement Learning. On the other hand, the platforms also needs to satisfy the constraint of accommodating the responses of multiple user interactions (auxiliary goals) such like, follow, share etc. In this paper, we formulate the problem of short video recommendation as a Constrained Markov Decision Process (CMDP). We find that traditional constrained reinforcement learning algorithms can not work well in this setting. We propose a novel two-stage constrained actor-critic method: At stage one, we learn individual policies to optimize each auxiliary signal. At stage two, we learn a policy to (i) optimize the main signal and (ii) stay close to policies learned at the first stage, which effectively guarantees the performance of this main policy on the auxiliaries. Through extensive offline evaluations, we demonstrate effectiveness of our method over alternatives in both optimizing the main goal as well as balancing the others. We further show the advantage of our method in live experiments of short video recommendations, where it significantly outperforms other baselines in terms of both watch time and interactions. Our approach has been fully launched in the production system to optimize user experiences on the platform. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2302_01680 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Two-Stage Constrained Actor-Critic for Short Video Recommendation Cai, Qingpeng Xue, Zhenghai Zhang, Chi Xue, Wanqi Liu, Shuchang Zhan, Ruohan Wang, Xueliang Zuo, Tianyou Xie, Wentao Zheng, Dong Jiang, Peng Gai, Kun Machine Learning Information Retrieval The wide popularity of short videos on social media poses new opportunities and challenges to optimize recommender systems on the video-sharing platforms. Users sequentially interact with the system and provide complex and multi-faceted responses, including watch time and various types of interactions with multiple videos. One the one hand, the platforms aims at optimizing the users' cumulative watch time (main goal) in long term, which can be effectively optimized by Reinforcement Learning. On the other hand, the platforms also needs to satisfy the constraint of accommodating the responses of multiple user interactions (auxiliary goals) such like, follow, share etc. In this paper, we formulate the problem of short video recommendation as a Constrained Markov Decision Process (CMDP). We find that traditional constrained reinforcement learning algorithms can not work well in this setting. We propose a novel two-stage constrained actor-critic method: At stage one, we learn individual policies to optimize each auxiliary signal. At stage two, we learn a policy to (i) optimize the main signal and (ii) stay close to policies learned at the first stage, which effectively guarantees the performance of this main policy on the auxiliaries. Through extensive offline evaluations, we demonstrate effectiveness of our method over alternatives in both optimizing the main goal as well as balancing the others. We further show the advantage of our method in live experiments of short video recommendations, where it significantly outperforms other baselines in terms of both watch time and interactions. Our approach has been fully launched in the production system to optimize user experiences on the platform. |
| title | Two-Stage Constrained Actor-Critic for Short Video Recommendation |
| topic | Machine Learning Information Retrieval |
| url | https://arxiv.org/abs/2302.01680 |