Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.16186 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912496452894720 |
|---|---|
| author | Li, Kaiyuan Wang, Pengyu Peng, Yunshan Yuan, Pengjia Zeng, Yanxiang Xiang, Rui Cheng, Yanhua Liu, Xialong Jiang, Peng |
| author_facet | Li, Kaiyuan Wang, Pengyu Peng, Yunshan Yuan, Pengjia Zeng, Yanxiang Xiang, Rui Cheng, Yanhua Liu, Xialong Jiang, Peng |
| contents | Reinforcement learning has been widely applied in automated bidding. Traditional approaches model bidding as a Markov Decision Process (MDP). Recently, some studies have explored using generative reinforcement learning methods to address long-term dependency issues in bidding environments. Although effective, these methods typically rely on supervised learning approaches, which are vulnerable to low data quality due to the amount of sub-optimal bids and low probability rewards resulting from the low click and conversion rates. Unfortunately, few studies have addressed these challenges.
In this paper, we formalize the automated bidding as a sequence decision-making problem and propose a novel Expert-guided Bag Reward Transformer (EBaReT) to address concerns related to data quality and uncertainty rewards. Specifically, to tackle data quality issues, we generate a set of expert trajectories to serve as supplementary data in the training process and employ a Positive-Unlabeled (PU) learning-based discriminator to identify expert transitions. To ensure the decision also meets the expert level, we further design a novel expert-guided inference strategy. Moreover, to mitigate the uncertainty of rewards, we consider the transitions within a certain period as a "bag" and carefully design a reward function that leads to a smoother acquisition of rewards. Extensive experiments demonstrate that our model achieves superior performance compared to state-of-the-art bidding methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_16186 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EBaReT: Expert-guided Bag Reward Transformer for Auto Bidding Li, Kaiyuan Wang, Pengyu Peng, Yunshan Yuan, Pengjia Zeng, Yanxiang Xiang, Rui Cheng, Yanhua Liu, Xialong Jiang, Peng Machine Learning Information Retrieval Reinforcement learning has been widely applied in automated bidding. Traditional approaches model bidding as a Markov Decision Process (MDP). Recently, some studies have explored using generative reinforcement learning methods to address long-term dependency issues in bidding environments. Although effective, these methods typically rely on supervised learning approaches, which are vulnerable to low data quality due to the amount of sub-optimal bids and low probability rewards resulting from the low click and conversion rates. Unfortunately, few studies have addressed these challenges. In this paper, we formalize the automated bidding as a sequence decision-making problem and propose a novel Expert-guided Bag Reward Transformer (EBaReT) to address concerns related to data quality and uncertainty rewards. Specifically, to tackle data quality issues, we generate a set of expert trajectories to serve as supplementary data in the training process and employ a Positive-Unlabeled (PU) learning-based discriminator to identify expert transitions. To ensure the decision also meets the expert level, we further design a novel expert-guided inference strategy. Moreover, to mitigate the uncertainty of rewards, we consider the transitions within a certain period as a "bag" and carefully design a reward function that leads to a smoother acquisition of rewards. Extensive experiments demonstrate that our model achieves superior performance compared to state-of-the-art bidding methods. |
| title | EBaReT: Expert-guided Bag Reward Transformer for Auto Bidding |
| topic | Machine Learning Information Retrieval |
| url | https://arxiv.org/abs/2507.16186 |