Towards a Pretrained Model for Restless Bandits via Multi-arm Generalization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916109636075520 |
|---|---|
| author | Zhao, Yunfan Behari, Nikhil Hughes, Edward Zhang, Edwin Nagaraj, Dheeraj Tuyls, Karl Taneja, Aparna Tambe, Milind |
| author_facet | Zhao, Yunfan Behari, Nikhil Hughes, Edward Zhang, Edwin Nagaraj, Dheeraj Tuyls, Karl Taneja, Aparna Tambe, Milind |
| contents | Restless multi-arm bandits (RMABs), a class of resource allocation problems with broad application in areas such as healthcare, online advertising, and anti-poaching, have recently been studied from a multi-agent reinforcement learning perspective. Prior RMAB research suffers from several limitations, e.g., it fails to adequately address continuous states, and requires retraining from scratch when arms opt-in and opt-out over time, a common challenge in many real world applications. We address these limitations by developing a neural network-based pre-trained model (PreFeRMAB) that has general zero-shot ability on a wide range of previously unseen RMABs, and which can be fine-tuned on specific instances in a more sample-efficient way than retraining from scratch. Our model also accommodates general multi-action settings and discrete or continuous state spaces. To enable fast generalization, we learn a novel single policy network model that utilizes feature information and employs a training procedure in which arms opt-in and out over time. We derive a new update rule for a crucial $λ$-network with theoretical convergence guarantees and empirically demonstrate the advantages of our approach on several challenging, real-world inspired problems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_14526 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Towards a Pretrained Model for Restless Bandits via Multi-arm Generalization Zhao, Yunfan Behari, Nikhil Hughes, Edward Zhang, Edwin Nagaraj, Dheeraj Tuyls, Karl Taneja, Aparna Tambe, Milind Machine Learning Artificial Intelligence Restless multi-arm bandits (RMABs), a class of resource allocation problems with broad application in areas such as healthcare, online advertising, and anti-poaching, have recently been studied from a multi-agent reinforcement learning perspective. Prior RMAB research suffers from several limitations, e.g., it fails to adequately address continuous states, and requires retraining from scratch when arms opt-in and opt-out over time, a common challenge in many real world applications. We address these limitations by developing a neural network-based pre-trained model (PreFeRMAB) that has general zero-shot ability on a wide range of previously unseen RMABs, and which can be fine-tuned on specific instances in a more sample-efficient way than retraining from scratch. Our model also accommodates general multi-action settings and discrete or continuous state spaces. To enable fast generalization, we learn a novel single policy network model that utilizes feature information and employs a training procedure in which arms opt-in and out over time. We derive a new update rule for a crucial $λ$-network with theoretical convergence guarantees and empirically demonstrate the advantages of our approach on several challenging, real-world inspired problems. |
| title | Towards a Pretrained Model for Restless Bandits via Multi-arm Generalization |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2310.14526 |