Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.01939 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916615756447744 |
|---|---|
| author | Farias, Vivek Gijsbrechts, Joren Khojandi, Aryan Peng, Tianyi Zheng, Andrew |
| author_facet | Farias, Vivek Gijsbrechts, Joren Khojandi, Aryan Peng, Tianyi Zheng, Andrew |
| contents | Simulating a single trajectory of a dynamical system under some state-dependent policy is a core bottleneck in policy optimization (PO) algorithms. The many inherently serial policy evaluations that must be performed in a single simulation constitute the bulk of this bottleneck. In applying PO to supply chain optimization (SCO) problems, simulating a single sample path corresponding to one month of a supply chain can take several hours. We present an iterative algorithm to accelerate policy simulation, dubbed Picard Iteration. This scheme carefully assigns policy evaluation tasks to independent processes. Within an iteration, any given process evaluates the policy only on its assigned tasks while assuming a certain "cached" evaluation for other tasks; the cache is updated at the end of the iteration. Implemented on GPUs, this scheme admits batched evaluation of the policy across a single trajectory. We prove that the structure afforded by many SCO problems allows convergence in a small number of iterations independent of the horizon. We demonstrate practical speedups of 400x on large-scale SCO problems even with a single GPU, and also demonstrate practical efficacy in other RL environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_01939 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Speeding up Policy Simulation in Supply Chain RL Farias, Vivek Gijsbrechts, Joren Khojandi, Aryan Peng, Tianyi Zheng, Andrew Artificial Intelligence Distributed, Parallel, and Cluster Computing Machine Learning Simulating a single trajectory of a dynamical system under some state-dependent policy is a core bottleneck in policy optimization (PO) algorithms. The many inherently serial policy evaluations that must be performed in a single simulation constitute the bulk of this bottleneck. In applying PO to supply chain optimization (SCO) problems, simulating a single sample path corresponding to one month of a supply chain can take several hours. We present an iterative algorithm to accelerate policy simulation, dubbed Picard Iteration. This scheme carefully assigns policy evaluation tasks to independent processes. Within an iteration, any given process evaluates the policy only on its assigned tasks while assuming a certain "cached" evaluation for other tasks; the cache is updated at the end of the iteration. Implemented on GPUs, this scheme admits batched evaluation of the policy across a single trajectory. We prove that the structure afforded by many SCO problems allows convergence in a small number of iterations independent of the horizon. We demonstrate practical speedups of 400x on large-scale SCO problems even with a single GPU, and also demonstrate practical efficacy in other RL environments. |
| title | Speeding up Policy Simulation in Supply Chain RL |
| topic | Artificial Intelligence Distributed, Parallel, and Cluster Computing Machine Learning |
| url | https://arxiv.org/abs/2406.01939 |