Actor-Critic or Critic-Actor? A Tale of Two Time Scales
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2022
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911924583661568 |
|---|---|
| author | Bhatnagar, Shalabh Borkar, Vivek S. Guin, Soumyajit |
| author_facet | Bhatnagar, Shalabh Borkar, Vivek S. Guin, Soumyajit |
| contents | We revisit the standard formulation of tabular actor-critic algorithm as a two time-scale stochastic approximation with value function computed on a faster time-scale and policy computed on a slower time-scale. This emulates policy iteration. We observe that reversal of the time scales will in fact emulate value iteration and is a legitimate algorithm. We provide a proof of convergence and compare the two empirically with and without function approximation (with both linear and nonlinear function approximators) and observe that our proposed critic-actor algorithm performs on par with actor-critic in terms of both accuracy and computational effort. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2210_04470 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Actor-Critic or Critic-Actor? A Tale of Two Time Scales Bhatnagar, Shalabh Borkar, Vivek S. Guin, Soumyajit Machine Learning We revisit the standard formulation of tabular actor-critic algorithm as a two time-scale stochastic approximation with value function computed on a faster time-scale and policy computed on a slower time-scale. This emulates policy iteration. We observe that reversal of the time scales will in fact emulate value iteration and is a legitimate algorithm. We provide a proof of convergence and compare the two empirically with and without function approximation (with both linear and nonlinear function approximators) and observe that our proposed critic-actor algorithm performs on par with actor-critic in terms of both accuracy and computational effort. |
| title | Actor-Critic or Critic-Actor? A Tale of Two Time Scales |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2210.04470 |