Finite Sample Analysis of Linear Temporal Difference Learning with Arbitrary Features
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917012708524032 |
|---|---|
| author | Xie, Zixuan Liu, Xinyu Chandra, Rohan Zhang, Shangtong |
| author_facet | Xie, Zixuan Liu, Xinyu Chandra, Rohan Zhang, Shangtong |
| contents | Linear TD($λ$) is one of the most fundamental reinforcement learning algorithms for policy evaluation. Previously, convergence rates are typically established under the assumption of linearly independent features, which does not hold in many practical scenarios. This paper instead establishes the first $L^2$ convergence rates for linear TD($λ$) operating under arbitrary features, without making any algorithmic modification or additional assumptions. Our results apply to both the discounted and average-reward settings. To address the potential non-uniqueness of solutions resulting from arbitrary features, we develop a novel stochastic approximation result featuring convergence rates to the solution set instead of a single point. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_21391 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Finite Sample Analysis of Linear Temporal Difference Learning with Arbitrary Features Xie, Zixuan Liu, Xinyu Chandra, Rohan Zhang, Shangtong Machine Learning Artificial Intelligence Linear TD($λ$) is one of the most fundamental reinforcement learning algorithms for policy evaluation. Previously, convergence rates are typically established under the assumption of linearly independent features, which does not hold in many practical scenarios. This paper instead establishes the first $L^2$ convergence rates for linear TD($λ$) operating under arbitrary features, without making any algorithmic modification or additional assumptions. Our results apply to both the discounted and average-reward settings. To address the potential non-uniqueness of solutions resulting from arbitrary features, we develop a novel stochastic approximation result featuring convergence rates to the solution set instead of a single point. |
| title | Finite Sample Analysis of Linear Temporal Difference Learning with Arbitrary Features |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2505.21391 |