R-GTD: A Geometric Analysis of Gradient Temporal-Difference Learning in Singular Regimes
Fuente:
arXiv
Salvato in:
| Autori principali: | , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910199882711040 |
|---|---|
| author | Na, Hyunjun Lee, Donghwan |
| author_facet | Na, Hyunjun Lee, Donghwan |
| contents | Gradient temporal-difference (GTD) learning algorithms are widely used for off-policy policy evaluation with function approximation. However, existing convergence analyses rely on the restrictive assumption that the so-called feature interaction matrix (FIM) is nonsingular. In practice, the FIM can become singular and leads to instability or degraded performance. While some prior works have applied regularization to relax the nonsingularity assumption, their theoretical guarantees inevitably rely on other restrictive conditions. In this paper, we propose a regularized optimization objective by reformulating the mean-square projected Bellman error minimization. This formulation naturally yields a regularized GTD algorithms, referred to as R-GTD, which guarantees convergence to a unique solution even when the FIM is singular. We conduct a geometric analysis to establish theoretical convergence guarantees and explicit error bounds for the proposed method, and validate its effectiveness through empirical experiments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_20599 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | R-GTD: A Geometric Analysis of Gradient Temporal-Difference Learning in Singular Regimes Na, Hyunjun Lee, Donghwan Machine Learning Artificial Intelligence Gradient temporal-difference (GTD) learning algorithms are widely used for off-policy policy evaluation with function approximation. However, existing convergence analyses rely on the restrictive assumption that the so-called feature interaction matrix (FIM) is nonsingular. In practice, the FIM can become singular and leads to instability or degraded performance. While some prior works have applied regularization to relax the nonsingularity assumption, their theoretical guarantees inevitably rely on other restrictive conditions. In this paper, we propose a regularized optimization objective by reformulating the mean-square projected Bellman error minimization. This formulation naturally yields a regularized GTD algorithms, referred to as R-GTD, which guarantees convergence to a unique solution even when the FIM is singular. We conduct a geometric analysis to establish theoretical convergence guarantees and explicit error bounds for the proposed method, and validate its effectiveness through empirical experiments. |
| title | R-GTD: A Geometric Analysis of Gradient Temporal-Difference Learning in Singular Regimes |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2601.20599 |