Cover Image

Constant Stepsize Q-learning: Distributional Convergence, Bias and Extrapolation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhang, Yixuan, Xie, Qiaomin
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Optimization and Control
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Bias and Extrapolation in Markovian Linear Stochastic Approximation with Constant Stepsizes
by: Huo, Dongyan, et al.
Published: (2022)

The Collusion of Memory and Nonlinearity in Stochastic Approximation With Constant Stepsize
by: Huo, Dongyan, et al.
Published: (2024)

Coupling-based Convergence Diagnostic and Stepsize Scheme for Stochastic Gradient Descent
by: Li, Xiang, et al.
Published: (2024)

Two-Timescale Linear Stochastic Approximation: Constant Stepsizes Go a Long Way
by: Kwon, Jeongyeol, et al.
Published: (2024)

Prelimit Coupling and Steady-State Convergence of Constant-stepsize Nonsmooth Contractive SA
by: Zhang, Yixuan, et al.
Published: (2024)

Simple Stepsize for Quasi-Newton Methods with Global Convergence Guarantees
by: Agafonov, Artem, et al.
Published: (2025)

A Piecewise Lyapunov Analysis of Sub-quadratic SGD: Applications to Robust and Quantile Regression
by: Zhang, Yixuan, et al.
Published: (2025)

Adaptive SGD with Line-Search and Polyak Stepsizes: Nonconvex Convergence and Accelerated Rates
by: Wu, Haotian
Published: (2025)

Achieving Near-Optimal Convergence for Distributed Minimax Optimization with Adaptive Stepsizes
by: Huang, Yan, et al.
Published: (2024)

Convergence and stability of Q-learning in Hierarchical Reinforcement Learning
by: Manenti, Massimiliano, et al.
Published: (2025)

Negative Stepsizes Make Gradient-Descent-Ascent Converge
by: Shugart, Henry, et al.
Published: (2025)

AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updates
by: Zhang, Minxin, et al.
Published: (2025)

Stochastic Approximation with Block Coordinate Optimal Stepsizes
by: Jiang, Tao, et al.
Published: (2025)

Sample Complexity of Variance-reduced Distributionally Robust Q-learning
by: Wang, Shengbo, et al.
Published: (2023)

New Perspectives on the Polyak Stepsize: Surrogate Functions and Negative Results
by: Orabona, Francesco, et al.
Published: (2025)

Convergence of Spectral Descent for Non-smooth Optimization
by: Yang, Yixuan, et al.
Published: (2026)

Computing the Bias of Constant-step Stochastic Approximation with Markovian Noise
by: Allmeier, Sebastian, et al.
Published: (2024)

An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes
by: Orvieto, Antonio, et al.
Published: (2024)

MARINA-P: Superior Performance in Non-smooth Federated Optimization with Adaptive Stepsizes
by: Sokolov, Igor, et al.
Published: (2024)

Enhancing Stochastic Optimization for Statistical Efficiency Using ROOT-SGD with Diminishing Stepsize
by: Li, Chris Junchi
Published: (2024)

Implicit Bias and Fast Convergence Rates for Self-attention
by: Vasudeva, Bhavya, et al.
Published: (2024)

Implicit Bias and Convergence of Matrix Stochastic Mirror Descent
by: Akhtiamov, Danil, et al.
Published: (2026)

Convergence and Implicit Bias of Gradient Descent on Continual Linear Classification
by: Jung, Hyunji, et al.
Published: (2025)

Convergence Analysis of the Lion Optimizer in Centralized and Distributed Settings
by: Jiang, Wei, et al.
Published: (2025)

On the Interplay Between Stepsize Tuning and Progressive Sharpening
by: Roulet, Vincent, et al.
Published: (2023)

From PowerSGD to PowerSGD+: Low-Rank Gradient Compression for Distributed Optimization with Convergence Guarantees
by: Xie, Shengping, et al.
Published: (2025)

Q-Measure-Learning for Continuous State RL: Efficient Implementation and Convergence
by: Wang, Shengbo
Published: (2026)

Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization
by: Cai, Yuhang, et al.
Published: (2024)

A Theoretical and Empirical Study on the Convergence of Adam with an "Exact" Constant Step Size in Non-Convex Settings
by: Mazumder, Alokendu, et al.
Published: (2023)

On Convergence of Average-Reward Q-Learning in Weakly Communicating Markov Decision Processes
by: Wan, Yi, et al.
Published: (2024)

Regularized Q-learning through Robust Averaging
by: Schmitt-Förster, Peter, et al.
Published: (2024)

Convergence of Distributed Adaptive Optimization with Local Updates
by: Cheng, Ziheng, et al.
Published: (2024)

Mitigating Gradient Bias in Multi-objective Learning: A Provably Convergent Stochastic Approach
by: Fernando, Heshan, et al.
Published: (2022)

ADDQ: Adaptive Distributional Double Q-Learning
by: Döring, Leif, et al.
Published: (2025)

Convergence of gradient flow for learning convolutional neural networks
by: Diederen, Jona-Maria, et al.
Published: (2026)

Tractable Representations for Convergent Approximation of Distributional HJB Equations
by: Alhosh, Julie, et al.
Published: (2025)

Risk-Sensitive Q-Learning in Continuous Time with Application to Dynamic Portfolio Selection
by: Xie, Chuhan
Published: (2025)

RAMPAGE: RAndomized Mid-Point for debiAsed Gradient Extrapolation
by: Luo, Zhankun, et al.
Published: (2026)

Almost Sure Convergence Rates of Stochastic Approximation and Reinforcement Learning via a Poisson-Moreau Drift
by: Liu, Xinyu, et al.
Published: (2026)

Variance-Reduced Cascade Q-learning: Algorithms and Sample Complexity
by: Boveiri, Mohammad, et al.
Published: (2024)