Cover Image

Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedback

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhou, Runlong, Fazel, Maryam, Du, Simon S.
Format:	Preprint
Published:	2025
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Unregularized Linear Convergence in Zero-Sum Game from Preference Feedback
by: Chen, Shulun, et al.
Published: (2025)

Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO
by: Shi, Ruizhe, et al.
Published: (2025)

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration
by: Bose, Avinandan, et al.
Published: (2024)

Sharp Gap-Dependent Variance-Aware Regret Bounds for Tabular MDPs
by: Chen, Shulun, et al.
Published: (2025)

The Crucial Role of Samplers in Online Direct Preference Optimization
by: Shi, Ruizhe, et al.
Published: (2024)

Global Convergence of Gradient EM for Over-Parameterized Gaussian Mixtures
by: Zhou, Mo, et al.
Published: (2025)

Toward Global Convergence of Gradient EM for Over-Parameterized Gaussian Mixture Models
by: Xu, Weihang, et al.
Published: (2024)

Convergence Dynamics of Over-Parameterized Score Matching for a Single Gaussian
by: Zhang, Yiran, et al.
Published: (2025)

Efficient Exploration for Iterative Nash Preference Optimization
by: Nan, Tianlong, et al.
Published: (2026)

The Harder Path: Last Iterate Convergence for Uncoupled Learning in Zero-Sum Games with Bandit Feedback
by: Fiegel, Côme, et al.
Published: (2026)

On the Last-Iterate Convergence of Shuffling Gradient Methods
by: Liu, Zijian, et al.
Published: (2024)

Global Convergence of Four-Layer Matrix Factorization under Random Initialization
by: Luo, Minrui, et al.
Published: (2025)

Improved Last-Iterate Convergence of Shuffling Gradient Methods for Nonsmooth Convex Optimization
by: Liu, Zijian, et al.
Published: (2025)

Online Iterative Reinforcement Learning from Human Feedback with General Preference Model
by: Ye, Chenlu, et al.
Published: (2024)

Learning Optimal Tax Design in Nonatomic Congestion Games
by: Cui, Qiwen, et al.
Published: (2024)

Revisiting the Last-Iterate Convergence of Stochastic Gradient Methods
by: Liu, Zijian, et al.
Published: (2023)

Proximal Point Nash Learning from Human Feedback
by: Tiapkin, Daniil, et al.
Published: (2025)

Near-Optimal Last-Iterate Convergence for Zero-Sum Games with Bandit Feedback and Opponent Actions
by: Hait, Soumita, et al.
Published: (2026)

Offline Multi-task Transfer RL with Representational Penalization
by: Bose, Avinandan, et al.
Published: (2024)

Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning
by: Montenegro, Alessandro, et al.
Published: (2024)

Last Iterate Convergence of Incremental Methods and Applications in Continual Learning
by: Cai, Xufeng, et al.
Published: (2024)

Dual Approximation Policy Optimization
by: Xiong, Zhihan, et al.
Published: (2024)

Nash Learning from Human Feedback
by: Munos, Rémi, et al.
Published: (2023)

Reflect-RL: Two-Player Online RL Fine-Tuning for LMs
by: Zhou, Runlong, et al.
Published: (2024)

Augmented Lagrangian Method for Last-Iterate Convergence for Constrained MDPs
by: Lu, Michael, et al.
Published: (2026)

Last-Iterate Convergence of No-Regret Learning for Equilibria in Bargaining Games
by: Kamp, Serafina, et al.
Published: (2025)

On Separation Between Best-Iterate, Random-Iterate, and Last-Iterate Convergence of Learning in Games
by: Cai, Yang, et al.
Published: (2025)

Convergence Rate of the Last Iterate of Stochastic Proximal Algorithms
by: Vaidyan, Kevin Kurian Thomas, et al.
Published: (2026)

Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning
by: Zhang, Yuheng, et al.
Published: (2024)

Iteratively reweighted kernel machines efficiently learn sparse functions
by: Zhu, Libin, et al.
Published: (2025)

Iterate to Accelerate: A Unified Framework for Iterative Reasoning and Feedback Convergence
by: Fein-Ashley, Jacob
Published: (2025)

Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
by: Xiong, Wei, et al.
Published: (2023)

Fast Last-Iterate Convergence of SGD in the Smooth Interpolation Regime
by: Attia, Amit, et al.
Published: (2025)

Last-Iterate Convergence of General Parameterized Policies in Constrained MDPs
by: Mondal, Washim Uddin, et al.
Published: (2024)

Iterative Linear Quadratic Optimization for Nonlinear Control: Differentiable Programming Algorithmic Templates
by: Roulet, Vincent, et al.
Published: (2022)

Last-Iterate Convergence: Zero-Sum Games and Constrained Min-Max Optimization
by: Daskalakis, Constantinos, et al.
Published: (2018)

Online SuBmodular + SuPermodular (BP) Maximization with Bandit Feedback
by: Narang, Adhyyan, et al.
Published: (2022)

Preference-Based Multi-Agent Reinforcement Learning: Data Coverage and Algorithmic Techniques
by: Zhang, Natalia, et al.
Published: (2024)

Fast Last-Iterate Convergence of Learning in Games Requires Forgetful Algorithms
by: Cai, Yang, et al.
Published: (2024)

Multi-turn Reinforcement Learning from Preference Human Feedback
by: Shani, Lior, et al.
Published: (2024)