Cover Image

Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approach

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhang, Xinnan, Li, Chenliang, Zeng, Siliang, Li, Jiaxiang, Wang, Zhongruo, Lin, Kaixiang, Lu, Songtao, Garcia, Alfredo, Hong, Mingyi
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning
by: Zeng, Siliang, et al.
Published: (2023)

Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment
by: Li, Jiaxiang, et al.
Published: (2024)

Understanding Inverse Reinforcement Learning under Overparameterization: Non-Asymptotic Analysis and Global Optimality
by: Zhang, Ruijia, et al.
Published: (2025)

A Bayesian Approach to Robust Inverse Reinforcement Learning
by: Wei, Ran, et al.
Published: (2023)

Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization
by: Li, Chenliang, et al.
Published: (2025)

Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignment
by: Li, Chenliang, et al.
Published: (2024)

Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs
by: Zhao, Siyan, et al.
Published: (2025)

Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guarantees
by: Zeng, Siliang, et al.
Published: (2022)

A Framework for Quantifying How Pre-Training and Context Benefit In-Context Learning
by: Song, Bingqing, et al.
Published: (2025)

The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimation
by: Liu, Mingyi
Published: (2026)

Learning Evidence Highlighting for Frozen LLMs
by: Li, Shaoang, et al.
Published: (2026)

DARO: Difficulty-Aware Reweighting Policy Optimization
by: Zhou, Jingyu, et al.
Published: (2025)

A Note on the Convergence of Muon
by: Li, Jiaxiang, et al.
Published: (2025)

Few-Shot VQA with Frozen LLMs: A Tale of Two Approaches
by: Sterner, Igor, et al.
Published: (2024)

RIVAL: Reinforcement Learning with Iterative and Adversarial Optimization for Machine Translation
by: Li, Tianjiao, et al.
Published: (2025)

Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning
by: Zhang, Yuheng, et al.
Published: (2024)

StitchCUDA: An Automated Multi-Agents End-to-End GPU Programing Framework with Rubric-based Agentic Reinforcement Learning
by: Li, Shiyang, et al.
Published: (2026)

A Barrier Function Approach for Bilevel Optimization with Coupled Lower-Level Constraints: Formulation, Approximation and Algorithms
by: Jiang, Xiaotian, et al.
Published: (2024)

Learning to Edit: Aligning LLMs with Knowledge Editing
by: Jiang, Yuxin, et al.
Published: (2024)

StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning
by: Wang, Daoyu, et al.
Published: (2026)

Optimizing Class-Level Probability Reweighting Coefficients for Equitable Prompting Accuracy
by: Lin, Ruixi, et al.
Published: (2024)

FreezeEmpath: Efficient Training for Empathetic Spoken Chatbots with Frozen LLMs
by: Hong, Yun, et al.
Published: (2026)

Refining Translations with LLMs: A Constraint-Aware Iterative Prompting Approach
by: Chen, Shangfeng, et al.
Published: (2024)

Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Reward Design
by: Wei, Quan, et al.
Published: (2025)

Learning to Align, Aligning to Learn: A Unified Approach for Self-Optimized Alignment
by: Wang, Haowen, et al.
Published: (2025)

LIONs: An Empirically Optimized Approach to Align Language Models
by: Yu, Xiao, et al.
Published: (2024)

Rectify Evaluation Preference: Improving LLMs' Critique on Math Reasoning via Perplexity-aware Reinforcement Learning
by: Tian, Changyuan, et al.
Published: (2025)

IterAlign: Iterative Constitutional Alignment of Large Language Models
by: Chen, Xiusi, et al.
Published: (2024)

Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approach
by: Li, Xuying, et al.
Published: (2025)

Aligning Paralinguistic Understanding and Generation in Speech LLMs via Multi-Task Reinforcement Learning
by: Chen, Jingxiang, et al.
Published: (2026)

XDoGE: Multilingual Data Reweighting to Enhance Language Inclusivity in LLMs
by: Lacunza, Iñaki, et al.
Published: (2025)

Problem-Parameter-Free Decentralized Nonconvex Stochastic Optimization
by: Li, Jiaxiang, et al.
Published: (2024)

Memory-Efficient LLM Pretraining via Minimalist Optimizer Design
by: Glentis, Athanasios, et al.
Published: (2025)

SIRI: Scaling Iterative Reinforcement Learning with Interleaved Compression
by: Wen, Haoming, et al.
Published: (2025)

FLAME: Empowering Frozen LLMs for Knowledge Graph Completion
by: Xue, Bo, et al.
Published: (2024)

Divide, Reweight, and Conquer: A Logit Arithmetic Approach for In-Context Learning
by: Huang, Chengsong, et al.
Published: (2024)

Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark
by: Zhang, Yihua, et al.
Published: (2024)

KV Cache Steering for Controlling Frozen LLMs
by: Belitsky, Max, et al.
Published: (2025)

Do LLMs Recognize Your Latent Preferences? A Benchmark for Latent Information Discovery in Personalized Interaction
by: Tsaknakis, Ioannis, et al.
Published: (2025)

Reward Difference Optimization For Sample Reweighting In Offline RLHF
by: Wang, Shiqi, et al.
Published: (2024)