Cover Image

Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Tajwar, Fahim, Singh, Anikait, Sharma, Archit, Rafailov, Rafael, Schneider, Jeff, Xie, Tengyang, Ermon, Stefano, Finn, Chelsea, Kumar, Aviral
Format:	Preprint
Published:	2024
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Direct Preference Optimization: Your Language Model is Secretly a Reward Model
by: Rafailov, Rafael, et al.
Published: (2023)

Disentangling Length from Quality in Direct Preference Optimization
by: Park, Ryan, et al.
Published: (2024)

FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Users
by: Singh, Anikait, et al.
Published: (2025)

Personalized Preference Fine-tuning of Diffusion Models
by: Dang, Meihua, et al.
Published: (2025)

Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generation
by: Manvi, Rohin, et al.
Published: (2024)

Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone
by: Mark, Max Sobol, et al.
Published: (2024)

Aligning Modalities in Vision Large Language Models via Preference Fine-tuning
by: Zhou, Yiyang, et al.
Published: (2024)

Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
by: Nakamoto, Mitsuhiko, et al.
Published: (2023)

RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems
by: Qu, Yuxiao, et al.
Published: (2025)

Test-Time Alignment via Hypothesis Reweighting
by: Lee, Yoonho, et al.
Published: (2024)

Grounding by Trying: LLMs with Reinforcement Learning-Enhanced Retrieval
by: Hsu, Sheryl, et al.
Published: (2024)

Reasoning as an Adaptive Defense for Safety
by: Kim, Taeyoun, et al.
Published: (2025)

D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning
by: Rafailov, Rafael, et al.
Published: (2024)

From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function
by: Rafailov, Rafael, et al.
Published: (2024)

Conservative Prediction via Data-Driven Confidence Minimization
by: Choi, Caroline, et al.
Published: (2023)

Can Large Reasoning Models Self-Train?
by: Shafayat, Sheikh, et al.
Published: (2025)

MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning
by: Rafailov, Rafael, et al.
Published: (2024)

Contrastive Preference Learning: Learning from Human Feedback without RL
by: Hejna, Joey, et al.
Published: (2023)

Efficient Imitation Learning with Conservative World Models
by: Kolev, Victor, et al.
Published: (2024)

Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thought
by: Xiang, Violet, et al.
Published: (2025)

PERSONA: A Reproducible Testbed for Pluralistic Alignment
by: Castricato, Louis, et al.
Published: (2024)

State Combinatorial Generalization In Decision Making With Conditional Diffusion Models
by: Duan, Xintong, et al.
Published: (2025)

Accelerating Diffusion Planners in Offline RL via Reward-Aware Consistency Trajectory Distillation
by: Duan, Xintong, et al.
Published: (2025)

Sample Efficient Preference Alignment in LLMs via Active Exploration
by: Mehta, Viraj, et al.
Published: (2023)

Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
by: Kim, Moo Jin, et al.
Published: (2025)

MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generation
by: Tong, Haibo, et al.
Published: (2025)

A Critical Evaluation of AI Feedback for Aligning Large Language Models
by: Sharma, Archit, et al.
Published: (2024)

Splats under Pressure: Exploring Performance-Energy Trade-offs in Real-Time 3D Gaussian Splatting under Constrained GPU Budgets
by: Tajwar, Muhammad Fahim, et al.
Published: (2026)

Retrospective In-Context Learning for Temporal Credit Assignment with Large Language Models
by: Chen, Wen-Tse, et al.
Published: (2026)

Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning
by: Xiang, Violet, et al.
Published: (2025)

Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents
by: Putta, Pranav, et al.
Published: (2024)

Scaling Test-Time Compute Without Verification or RL is Suboptimal
by: Setlur, Amrith, et al.
Published: (2025)

Training a Generally Curious Agent
by: Tajwar, Fahim, et al.
Published: (2025)

AutoFT: Learning an Objective for Robust Fine-Tuning
by: Choi, Caroline, et al.
Published: (2024)

Privacy-Constrained Policies via Mutual Information Regularized Policy Gradients
by: Cundy, Chris, et al.
Published: (2020)

Adapt On-the-Go: Behavior Modulation for Single-Life Robot Deployment
by: Chen, Annie S., et al.
Published: (2023)

RLVF: Learning from Verbal Feedback without Overgeneralization
by: Stephan, Moritz, et al.
Published: (2024)

Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
by: Rafailov, Rafael, et al.
Published: (2024)

CaRT: Teaching LLM Agents to Know When They Know Enough
by: Liu, Grace, et al.
Published: (2025)

Parameter Efficient Fine-Tuning of Segment Anything Model for Biomedical Imaging
by: Teuber, Carolin, et al.
Published: (2025)