Cover Image

From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Rafailov, Rafael, Hejna, Joey, Park, Ryan, Finn, Chelsea
Format:	Preprint
Published:	2024
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Direct Preference Optimization: Your Language Model is Secretly a Reward Model
by: Rafailov, Rafael, et al.
Published: (2023)

Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
by: Rafailov, Rafael, et al.
Published: (2024)

Disentangling Length from Quality in Direct Preference Optimization
by: Park, Ryan, et al.
Published: (2024)

Contrastive Preference Learning: Learning from Human Feedback without RL
by: Hejna, Joey, et al.
Published: (2023)

Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents
by: Putta, Pranav, et al.
Published: (2024)

Efficient Imitation Learning with Conservative World Models
by: Kolev, Victor, et al.
Published: (2024)

Aligning Modalities in Vision Large Language Models via Preference Fine-tuning
by: Zhou, Yiyang, et al.
Published: (2024)

TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapse
by: Dong, Perry, et al.
Published: (2026)

Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning
by: Hong, Joey, et al.
Published: (2024)

MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning
by: Rafailov, Rafael, et al.
Published: (2024)

Strategically Conservative Q-Learning
by: Shimizu, Yutaka, et al.
Published: (2024)

Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning
by: Xiang, Violet, et al.
Published: (2025)

Generative Reward Models
by: Mahan, Dakota, et al.
Published: (2024)

Aligning Flow Map Policies with Optimal Q-Guidance
by: Ziakas, Christos, et al.
Published: (2026)

Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data
by: Tajwar, Fahim, et al.
Published: (2024)

Universal Neural Functionals
by: Zhou, Allan, et al.
Published: (2024)

Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learning
by: Hejna, Joey, et al.
Published: (2024)

Digi-Q: Learning Q-Value Functions for Training Device-Control Agents
by: Bai, Hao, et al.
Published: (2025)

rQdia: Regularizing Q-Value Distributions With Image Augmentation
by: Lerman, Sam, et al.
Published: (2025)

Multiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generation
by: Yang, Xinyu, et al.
Published: (2025)

Flow Q-Learning
by: Park, Seohong, et al.
Published: (2025)

Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph
by: Liu, Ning, et al.
Published: (2026)

ADMM-Q: An Improved Hessian-based Weight Quantizer for Post-Training Quantization of Large Language Models
by: Lucas, Ryan, et al.
Published: (2026)

Decoupled Q-Chunking
by: Li, Qiyang, et al.
Published: (2025)

Yell At Your Robot: Improving On-the-Fly from Language Corrections
by: Shi, Lucy Xiaoyang, et al.
Published: (2024)

Your Policy Regularizer is Secretly an Adversary
by: Brekelmans, Rob, et al.
Published: (2022)

Semiparametric Preference Optimization: Your Language Model is Secretly a Single-Index Model
by: Kallus, Nathan
Published: (2025)

Chunk-Guided Q-Learning
by: Song, Gwanwoo, et al.
Published: (2026)

Gaussian-Mixture-Model Q-Functions for Reinforcement Learning by Riemannian Optimization
by: Vu, Minh, et al.
Published: (2024)

Gaussian-Mixture-Model Q-Functions for Policy Iteration in Reinforcement Learning
by: Vu, Minh, et al.
Published: (2025)

Feedback Descent: Open-Ended Text Optimization via Pairwise Comparison
by: Lee, Yoonho, et al.
Published: (2025)

Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning
by: Park, Kwanyoung, et al.
Published: (2024)

MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?
by: Chen, Zhaorun, et al.
Published: (2024)

Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
by: Kim, Moo Jin, et al.
Published: (2025)

Diffusion Fine-Tuning via Reparameterized Policy Gradient of the Soft Q-Function
by: Kang, Hyeongyu, et al.
Published: (2025)

HyperQ-Opt: Q-learning for Hyperparameter Optimization
by: Hasan, Md. Tarek
Published: (2024)

OpenVLA: An Open-Source Vision-Language-Action Model
by: Kim, Moo Jin, et al.
Published: (2024)

Calibrating Language Models with Adaptive Temperature Scaling
by: Xie, Johnathan, et al.
Published: (2024)

Enhancing Q-Learning with Large Language Model Heuristics
by: Wu, Xiefeng
Published: (2024)

Q-Probe: A Lightweight Approach to Reward Maximization for Language Models
by: Li, Kenneth, et al.
Published: (2024)