Image de couverture de livre

Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Chen, Zhuotong, Liu, Fang, Zhu, Jennifer, Du, Wanyu, Qi, Yanjun
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning Artificial Intelligence Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Optimizing Anytime Reasoning via Budget Relative Policy Optimization
par: Qi, Penghui, et autres
Publié: (2025)

Dynamic Model Merging Made Slim
par: Du, Guodong, et autres
Publié: (2026)

Preference Optimization via Contrastive Divergence: Your Reward Model is Secretly an NLL Estimator
par: Chen, Zhuotong, et autres
Publié: (2025)

T-REG: Preference Optimization with Token-Level Reward Regularization
par: Zhou, Wenxuan, et autres
Publié: (2024)

Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level
par: Liu, Jie, et autres
Publié: (2024)

Less is More for Improving Automatic Evaluation of Factual Consistency
par: Wang, Tong, et autres
Publié: (2024)

TSO: Self-Training with Scaled Preference Optimization
par: Chen, Kaihui, et autres
Publié: (2024)

Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences
par: Rosset, Corby, et autres
Publié: (2024)

Entropy Controllable Direct Preference Optimization
par: Omura, Motoki, et autres
Publié: (2024)

Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization
par: Yang, Junming, et autres
Publié: (2025)

Preference Optimization by Estimating the Ratio of the Data Distribution
par: Kim, Yeongmin, et autres
Publié: (2025)

APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay
par: Prabhakar, Akshara, et autres
Publié: (2025)

Token-Budget-Aware LLM Reasoning
par: Han, Tingxu, et autres
Publié: (2024)

Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization
par: Wu, Junkang, et autres
Publié: (2024)

TaeBench: Improving Quality of Toxic Adversarial Examples
par: Zhu, Xuan, et autres
Publié: (2024)

Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization
par: Huang, Audrey, et autres
Publié: (2024)

On the Role of Preference Variance in Preference Optimization
par: Guo, Jiacheng, et autres
Publié: (2025)

TB or Not TB: Coverage-Driven Direct Preference Optimization for Verilog Stimulus Generation
par: Nadimi, Bardia, et autres
Publié: (2025)

Less is More: Improving LLM Alignment via Preference Data Selection
par: Deng, Xun, et autres
Publié: (2025)

Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison
par: Shen, Judy Hanwen, et autres
Publié: (2024)

From Noisy Traces to Stable Gradients: Bias-Variance Optimized Preference Optimization for Aligning Large Reasoning Models
par: Zhu, Mingkang, et autres
Publié: (2025)

MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples
par: Xie, Shuo, et autres
Publié: (2024)

Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
par: Qi, Xuan, et autres
Publié: (2025)

Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shift
par: Jian, Chengtao, et autres
Publié: (2025)

Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocation
par: Li, Ziniu, et autres
Publié: (2025)

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
par: Zhu, Mingkang, et autres
Publié: (2025)

WPO: Enhancing RLHF with Weighted Preference Optimization
par: Zhou, Wenxuan, et autres
Publié: (2024)

PerPO: Perceptual Preference Optimization via Discriminative Rewarding
par: Zhu, Zining, et autres
Publié: (2025)

Geometric-Averaged Preference Optimization for Soft Preference Labels
par: Furuta, Hiroki, et autres
Publié: (2024)

FLAMES: Improving LLM Math Reasoning via a Fine-Grained Analysis of the Data Synthesis Pipeline
par: Seegmiller, Parker, et autres
Publié: (2025)

Beyond Scalar Reward Model: Learning Generative Judge from Preference Data
par: Ye, Ziyi, et autres
Publié: (2024)

Orthogonal Finetuning for Direct Preference Optimization
par: Yang, Chenxu, et autres
Publié: (2024)

Improving LLM General Preference Alignment via Optimistic Online Mirror Descent
par: Zhang, Yuheng, et autres
Publié: (2025)

Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing
par: Qi, Biqing, et autres
Publié: (2024)

Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gain
par: Liu, Wei, et autres
Publié: (2026)

Zero-knowledge LLM hallucination detection and mitigation through fine-grained cross-model consistency
par: Goel, Aman, et autres
Publié: (2025)

IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining
par: Li, Yixiao, et autres
Publié: (2025)

Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts
par: Gupta, Taneesh, et autres
Publié: (2024)

The Larger the Better? Improved LLM Code-Generation via Budget Reallocation
par: Hassid, Michael, et autres
Publié: (2024)

Filtered Direct Preference Optimization
par: Morimura, Tetsuro, et autres
Publié: (2024)