Cover Image

Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Liu, Jie, Zhou, Zhanhui, Liu, Jiaheng, Bu, Xingyuan, Yang, Chao, Zhong, Han-Sen, Ouyang, Wanli
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization
by: Zhou, Zhanhui, et al.
Published: (2023)

Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!
by: Zhou, Zhanhui, et al.
Published: (2024)

MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues
by: Bai, Ge, et al.
Published: (2024)

ConceptMath: A Bilingual Concept-wise Benchmark for Measuring Mathematical Reasoning of Large Language Models
by: Wu, Yanan, et al.
Published: (2024)

2D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervision
by: Li, Shilong, et al.
Published: (2024)

Length Desensitization in Direct Preference Optimization
by: Liu, Wei, et al.
Published: (2024)

GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Models
by: Li, Shilong, et al.
Published: (2024)

Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization
by: Fu, Yuhan, et al.
Published: (2024)

Aligning CodeLLMs with Direct Preference Optimization
by: Miao, Yibo, et al.
Published: (2024)

Efficient and Universal Neural-Network Decoder for Stabilizer-Based Quantum Error Correction
by: Hu, Gengyuan, et al.
Published: (2025)

Reg-DPO: SFT-Regularized Direct Preference Optimization with GT-Pair for Improving Video Generation
by: Du, Jie, et al.
Published: (2025)

Disentangling Length from Quality in Direct Preference Optimization
by: Park, Ryan, et al.
Published: (2024)

Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models
by: Zhou, Zhanhui, et al.
Published: (2024)

CrystalX: High-accuracy Crystal Structure Analysis Using Deep Learning
by: Zheng, Kaipeng, et al.
Published: (2024)

AIPO: Improving Training Objective for Iterative Preference Optimization
by: Shen, Yaojie, et al.
Published: (2024)

T-REG: Preference Optimization with Token-Level Reward Regularization
by: Zhou, Wenxuan, et al.
Published: (2024)

Improving Variable-Length Generation in Diffusion Language Models via Length Regularization
by: Cheng, Zicong, et al.
Published: (2026)

SDPO: Segment-Level Direct Preference Optimization for Social Agents
by: Kong, Aobo, et al.
Published: (2025)

BPO: Revisiting Preference Modeling in Direct Preference Optimization
by: Sun, Lin, et al.
Published: (2025)

APT: Improving Specialist LLM Performance with Weakness Case Acquisition and Iterative Preference Training
by: Rao, Jun, et al.
Published: (2025)

An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4
by: Huang, Hui, et al.
Published: (2024)

Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Models
by: Tan, Yingshui, et al.
Published: (2025)

TPMM-DPO: Trajectory-aware Preference-guided Model Merging for Iterative Direct Preference Optimization
by: Fu, Lingling, et al.
Published: (2026)

Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization
by: Chen, Zhuotong, et al.
Published: (2024)

On the Convergence Rates of Iterative Regularization Algorithms for Composite Bi-Level Optimization
by: Shtern, Shimrit, et al.
Published: (2025)

Understanding Reference Policies in Direct Preference Optimization
by: Liu, Yixin, et al.
Published: (2024)

Iterated relation systems on Riemannian manifolds
by: Liu, Jie, et al.
Published: (2024)

Flow-GRPO: Training Flow Matching Models via Online RL
by: Liu, Jie, et al.
Published: (2025)

Robust Preference Alignment via Directional Neighborhood Consensus
by: Mao, Ruochen, et al.
Published: (2025)

LOCR: Location-Guided Transformer for Optical Character Recognition
by: Sun, Yu, et al.
Published: (2024)

SynCast: Synergizing Contradictions in Precipitation Nowcasting via Diffusion Sequential Preference Optimization
by: Xu, Kaiyi, et al.
Published: (2025)

DocReLM: Mastering Document Retrieval with Language Model
by: Wei, Gengchen, et al.
Published: (2024)

FGBench: A Dataset and Benchmark for Molecular Property Reasoning at Functional Group-Level in Large Language Models
by: Liu, Xuan, et al.
Published: (2025)

Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences
by: Rosset, Corby, et al.
Published: (2024)

Iterative Reasoning Preference Optimization
by: Pang, Richard Yuanzhe, et al.
Published: (2024)

Token-Importance Guided Direct Preference Optimization
by: Yang, Ning, et al.
Published: (2025)

SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment
by: Zhu, Wenqiao, et al.
Published: (2025)

SafeCoT: Improving VLM Safety with Minimal Reasoning
by: Ma, Jiachen, et al.
Published: (2025)

GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?
by: Wu, Wenhao, et al.
Published: (2023)

Instruction-Driven Game Engine: A Poker Case Study
by: Wu, Hongqiu, et al.
Published: (2024)