Cover Image

Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Lu, Junru, Li, Jiazheng, An, Siyu, Zhao, Meng, He, Yulan, Yin, Di, Sun, Xing
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

FIPO: Free-form Instruction-oriented Prompt Optimization with Preference Dataset and Modular Fine-tuning Schema
by: Lu, Junru, et al.
Published: (2024)

RoleMRC: A Fine-Grained Composite Benchmark for Role-Playing and Instruction-Following
by: Lu, Junru, et al.
Published: (2025)

SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space
by: Shen, Zhenyi, et al.
Published: (2025)

Calibrating LLMs with Preference Optimization on Thought Trees for Generating Rationale in Science Question Scoring
by: Li, Jiazheng, et al.
Published: (2024)

Two Heads Are Better Than One: Dual-Model Verbal Reflection at Inference-Time
by: Li, Jiazheng, et al.
Published: (2025)

Length Desensitization in Direct Preference Optimization
by: Liu, Wei, et al.
Published: (2024)

ExDDI: Explaining Drug-Drug Interaction Predictions with Natural Language
by: Sun, Zhaoyue, et al.
Published: (2024)

Disentangling Length from Quality in Direct Preference Optimization
by: Park, Ryan, et al.
Published: (2024)

Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization
by: Huang, Audrey, et al.
Published: (2024)

DGPO: Beyond Pairwise Preferences with Directional Consistent Groupwise Optimization
by: Deng, Mengyi, et al.
Published: (2026)

BPO: Revisiting Preference Modeling in Direct Preference Optimization
by: Sun, Lin, et al.
Published: (2025)

AERA Chat: An Interactive Platform for Automated Explainable Student Answer Assessment
by: Li, Jiazheng, et al.
Published: (2024)

TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights
by: Liu, Aiwei, et al.
Published: (2024)

New Desiderata for Direct Preference Optimization
by: Hu, Xiangkun, et al.
Published: (2024)

Ambiguity Awareness Optimization: Towards Semantic Disambiguation for Direct Preference Optimization
by: Li, Jian, et al.
Published: (2025)

An Automated Explainable Educational Assessment System Built on LLMs
by: Li, Jiazheng, et al.
Published: (2024)

Unveiling Divergent Inductive Biases of LLMs on Temporal Data
by: Kishore, Sindhu, et al.
Published: (2024)

FinVerse: An Autonomous Agent System for Versatile Financial Analysis
by: An, Siyu, et al.
Published: (2024)

Enhancing LLM Safety via Constrained Direct Preference Optimization
by: Liu, Zixuan, et al.
Published: (2024)

LearnLens: LLM-Enabled Personalised, Curriculum-Grounded Feedback with Educators in the Loop
by: Zhao, Runcong, et al.
Published: (2025)

Orthogonal Finetuning for Direct Preference Optimization
by: Yang, Chenxu, et al.
Published: (2024)

RKLD: Reverse KL-Divergence-based Knowledge Distillation for Unlearning Personal Information in Large Language Models
by: Wang, Bichen, et al.
Published: (2024)

Context-aware Biases for Length Extrapolation
by: Veisi, Ali, et al.
Published: (2025)

In-Context Learning (and Unlearning) of Length Biases
by: Schoch, Stephanie, et al.
Published: (2025)

The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis
by: Zhou, Yuxiang, et al.
Published: (2023)

Direct Judgement Preference Optimization
by: Wang, Peifeng, et al.
Published: (2024)

Statistical Rejection Sampling Improves Preference Optimization
by: Liu, Tianqi, et al.
Published: (2023)

SNFinLLM: Systematic and Nuanced Financial Domain Adaptation of Chinese Large Language Models
by: Zhao, Shujuan, et al.
Published: (2024)

DPO-Shift: Shifting the Distribution of Direct Preference Optimization
by: Yang, Xiliang, et al.
Published: (2025)

Importance Sampling for Multi-Negative Multimodal Direct Preference Optimization
by: Li, Xintong, et al.
Published: (2025)

Speech-Worthy Alignment for Japanese SpeechLLMs via Direct Preference Optimization
by: Zhao, Mengjie, et al.
Published: (2026)

Large Language Models Fall Short: Understanding Complex Relationships in Detective Narratives
by: Zhao, Runcong, et al.
Published: (2024)

Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning
by: Wang, Tianduo, et al.
Published: (2024)

ReCUT: Balancing Reasoning Length and Accuracy in LLMs via Stepwise Trails and Preference Optimization
by: Jin, Zhensheng, et al.
Published: (2025)

Preference Optimization with Multi-Sample Comparisons
by: Wang, Chaoqi, et al.
Published: (2024)

Mitigating Biases for Instruction-following Language Models via Bias Neurons Elimination
by: Yang, Nakyeong, et al.
Published: (2023)

Length-Controlled Margin-Based Preference Optimization without Reference Model
by: Li, Gengxu, et al.
Published: (2025)

DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization
by: Das, Amitava, et al.
Published: (2025)

POPI: Personalizing LLMs via Optimized Natural Language Preference Inference
by: Chen, Yizhuo, et al.
Published: (2025)

Robust Preference Optimization via Dynamic Target Margins
by: Sun, Jie, et al.
Published: (2025)