Cover Image

Small-Margin Preferences Still Matter-If You Train Them Right

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Pang, Jinlong, Zhu, Zhaowei, Di, Na, Zhang, Yichi, Wang, Yaxuan, Qian, Chen, Liu, Yang
Format:	Preprint
Published:	2026
Subjects:	Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Evaluating LLM-Contaminated Crowdsourcing Data Without Ground Truth
by: Zhang, Yichi, et al.
Published: (2025)

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning
by: Pang, Jinlong, et al.
Published: (2025)

Fairness Without Harm: An Influence-Guided Active Sampling Approach
by: Pang, Jinlong, et al.
Published: (2024)

Improving Data Efficiency via Curating LLM-Driven Rating Systems
by: Pang, Jinlong, et al.
Published: (2024)

SPPD: Self-training with Process Preference Learning Using Dynamic Value Margin
by: Yi, Hao, et al.
Published: (2025)

Are You Sure? Rank Them Again: Repeated Ranking For Better Preference Datasets
by: Devine, Peter
Published: (2024)

RLPO: Residual Listwise Preference Optimization for Long-Context Review Ranking
by: Jiang, Hao, et al.
Published: (2026)

If You Can't Use Them, Recycle Them: Optimizing Merging at Scale Mitigates Performance Tradeoffs
by: Khalifa, Muhammad, et al.
Published: (2024)

Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Models
by: Zhu, Zhaowei, et al.
Published: (2023)

Conditioning Matters: Training Diffusion Policies is Faster Than You Think
by: Dong, Zibin, et al.
Published: (2025)

Incentivizing High-quality Participation From Federated Learning Agents
by: Pang, Jinlong, et al.
Published: (2025)

Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
by: Dong, Zhijin
Published: (2025)

Transcendence: Generative Models Can Outperform The Experts That Train Them
by: Zhang, Edwin, et al.
Published: (2024)

Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering
by: Zhang, Yichi, et al.
Published: (2023)

LLM Unlearning via Loss Adjustment with Only Forget Data
by: Wang, Yaxuan, et al.
Published: (2024)

Towards Understanding the Influence of Reward Margin on Preference Model Performance
by: Qin, Bowen, et al.
Published: (2024)

Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimization
by: Xu, Ruijie, et al.
Published: (2024)

Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasets
by: Feng, Duanyu, et al.
Published: (2024)

Larger or Smaller Reward Margins to Select Preferences for Alignment?
by: Huang, Kexin, et al.
Published: (2025)

Adaptive Margin RLHF via Preference over Preferences
by: Chittepu, Yaswanth, et al.
Published: (2025)

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
by: Wu, Junkang, et al.
Published: (2024)

Training on the Benchmark Is Not All You Need
by: Ni, Shiwen, et al.
Published: (2024)

Preference Consistency Matters: Enhancing Preference Learning in Language Models with Automated Self-Curation of Training Corpora
by: Lee, JoonHo, et al.
Published: (2024)

Sponsored Questions and How to Auction Them
by: Bhawalkar, Kshipra, et al.
Published: (2025)

CHILL at SemEval-2025 Task 2: You Can't Just Throw Entities and Hope -- Make Your LLM to Get Them Right
by: Lee, Jaebok, et al.
Published: (2025)

$ξ$-DPO: Direct Preference Optimization via Ratio Reward Margin
by: Fan, Zhengyuan, et al.
Published: (2026)

Learning What Matters Now: Dynamic Preference Inference under Contextual Shifts
by: Cao, Xianwei, et al.
Published: (2026)

Using Large Language Models to Assess Teachers' Pedagogical Content Knowledge
by: Yang, Yaxuan, et al.
Published: (2025)

What LLMs Think When You Don't Tell Them What to Think About?
by: Kwon, Yongchan, et al.
Published: (2026)

Look Before You Decide: Prompting Active Deduction of MLLMs for Assumptive Reasoning
by: Li, Yian, et al.
Published: (2024)

Automatic Dataset Construction (ADC): Sample Collection, Data Curation, and Beyond
by: Liu, Minghao, et al.
Published: (2024)

Observations and Remedies for Large Language Model Bias in Self-Consuming Performative Loop
by: Wang, Yaxuan, et al.
Published: (2026)

LRR-Bench: Left, Right or Rotate? Vision-Language models Still Struggle With Spatial Understanding Tasks
by: Kong, Fei, et al.
Published: (2025)

You Only Forward Once: An Efficient Compositional Judging Paradigm
by: Zhang, Tianlong, et al.
Published: (2025)

POST: Prior-Observation Adversarial Learning of Spatio-Temporal Associations for Multivariate Time Series Anomaly Detection
by: Zhang, Suofei, et al.
Published: (2026)

DRAGON: Guard LLM Unlearning in Context via Negative Detection and Reasoning
by: Wang, Yaxuan, et al.
Published: (2025)

Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization
by: Hong, Bin, et al.
Published: (2025)

STAS: Spatial-Temporal Return Decomposition for Multi-agent Reinforcement Learning
by: Chen, Sirui, et al.
Published: (2023)

TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them
by: Wang, Yidong, et al.
Published: (2025)

Catch Me If You Can? Not Yet: LLMs Still Struggle to Imitate the Implicit Writing Styles of Everyday Authors
by: Wang, Zhengxiang, et al.
Published: (2025)