Cover Image

Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wang, Bo, Cheng, Qinyuan, Peng, Runyu, Bao, Rong, Li, Peiji, Guo, Qipeng, Li, Linyang, Zeng, Zhiyuan, Zhou, Yunhua, Qiu, Xipeng
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective
by: Peng, Runyu, et al.
Published: (2026)

Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective
by: Zeng, Zhiyuan, et al.
Published: (2024)

Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?
by: Zeng, Zhiyuan, et al.
Published: (2025)

Explicit Multi-head Attention for Inter-head Interaction in Large Language Models
by: Peng, Runyu, et al.
Published: (2026)

UnitCoder: Scalable Iterative Code Synthesis with Unit Test Guidance
by: Ma, Yichuan, et al.
Published: (2025)

FastMCTS: A Simple Sampling Strategy for Data Synthesis
by: Li, Peiji, et al.
Published: (2025)

Data-free Weight Compress and Denoise for Large Language Models
by: Peng, Runyu, et al.
Published: (2024)

Dynamic and Generalizable Process Reward Modeling
by: Yin, Zhangyue, et al.
Published: (2025)

UFT: Unifying Fine-Tuning of SFT and RLHF/DPO/UNA through a Generalized Implicit Reward Function
by: Wang, Zhichao, et al.
Published: (2024)

Turn Waste into Worth: Rectifying Top-$k$ Router of MoE
by: Zeng, Zhiyuan, et al.
Published: (2024)

Case2Code: Scalable Synthetic Data for Code Generation
by: Shao, Yunfan, et al.
Published: (2024)

How to Set the Learning Rate for Large-Scale Pre-training?
by: Zhou, Yunhua, et al.
Published: (2026)

How to Mitigate Overfitting in Weak-to-strong Generalization?
by: Shi, Junhao, et al.
Published: (2025)

Unearthing Large Scale Domain-Specific Knowledge from Public Corpora
by: Fei, Zhaoye, et al.
Published: (2024)

Unified Active Retrieval for Retrieval Augmented Generation
by: Cheng, Qinyuan, et al.
Published: (2024)

How to Set the Batch Size for Large-Scale Pre-training?
by: Zhou, Yunhua, et al.
Published: (2026)

Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimization
by: Zhang, Daoan, et al.
Published: (2024)

Timely Machine: Awareness of Time Makes Test-Time Scaling Agentic
by: Ma, Yichuan, et al.
Published: (2026)

Agent Alignment in Evolving Social Norms
by: Li, Shimin, et al.
Published: (2024)

Mixing Expert Knowledge: Bring Human Thoughts Back To the Game of Go
by: Ma, Yichuan, et al.
Published: (2026)

ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Models
by: Yin, Zhangyue, et al.
Published: (2025)

Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data
by: Guo, Xu, et al.
Published: (2026)

Bootstrapping Language Models with DPO Implicit Rewards
by: Chen, Changyu, et al.
Published: (2024)

What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents
by: Li, Xiaozhe, et al.
Published: (2026)

Aggregation of Reasoning: A Hierarchical Framework for Enhancing Answer Selection in Large Language Models
by: Yin, Zhangyue, et al.
Published: (2024)

F-Eval: Assessing Fundamental Abilities with Refined Evaluation Methods
by: Sun, Yu, et al.
Published: (2024)

VisuoThink: Empowering LVLM Reasoning with Multimodal Tree Search
by: Wang, Yikun, et al.
Published: (2025)

DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with Dynamic Multi-Sequence Drafting
by: Lv, Kai, et al.
Published: (2025)

AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models
by: Zhu, Qin, et al.
Published: (2025)

Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment
by: Li, Jiaxiang, et al.
Published: (2024)

Can AI Assistants Know What They Don't Know?
by: Cheng, Qinyuan, et al.
Published: (2024)

Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning
by: Zhang, Yechen, et al.
Published: (2026)

Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
by: Qi, Xuan, et al.
Published: (2025)

Reg-DPO: SFT-Regularized Direct Preference Optimization with GT-Pair for Improving Video Generation
by: Du, Jie, et al.
Published: (2025)

Scaling Laws for Fact Memorization of Large Language Models
by: Lu, Xingyu, et al.
Published: (2024)

TL-GRPO: Turn-Level RL for Reasoning-Guided Iterative Optimization
by: Li, Peiji, et al.
Published: (2026)

An Empirical Study of SFT-DPO Interaction and Parameterization in Small Language Models
by: Feng, Yuming, et al.
Published: (2026)

The Past Is Not Past: Memory-Enhanced Dynamic Reward Shaping
by: Liu, Yang, et al.
Published: (2026)

LLatrieval: LLM-Verified Retrieval for Verifiable Generation
by: Li, Xiaonan, et al.
Published: (2023)

Inference-Time Decontamination: Reusing Leaked Benchmarks for Large Language Model Evaluation
by: Zhu, Qin, et al.
Published: (2024)