Cover Image

The Importance of Online Data: Understanding Preference Fine-tuning via Coverage

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Song, Yuda, Swamy, Gokul, Singh, Aarti, Bagnell, J. Andrew, Sun, Wen
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Hybrid Reinforcement Learning from Offline Observation Alone
by: Song, Yuda, et al.
Published: (2024)

To Distill or Decide? Understanding the Algorithmic Trade-off in Partially Observable Reinforcement Learning
by: Song, Yuda, et al.
Published: (2025)

Hybrid Inverse Reinforcement Learning
by: Ren, Juntao, et al.
Published: (2024)

All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning
by: Swamy, Gokul, et al.
Published: (2025)

Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay
by: Sun, Yifan, et al.
Published: (2025)

Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF
by: Gao, Zhaolin, et al.
Published: (2024)

Explain Less, Understand More: Jargon Detection via Personalized Parameter-Efficient Fine-tuning
by: Wu, Bohao, et al.
Published: (2025)

Efficient Imitation under Misspecification
by: Espinosa-Dice, Nicolas, et al.
Published: (2025)

Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuning
by: Zou, Heming, et al.
Published: (2025)

Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Reward
by: Misra, Dipendra, et al.
Published: (2026)

SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection
by: Shen, Han, et al.
Published: (2024)

LLM-AutoDP: Automatic Data Processing via LLM Agents for Model Fine-tuning
by: Huang, Wei, et al.
Published: (2026)

Parameter Efficient Fine-tuning via Explained Variance Adaptation
by: Paischer, Fabian, et al.
Published: (2024)

Improving LLM General Preference Alignment via Optimistic Online Mirror Descent
by: Zhang, Yuheng, et al.
Published: (2025)

Sparse is Enough in Fine-tuning Pre-trained Large Language Models
by: Song, Weixi, et al.
Published: (2023)

Mechanistic Fine-tuning for In-context Learning
by: Cho, Hakaze, et al.
Published: (2025)

Improving Low-Resource Knowledge Tracing Tasks by Supervised Pre-training and Importance Mechanism Fine-tuning
by: Zhang, Hengyuan, et al.
Published: (2024)

Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuning
by: Song, Haobo, et al.
Published: (2024)

Selecting Large Language Model to Fine-tune via Rectified Scaling Law
by: Lin, Haowei, et al.
Published: (2024)

Fine-tuning can Help Detect Pretraining Data from Large Language Models
by: Zhang, Hengxiang, et al.
Published: (2024)

How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition
by: Dong, Guanting, et al.
Published: (2023)

Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide
by: Szep, Marton, et al.
Published: (2024)

AMAQ: Adaptive Mixed-bit Activation Quantization for Collaborative Parameter Efficient Fine-tuning
by: Song, Yurun, et al.
Published: (2025)

Data Scaling as Progressive Coverage of a Predictive Contribution Spectrum
by: Song, Zihui, et al.
Published: (2026)

Exploring Memorization in Fine-tuned Language Models
by: Zeng, Shenglai, et al.
Published: (2023)

Entity-level Factual Adaptiveness of Fine-tuning based Abstractive Summarization Models
by: Song, Jongyoon, et al.
Published: (2024)

LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
by: Chen, Yukang, et al.
Published: (2023)

TB or Not TB: Coverage-Driven Direct Preference Optimization for Verilog Stimulus Generation
by: Nadimi, Bardia, et al.
Published: (2025)

A Fine-tuning Dataset and Benchmark for Large Language Models for Protein Understanding
by: Shen, Yiqing, et al.
Published: (2024)

Information Guided Regularization for Fine-tuning Language Models
by: Sharma, Mandar, et al.
Published: (2024)

Efficient Model Development through Fine-tuning Transfer
by: Lin, Pin-Jie, et al.
Published: (2025)

Outlier-weighed Layerwise Sampling for LLM Fine-tuning
by: Li, Pengxiang, et al.
Published: (2024)

Inverse Reinforcement Learning without Reinforcement Learning
by: Swamy, Gokul, et al.
Published: (2023)

Teaching Your Models to Understand Code via Focal Preference Alignment
by: Wu, Jie, et al.
Published: (2025)

Filter-then-Weight: Online Data Selection and Reweighting for LLM Fine-Tuning
by: Wang, Fangxin, et al.
Published: (2026)

Fine-tuning Language Models with Generative Adversarial Reward Modelling
by: Yu, Zhang Ze, et al.
Published: (2023)

Efficient End-to-end Language Model Fine-tuning on Graphs
by: Xue, Rui, et al.
Published: (2023)

Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation
by: Dong, Guanting, et al.
Published: (2024)

Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing
by: Qi, Biqing, et al.
Published: (2024)

CaRT: Teaching LLM Agents to Know When They Know Enough
by: Liu, Grace, et al.
Published: (2025)