Cover Image

SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Li, Tianjian, Khashabi, Daniel
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Verifiable by Design: Aligning Language Models to Quote from Pre-Training Data
by: Zhang, Jingyu, et al.
Published: (2024)

The Flaw of Averages: Quantifying Uniformity of Performance on Benchmarks
by: Uzunoglu, Arda, et al.
Published: (2025)

Error Norm Truncation: Robust Training in the Presence of Data Noise for Text Generation Models
by: Li, Tianjian, et al.
Published: (2023)

Benchmarking Language Model Creativity: A Case Study on Code Generation
by: Lu, Yining, et al.
Published: (2024)

The Translation Barrier Hypothesis: Multilingual Generation with Large Language Models Suffers from Implicit Translation Failure
by: Bafna, Niyati, et al.
Published: (2025)

Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasets
by: Li, Tianjian, et al.
Published: (2024)

Jointly Reinforcing Diversity and Quality in Language Model Generations
by: Li, Tianjian, et al.
Published: (2025)

Many-Tier Instruction Hierarchy in LLM Agents
by: Zhang, Jingyu, et al.
Published: (2026)

Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG
by: Ki, Dayeon, et al.
Published: (2025)

Dated Data: Tracing Knowledge Cutoffs in Large Language Models
by: Cheng, Jeffrey, et al.
Published: (2024)

Frustratingly Simple Prompting-based Text Denoising
by: Park, Jungyeul, et al.
Published: (2024)

GOLD PANNING: Strategic Context Shuffling for Needle-in-Haystack Reasoning
by: Byerly, Adam, et al.
Published: (2025)

Challenging the Evaluator: LLM Sycophancy Under User Rebuttal
by: Kim, Sungwon, et al.
Published: (2025)

Self-Consistency Falls Short! The Adverse Effects of Positional Bias on Long-Context Problems
by: Byerly, Adam, et al.
Published: (2024)

A Frustratingly Simple Decoding Method for Neural Text Generation
by: Yang, Haoran, et al.
Published: (2023)

Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model
by: Gou, Qi, et al.
Published: (2024)

Enabling Language Models to Implicitly Learn Self-Improvement
by: Wang, Ziqi, et al.
Published: (2023)

"According to ...": Prompting Language Models Improves Quoting from Pre-Training Data
by: Weller, Orion, et al.
Published: (2023)

Trust Functions: Near-Lossless Weak-to-Strong Generalization by Learning When to Trust the Weak Teacher
by: Uzunoglu, Arda, et al.
Published: (2026)

Evaluating the Evaluators: Are readability metrics good measures of readability?
by: Cachola, Isabel, et al.
Published: (2025)

Do pretrained Transformers Learn In-Context by Gradient Descent?
by: Shen, Lingfeng, et al.
Published: (2023)

ICL CIPHERS: Quantifying "Learning" in In-Context Learning via Substitution Ciphers
by: Fang, Zhouxiang, et al.
Published: (2025)

Frustratingly Simple Retrieval Improves Challenging, Reasoning-Intensive Benchmarks
by: Lyu, Xinxi, et al.
Published: (2025)

Formality is Favored: Unraveling the Learning Preferences of Large Language Models on Data with Conflicting Knowledge
by: Li, Jiahuan, et al.
Published: (2024)

Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment
by: Yin, Yueqin, et al.
Published: (2024)

SOUP: Token-level Single-sample Mix-policy Reinforcement Learning for Large Language Models
by: Yang, Lei, et al.
Published: (2026)

Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time
by: Hu, Michael Y., et al.
Published: (2026)

A Survey on Human Preference Learning for Large Language Models
by: Jiang, Ruili, et al.
Published: (2024)

Frustratingly Easy Data Augmentation for Low-Resource ASR
by: Ibaraki, Katsumi, et al.
Published: (2025)

Frustratingly Easy Task-aware Pruning for Large Language Models
by: Tian, Yuanhe, et al.
Published: (2025)

Self-Boosting Large Language Models with Synthetic Preference Data
by: Dong, Qingxiu, et al.
Published: (2024)

Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis
by: Tao, Leitian, et al.
Published: (2025)

IA2: Alignment with ICL Activations Improves Supervised Fine-Tuning
by: Mishra, Aayush, et al.
Published: (2025)

WorldAPIs: The World Is Worth How Many APIs? A Thought Experiment
by: Ou, Jiefu, et al.
Published: (2024)

Are Finer Citations Always Better? Rethinking Granularity for Attributed Generation
by: Wang, Hexuan, et al.
Published: (2026)

Establishing Knowledge Preference in Language Models
by: Zhou, Sizhe, et al.
Published: (2024)

Off-Policy Value-Based Reinforcement Learning for Large Language Models
by: Wang, Peng-Yuan, et al.
Published: (2026)

Pluralistic Off-policy Evaluation and Alignment
by: Huang, Chengkai, et al.
Published: (2025)

Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data
by: Xia, Han, et al.
Published: (2024)

Bayesian Preference Elicitation with Language Models
by: Handa, Kunal, et al.
Published: (2024)