On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Xiao, Jiancong, Li, Ziniu, Xie, Xingyu, Getzen, Emily, Fang, Cong, Long, Qi, Su, Weijie J.
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Methodology
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Do Large Language Models (Really) Need Statistical Foundations?
by: Su, Weijie
Published: (2025)

Robust Detection of Watermarks for Large Language Models Under Human Edits
by: Li, Xiang, et al.
Published: (2024)

Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching
by: Shi, Zhekun, et al.
Published: (2025)

Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium
by: Liu, Kaizhao, et al.
Published: (2025)

Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theory
by: Xiao, Jiancong, et al.
Published: (2025)

Debiasing Watermarks for Large Language Models via Maximal Coupling
by: Xie, Yangxinyu, et al.
Published: (2024)

DISCRET: Synthesizing Faithful Explanations For Treatment Effect Estimation
by: Wu, Yinjun, et al.
Published: (2024)

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
by: Xiao, Jiancong, et al.
Published: (2025)

Reconciling Overt Bias and Hidden Bias in Sensitivity Analysis for Matched Observational Studies
by: Heng, Siyu, et al.
Published: (2023)

Reducing Symbiosis Bias Through Better A/B Tests of Recommendation Algorithms
by: Brennan, Jennifer, et al.
Published: (2023)

Bias Mitigation in Matched Observational Studies with Continuous Treatments: Calipered Non-Bipartite Matching and Bias-Corrected Estimation and Inference
by: Frazier, Anthony, et al.
Published: (2024)

Reward Collapse in Aligning Large Language Models
by: Song, Ziang, et al.
Published: (2023)

Mitigating Privacy-Utility Trade-off in Decentralized Federated Learning via $f$-Differential Privacy
by: Li, Xiang, et al.
Published: (2025)

Evaluating the Unseen Capabilities: How Many Theorems Do LLMs Know?
by: Li, Xiang, et al.
Published: (2025)

Bias-Variance Tradeoff of Matching Prior to Difference-in-Differences When Parallel Trends is Violated
by: Ge, Mingxuan, et al.
Published: (2025)

Optimal Estimation of Watermark Proportions in Hybrid AI-Human Texts
by: Li, Xiang, et al.
Published: (2025)

Causal Effects in Matching Mechanisms with Strategically Reported Preferences
by: Bertanha, Marinho, et al.
Published: (2023)

Online LLM watermark detection via e-processes
by: Su, Weijie, et al.
Published: (2026)

Fundamental Computational Limits in Pursuing Invariant Causal Prediction and Invariance-Guided Regularization
by: Gu, Yihong, et al.
Published: (2025)

A Comparative Study of DSPy Teleprompter Algorithms for Aligning Large Language Models Evaluation Metrics to Human Evaluation
by: Sarmah, Bhaskarjit, et al.
Published: (2024)

Bias in Meta-Analytic Modeling of Surrogate Endpoints in Cancer Screening Trials
by: Long, James P., et al.
Published: (2025)

Graph-Aligned Random Partition Model (GARP)
by: Rebaudo, Giovanni, et al.
Published: (2023)

Bias Correction for Semiparametric Regression Models
by: Zhang, Yuming, et al.
Published: (2026)

Transfer Learning for Moderate-Dimensional Ridge-Regularized Robust Linear Regression
by: Lyu, Lingfeng, et al.
Published: (2026)

Matrix Completion via Residual Spectral Matching
by: Chen, Ziyuan, et al.
Published: (2024)

You Are the Best Reviewer of Your Own Papers: The Isotonic Mechanism
by: Su, Weijie
Published: (2022)

Bridging the Gap: Rademacher Complexity in Robust and Standard Generalization
by: Xiao, Jiancong, et al.
Published: (2024)

Likelihood Matching for Diffusion Models
by: Qian, Lei, et al.
Published: (2025)

A Utility Score Framework for Dose Optimization Studies with Binary Efficacy-Safety Endpoints: Sample Size Determination and Bias Characterization
by: Gu, Xuemin, et al.
Published: (2026)

Correction for Weak IV Bias and Winner's Curse in Mendelian Randomization Egger Regression: Rerandomized Egger estimator
by: Su, Youpeng, et al.
Published: (2025)

Ridge Regularized Estimation of VAR Models for Inference
by: Ballarin, Giovanni
Published: (2021)

Collapsible Kernel Machine Regression for Exposomic Analyses
by: McGee, Glen, et al.
Published: (2024)

Confidence Diagram of Nonparametric Ranking for Uncertainty Assessment in Large Language Models Evaluation
by: Wang, Zebin, et al.
Published: (2024)

Large Scale Partial Correlation Screening with Uncertainty Quantification
by: Neo, Emily, et al.
Published: (2025)

Large-scale Data Integration using Matrix Denoising and Geometric Factor Matching
by: Held, Felix
Published: (2024)

High-Dimensional Regularized Additive Matrix Autoregressive Model
by: Ghosh, Debika, et al.
Published: (2025)

Robust Inference Under Heteroskedasticity via the Hadamard Estimator
by: Dobriban, Edgar, et al.
Published: (2018)

The Exact Risks of Reference Panel-based Regularized Estimators
by: Su, Buxin, et al.
Published: (2024)

Collapsed Structured Block Models for Community Detection in Complex Networks
by: Papamichalis, Marios, et al.
Published: (2026)

Deep learning for interval-censored failure time data from case-cohort studies
by: Xiao, Yeyu, et al.
Published: (2025)