Imagen de Portada

A Common Pitfall of Margin-based Language Model Alignment: Gradient Entanglement

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Yuan, Hui, Zeng, Yifan, Wu, Yue, Wang, Huazheng, Wang, Mengdi, Leqi, Liu
Formato:	Preprint
Publicado:	2024
Materias:	Machine Learning Artificial Intelligence Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling
por: Qiu, Jiahao, et al.
Publicado: (2024)

Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
por: Ji, Xiang, et al.
Publicado: (2024)

Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
por: Zhang, Yifan, et al.
Publicado: (2024)

Larger or Smaller Reward Margins to Select Preferences for Alignment?
por: Huang, Kexin, et al.
Publicado: (2025)

SAIL: Self-Improving Efficient Online Alignment of Large Language Models
por: Ding, Mucong, et al.
Publicado: (2024)

Linear Representation Transferability Hypothesis: Leveraging Small Models to Steer Large Models
por: Bello, Femi, et al.
Publicado: (2025)

Self-Play Preference Optimization for Language Model Alignment
por: Wu, Yue, et al.
Publicado: (2024)

Personalized Language Modeling from Personalized Human Feedback
por: Li, Xinyu, et al.
Publicado: (2024)

MaxMin-RLHF: Alignment with Diverse Human Preferences
por: Chakraborty, Souradip, et al.
Publicado: (2024)

Benefits and Pitfalls of Reinforcement Learning for Language Model Planning: A Theoretical Perspective
por: Wang, Siwei, et al.
Publicado: (2025)

Higher-order Linear Attention
por: Zhang, Yifan, et al.
Publicado: (2025)

AlignBench: Benchmarking Chinese Alignment of Large Language Models
por: Liu, Xiao, et al.
Publicado: (2023)

Prompting Fairness: Integrating Causality to Debias Large Language Models
por: Li, Jingling, et al.
Publicado: (2024)

Probabilistic Token Alignment for Large Language Model Fusion
por: Zeng, Runjia, et al.
Publicado: (2025)

CARE-RFT: Confidence-Anchored Reinforcement Finetuning for Reliable Reasoning in Large Language Models
por: Li, Shuozhe, et al.
Publicado: (2026)

On the Role of Preference Variance in Preference Optimization
por: Guo, Jiacheng, et al.
Publicado: (2025)

Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models
por: Abbes, Istabrak, et al.
Publicado: (2025)

Training and Evaluating Language Models with Template-based Data Generation
por: Zhang, Yifan
Publicado: (2024)

CoRA: Optimizing Low-Rank Adaptation with Common Subspace of Large Language Models
por: Xiao, Xiaojun, et al.
Publicado: (2024)

Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models
por: Li, Chengao, et al.
Publicado: (2025)

SEE: Sememe Entanglement Encoding for Transformer-bases Models Compression
por: Zhang, Jing, et al.
Publicado: (2024)

Accelerated Preference Optimization for Large Language Model Alignment
por: He, Jiafan, et al.
Publicado: (2024)

Curriculum Learning-Guided Progressive Distillation in Large Language Models
por: Cao, Jincheng, et al.
Publicado: (2026)

On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning
por: Zhang, Yifan, et al.
Publicado: (2025)

Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization
por: Wu, Junkang, et al.
Publicado: (2024)

FlashSampling: Fast and Memory-Efficient Exact Sampling
por: Ruiz, Tomas, et al.
Publicado: (2026)

Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks
por: Hua, Wenyue, et al.
Publicado: (2024)

What Makes an Evaluation Useful? Common Pitfalls and Best Practices
por: Gekker, Gil, et al.
Publicado: (2025)

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
por: Wu, Junkang, et al.
Publicado: (2024)

Interactive Benchmarks
por: Yue, Baoqing, et al.
Publicado: (2026)

Temporal Consistency for LLM Reasoning Process Error Identification
por: Guo, Jiacheng, et al.
Publicado: (2025)

Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications
por: Wei, Boyi, et al.
Publicado: (2024)

Deep Delta Learning
por: Zhang, Yifan, et al.
Publicado: (2026)

SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths
por: Huang, Kaixuan, et al.
Publicado: (2024)

Wanda++: Pruning Large Language Models via Regional Gradients
por: Yang, Yifan, et al.
Publicado: (2025)

From Instructions to Constraints: Language Model Alignment with Automatic Constraint Verification
por: Wang, Fei, et al.
Publicado: (2024)

When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffs
por: Zeng, Yifan, et al.
Publicado: (2026)

Can Brain Signals Reveal Inner Alignment with Human Languages?
por: Han, William, et al.
Publicado: (2022)

LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings
por: Wang, Duo, et al.
Publicado: (2024)

A Markov Categorical Framework for Language Modeling
por: Zhang, Yifan
Publicado: (2025)