Cover Image

When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Afzali, Amirabbas, Jeon, Myeongho, Brbic, Maria
Format:	Preprint
Published:	2026
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Weak-to-Strong Generalization under Distribution Shifts
by: Jeon, Myeongho, et al.
Published: (2025)

Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models
by: Kim, Seungduk, et al.
Published: (2024)

Retracing the Past: LLMs Emit Training Data When They Get Lost
by: Ko, Myeongseob, et al.
Published: (2025)

When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models
by: Sanyal, Sunny, et al.
Published: (2024)

No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping
by: Le, Thanh-Long V., et al.
Published: (2025)

MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization
by: Lyu, Yougang, et al.
Published: (2024)

Debate Helps Weak Judges Reward Stronger Models
by: Elasky, Ethan, et al.
Published: (2026)

Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering
by: Zhang, Yichi, et al.
Published: (2023)

Stronger Models are NOT Stronger Teachers for Instruction Tuning
by: Xu, Zhangchen, et al.
Published: (2024)

Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding
by: Song, Feifan, et al.
Published: (2025)

CAPO: Confidence Aware Preference Optimization Learning for Multilingual Preferences
by: Pokharel, Rhitabrat, et al.
Published: (2025)

RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs
by: Dang, John, et al.
Published: (2024)

Multiple Choice Questions: Reasoning Makes Large Language Models (LLMs) More Self-Confident, Especially When They are Wrong
by: Fu, Tairan, et al.
Published: (2025)

Reverse Thinking Makes LLMs Stronger Reasoners
by: Chen, Justin Chih-Yao, et al.
Published: (2024)

A Systematic Evaluation of Preference Aggregation in Federated RLHF for Pluralistic Alignment of LLMs
by: Srewa, Mahmoud, et al.
Published: (2025)

SpeakRL: Synergizing Reasoning, Speaking, and Acting in Language Models with Reinforcement Learning
by: Acikgoz, Emre Can, et al.
Published: (2025)

PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference
by: Ji, Jiaming, et al.
Published: (2024)

Confident or Seek Stronger: Exploring Uncertainty-Based On-device LLM Routing From Benchmarking to Generalization
by: Chuang, Yu-Neng, et al.
Published: (2025)

When Quantization Affects Confidence of Large Language Models?
by: Proskurina, Irina, et al.
Published: (2024)

Confidence Improves Self-Consistency in LLMs
by: Taubenfeld, Amir, et al.
Published: (2025)

MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning
by: Jiang, Yulun, et al.
Published: (2025)

Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards
by: Wang, Haoxiang, et al.
Published: (2024)

One Goal, Many Challenges: Robust Preference Optimization Amid Content-Aware and Multi-Source Noise
by: Afzali, Amirabbas, et al.
Published: (2025)

TokDrift: When LLM Speaks in Subwords but Code Speaks in Grammar
by: Li, Yinxi, et al.
Published: (2025)

Effectively Steer LLM To Follow Preference via Building Confident Directions
by: Song, Bingqing, et al.
Published: (2025)

Zero-Shot Confidence Estimation for Small LLMs: When Supervised Baselines Aren't Worth Training
by: Nguyen, Luong N.
Published: (2026)

Word Alignment as Preference for Machine Translation
by: Wu, Qiyu, et al.
Published: (2024)

Preference Ranking Optimization for Human Alignment
by: Song, Feifan, et al.
Published: (2023)

Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
by: Zhang, Jianfei, et al.
Published: (2024)

The Alignment Floor: How Persona Customization Breaks Safety in Weakly-Aligned LLMs
by: Zhang, Xing, et al.
Published: (2026)

Dropouts in Confidence: Moral Uncertainty in Human-LLM Alignment
by: Kwon, Jea, et al.
Published: (2025)

Confident RAG: Enhancing the Performance of LLMs for Mathematics Question Answering through Multi-Embedding and Confidence Scoring
by: Chen, Shiting, et al.
Published: (2025)

Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data
by: Zhang, Xuemiao, et al.
Published: (2025)

SparsePO: Controlling Preference Alignment of LLMs via Sparse Token Masks
by: Christopoulou, Fenia, et al.
Published: (2024)

Easy Problems That LLMs Get Wrong
by: Williams, Sean, et al.
Published: (2024)

The Political Preferences of LLMs
by: Rozado, David
Published: (2024)

Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
by: Liu, Geng, et al.
Published: (2026)

Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models
by: Kumar, Abhishek, et al.
Published: (2024)

Beyond Words: Multimodal LLM Knows When to Speak
by: Liao, Zikai, et al.
Published: (2025)

Can LLMs Capture Human Preferences?
by: Goli, Ali, et al.
Published: (2023)