Cover Image

Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhu, Wenhong, He, Zhiwei, Wang, Xiaofeng, Liu, Pengfei, Wang, Rui
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Weak-to-Strong Reasoning
by: Yang, Yuqing, et al.
Published: (2024)

Do Large Language Models Truly Understand Geometric Structures?
by: Wang, Xiaofeng, et al.
Published: (2025)

When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shift
by: Le, Khoi, et al.
Published: (2026)

MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization
by: Lyu, Yougang, et al.
Published: (2024)

VCap: Hypergeometric Rewards for Weak-to-Strong Visual Captioning
by: Lu, Xingyu, et al.
Published: (2026)

Weak-to-Strong GraphRAG: Aligning Weak Retrievers with Large Language Models for Graph-based Retrieval Augmented Generation
by: Zou, Deyu, et al.
Published: (2025)

Weak-to-Strong Jailbreaking on Large Language Models
by: Zhao, Xuandong, et al.
Published: (2024)

MrRoPE: Mixed-radix Rotary Position Embedding
by: Tian, Qingyuan, et al.
Published: (2026)

Improving Open-Ended Text Generation via Adaptive Decoding
by: Zhu, Wenhong, et al.
Published: (2024)

Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding
by: Song, Feifan, et al.
Published: (2025)

Hybrid Policy Distillation for LLMs
by: Zhu, Wenhong, et al.
Published: (2026)

Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization
by: Yang, Wenkai, et al.
Published: (2024)

Explanation, Debate, Align: A Weak-to-Strong Framework for Language Model Generalization
by: Zakershahrak, Mehrdad, et al.
Published: (2024)

Reward Modeling with Weak Supervision for Language Models
by: Hauptvogel, Ben, et al.
Published: (2024)

GenTool: Enhancing Tool Generalization in Language Models through Zero-to-One and Weak-to-Strong Simulation
by: He, Jie, et al.
Published: (2025)

Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner
by: Li, Bolian, et al.
Published: (2025)

Adding Alignment Control to Language Models
by: Zhu, Wenhong, et al.
Published: (2025)

W2S-AlignTree: Weak-to-Strong Inference-Time Alignment for Large Language Models via Monte Carlo Tree Search
by: Ding, Zhenyu, et al.
Published: (2025)

Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models
by: Zhou, Zhanhui, et al.
Published: (2024)

Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning
by: Li, Ming, et al.
Published: (2024)

Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learning
by: Sang, Jitao, et al.
Published: (2024)

Incentivizing Strong Reasoning from Weak Supervision
by: Yuan, Yige, et al.
Published: (2025)

The Alignment Floor: How Persona Customization Breaks Safety in Weakly-Aligned LLMs
by: Zhang, Xing, et al.
Published: (2026)

Weak Reward Model Transforms Generative Models into Robust Causal Event Extraction Systems
by: da Silva, Italo Luis, et al.
Published: (2024)

Optimizing Language Model's Reasoning Abilities with Weak Supervision
by: Tong, Yongqi, et al.
Published: (2024)

Selective Weak-to-Strong Generalization
by: Lang, Hao, et al.
Published: (2025)

Is Self-knowledge and Action Consistent or Not: Investigating Large Language Model's Personality
by: Ai, Yiming, et al.
Published: (2024)

User Preference Modeling for Conversational LLM Agents: Weak Rewards from Retrieval-Augmented Interaction
by: Hao, Yuren, et al.
Published: (2026)

CRPO: Confidence-Reward Driven Preference Optimization for Machine Translation
by: Cui, Guofeng, et al.
Published: (2025)

Theoretical Analysis of Weak-to-Strong Generalization
by: Lang, Hunter, et al.
Published: (2024)

Debate Helps Weak-to-Strong Generalization
by: Lang, Hao, et al.
Published: (2025)

How Do Latent Reasoning Methods Perform Under Weak and Strong Supervision?
by: Cui, Yingqian, et al.
Published: (2026)

Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences
by: Jin, Zhuoran, et al.
Published: (2025)

AlignSum: Data Pyramid Hierarchical Fine-tuning for Aligning with Human Summarization Preference
by: Han, Yang, et al.
Published: (2024)

Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation
by: Liu, Kaiyuan, et al.
Published: (2026)

Proximal Supervised Fine-Tuning
by: Zhu, Wenhong, et al.
Published: (2025)

Flexible Realignment of Language Models
by: Zhu, Wenhong, et al.
Published: (2025)

Safety Alignment in NLP Tasks: Weakly Aligned Summarization as an In-Context Attack
by: Fu, Yu, et al.
Published: (2023)

CLEAN-EVAL: Clean Evaluation on Contaminated Large Language Models
by: Zhu, Wenhong, et al.
Published: (2023)

Synthesizing Text-to-SQL Data from Weak and Strong LLMs
by: Yang, Jiaxi, et al.
Published: (2024)