Cover Image

Consistency Training Helps Stop Sycophancy and Jailbreaks

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Irpan, Alex, Turner, Alexander Matt, Kurzeja, Mark, Elson, David K., Shah, Rohin
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking
by: Farquhar, Sebastian, et al.
Published: (2025)

Stop Regressing: Training Value Functions via Classification for Scalable Deep RL
by: Farebrother, Jesse, et al.
Published: (2024)

Aligned, Orthogonal or In-conflict: When can we safely optimize Chain-of-Thought?
by: Kaufmann, Max, et al.
Published: (2026)

Detecting Jailbreak Attempts in Clinical Training LLMs Through Automated Linguistic Feature Extraction
by: Nguyen, Tri, et al.
Published: (2026)

Gradient Routing: Masking Gradients to Localize Computation in Neural Networks
by: Cloud, Alex, et al.
Published: (2024)

Optimal Bayesian Stopping for Efficient Inference of Consistent LLM Answers
by: Huang, Jingkai, et al.
Published: (2026)

Output Supervision Can Obfuscate the Chain of Thought
by: Drori, Jacob, et al.
Published: (2025)

The Price of Agreement: Measuring LLM Sycophancy in Agentic Financial Applications
by: Zhao, Zhenyu, et al.
Published: (2026)

A Pragmatic Way to Measure Chain-of-Thought Monitorability
by: Emmons, Scott, et al.
Published: (2025)

Jailbreaking in the Haystack
by: Shah, Rishi Rajesh, et al.
Published: (2025)

Not Just RLHF: Why Alignment Alone Won't Fix Multi-Agent Sycophancy
by: Kumarappan, Adarsh, et al.
Published: (2026)

Rethinking Early Stopping: Refine, Then Calibrate
by: Berta, Eugène, et al.
Published: (2025)

Distillation Robustifies Unlearning
by: Lee, Bruce W., et al.
Published: (2025)

Improving Dictionary Learning with Gated Sparse Autoencoders
by: Rajamanoharan, Senthooran, et al.
Published: (2024)

An Approach to Technical AGI Safety and Security
by: Shah, Rohin, et al.
Published: (2025)

Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generation
by: Manvi, Rohin, et al.
Published: (2024)

Recontextualization Mitigates Specification Gaming without Modifying the Specification
by: Azarbal, Ariana, et al.
Published: (2025)

BrokenMath: A Benchmark for Sycophancy in Theorem Proving with LLMs
by: Petrov, Ivo, et al.
Published: (2025)

Towards Understanding Sycophancy in Language Models
by: Sharma, Mrinank, et al.
Published: (2023)

SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks
by: Robey, Alexander, et al.
Published: (2023)

It's Not Always Sycophancy: Measuring LLM Conformity as a Function of Epistemic Uncertainty
by: Guo, Kevin H., et al.
Published: (2026)

Evaluating the Goal-Directedness of Large Language Models
by: Everitt, Tom, et al.
Published: (2025)

Building Production-Ready Probes For Gemini
by: Kramár, János, et al.
Published: (2026)

Jailbreaking Black Box Large Language Models in Twenty Queries
by: Chao, Patrick, et al.
Published: (2023)

Yes, Q-learning Helps Offline In-Context RL
by: Tarasov, Denis, et al.
Published: (2025)

Use of What-if Scenarios to Help Explain Artificial Intelligence Models for Neonatal Health
by: Mamun, Abdullah, et al.
Published: (2024)

A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
by: Li, Mengqi, et al.
Published: (2025)

Statistical Early Stopping for Reasoning Models
by: Xie, Yangxinyu, et al.
Published: (2026)

Adversarial Reasoning at Jailbreaking Time
by: Sabbaghi, Mahdi, et al.
Published: (2025)

Enforcing Orderedness to Improve Feature Consistency
by: Wang, Sophie L., et al.
Published: (2025)

Fresh in memory: Training-order recency is linearly encoded in language model activations
by: Krasheninnikov, Dmitrii, et al.
Published: (2025)

ESPO: Early-Stopping Proximal Policy Optimization
by: Li, Zihang, et al.
Published: (2026)

Distributional Training Data Attribution: What do Influence Functions Sample?
by: Mlodozeniec, Bruno, et al.
Published: (2025)

More Consistent Accuracy PINN via Alternating Easy-Hard Training
by: Gao, Zhaoqian, et al.
Published: (2025)

Training Verification-Friendly Neural Networks via Neuron Behavior Consistency
by: Liu, Zongxin, et al.
Published: (2024)

Towards Faster Training of Diffusion Models: An Inspiration of A Consistency Phenomenon
by: Xu, Tianshuo, et al.
Published: (2024)

The Jailbreak Tax: How Useful are Your Jailbreak Outputs?
by: Nikolić, Kristina, et al.
Published: (2025)

BEACON: Bayesian Optimal Stopping for Efficient LLM Sampling
by: Wan, Guangya, et al.
Published: (2025)

A Systematic Investigation of The RL-Jailbreaker in LLMs
by: Mohammedalamen, Montaser, et al.
Published: (2026)

Steering Llama 2 via Contrastive Activation Addition
by: Panickssery, Nina, et al.
Published: (2023)