I Can't Believe It's Not Robust: Catastrophic Collapse of Safety Classifiers under Embedding Drift

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Sahoo, Subramanyam, Jain, Vinija, Chaudhary, Divya, Chadha, Aman
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866918364324036608
author	Sahoo, Subramanyam Jain, Vinija Chaudhary, Divya Chadha, Aman
author_facet	Sahoo, Subramanyam Jain, Vinija Chaudhary, Divya Chadha, Aman
contents	Instruction tuned reasoning models are increasingly deployed with safety classifiers trained on frozen embeddings, assuming representation stability across model updates. We systematically investigate this assumption and find it fails: normalized perturbations of magnitude $σ=0.02$ (corresponding to $\approx 1^\circ$ angular drift on the embedding sphere) reduce classifier performance from $85\%$ to $50\%$ ROC-AUC. Critically, mean confidence only drops $14\%$, producing dangerous silent failures where $72\%$ of misclassifications occur with high confidence, defeating standard monitoring. We further show that instruction-tuned models exhibit 20$\%$ worse class separability than base models, making aligned systems paradoxically harder to safeguard. Our findings expose a fundamental fragility in production AI safety architectures and challenge the assumption that safety mechanisms transfer across model versions.
format	Preprint
id	arxiv_https___arxiv_org_abs_2603_01297
institution	arXiv
publishDate	2026
record_format	arxiv
spellingShingle	I Can't Believe It's Not Robust: Catastrophic Collapse of Safety Classifiers under Embedding Drift Sahoo, Subramanyam Jain, Vinija Chaudhary, Divya Chadha, Aman Machine Learning Computation and Language Instruction tuned reasoning models are increasingly deployed with safety classifiers trained on frozen embeddings, assuming representation stability across model updates. We systematically investigate this assumption and find it fails: normalized perturbations of magnitude $σ=0.02$ (corresponding to $\approx 1^\circ$ angular drift on the embedding sphere) reduce classifier performance from $85\%$ to $50\%$ ROC-AUC. Critically, mean confidence only drops $14\%$, producing dangerous silent failures where $72\%$ of misclassifications occur with high confidence, defeating standard monitoring. We further show that instruction-tuned models exhibit 20$\%$ worse class separability than base models, making aligned systems paradoxically harder to safeguard. Our findings expose a fundamental fragility in production AI safety architectures and challenge the assumption that safety mechanisms transfer across model versions.
title	I Can't Believe It's Not Robust: Catastrophic Collapse of Safety Classifiers under Embedding Drift
topic	Machine Learning Computation and Language
url	https://arxiv.org/abs/2603.01297