Imagen de Portada

A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Schwinn, Leo, Ladenburger, Moritz, Beyer, Tim, Mofakhami, Mehrnaz, Gidel, Gauthier, Günnemann, Stephan
Formato:	Preprint
Publicado:	2026
Materias:	Computation and Language Artificial Intelligence
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

A Generative Approach to LLM Harmfulness Mitigation with Red Flag Tokens
por: Dobre, David, et al.
Publicado: (2025)

LLM-Safety Evaluations Lack Robustness
por: Beyer, Tim, et al.
Publicado: (2025)

AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Research
por: Beyer, Tim, et al.
Publicado: (2025)

Fast Proxies for LLM Robustness Evaluation
por: Beyer, Tim, et al.
Publicado: (2025)

Extracting Unlearned Information from LLMs with Activation Steering
por: Seyitoğlu, Atakan, et al.
Publicado: (2024)

Tight Lower Bounds and Improved Convergence in Performative Prediction
por: Khorsandi, Pedram, et al.
Publicado: (2024)

Efficient Adversarial Training in LLMs with Continuous Attacks
por: Xhonneux, Sophie, et al.
Publicado: (2024)

Performative Prediction with Neural Networks
por: Mofakhami, Mehrnaz, et al.
Publicado: (2023)

A Probabilistic Perspective on Unlearning and Alignment for Large Language Models
por: Scholten, Yan, et al.
Publicado: (2024)

Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space
por: Schwinn, Leo, et al.
Publicado: (2024)

Sampling-aware Adversarial Attacks Against Large Language Models
por: Beyer, Tim, et al.
Publicado: (2025)

Assessing Robustness via Score-Based Adversarial Image Generation
por: Kollovieh, Marcel, et al.
Publicado: (2023)

Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectives
por: Schwinn, Leo, et al.
Publicado: (2025)

Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMs
por: Scholten, Yan, et al.
Publicado: (2025)

Closing the Distribution Gap in Adversarial Training for LLMs
por: Hu, Chengzhi, et al.
Publicado: (2026)

Enough Coin Flips Can Make LLMs Act Bayesian
por: Gupta, Ritwik, et al.
Publicado: (2025)

Flow Matching with Gaussian Process Priors for Probabilistic Time Series Forecasting
por: Kollovieh, Marcel, et al.
Publicado: (2024)

Prompt Optimization Is a Coin Flip: Diagnosing When It Helps in Compound AI Systems
por: Zhang, Xing, et al.
Publicado: (2026)

Evaluating Metrics for Safety with LLM-as-Judges
por: Clegg, Kester, et al.
Publicado: (2025)

VERT: Reliable LLM Judges for Radiology Report Evaluation
por: Bologna, Federica, et al.
Publicado: (2026)

When LLM Judge Scores Look Good but Best-of-N Decisions Fail
por: Landesberg, Eddie
Publicado: (2026)

Benchmarking Adversarial Robustness to Bias Elicitation in Large Language Models: Scalable Automated Assessment with LLM-as-a-Judge
por: Cantini, Riccardo, et al.
Publicado: (2025)

R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
por: Yuan, Tongxin, et al.
Publicado: (2024)

When Refusals Fail: Unstable Safety Mechanisms in Long-Context LLM Agents
por: Hadeliya, Tsimur, et al.
Publicado: (2025)

Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations
por: Gupta, Manan, et al.
Publicado: (2026)

Investigating Non-Transitivity in LLM-as-a-Judge
por: Xu, Yi, et al.
Publicado: (2025)

YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering
por: D'Souza, Jennifer, et al.
Publicado: (2025)

Performance Control in Early Exiting to Deploy Large Models at the Same Cost of Smaller Ones
por: Mofakhami, Mehrnaz, et al.
Publicado: (2024)

Adversarial Robustness of Graph Transformers
por: Foth, Philipp, et al.
Publicado: (2024)

Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
por: Shi, Lin, et al.
Publicado: (2024)

From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges
por: Hong, Yihan, et al.
Publicado: (2026)

LLM Self-Explanations Fail Semantic Invariance
por: Szeider, Stefan
Publicado: (2026)

Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safety
por: Galisai, Marcello, et al.
Publicado: (2026)

ADVERSA: Measuring Multi-Turn Guardrail Degradation and Judge Reliability in Large Language Models
por: Owiredu-Ashley, Harry
Publicado: (2026)

Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge
por: Spiliopoulou, Evangelia, et al.
Publicado: (2025)

Through the Judge's Eyes: Inferred Thinking Traces Improve Reliability of LLM Raters
por: Zhang, Xingjian, et al.
Publicado: (2025)

TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them
por: Wang, Yidong, et al.
Publicado: (2025)

Diffusion LLMs are Natural Adversaries for any LLM
por: Lüdke, David, et al.
Publicado: (2025)

MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safety
por: Wen, Xiaoyu, et al.
Publicado: (2026)

A Survey on LLM-as-a-Judge
por: Gu, Jiawei, et al.
Publicado: (2024)