Cover Image

Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Chouldechova, Alexandra, Cooper, A. Feder, Barocas, Solon, Palia, Abhinav, Vann, Dan, Wallach, Hanna
Format:	Preprint
Published:	2026
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

A Framework for Evaluating LLMs Under Task Indeterminacy
by: Guerdan, Luke, et al.
Published: (2024)

Validating LLM-as-a-Judge Systems under Rating Indeterminacy
by: Guerdan, Luke, et al.
Published: (2025)

Dimensions of Generative AI Evaluation Design
by: Dow, P. Alex, et al.
Published: (2024)

Effects of Generative AI Errors on User Reliance Across Task Difficulty
by: Anthis, Jacy Reese, et al.
Published: (2026)

A Shared Standard for Valid Measurement of Generative AI Systems' Capabilities, Risks, and Impacts
by: Chouldechova, Alexandra, et al.
Published: (2024)

AI-Assisted Systematization for Evaluating GenAI Systems
by: Agarwal, Dhruv, et al.
Published: (2026)

Evaluating Generative AI Systems is a Social Science Measurement Challenge
by: Wallach, Hanna, et al.
Published: (2024)

Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge
by: Wallach, Hanna, et al.
Published: (2025)

What Constitutes a Less Discriminatory Algorithm?
by: Laufer, Benjamin, et al.
Published: (2024)

Statistical Guarantees in the Search for Less Discriminatory Algorithms
by: Hays, Chris, et al.
Published: (2025)

Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification
by: Cooper, A. Feder, et al.
Published: (2023)

Between Randomness and Arbitrariness: Some Lessons for Reliable Machine Learning at Scale
by: Cooper, A. Feder
Published: (2024)

Learning diverse attacks on large language models for robust red-teaming and safety tuning
by: Lee, Seanie, et al.
Published: (2024)

Distinguishing Task-Specific and General-Purpose AI in Regulation
by: Wang, Jennifer, et al.
Published: (2025)

Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor
by: Olteanu, Alexandra, et al.
Published: (2025)

Taxonomizing Representational Harms using Speech Act Theory
by: Corvi, Emily, et al.
Published: (2025)

Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems
by: Harvey, Emma, et al.
Published: (2024)

Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems
by: Harvey, Emma, et al.
Published: (2025)

The Impact of Differential Feature Under-reporting on Algorithmic Fairness
by: Akpinar, Nil-Jana, et al.
Published: (2024)

A structured regression approach for evaluating model performance across intersectional subgroups
by: Herlihy, Christine, et al.
Published: (2024)

SureMap: Simultaneous Mean Estimation for Single-Task and Multi-Task Disaggregated Evaluation
by: Khodak, Mikhail, et al.
Published: (2024)

AI Automatons: AI Systems Intended to Imitate Humans
by: Olteanu, Alexandra, et al.
Published: (2025)

Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Research
by: Cooper, A. Feder, et al.
Published: (2024)

Leveraging Expert Consistency to Improve Algorithmic Decision Support
by: De-Arteaga, Maria, et al.
Published: (2021)

Designing Algorithmic Delegates: The Role of Indistinguishability in Human-AI Handoff
by: Greenwood, Sophie, et al.
Published: (2025)

Non-Determinism and the Lawlessness of Machine Learning Code
by: Cooper, A. Feder, et al.
Published: (2022)

Rethinking external validation for the target population: Capturing patient-level similarity with a generative model
by: Azizmalayeri, Mohammad, et al.
Published: (2026)

Supporting Industry Computing Researchers in Assessing, Articulating, and Addressing the Potential Negative Societal Impact of Their Work
by: Deng, Wesley Hanwen, et al.
Published: (2024)

The Files are in the Computer: On Copyright, Memorization, and Generative AI
by: Cooper, A. Feder, et al.
Published: (2024)

Do Responsible AI Artifacts Advance Stakeholder Goals? Four Key Barriers Perceived by Legal and Civil Stakeholders
by: Kawakami, Anna, et al.
Published: (2024)

Extracting books from production language models
by: Ahmed, Ahmed, et al.
Published: (2026)

Measuring Machine Learning Harms from Stereotypes Requires Understanding Who Is Harmed by Which Errors in What Ways
by: Wang, Angelina, et al.
Published: (2024)

A team of teams creates success from successes
by: Stan Skipworth
Published: (2024)

A team of teams creates success from successes
by: Stan Skipworth
Published: (2024)

Exploring the limits of strong membership inference attacks on large language models
by: Hayes, Jamie, et al.
Published: (2025)

Error Exponent in Agnostic PAC Learning
by: Hendel, Adi, et al.
Published: (2024)

Toward an Evaluation Science for Generative AI Systems
by: Weidinger, Laura, et al.
Published: (2025)

Talkin' 'Bout AI Generation: Copyright and the Generative-AI Supply Chain
by: Lee, Katherine, et al.
Published: (2023)

Universal Batch Learning Under The Misspecification Setting
by: Vituri, Shlomi, et al.
Published: (2024)

Minimax rates of convergence for nonparametric regression under adversarial attacks
by: Peng, Jingfu, et al.
Published: (2024)