Cover Image

When prompt perturbations break your A/B test: A valid statistical test for generative surveying

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Helm, Hayden, Priebe, Carey
Format:	Preprint
Published:	2026
Subjects:	Methodology Artificial Intelligence Applications
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Testing for LLM response differences: the case of a composite null consisting of semantically irrelevant query perturbations
by: Acharyya, Aranyak, et al.
Published: (2025)

Query-efficient model evaluation using cached responses
by: Helm, Hayden, et al.
Published: (2026)

Comparing Foundation Models using Data Kernels
by: Duderstadt, Brandon, et al.
Published: (2023)

Approximately optimal domain adaptation with Fisher's Linear Discriminant
by: Helm, Hayden S., et al.
Published: (2023)

A Distribution-Free Framework for Rewrite-Based Human-text Detection via Knockoff Filtering
by: Liu, Yi
Published: (2026)

A non-parametric U-statistic testing approach for multi-arm clinical trials with multivariate longitudinal data
by: Ghosh, Dhrubajyoti, et al.
Published: (2024)

Does AI help humans make better decisions? A statistical evaluation framework for experimental and observational studies
by: Ben-Michael, Eli, et al.
Published: (2024)

Robustifying and Selecting Cohort-Appropriate Prognostic Models under Distributional Shifts
by: Bertsimas, Dimitris, et al.
Published: (2026)

Observing the unobserved confounding through its effects: toward randomized trial-like estimates from real-world survival data
by: Stoumpou, Vasiliki, et al.
Published: (2026)

Eligibility-Aware Evidence Synthesis: An Agentic Framework for Clinical Trial Meta-Analysis
by: Zhao, Yao, et al.
Published: (2026)

A new test for assessing the covariate effect in ROC curves
by: Fanjul-Hevia, Arís, et al.
Published: (2024)

Bootstrap Matching: a robust and efficient correction for non-random A/B test, and its applications
by: Zheng, Zihao, et al.
Published: (2024)

Mesoscale two-sample testing for networks
by: MacDonald, Peter W., et al.
Published: (2024)

Planning for gold: Hypothesis screening with split samples for valid powerful testing in matched observational studies
by: Bekerman, William, et al.
Published: (2024)

A Tutorial on Asymptotic Properties for Biostatisticians with Applications to COVID-19 Data
by: Cui, Elvis Han
Published: (2022)

Robust distribution-free tests for the linear model
by: Hilbert, Torey, et al.
Published: (2024)

Efficiency of nonparametric superiority tests based on restricted mean survival time versus the log-rank test under proportional hazards
by: Magirr, Dominic, et al.
Published: (2024)

A multi-arm multi-stage design for trials with all pairwise testing
by: Greenstreet, Peter, et al.
Published: (2025)

An Algorithmic Approach for Causal Health Equity: A Look at Race Differentials in Intensive Care Unit (ICU) Outcomes
by: Plecko, Drago, et al.
Published: (2025)

Gotta match 'em all: Solution diversification in graph matching matched filters
by: Li, Zhirui, et al.
Published: (2023)

Bayes factor functions for testing partial correlation coefficients
by: Datta, Saptati
Published: (2025)

The permuted score test for robust differential expression analysis
by: Barry, Timothy, et al.
Published: (2025)

Improved prediction of future user activity in online A/B testing
by: Masoero, Lorenzo, et al.
Published: (2024)

Detecting Perspective Shifts in Multi-agent Systems
by: Bridgeford, Eric, et al.
Published: (2025)

Control Charts for Multi-agent Systems
by: Helm, Hayden, et al.
Published: (2026)

Meta-analysis of networks of diagnostic tests with binary and continuous results
by: Derezea, Efthymia, et al.
Published: (2026)

A novel smoothing-based goodness-of-fit test of covariance for multivariate sparse functional data
by: Ghosh, Dhrubajyoti, et al.
Published: (2025)

Nonparametric Bayes multiresolution testing for high-dimensional rare events
by: Datta, Jyotishka, et al.
Published: (2023)

Bridging the Unavoidable A Priori: A Framework for Comparative Causal Modeling
by: Hovmand, Peter S., et al.
Published: (2025)

A group testing based exploration of age-varying factors in chlamydia infections among Iowa residents
by: Li, Yizeng, et al.
Published: (2024)

A new composite Mann-Whitney test for two-sample survival comparisons with right-censored data
by: Hussain, Abid, et al.
Published: (2025)

The conditional saddlepoint approximation for fast and accurate large-scale hypothesis testing
by: Niu, Ziang, et al.
Published: (2024)

The general linear hypothesis testing problem for multivariate functional data with applications
by: Zhu, Tianming
Published: (2023)

ConfHit: Conformal Generative Design with Oracle Free Guarantees
by: Laghuvarapu, Siddhartha, et al.
Published: (2026)

Estimating Item Difficulty with Large Language Models as Experts
by: Kolesnikova, Diana, et al.
Published: (2026)

Cross-Domain Offshore Wind Power Forecasting: Transfer Learning Through Meteorological Clusters
by: Weisser, Dominic, et al.
Published: (2026)

Fairboard: a quantitative framework for equity assessment of healthcare models
by: Ruffle, James K., et al.
Published: (2026)

Super-Level-Set Regression: Conditional Quantiles via Volume Minimization
by: Braun, Sacha, et al.
Published: (2026)

Detecting Structural Heart Disease from Electrocardiograms via a Generalized Additive Model of Interpretable Foundation-Model Predictors
by: Zhou, Ya, et al.
Published: (2026)

Is Machine Learning Unsafe and Irresponsible in Social Sciences? Paradoxes and Reconsidering from Recidivism Prediction Tasks
by: Liu, Jianhong, et al.
Published: (2023)