Under-coverage in high-statistics counting experiments with finite MC samples
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914309171314688 |
|---|---|
| author | Alexe, Cristina-Andreea Bendavid, Joshua Bianchini, Lorenzo Bruschini, Davide |
| author_facet | Alexe, Cristina-Andreea Bendavid, Joshua Bianchini, Lorenzo Bruschini, Davide |
| contents | We consider the problem of setting confidence intervals on a parameter of interest from the maximum-likelihood fit of a physics model to a binned data set with a large number of bins, large event-counts per bin, and in the presence of systematic uncertainties modeled as nuisance parameters. We use the profile-likelihood ratio for statistical inference and focus on the case in which the model is determined from Monte Carlo simulated samples of finite size. We start by presenting a toy model in which the properties of widely used approximations of the profile-likelihood ratio in the asymptotic limit, which are commonly expected to hold in the high-statistics regime, are manifestly broken even if the numbers of events per bin in both the data and simulated samples are seemingly large enough to warrant their validity. We then move to the general setting to show how statistical uncertainties in the Monte Carlo predictions can affect the coverage of confidence intervals constructed in the asymptotic approximation always in the same direction, namely they lead to systematic under-coverage. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_10542 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Under-coverage in high-statistics counting experiments with finite MC samples Alexe, Cristina-Andreea Bendavid, Joshua Bianchini, Lorenzo Bruschini, Davide Data Analysis, Statistics and Probability High Energy Physics - Experiment We consider the problem of setting confidence intervals on a parameter of interest from the maximum-likelihood fit of a physics model to a binned data set with a large number of bins, large event-counts per bin, and in the presence of systematic uncertainties modeled as nuisance parameters. We use the profile-likelihood ratio for statistical inference and focus on the case in which the model is determined from Monte Carlo simulated samples of finite size. We start by presenting a toy model in which the properties of widely used approximations of the profile-likelihood ratio in the asymptotic limit, which are commonly expected to hold in the high-statistics regime, are manifestly broken even if the numbers of events per bin in both the data and simulated samples are seemingly large enough to warrant their validity. We then move to the general setting to show how statistical uncertainties in the Monte Carlo predictions can affect the coverage of confidence intervals constructed in the asymptotic approximation always in the same direction, namely they lead to systematic under-coverage. |
| title | Under-coverage in high-statistics counting experiments with finite MC samples |
| topic | Data Analysis, Statistics and Probability High Energy Physics - Experiment |
| url | https://arxiv.org/abs/2401.10542 |