Anomaly Detection for a Large Number of Streams: A Permutation-Based Higher Criticism Approach
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2020
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915679532220416 |
|---|---|
| author | Stoepker, Ivo V. Castro, Rui M. Arias-Castro, Ery Heuvel, Edwin van den |
| author_facet | Stoepker, Ivo V. Castro, Rui M. Arias-Castro, Ery Heuvel, Edwin van den |
| contents | Anomaly detection when observing a large number of data streams is essential in a variety of applications, ranging from epidemiological studies to monitoring of complex systems. High-dimensional scenarios are usually tackled with scan-statistics and related methods, requiring stringent modeling assumptions for proper calibration. In this work we take a non-parametric stance, and propose a permutation-based variant of the higher criticism statistic not requiring knowledge of the null distribution. This results in an exact test in finite samples which is asymptotically optimal in the wide class of exponential models. We demonstrate the power loss in finite samples is minimal with respect to the oracle test. Furthermore, since the proposed statistic does not rely on asymptotic approximations it typically performs better than popular variants of higher criticism that rely on such approximations. We include recommendations such that the test can be readily applied in practice, and demonstrate its applicability in monitoring the content uniformity of an active ingredient for a batch-produced drug product. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2009_03117 |
| institution | arXiv |
| publishDate | 2020 |
| record_format | arxiv |
| spellingShingle | Anomaly Detection for a Large Number of Streams: A Permutation-Based Higher Criticism Approach Stoepker, Ivo V. Castro, Rui M. Arias-Castro, Ery Heuvel, Edwin van den Methodology Statistics Theory Anomaly detection when observing a large number of data streams is essential in a variety of applications, ranging from epidemiological studies to monitoring of complex systems. High-dimensional scenarios are usually tackled with scan-statistics and related methods, requiring stringent modeling assumptions for proper calibration. In this work we take a non-parametric stance, and propose a permutation-based variant of the higher criticism statistic not requiring knowledge of the null distribution. This results in an exact test in finite samples which is asymptotically optimal in the wide class of exponential models. We demonstrate the power loss in finite samples is minimal with respect to the oracle test. Furthermore, since the proposed statistic does not rely on asymptotic approximations it typically performs better than popular variants of higher criticism that rely on such approximations. We include recommendations such that the test can be readily applied in practice, and demonstrate its applicability in monitoring the content uniformity of an active ingredient for a batch-produced drug product. |
| title | Anomaly Detection for a Large Number of Streams: A Permutation-Based Higher Criticism Approach |
| topic | Methodology Statistics Theory |
| url | https://arxiv.org/abs/2009.03117 |