YEAST: Yet Another Sequential Test
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915535321563136 |
|---|---|
| author | Kurennoy, Alexey Dodin, Majed Gurbanov, Tural Ramallo, Ana Peleteiro |
| author_facet | Kurennoy, Alexey Dodin, Majed Gurbanov, Tural Ramallo, Ana Peleteiro |
| contents | Online evaluation of machine learning models is typically conducted through A/B experiments. Sequential statistical tests are valuable tools for analysing these experiments, as they enable researchers to stop data collection early without increasing the risk of false discoveries. However, existing sequential tests either limit the number of interim analyses or suffer from low statistical power. In this paper, we introduce a novel sequential test designed for continuous monitoring of A/B experiments. We validate our method using semi-synthetic simulations and demonstrate that it outperforms current state-of-the-art sequential testing approaches. Our method is derived using a new technique that inverts a bound on the probability of threshold crossing, based on a classical maximal inequality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_16523 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | YEAST: Yet Another Sequential Test Kurennoy, Alexey Dodin, Majed Gurbanov, Tural Ramallo, Ana Peleteiro Methodology Online evaluation of machine learning models is typically conducted through A/B experiments. Sequential statistical tests are valuable tools for analysing these experiments, as they enable researchers to stop data collection early without increasing the risk of false discoveries. However, existing sequential tests either limit the number of interim analyses or suffer from low statistical power. In this paper, we introduce a novel sequential test designed for continuous monitoring of A/B experiments. We validate our method using semi-synthetic simulations and demonstrate that it outperforms current state-of-the-art sequential testing approaches. Our method is derived using a new technique that inverts a bound on the probability of threshold crossing, based on a classical maximal inequality. |
| title | YEAST: Yet Another Sequential Test |
| topic | Methodology |
| url | https://arxiv.org/abs/2406.16523 |