Oops, I Sampled it Again: Reinterpreting Confidence Intervals in Few-Shot Learning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929489222565888 |
|---|---|
| author | Lafargue, Raphael Smith, Luke Vermet, Franck Löwe, Mathias Reid, Ian Gripon, Vincent Valmadre, Jack |
| author_facet | Lafargue, Raphael Smith, Luke Vermet, Franck Löwe, Mathias Reid, Ian Gripon, Vincent Valmadre, Jack |
| contents | The predominant method for computing confidence intervals (CI) in few-shot learning (FSL) is based on sampling the tasks with replacement, i.e.\ allowing the same samples to appear in multiple tasks. This makes the CI misleading in that it takes into account the randomness of the sampler but not the data itself. To quantify the extent of this problem, we conduct a comparative analysis between CIs computed with and without replacement. These reveal a notable underestimation by the predominant method. This observation calls for a reevaluation of how we interpret confidence intervals and the resulting conclusions in FSL comparative studies. Our research demonstrates that the use of paired tests can partially address this issue. Additionally, we explore methods to further reduce the (size of the) CI by strategically sampling tasks of a specific size. We also introduce a new optimized benchmark, which can be accessed at https://github.com/RafLaf/FSL-benchmark-again |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_02850 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Oops, I Sampled it Again: Reinterpreting Confidence Intervals in Few-Shot Learning Lafargue, Raphael Smith, Luke Vermet, Franck Löwe, Mathias Reid, Ian Gripon, Vincent Valmadre, Jack Machine Learning Artificial Intelligence 68T06 I.2; I.4; I.5; G.3 The predominant method for computing confidence intervals (CI) in few-shot learning (FSL) is based on sampling the tasks with replacement, i.e.\ allowing the same samples to appear in multiple tasks. This makes the CI misleading in that it takes into account the randomness of the sampler but not the data itself. To quantify the extent of this problem, we conduct a comparative analysis between CIs computed with and without replacement. These reveal a notable underestimation by the predominant method. This observation calls for a reevaluation of how we interpret confidence intervals and the resulting conclusions in FSL comparative studies. Our research demonstrates that the use of paired tests can partially address this issue. Additionally, we explore methods to further reduce the (size of the) CI by strategically sampling tasks of a specific size. We also introduce a new optimized benchmark, which can be accessed at https://github.com/RafLaf/FSL-benchmark-again |
| title | Oops, I Sampled it Again: Reinterpreting Confidence Intervals in Few-Shot Learning |
| topic | Machine Learning Artificial Intelligence 68T06 I.2; I.4; I.5; G.3 |
| url | https://arxiv.org/abs/2409.02850 |