When Does Confidence-Based Cascade Deferral Suffice?
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929218931130368 |
|---|---|
| author | Jitkrittum, Wittawat Gupta, Neha Menon, Aditya Krishna Narasimhan, Harikrishna Rawat, Ankit Singh Kumar, Sanjiv |
| author_facet | Jitkrittum, Wittawat Gupta, Neha Menon, Aditya Krishna Narasimhan, Harikrishna Rawat, Ankit Singh Kumar, Sanjiv |
| contents | Cascades are a classical strategy to enable inference cost to vary adaptively across samples, wherein a sequence of classifiers are invoked in turn. A deferral rule determines whether to invoke the next classifier in the sequence, or to terminate prediction. One simple deferral rule employs the confidence of the current classifier, e.g., based on the maximum predicted softmax probability. Despite being oblivious to the structure of the cascade -- e.g., not modelling the errors of downstream models -- such confidence-based deferral often works remarkably well in practice. In this paper, we seek to better understand the conditions under which confidence-based deferral may fail, and when alternate deferral strategies can perform better. We first present a theoretical characterisation of the optimal deferral rule, which precisely characterises settings under which confidence-based deferral may suffer. We then study post-hoc deferral mechanisms, and demonstrate they can significantly improve upon confidence-based deferral in settings where (i) downstream models are specialists that only work well on a subset of inputs, (ii) samples are subject to label noise, and (iii) there is distribution shift between the train and test set. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_02764 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | When Does Confidence-Based Cascade Deferral Suffice? Jitkrittum, Wittawat Gupta, Neha Menon, Aditya Krishna Narasimhan, Harikrishna Rawat, Ankit Singh Kumar, Sanjiv Machine Learning Cascades are a classical strategy to enable inference cost to vary adaptively across samples, wherein a sequence of classifiers are invoked in turn. A deferral rule determines whether to invoke the next classifier in the sequence, or to terminate prediction. One simple deferral rule employs the confidence of the current classifier, e.g., based on the maximum predicted softmax probability. Despite being oblivious to the structure of the cascade -- e.g., not modelling the errors of downstream models -- such confidence-based deferral often works remarkably well in practice. In this paper, we seek to better understand the conditions under which confidence-based deferral may fail, and when alternate deferral strategies can perform better. We first present a theoretical characterisation of the optimal deferral rule, which precisely characterises settings under which confidence-based deferral may suffer. We then study post-hoc deferral mechanisms, and demonstrate they can significantly improve upon confidence-based deferral in settings where (i) downstream models are specialists that only work well on a subset of inputs, (ii) samples are subject to label noise, and (iii) there is distribution shift between the train and test set. |
| title | When Does Confidence-Based Cascade Deferral Suffice? |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2307.02764 |