An Efficient Plugin Method for Metric Optimization of Black-Box Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915181349568512 |
|---|---|
| author | Devic, Siddartha Choudhary, Nurendra Srinivasan, Anirudh Genc, Sahika Kveton, Branislav Hiranandani, Gaurush |
| author_facet | Devic, Siddartha Choudhary, Nurendra Srinivasan, Anirudh Genc, Sahika Kveton, Branislav Hiranandani, Gaurush |
| contents | Many machine learning algorithms and classifiers are available only via API queries as a ``black-box'' -- that is, the downstream user has no ability to change, re-train, or fine-tune the model on a particular target distribution. Indeed, the downstream user may not even have knowledge of the \emph{original} training distribution or performance metric used to construct and optimize the black-box model. We propose a simple and efficient method, Plugin, which \emph{post-processes} arbitrary multiclass predictions from any black-box classifier in order to simultaneously (1) adapt these predictions to a target distribution; and (2) optimize a particular metric of the confusion matrix. Importantly, Plugin is a completely \textit{post-hoc} method which does not rely on feature information, only requires a small amount of probabilistic predictions along with their corresponding true label, and optimizes metrics by querying. We empirically demonstrate that Plugin is both broadly applicable and has performance competitive with related methods on a variety of tabular and language tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_02119 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | An Efficient Plugin Method for Metric Optimization of Black-Box Models Devic, Siddartha Choudhary, Nurendra Srinivasan, Anirudh Genc, Sahika Kveton, Branislav Hiranandani, Gaurush Machine Learning Many machine learning algorithms and classifiers are available only via API queries as a ``black-box'' -- that is, the downstream user has no ability to change, re-train, or fine-tune the model on a particular target distribution. Indeed, the downstream user may not even have knowledge of the \emph{original} training distribution or performance metric used to construct and optimize the black-box model. We propose a simple and efficient method, Plugin, which \emph{post-processes} arbitrary multiclass predictions from any black-box classifier in order to simultaneously (1) adapt these predictions to a target distribution; and (2) optimize a particular metric of the confusion matrix. Importantly, Plugin is a completely \textit{post-hoc} method which does not rely on feature information, only requires a small amount of probabilistic predictions along with their corresponding true label, and optimizes metrics by querying. We empirically demonstrate that Plugin is both broadly applicable and has performance competitive with related methods on a variety of tabular and language tasks. |
| title | An Efficient Plugin Method for Metric Optimization of Black-Box Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2503.02119 |