Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.16188 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913328799940608 |
|---|---|
| author | Vishwakarma, Harit Reid Chen Tay, Sui Jiet Namburi, Satya Sai Srinath Sala, Frederic Vinayak, Ramya Korlakai |
| author_facet | Vishwakarma, Harit Reid Chen Tay, Sui Jiet Namburi, Satya Sai Srinath Sala, Frederic Vinayak, Ramya Korlakai |
| contents | Auto-labeling is an important family of techniques that produce labeled training sets with minimum manual labeling. A prominent variant, threshold-based auto-labeling (TBAL), works by finding a threshold on a model's confidence scores above which it can accurately label unlabeled data points. However, many models are known to produce overconfident scores, leading to poor TBAL performance. While a natural idea is to apply off-the-shelf calibration methods to alleviate the overconfidence issue, such methods still fall short. Rather than experimenting with ad-hoc choices of confidence functions, we propose a framework for studying the \emph{optimal} TBAL confidence function. We develop a tractable version of the framework to obtain \texttt{Colander} (Confidence functions for Efficient and Reliable Auto-labeling), a new post-hoc method specifically designed to maximize performance in TBAL systems. We perform an extensive empirical evaluation of our method \texttt{Colander} and compare it against methods designed for calibration. \texttt{Colander} achieves up to 60\% improvements on coverage over the baselines while maintaining auto-labeling error below $5\%$ and using the same amount of labeled data as the baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_16188 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Pearls from Pebbles: Improved Confidence Functions for Auto-labeling Vishwakarma, Harit Reid Chen Tay, Sui Jiet Namburi, Satya Sai Srinath Sala, Frederic Vinayak, Ramya Korlakai Machine Learning Artificial Intelligence Auto-labeling is an important family of techniques that produce labeled training sets with minimum manual labeling. A prominent variant, threshold-based auto-labeling (TBAL), works by finding a threshold on a model's confidence scores above which it can accurately label unlabeled data points. However, many models are known to produce overconfident scores, leading to poor TBAL performance. While a natural idea is to apply off-the-shelf calibration methods to alleviate the overconfidence issue, such methods still fall short. Rather than experimenting with ad-hoc choices of confidence functions, we propose a framework for studying the \emph{optimal} TBAL confidence function. We develop a tractable version of the framework to obtain \texttt{Colander} (Confidence functions for Efficient and Reliable Auto-labeling), a new post-hoc method specifically designed to maximize performance in TBAL systems. We perform an extensive empirical evaluation of our method \texttt{Colander} and compare it against methods designed for calibration. \texttt{Colander} achieves up to 60\% improvements on coverage over the baselines while maintaining auto-labeling error below $5\%$ and using the same amount of labeled data as the baselines. |
| title | Pearls from Pebbles: Improved Confidence Functions for Auto-labeling |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2404.16188 |