How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Ranking
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913236113162240 |
|---|---|
| author | Hu, Zhengyu Zhang, Jieyu Yu, Yue Zhuang, Yuchen Xiong, Hui |
| author_facet | Hu, Zhengyu Zhang, Jieyu Yu, Yue Zhuang, Yuchen Xiong, Hui |
| contents | This paper presents LEMR (Label-Efficient Model Ranking) and introduces the MoraBench Benchmark. LEMR is a novel framework that minimizes the need for costly annotations in model selection by strategically annotating instances from an unlabeled validation set. To evaluate LEMR, we leverage the MoraBench Benchmark, a comprehensive collection of model outputs across diverse scenarios. Our extensive evaluation across 23 different NLP tasks in semi-supervised learning, weak supervision, and prompt selection tasks demonstrates LEMR's effectiveness in significantly reducing labeling costs. Key findings highlight the impact of suitable ensemble methods, uncertainty sampling strategies, and model committee selection in enhancing model ranking accuracy. LEMR, supported by the insights from MoraBench, provides a cost-effective and accurate solution for model selection, especially valuable in resource-constrained environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_01619 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Ranking Hu, Zhengyu Zhang, Jieyu Yu, Yue Zhuang, Yuchen Xiong, Hui Machine Learning This paper presents LEMR (Label-Efficient Model Ranking) and introduces the MoraBench Benchmark. LEMR is a novel framework that minimizes the need for costly annotations in model selection by strategically annotating instances from an unlabeled validation set. To evaluate LEMR, we leverage the MoraBench Benchmark, a comprehensive collection of model outputs across diverse scenarios. Our extensive evaluation across 23 different NLP tasks in semi-supervised learning, weak supervision, and prompt selection tasks demonstrates LEMR's effectiveness in significantly reducing labeling costs. Key findings highlight the impact of suitable ensemble methods, uncertainty sampling strategies, and model committee selection in enhancing model ranking accuracy. LEMR, supported by the insights from MoraBench, provides a cost-effective and accurate solution for model selection, especially valuable in resource-constrained environments. |
| title | How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Ranking |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2312.01619 |