Prompt Candidates, then Distill: A Teacher-Student Framework for LLM-driven Data Annotation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913875054559232 |
|---|---|
| author | Xia, Mingxuan Wang, Haobo Li, Yixuan Yu, Zewei Wang, Jindong Zhao, Junbo Wu, Runze |
| author_facet | Xia, Mingxuan Wang, Haobo Li, Yixuan Yu, Zewei Wang, Jindong Zhao, Junbo Wu, Runze |
| contents | Recently, Large Language Models (LLMs) have demonstrated significant potential for data annotation, markedly reducing the labor costs associated with downstream applications. However, existing methods mostly adopt an aggressive strategy by prompting LLM to determine a single gold label for each unlabeled sample. Due to the inherent uncertainty within LLMs, they often produce incorrect labels for difficult samples, severely compromising the data quality for downstream applications. Motivated by ambiguity aversion in human behaviors, we propose a novel candidate annotation paradigm wherein large language models are encouraged to output all possible labels when incurring uncertainty. To ensure unique labels are provided for downstream tasks, we develop a teacher-student framework CanDist that distills candidate annotations with a Small Language Model (SLM). We further provide a rigorous justification demonstrating that distilling candidate annotations from the teacher LLM offers superior theoretical guarantees compared to directly using single annotations. Extensive experiments across six text classification tasks validate the effectiveness of our proposed method. The source code is available at https://github.com/MingxuanXia/CanDist. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_03857 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Prompt Candidates, then Distill: A Teacher-Student Framework for LLM-driven Data Annotation Xia, Mingxuan Wang, Haobo Li, Yixuan Yu, Zewei Wang, Jindong Zhao, Junbo Wu, Runze Machine Learning Computation and Language Recently, Large Language Models (LLMs) have demonstrated significant potential for data annotation, markedly reducing the labor costs associated with downstream applications. However, existing methods mostly adopt an aggressive strategy by prompting LLM to determine a single gold label for each unlabeled sample. Due to the inherent uncertainty within LLMs, they often produce incorrect labels for difficult samples, severely compromising the data quality for downstream applications. Motivated by ambiguity aversion in human behaviors, we propose a novel candidate annotation paradigm wherein large language models are encouraged to output all possible labels when incurring uncertainty. To ensure unique labels are provided for downstream tasks, we develop a teacher-student framework CanDist that distills candidate annotations with a Small Language Model (SLM). We further provide a rigorous justification demonstrating that distilling candidate annotations from the teacher LLM offers superior theoretical guarantees compared to directly using single annotations. Extensive experiments across six text classification tasks validate the effectiveness of our proposed method. The source code is available at https://github.com/MingxuanXia/CanDist. |
| title | Prompt Candidates, then Distill: A Teacher-Student Framework for LLM-driven Data Annotation |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2506.03857 |