HyPAC: Cost-Efficient LLMs-Human Hybrid Annotation with PAC Error Guarantees
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917243689893888 |
|---|---|
| author | Zeng, Hao Huang, Huipeng Qu, Xinhao Huang, Jianguo Jing, Bingyi Wei, Hongxin |
| author_facet | Zeng, Hao Huang, Huipeng Qu, Xinhao Huang, Jianguo Jing, Bingyi Wei, Hongxin |
| contents | Data annotation often involves multiple sources with different cost-quality trade-offs, such as fast large language models (LLMs), slow reasoning models, and human experts. In this work, we study the problem of routing inputs to the most cost-efficient annotation source while controlling the labeling error on test instances. We propose \textbf{HyPAC}, a method that adaptively labels inputs to the most cost-efficient annotation source while providing distribution-free guarantees on annotation error. HyPAC calibrates two decision thresholds using importance sampling and upper confidence bounds, partitioning inputs into three regions based on uncertainty and routing each to the appropriate annotation source. We prove that HyPAC achieves the minimum expected cost with a probably approximately correct (PAC) guarantee on the annotation error, free of data distribution and pre-trained models. Experiments on common benchmarks demonstrate the effectiveness of our method, reducing the annotation cost by 78.51\% while tightly controlling the annotation error. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_02550 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | HyPAC: Cost-Efficient LLMs-Human Hybrid Annotation with PAC Error Guarantees Zeng, Hao Huang, Huipeng Qu, Xinhao Huang, Jianguo Jing, Bingyi Wei, Hongxin Machine Learning Artificial Intelligence Data annotation often involves multiple sources with different cost-quality trade-offs, such as fast large language models (LLMs), slow reasoning models, and human experts. In this work, we study the problem of routing inputs to the most cost-efficient annotation source while controlling the labeling error on test instances. We propose \textbf{HyPAC}, a method that adaptively labels inputs to the most cost-efficient annotation source while providing distribution-free guarantees on annotation error. HyPAC calibrates two decision thresholds using importance sampling and upper confidence bounds, partitioning inputs into three regions based on uncertainty and routing each to the appropriate annotation source. We prove that HyPAC achieves the minimum expected cost with a probably approximately correct (PAC) guarantee on the annotation error, free of data distribution and pre-trained models. Experiments on common benchmarks demonstrate the effectiveness of our method, reducing the annotation cost by 78.51\% while tightly controlling the annotation error. |
| title | HyPAC: Cost-Efficient LLMs-Human Hybrid Annotation with PAC Error Guarantees |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2602.02550 |