A Case Study of Scalable Content Annotation Using Multi-LLM Consensus and Human Review
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909595430027264 |
|---|---|
| author | Yuan, Mingyue Chen, Jieshan Xing, Zhenchang Mohammadi, Gelareh Quigley, Aaron |
| author_facet | Yuan, Mingyue Chen, Jieshan Xing, Zhenchang Mohammadi, Gelareh Quigley, Aaron |
| contents | Content annotation at scale remains challenging, requiring substantial human expertise and effort. This paper presents a case study in code documentation analysis, where we explore the balance between automation efficiency and annotation accuracy. We present MCHR (Multi-LLM Consensus with Human Review), a novel semi-automated framework that enhances annotation scalability through the systematic integration of multiple LLMs and targeted human review. Our framework introduces a structured consensus-building mechanism among LLMs and an adaptive review protocol that strategically engages human expertise. Through our case study, we demonstrate that MCHR reduces annotation time by 32% to 100% compared to manual annotation while maintaining high accuracy (85.5% to 98%) across different difficulty levels, from basic binary classification to challenging open-set scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_17620 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Case Study of Scalable Content Annotation Using Multi-LLM Consensus and Human Review Yuan, Mingyue Chen, Jieshan Xing, Zhenchang Mohammadi, Gelareh Quigley, Aaron Human-Computer Interaction Content annotation at scale remains challenging, requiring substantial human expertise and effort. This paper presents a case study in code documentation analysis, where we explore the balance between automation efficiency and annotation accuracy. We present MCHR (Multi-LLM Consensus with Human Review), a novel semi-automated framework that enhances annotation scalability through the systematic integration of multiple LLMs and targeted human review. Our framework introduces a structured consensus-building mechanism among LLMs and an adaptive review protocol that strategically engages human expertise. Through our case study, we demonstrate that MCHR reduces annotation time by 32% to 100% compared to manual annotation while maintaining high accuracy (85.5% to 98%) across different difficulty levels, from basic binary classification to challenging open-set scenarios. |
| title | A Case Study of Scalable Content Annotation Using Multi-LLM Consensus and Human Review |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2503.17620 |