From Fallback to Frontline: When Can LLMs be Superior Annotators of Human Perspectives?
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918456441438208 |
|---|---|
| author | Amin, Hasan Tian, Harry Yizhou Duan, Xiaoni Ho, Chien-Ju Khanna, Rajiv Yin, Ming |
| author_facet | Amin, Hasan Tian, Harry Yizhou Duan, Xiaoni Ho, Chien-Ju Khanna, Rajiv Yin, Ming |
| contents | Although large language models (LLMs) are increasingly used as annotators at scale, they are typically treated as a pragmatic fallback rather than a faithful estimator of human perspectives. This work challenges that presumption. By framing perspective-taking as the estimation of a latent group-level judgment, we characterize the conditions under which modern LLMs can outperform human annotators, including in-group humans, when predicting aggregate subgroup opinions on subjective tasks, and show that these conditions are common in practice. This advantage arises from structural properties of LLMs as estimators, including low variance and reduced coupling between representation and processing biases, rather than any claim of lived experience. Our analysis identifies clear regimes where LLMs act as statistically superior frontline estimators, as well as principled limits where human judgment remains essential. These findings reposition LLMs from a cost-saving compromise to a principled tool for estimating collective human perspectives. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_17968 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | From Fallback to Frontline: When Can LLMs be Superior Annotators of Human Perspectives? Amin, Hasan Tian, Harry Yizhou Duan, Xiaoni Ho, Chien-Ju Khanna, Rajiv Yin, Ming Artificial Intelligence Computation and Language Although large language models (LLMs) are increasingly used as annotators at scale, they are typically treated as a pragmatic fallback rather than a faithful estimator of human perspectives. This work challenges that presumption. By framing perspective-taking as the estimation of a latent group-level judgment, we characterize the conditions under which modern LLMs can outperform human annotators, including in-group humans, when predicting aggregate subgroup opinions on subjective tasks, and show that these conditions are common in practice. This advantage arises from structural properties of LLMs as estimators, including low variance and reduced coupling between representation and processing biases, rather than any claim of lived experience. Our analysis identifies clear regimes where LLMs act as statistically superior frontline estimators, as well as principled limits where human judgment remains essential. These findings reposition LLMs from a cost-saving compromise to a principled tool for estimating collective human perspectives. |
| title | From Fallback to Frontline: When Can LLMs be Superior Annotators of Human Perspectives? |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2604.17968 |