LLM Confidence Evaluation Measures in Zero-Shot CSS Classification
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915002432094208 |
|---|---|
| author | Farr, David Cruickshank, Iain Manzonelli, Nico Clark, Nicholas Starbird, Kate West, Jevin |
| author_facet | Farr, David Cruickshank, Iain Manzonelli, Nico Clark, Nicholas Starbird, Kate West, Jevin |
| contents | Assessing classification confidence is critical for leveraging large language models (LLMs) in automated labeling tasks, especially in the sensitive domains presented by Computational Social Science (CSS) tasks. In this paper, we make three key contributions: (1) we propose an uncertainty quantification (UQ) performance measure tailored for data annotation tasks, (2) we compare, for the first time, five different UQ strategies across three distinct LLMs and CSS data annotation tasks, (3) we introduce a novel UQ aggregation strategy that effectively identifies low-confidence LLM annotations and disproportionately uncovers data incorrectly labeled by the LLMs. Our results demonstrate that our proposed UQ aggregation strategy improves upon existing methods andcan be used to significantly improve human-in-the-loop data annotation processes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_13047 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLM Confidence Evaluation Measures in Zero-Shot CSS Classification Farr, David Cruickshank, Iain Manzonelli, Nico Clark, Nicholas Starbird, Kate West, Jevin Human-Computer Interaction Computation and Language Information Retrieval Assessing classification confidence is critical for leveraging large language models (LLMs) in automated labeling tasks, especially in the sensitive domains presented by Computational Social Science (CSS) tasks. In this paper, we make three key contributions: (1) we propose an uncertainty quantification (UQ) performance measure tailored for data annotation tasks, (2) we compare, for the first time, five different UQ strategies across three distinct LLMs and CSS data annotation tasks, (3) we introduce a novel UQ aggregation strategy that effectively identifies low-confidence LLM annotations and disproportionately uncovers data incorrectly labeled by the LLMs. Our results demonstrate that our proposed UQ aggregation strategy improves upon existing methods andcan be used to significantly improve human-in-the-loop data annotation processes. |
| title | LLM Confidence Evaluation Measures in Zero-Shot CSS Classification |
| topic | Human-Computer Interaction Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2410.13047 |