Reliable Decision Support with LLMs: A Framework for Evaluating Consistency in Binary Text Classification Applications
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915687448969216 |
|---|---|
| author | Megahed, Fadel M. Chen, Ying-Ju Jones-Farmer, L. Allision Lee, Younghwa Wang, Jiawei Brooke Zwetsloot, Inez M. |
| author_facet | Megahed, Fadel M. Chen, Ying-Ju Jones-Farmer, L. Allision Lee, Younghwa Wang, Jiawei Brooke Zwetsloot, Inez M. |
| contents | This study introduces a framework for evaluating consistency in large language model (LLM) binary text classification, addressing the lack of established reliability assessment methods. Adapting psychometric principles, we determine sample size requirements, develop metrics for invalid responses, and evaluate intra- and inter-rater reliability. Our case study examines financial news sentiment classification across 14 LLMs (including claude-3-7-sonnet, gpt-4o, deepseek-r1, gemma3, llama3.2, phi4, and command-r-plus), with five replicates per model on 1,350 articles. Models demonstrated high intra-rater consistency, achieving perfect agreement on 90-98% of examples, with minimal differences between expensive and economical models from the same families. When validated against StockNewsAPI labels, models achieved strong performance (accuracy 0.76-0.88), with smaller models like gemma3:1B, llama3.2:3B, and claude-3-5-haiku outperforming larger counterparts. All models performed at chance when predicting actual market movements, indicating task constraints rather than model limitations. Our framework provides systematic guidance for LLM selection, sample size planning, and reliability assessment, enabling organizations to optimize resources for classification tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_14918 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Reliable Decision Support with LLMs: A Framework for Evaluating Consistency in Binary Text Classification Applications Megahed, Fadel M. Chen, Ying-Ju Jones-Farmer, L. Allision Lee, Younghwa Wang, Jiawei Brooke Zwetsloot, Inez M. Computation and Language Machine Learning This study introduces a framework for evaluating consistency in large language model (LLM) binary text classification, addressing the lack of established reliability assessment methods. Adapting psychometric principles, we determine sample size requirements, develop metrics for invalid responses, and evaluate intra- and inter-rater reliability. Our case study examines financial news sentiment classification across 14 LLMs (including claude-3-7-sonnet, gpt-4o, deepseek-r1, gemma3, llama3.2, phi4, and command-r-plus), with five replicates per model on 1,350 articles. Models demonstrated high intra-rater consistency, achieving perfect agreement on 90-98% of examples, with minimal differences between expensive and economical models from the same families. When validated against StockNewsAPI labels, models achieved strong performance (accuracy 0.76-0.88), with smaller models like gemma3:1B, llama3.2:3B, and claude-3-5-haiku outperforming larger counterparts. All models performed at chance when predicting actual market movements, indicating task constraints rather than model limitations. Our framework provides systematic guidance for LLM selection, sample size planning, and reliability assessment, enabling organizations to optimize resources for classification tasks. |
| title | Reliable Decision Support with LLMs: A Framework for Evaluating Consistency in Binary Text Classification Applications |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2505.14918 |