Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.07458 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915575862657024 |
|---|---|
| author | Jung, Yujin J. Tamaki, Eduardo Ryô Chatterley, Julia Mitchell, Grant Dzebo, Semir Sandoval, Cristóbal Littvay, Levente Hawkins, Kirk A. |
| author_facet | Jung, Yujin J. Tamaki, Eduardo Ryô Chatterley, Julia Mitchell, Grant Dzebo, Semir Sandoval, Cristóbal Littvay, Levente Hawkins, Kirk A. |
| contents | Measuring the ideational content of populism remains a challenge. Traditional strategies based on textual analysis have been critical for building the field's foundations and providing a valid, objective indicator of populist framing. Yet these approaches are costly, time consuming, and difficult to scale across languages, contexts, and large corpora. Here we present the results from a rubric and anchor guided chain of thought (CoT) prompting approach that mirrors human coder training. By leveraging the Global Populism Database (GPD), a comprehensive dataset of global leaders' speeches annotated for degrees of populism, we replicate the process used to train human coders by prompting the LLM with an adapted version of the same documentation to guide the model's reasoning. We then test multiple proprietary and open weight models by replicating scores in the GPD. Our findings reveal that this domain specific prompting strategy enables the LLM to achieve classification accuracy on par with expert human coders, demonstrating its ability to navigate the nuanced, context sensitive aspects of populism. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_07458 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Populism Meets AI: Advancing Populism Research with LLMs Jung, Yujin J. Tamaki, Eduardo Ryô Chatterley, Julia Mitchell, Grant Dzebo, Semir Sandoval, Cristóbal Littvay, Levente Hawkins, Kirk A. Computation and Language Measuring the ideational content of populism remains a challenge. Traditional strategies based on textual analysis have been critical for building the field's foundations and providing a valid, objective indicator of populist framing. Yet these approaches are costly, time consuming, and difficult to scale across languages, contexts, and large corpora. Here we present the results from a rubric and anchor guided chain of thought (CoT) prompting approach that mirrors human coder training. By leveraging the Global Populism Database (GPD), a comprehensive dataset of global leaders' speeches annotated for degrees of populism, we replicate the process used to train human coders by prompting the LLM with an adapted version of the same documentation to guide the model's reasoning. We then test multiple proprietary and open weight models by replicating scores in the GPD. Our findings reveal that this domain specific prompting strategy enables the LLM to achieve classification accuracy on par with expert human coders, demonstrating its ability to navigate the nuanced, context sensitive aspects of populism. |
| title | Populism Meets AI: Advancing Populism Research with LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.07458 |