Scalable and consistent few-shot classification of survey responses using text embeddings
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915466206773248 |
|---|---|
| author | Mjaaland, Jonas Timmann Kreutzer, Markus Fleten Tyseng, Halvor Fussell, Rebeckah K. Passante, Gina Holmes, N. G. Malthe-Sørenssen, Anders Odden, Tor Ole B. |
| author_facet | Mjaaland, Jonas Timmann Kreutzer, Markus Fleten Tyseng, Halvor Fussell, Rebeckah K. Passante, Gina Holmes, N. G. Malthe-Sørenssen, Anders Odden, Tor Ole B. |
| contents | Qualitative analysis of open-ended survey responses is a commonly-used research method in the social sciences, but traditional coding approaches are often time-consuming and prone to inconsistency. Existing solutions from Natural Language Processing such as supervised classifiers, topic modeling techniques, and generative large language models have limited applicability in qualitative analysis, since they demand extensive labeled data, disrupt established qualitative workflows, and/or yield variable results. In this paper, we introduce a text embedding-based classification framework that requires only a handful of examples per category and fits well with standard qualitative workflows. When benchmarked against human analysis of a conceptual physics survey consisting of 2899 open-ended responses, our framework achieves a Cohen's Kappa ranging from 0.74 to 0.83 as compared to expert human coders in an exhaustive coding scheme. We further show how performance of this framework improves with fine-tuning of the text embedding model, and how the method can be used to audit previously-analyzed datasets. These findings demonstrate that text embedding-assisted coding can flexibly scale to thousands of responses without sacrificing interpretability, opening avenues for deductive qualitative analysis at scale. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_19836 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scalable and consistent few-shot classification of survey responses using text embeddings Mjaaland, Jonas Timmann Kreutzer, Markus Fleten Tyseng, Halvor Fussell, Rebeckah K. Passante, Gina Holmes, N. G. Malthe-Sørenssen, Anders Odden, Tor Ole B. Computation and Language Physics Education Qualitative analysis of open-ended survey responses is a commonly-used research method in the social sciences, but traditional coding approaches are often time-consuming and prone to inconsistency. Existing solutions from Natural Language Processing such as supervised classifiers, topic modeling techniques, and generative large language models have limited applicability in qualitative analysis, since they demand extensive labeled data, disrupt established qualitative workflows, and/or yield variable results. In this paper, we introduce a text embedding-based classification framework that requires only a handful of examples per category and fits well with standard qualitative workflows. When benchmarked against human analysis of a conceptual physics survey consisting of 2899 open-ended responses, our framework achieves a Cohen's Kappa ranging from 0.74 to 0.83 as compared to expert human coders in an exhaustive coding scheme. We further show how performance of this framework improves with fine-tuning of the text embedding model, and how the method can be used to audit previously-analyzed datasets. These findings demonstrate that text embedding-assisted coding can flexibly scale to thousands of responses without sacrificing interpretability, opening avenues for deductive qualitative analysis at scale. |
| title | Scalable and consistent few-shot classification of survey responses using text embeddings |
| topic | Computation and Language Physics Education |
| url | https://arxiv.org/abs/2508.19836 |