Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.08646 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911456412303360 |
|---|---|
| author | Kreutner, Maximilian Rupprecht, Jens Ahnert, Georg Salem, Ahmed Strohmaier, Markus |
| author_facet | Kreutner, Maximilian Rupprecht, Jens Ahnert, Georg Salem, Ahmed Strohmaier, Markus |
| contents | We introduce QSTN, an open-source Python framework for systematically generating responses from questionnaire-style prompts to support in-silico surveys and annotation tasks with large language models (LLMs). QSTN enables robust evaluation of questionnaire presentation, prompt perturbations, and response generation methods. Our extensive evaluation (>40 million survey responses) shows that question structure and response generation methods have a significant impact on the alignment of generated survey responses with human answers. We also find that answers can be obtained for a fraction of the compute cost, by changing the presentation method. In addition, we offer a no-code user interface that allows researchers to set up robust experiments with LLMs \emph{without coding knowledge}. We hope that QSTN will support the reproducibility and reliability of LLM-based research in the future. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_08646 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | QSTN: A Modular Framework for Robust Questionnaire Inference with Large Language Models Kreutner, Maximilian Rupprecht, Jens Ahnert, Georg Salem, Ahmed Strohmaier, Markus Computation and Language Computers and Society We introduce QSTN, an open-source Python framework for systematically generating responses from questionnaire-style prompts to support in-silico surveys and annotation tasks with large language models (LLMs). QSTN enables robust evaluation of questionnaire presentation, prompt perturbations, and response generation methods. Our extensive evaluation (>40 million survey responses) shows that question structure and response generation methods have a significant impact on the alignment of generated survey responses with human answers. We also find that answers can be obtained for a fraction of the compute cost, by changing the presentation method. In addition, we offer a no-code user interface that allows researchers to set up robust experiments with LLMs \emph{without coding knowledge}. We hope that QSTN will support the reproducibility and reliability of LLM-based research in the future. |
| title | QSTN: A Modular Framework for Robust Questionnaire Inference with Large Language Models |
| topic | Computation and Language Computers and Society |
| url | https://arxiv.org/abs/2512.08646 |