Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.11381 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918146795896832 |
|---|---|
| author | Zhou, Yuxiang Xu, Hainiu Ong, Desmond C. Liakata, Maria Slovak, Petr He, Yulan |
| author_facet | Zhou, Yuxiang Xu, Hainiu Ong, Desmond C. Liakata, Maria Slovak, Petr He, Yulan |
| contents | As the utilization of language models in interdisciplinary, human-centered studies grow, expectations of their capabilities continue to evolve. Beyond excelling at conventional tasks, models are now expected to perform well on user-centric measurements involving confidence and human (dis)agreement-factors that reflect subjective preferences. While modeling subjectivity plays an essential role in cognitive science and has been extensively studied, its investigation at the intersection with NLP remains under-explored. In light of this gap, we explore how language models can quantify subjectivity in cognitive appraisal by conducting comprehensive experiments and analyses with both fine-tuned models and prompt-based large language models (LLMs). Our quantitative and qualitative results demonstrate that personality traits and demographic information are critical for measuring subjectivity, yet existing post-hoc calibration methods often fail to achieve satisfactory performance. Furthermore, our in-depth analysis provides valuable insights to guide future research at the intersection of NLP and cognitive science. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_11381 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Modeling Subjectivity in Cognitive Appraisal with Language Models Zhou, Yuxiang Xu, Hainiu Ong, Desmond C. Liakata, Maria Slovak, Petr He, Yulan Computation and Language As the utilization of language models in interdisciplinary, human-centered studies grow, expectations of their capabilities continue to evolve. Beyond excelling at conventional tasks, models are now expected to perform well on user-centric measurements involving confidence and human (dis)agreement-factors that reflect subjective preferences. While modeling subjectivity plays an essential role in cognitive science and has been extensively studied, its investigation at the intersection with NLP remains under-explored. In light of this gap, we explore how language models can quantify subjectivity in cognitive appraisal by conducting comprehensive experiments and analyses with both fine-tuned models and prompt-based large language models (LLMs). Our quantitative and qualitative results demonstrate that personality traits and demographic information are critical for measuring subjectivity, yet existing post-hoc calibration methods often fail to achieve satisfactory performance. Furthermore, our in-depth analysis provides valuable insights to guide future research at the intersection of NLP and cognitive science. |
| title | Modeling Subjectivity in Cognitive Appraisal with Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.11381 |