Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.05135 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908578878586880 |
|---|---|
| author | Kumar, Vanya Bannihatti Goyal, Divyanshu Eppa, Akhil Bhandari, Neel |
| author_facet | Kumar, Vanya Bannihatti Goyal, Divyanshu Eppa, Akhil Bhandari, Neel |
| contents | Modern large language models (LLMs) excel at objective tasks such as evaluating mathematical reasoning and factual accuracy, yet they falter when faced with the nuanced, subjective nature of assessing creativity. In this work, we propose a novel curiosity-driven LLM-as-a-judge for evaluating creative writing which is personlized to each individual's creative judgments. We use the Torrance Test of Creative Thinking(TTCW) benchmark introduced in Chakrabarty et al. (2024), which has stories annotated by expert humans across various subjective dimensions like Originality, to test our hypothesis. We show that our method enables models across various sizes, to learn the nuanced creative judgments of different individuals, by showing improvements over baseline supervised finetuning(SFT) method across various evaluation metrics like Pearson correlation, Cohen's and F1 values. Our method is especially useful in subjective evaluations where not all the annotators agree with each other.
|
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_05135 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Curiosity-Driven LLM-as-a-judge for Personalized Creative Judgment Kumar, Vanya Bannihatti Goyal, Divyanshu Eppa, Akhil Bhandari, Neel Computation and Language Machine Learning Modern large language models (LLMs) excel at objective tasks such as evaluating mathematical reasoning and factual accuracy, yet they falter when faced with the nuanced, subjective nature of assessing creativity. In this work, we propose a novel curiosity-driven LLM-as-a-judge for evaluating creative writing which is personlized to each individual's creative judgments. We use the Torrance Test of Creative Thinking(TTCW) benchmark introduced in Chakrabarty et al. (2024), which has stories annotated by expert humans across various subjective dimensions like Originality, to test our hypothesis. We show that our method enables models across various sizes, to learn the nuanced creative judgments of different individuals, by showing improvements over baseline supervised finetuning(SFT) method across various evaluation metrics like Pearson correlation, Cohen's and F1 values. Our method is especially useful in subjective evaluations where not all the annotators agree with each other. |
| title | Curiosity-Driven LLM-as-a-judge for Personalized Creative Judgment |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2510.05135 |