Automated Creativity Evaluation for Large Language Models: A Reference-Based Approach
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910916457529344 |
|---|---|
| author | Li, Ruizhe Zhu, Chiwei Xu, Benfeng Wang, Xiaorui Mao, Zhendong |
| author_facet | Li, Ruizhe Zhu, Chiwei Xu, Benfeng Wang, Xiaorui Mao, Zhendong |
| contents | Creative writing is a key capability of Large Language Models (LLMs), with potential applications in literature, storytelling, and various creative domains. However, evaluating the creativity of machine-generated texts remains a significant challenge, as existing methods either rely on costly manual annotations or fail to align closely with human assessments. In this paper, we propose an effective automated evaluation method based on the Torrance Test of Creative Writing (TTCW), which evaluates creativity as product. Our method employs a reference-based Likert-style approach, scoring generated creative texts relative to high-quality reference texts across various tests. Experimental results demonstrate that our method significantly improves the alignment between LLM evaluations and human assessments, achieving a pairwise accuracy of 0.75 (+15\%). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_15784 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Automated Creativity Evaluation for Large Language Models: A Reference-Based Approach Li, Ruizhe Zhu, Chiwei Xu, Benfeng Wang, Xiaorui Mao, Zhendong Computation and Language Artificial Intelligence Creative writing is a key capability of Large Language Models (LLMs), with potential applications in literature, storytelling, and various creative domains. However, evaluating the creativity of machine-generated texts remains a significant challenge, as existing methods either rely on costly manual annotations or fail to align closely with human assessments. In this paper, we propose an effective automated evaluation method based on the Torrance Test of Creative Writing (TTCW), which evaluates creativity as product. Our method employs a reference-based Likert-style approach, scoring generated creative texts relative to high-quality reference texts across various tests. Experimental results demonstrate that our method significantly improves the alignment between LLM evaluations and human assessments, achieving a pairwise accuracy of 0.75 (+15\%). |
| title | Automated Creativity Evaluation for Large Language Models: A Reference-Based Approach |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2504.15784 |