CreativityPrism: A Holistic Evaluation Framework for Large Language Model Creativity
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910025257058304 |
|---|---|
| author | Hou, Zhaoyi Joey Zhang, Bowei Alvin Lu, Yining Baghel, Bhiman Kumar Brei, Anneliese Lu, Ximing Jiang, Meng Brahman, Faeze Chaturvedi, Snigdha Chang, Haw-Shiuan Khashabi, Daniel Li, Xiang Lorraine |
| author_facet | Hou, Zhaoyi Joey Zhang, Bowei Alvin Lu, Yining Baghel, Bhiman Kumar Brei, Anneliese Lu, Ximing Jiang, Meng Brahman, Faeze Chaturvedi, Snigdha Chang, Haw-Shiuan Khashabi, Daniel Li, Xiang Lorraine |
| contents | Creativity is often seen as a hallmark of human intelligence. While large language models (LLMs) are increasingly perceived as generating creative text, there is still no holistic and scalable framework to evaluate their creativity across diverse scenarios. Existing methods of LLM creativity evaluation either heavily rely on humans, limiting speed and scalability, or are fragmented across different domains and different definitions of creativity. To address this gap, we propose CREATIVITYPRISM, an evaluation analysis framework that consolidates eight tasks from three domains, divergent thinking, creative writing, and logical reasoning, into a taxonomy of creativity that emphasizes three dimensions: quality, novelty, and diversity of LLM generations. The framework is designed to be scalable with reliable automatic evaluation judges that have been validated against human annotations. We evaluate 17 state-of-the-art (SoTA) proprietary and open-sourced LLMs on CREATIVITYPRISM and find that while proprietary LLMs dominate creative writing and logical reasoning tasks by a 15% lead over open-sourced ones, they offer no significant advantage in divergent thinking, a domain much less explored in existing post-training regimes. Our analysis also shows that high performance in one creative dimension or domain rarely generalizes to others; specifically, novelty metrics often show weak or negative correlations with other metrics. This fragmentation confirms that a holistic, multi-dimensional framework like CREATIVITYPRISM is essential for meaningful assessment of LLM creativity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_20091 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CreativityPrism: A Holistic Evaluation Framework for Large Language Model Creativity Hou, Zhaoyi Joey Zhang, Bowei Alvin Lu, Yining Baghel, Bhiman Kumar Brei, Anneliese Lu, Ximing Jiang, Meng Brahman, Faeze Chaturvedi, Snigdha Chang, Haw-Shiuan Khashabi, Daniel Li, Xiang Lorraine Computation and Language Artificial Intelligence Creativity is often seen as a hallmark of human intelligence. While large language models (LLMs) are increasingly perceived as generating creative text, there is still no holistic and scalable framework to evaluate their creativity across diverse scenarios. Existing methods of LLM creativity evaluation either heavily rely on humans, limiting speed and scalability, or are fragmented across different domains and different definitions of creativity. To address this gap, we propose CREATIVITYPRISM, an evaluation analysis framework that consolidates eight tasks from three domains, divergent thinking, creative writing, and logical reasoning, into a taxonomy of creativity that emphasizes three dimensions: quality, novelty, and diversity of LLM generations. The framework is designed to be scalable with reliable automatic evaluation judges that have been validated against human annotations. We evaluate 17 state-of-the-art (SoTA) proprietary and open-sourced LLMs on CREATIVITYPRISM and find that while proprietary LLMs dominate creative writing and logical reasoning tasks by a 15% lead over open-sourced ones, they offer no significant advantage in divergent thinking, a domain much less explored in existing post-training regimes. Our analysis also shows that high performance in one creative dimension or domain rarely generalizes to others; specifically, novelty metrics often show weak or negative correlations with other metrics. This fragmentation confirms that a holistic, multi-dimensional framework like CREATIVITYPRISM is essential for meaningful assessment of LLM creativity. |
| title | CreativityPrism: A Holistic Evaluation Framework for Large Language Model Creativity |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.20091 |