NoveltyBench: Evaluating Language Models for Humanlike Diversity
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915436103204864 |
|---|---|
| author | Zhang, Yiming Diddee, Harshita Holm, Susan Liu, Hanchen Liu, Xinyue Samuel, Vinay Wang, Barry Ippolito, Daphne |
| author_facet | Zhang, Yiming Diddee, Harshita Holm, Susan Liu, Hanchen Liu, Xinyue Samuel, Vinay Wang, Barry Ippolito, Daphne |
| contents | Language models have demonstrated remarkable capabilities on standard benchmarks, yet they struggle increasingly from mode collapse, the inability to generate diverse and novel outputs. Our work introduces NoveltyBench, a benchmark specifically designed to evaluate the ability of language models to produce multiple distinct and high-quality outputs. NoveltyBench utilizes prompts curated to elicit diverse answers and filtered real-world user queries. Evaluating 20 leading language models, we find that current state-of-the-art systems generate significantly less diversity than human writers. Notably, larger models within a family often exhibit less diversity than their smaller counterparts, challenging the notion that capability on standard benchmarks translates directly to generative utility. While prompting strategies like in-context regeneration can elicit diversity, our findings highlight a fundamental lack of distributional diversity in current models, reducing their utility for users seeking varied responses and suggesting the need for new training and evaluation paradigms that prioritize diversity alongside quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_05228 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | NoveltyBench: Evaluating Language Models for Humanlike Diversity Zhang, Yiming Diddee, Harshita Holm, Susan Liu, Hanchen Liu, Xinyue Samuel, Vinay Wang, Barry Ippolito, Daphne Computation and Language Language models have demonstrated remarkable capabilities on standard benchmarks, yet they struggle increasingly from mode collapse, the inability to generate diverse and novel outputs. Our work introduces NoveltyBench, a benchmark specifically designed to evaluate the ability of language models to produce multiple distinct and high-quality outputs. NoveltyBench utilizes prompts curated to elicit diverse answers and filtered real-world user queries. Evaluating 20 leading language models, we find that current state-of-the-art systems generate significantly less diversity than human writers. Notably, larger models within a family often exhibit less diversity than their smaller counterparts, challenging the notion that capability on standard benchmarks translates directly to generative utility. While prompting strategies like in-context regeneration can elicit diversity, our findings highlight a fundamental lack of distributional diversity in current models, reducing their utility for users seeking varied responses and suggesting the need for new training and evaluation paradigms that prioritize diversity alongside quality. |
| title | NoveltyBench: Evaluating Language Models for Humanlike Diversity |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.05228 |