WritingBench: A Comprehensive Benchmark for Generative Writing
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914172898377728 |
|---|---|
| author | Wu, Yuning Mei, Jiahao Yan, Ming Li, Chenliang Lai, Shaopeng Ren, Yuran Wang, Zijia Zhang, Ji Wu, Mengyue Jin, Qin Huang, Fei |
| author_facet | Wu, Yuning Mei, Jiahao Yan, Ming Li, Chenliang Lai, Shaopeng Ren, Yuran Wang, Zijia Zhang, Ji Wu, Mengyue Jin, Qin Huang, Fei |
| contents | Recent advancements in large language models (LLMs) have significantly enhanced text generation capabilities, yet evaluating their performance in generative writing remains a challenge. Existing benchmarks primarily focus on generic text generation or limited in writing tasks, failing to capture the diverse requirements of high-quality written contents across various domains. To bridge this gap, we present WritingBench, a comprehensive benchmark designed to evaluate LLMs across 6 core writing domains and 100 subdomains. We further propose a query-dependent evaluation framework that empowers LLMs to dynamically generate instance-specific assessment criteria. This framework is complemented by a fine-tuned critic model for criteria-aware scoring, enabling evaluations in style, format and length. The framework's validity is further demonstrated by its data curation capability, which enables a 7B-parameter model to outperform the performance of GPT-4o in writing. We open-source the benchmark, along with evaluation tools and modular framework components, to advance the development of LLMs in writing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_05244 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | WritingBench: A Comprehensive Benchmark for Generative Writing Wu, Yuning Mei, Jiahao Yan, Ming Li, Chenliang Lai, Shaopeng Ren, Yuran Wang, Zijia Zhang, Ji Wu, Mengyue Jin, Qin Huang, Fei Artificial Intelligence Computation and Language Recent advancements in large language models (LLMs) have significantly enhanced text generation capabilities, yet evaluating their performance in generative writing remains a challenge. Existing benchmarks primarily focus on generic text generation or limited in writing tasks, failing to capture the diverse requirements of high-quality written contents across various domains. To bridge this gap, we present WritingBench, a comprehensive benchmark designed to evaluate LLMs across 6 core writing domains and 100 subdomains. We further propose a query-dependent evaluation framework that empowers LLMs to dynamically generate instance-specific assessment criteria. This framework is complemented by a fine-tuned critic model for criteria-aware scoring, enabling evaluations in style, format and length. The framework's validity is further demonstrated by its data curation capability, which enables a 7B-parameter model to outperform the performance of GPT-4o in writing. We open-source the benchmark, along with evaluation tools and modular framework components, to advance the development of LLMs in writing. |
| title | WritingBench: A Comprehensive Benchmark for Generative Writing |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2503.05244 |