RAVEL: Reasoning Agents for Validating and Evaluating LLM Text Synthesis
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908858567360512 |
|---|---|
| author | Feng, Andrew Zhuoer Wang, Cunxiang Luo, Yu Wen, Bosi Wang, Yidong Fan, Lin Zhou, Yilin Wang, Zikang Yu, Wenbo Wu, Lindong Wang, Hongning Huang, Minlie |
| author_facet | Feng, Andrew Zhuoer Wang, Cunxiang Luo, Yu Wen, Bosi Wang, Yidong Fan, Lin Zhou, Yilin Wang, Zikang Yu, Wenbo Wu, Lindong Wang, Hongning Huang, Minlie |
| contents | Large Language Models have evolved from single-round generators into long-horizon agents, capable of complex text synthesis scenarios. However, current evaluation frameworks lack the ability to assess the actual synthesis operations, such as outlining, drafting, and editing. Consequently, they fail to evaluate the actual and detailed capabilities of LLMs. To bridge this gap, we introduce RAVEL, an agentic framework that enables the LLM testers to autonomously plan and execute typical synthesis operations, including outlining, drafting, reviewing, and refining. Complementing this framework, we present C3EBench, a comprehensive benchmark comprising 1,258 samples derived from professional human writings. We utilize a "reverse-engineering" pipeline to isolate specific capabilities across four tasks: Cloze, Edit, Expand, and End-to-End. Through our analysis of 14 LLMs, we uncover that most LLMs struggle with tasks that demand contextual understanding under limited or under-specified instructions. By augmenting RAVEL with SOTA LLMs as operators, we find that such agentic text synthesis is dominated by the LLM's reasoning capability rather than raw generative capacity. Furthermore, we find that a strong reasoner can guide a weaker generator to yield higher-quality results, whereas the inverse does not hold. Our code and data are available at this link: https://github.com/ZhuoerFeng/RAVEL-Reasoning-Agents-Text-Eval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_00686 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | RAVEL: Reasoning Agents for Validating and Evaluating LLM Text Synthesis Feng, Andrew Zhuoer Wang, Cunxiang Luo, Yu Wen, Bosi Wang, Yidong Fan, Lin Zhou, Yilin Wang, Zikang Yu, Wenbo Wu, Lindong Wang, Hongning Huang, Minlie Computation and Language Large Language Models have evolved from single-round generators into long-horizon agents, capable of complex text synthesis scenarios. However, current evaluation frameworks lack the ability to assess the actual synthesis operations, such as outlining, drafting, and editing. Consequently, they fail to evaluate the actual and detailed capabilities of LLMs. To bridge this gap, we introduce RAVEL, an agentic framework that enables the LLM testers to autonomously plan and execute typical synthesis operations, including outlining, drafting, reviewing, and refining. Complementing this framework, we present C3EBench, a comprehensive benchmark comprising 1,258 samples derived from professional human writings. We utilize a "reverse-engineering" pipeline to isolate specific capabilities across four tasks: Cloze, Edit, Expand, and End-to-End. Through our analysis of 14 LLMs, we uncover that most LLMs struggle with tasks that demand contextual understanding under limited or under-specified instructions. By augmenting RAVEL with SOTA LLMs as operators, we find that such agentic text synthesis is dominated by the LLM's reasoning capability rather than raw generative capacity. Furthermore, we find that a strong reasoner can guide a weaker generator to yield higher-quality results, whereas the inverse does not hold. Our code and data are available at this link: https://github.com/ZhuoerFeng/RAVEL-Reasoning-Agents-Text-Eval. |
| title | RAVEL: Reasoning Agents for Validating and Evaluating LLM Text Synthesis |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2603.00686 |