StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917381130944512 |
|---|---|
| author | Yang, Jialin Jiang, Dongfu He, Lipeng Siu, Sherman Zhang, Yuxuan Liao, Disen Li, Zhuofeng Zeng, Huaye Jia, Yiming Wang, Haozhe Schneider, Benjamin Ruan, Chi Ma, Wentao Lyu, Zhiheng Wang, Yifei Lu, Yi Do, Quy Duc Jiang, Ziyan Nie, Ping Chen, Wenhu |
| author_facet | Yang, Jialin Jiang, Dongfu He, Lipeng Siu, Sherman Zhang, Yuxuan Liao, Disen Li, Zhuofeng Zeng, Huaye Jia, Yiming Wang, Haozhe Schneider, Benjamin Ruan, Chi Ma, Wentao Lyu, Zhiheng Wang, Yifei Lu, Yi Do, Quy Duc Jiang, Ziyan Nie, Ping Chen, Wenhu |
| contents | As Large Language Models (LLMs) become integral to software development workflows, their ability to generate structured outputs has become critically important. We introduce StructEval, a comprehensive benchmark for evaluating LLMs' capabilities in producing both non-renderable (JSON, YAML, CSV) and renderable (HTML, React, SVG) structured formats. Unlike prior benchmarks, StructEval systematically evaluates structural fidelity across diverse formats through two paradigms: 1) generation tasks, producing structured output from natural language prompts, and \textbf{2)} conversion tasks, translating between structured formats. Our benchmark encompasses 18 formats and 44 types of task, with novel metrics for format adherence and structural correctness. Results reveal significant performance gaps-even state-of-the-art models like o1-mini achieve only 75.58 average score, with open-source alternatives lagging approximately 10 points behind. We find generation tasks more challenging than conversion tasks, and producing correct visual content more difficult than generating text-only structures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_20139 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs Yang, Jialin Jiang, Dongfu He, Lipeng Siu, Sherman Zhang, Yuxuan Liao, Disen Li, Zhuofeng Zeng, Huaye Jia, Yiming Wang, Haozhe Schneider, Benjamin Ruan, Chi Ma, Wentao Lyu, Zhiheng Wang, Yifei Lu, Yi Do, Quy Duc Jiang, Ziyan Nie, Ping Chen, Wenhu Software Engineering Artificial Intelligence Computation and Language As Large Language Models (LLMs) become integral to software development workflows, their ability to generate structured outputs has become critically important. We introduce StructEval, a comprehensive benchmark for evaluating LLMs' capabilities in producing both non-renderable (JSON, YAML, CSV) and renderable (HTML, React, SVG) structured formats. Unlike prior benchmarks, StructEval systematically evaluates structural fidelity across diverse formats through two paradigms: 1) generation tasks, producing structured output from natural language prompts, and \textbf{2)} conversion tasks, translating between structured formats. Our benchmark encompasses 18 formats and 44 types of task, with novel metrics for format adherence and structural correctness. Results reveal significant performance gaps-even state-of-the-art models like o1-mini achieve only 75.58 average score, with open-source alternatives lagging approximately 10 points behind. We find generation tasks more challenging than conversion tasks, and producing correct visual content more difficult than generating text-only structures. |
| title | StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs |
| topic | Software Engineering Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2505.20139 |