LLMStructBench: Benchmarking Large Language Model Structured Data Extraction
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911450593755136 |
|---|---|
| author | Tenckhoff, Sönke Koddenbrock, Mario Rodner, Erik |
| author_facet | Tenckhoff, Sönke Koddenbrock, Mario Rodner, Erik |
| contents | We present LLMStructBench, a novel benchmark for evaluating Large Language Models (LLMs) on extracting structured data and generating valid JavaScript Object Notation (JSON) outputs from natural-language text. Our open dataset comprises diverse, manually verified parsing scenarios of varying complexity and enables systematic testing across 22 models and five prompting strategies. We further introduce complementary performance metrics that capture both token-level accuracy and document-level validity, facilitating rigorous comparison of model, size, and prompting effects on parsing reliability.
In particular, we show that choosing the right prompting strategy is more important than standard attributes such as model size. This especially ensures structural validity for smaller or less reliable models but increase the number of semantic errors. Our benchmark suite is an step towards future research in the area of LLM applied to parsing or Extract, Transform and Load (ETL) applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_14743 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | LLMStructBench: Benchmarking Large Language Model Structured Data Extraction Tenckhoff, Sönke Koddenbrock, Mario Rodner, Erik Computation and Language Machine Learning E.2; H.1; H.3; H.4; I.2 We present LLMStructBench, a novel benchmark for evaluating Large Language Models (LLMs) on extracting structured data and generating valid JavaScript Object Notation (JSON) outputs from natural-language text. Our open dataset comprises diverse, manually verified parsing scenarios of varying complexity and enables systematic testing across 22 models and five prompting strategies. We further introduce complementary performance metrics that capture both token-level accuracy and document-level validity, facilitating rigorous comparison of model, size, and prompting effects on parsing reliability. In particular, we show that choosing the right prompting strategy is more important than standard attributes such as model size. This especially ensures structural validity for smaller or less reliable models but increase the number of semantic errors. Our benchmark suite is an step towards future research in the area of LLM applied to parsing or Extract, Transform and Load (ETL) applications. |
| title | LLMStructBench: Benchmarking Large Language Model Structured Data Extraction |
| topic | Computation and Language Machine Learning E.2; H.1; H.3; H.4; I.2 |
| url | https://arxiv.org/abs/2602.14743 |