Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915586102001664 |
|---|---|
| author | Zhong, Tianyun Mo, Guozhao Liu, Yanjiang Chen, Yihan Kong, Lingdi Chen, Xuanang Lu, Yaojie Lin, Hongyu Ye, Shiwei Han, Xianpei He, Ben Sun, Le |
| author_facet | Zhong, Tianyun Mo, Guozhao Liu, Yanjiang Chen, Yihan Kong, Lingdi Chen, Xuanang Lu, Yaojie Lin, Hongyu Ye, Shiwei Han, Xianpei He, Ben Sun, Le |
| contents | With the emergence of large language models (LLMs), there is an expectation that LLMs can effectively extract explicit information from complex real-world documents (e.g., papers, reports). However, most LLMs generate paragraph-style answers that are chaotic, disorganized, and untraceable. To bridge this gap, we introduce the Arranged and Organized Extraction Benchmark (AOE), a new bilingual benchmark with data and documents of varying lengths designed to systematically evaluate the ability of LLMs to comprehend fragmented documents and reconstruct isolated information into one organized table. Unlike conventional text-to-table tasks, which rely on fixed schema and narrow task domains, AOE includes 11 carefully crafted tasks across three diverse domains, requiring models to generate context-specific schema tailored to varied input queries. In the experiment, we evaluated both open-source and closed-source state-of-the-art LLMs. The results show that even the most advanced models struggled significantly. The benchmark is available at https://anonymous.4open.science/r/AOE-Benchmark/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_16271 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction Zhong, Tianyun Mo, Guozhao Liu, Yanjiang Chen, Yihan Kong, Lingdi Chen, Xuanang Lu, Yaojie Lin, Hongyu Ye, Shiwei Han, Xianpei He, Ben Sun, Le Computation and Language With the emergence of large language models (LLMs), there is an expectation that LLMs can effectively extract explicit information from complex real-world documents (e.g., papers, reports). However, most LLMs generate paragraph-style answers that are chaotic, disorganized, and untraceable. To bridge this gap, we introduce the Arranged and Organized Extraction Benchmark (AOE), a new bilingual benchmark with data and documents of varying lengths designed to systematically evaluate the ability of LLMs to comprehend fragmented documents and reconstruct isolated information into one organized table. Unlike conventional text-to-table tasks, which rely on fixed schema and narrow task domains, AOE includes 11 carefully crafted tasks across three diverse domains, requiring models to generate context-specific schema tailored to varied input queries. In the experiment, we evaluated both open-source and closed-source state-of-the-art LLMs. The results show that even the most advanced models struggled significantly. The benchmark is available at https://anonymous.4open.science/r/AOE-Benchmark/. |
| title | Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2507.16271 |