OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909550846672896 |
|---|---|
| author | Ouyang, Linke Qu, Yuan Zhou, Hongbin Zhu, Jiawei Zhang, Rui Lin, Qunshu Wang, Bin Zhao, Zhiyuan Jiang, Man Zhao, Xiaomeng Shi, Jin Wu, Fan Chu, Pei Liu, Minghao Li, Zhenxiang Xu, Chao Zhang, Bo Shi, Botian Tu, Zhongying He, Conghui |
| author_facet | Ouyang, Linke Qu, Yuan Zhou, Hongbin Zhu, Jiawei Zhang, Rui Lin, Qunshu Wang, Bin Zhao, Zhiyuan Jiang, Man Zhao, Xiaomeng Shi, Jin Wu, Fan Chu, Pei Liu, Minghao Li, Zhenxiang Xu, Chao Zhang, Bo Shi, Botian Tu, Zhongying He, Conghui |
| contents | Document content extraction is a critical task in computer vision, underpinning the data needs of large language models (LLMs) and retrieval-augmented generation (RAG) systems. Despite recent progress, current document parsing methods have not been fairly and comprehensively evaluated due to the narrow coverage of document types and the simplified, unrealistic evaluation procedures in existing benchmarks. To address these gaps, we introduce OmniDocBench, a novel benchmark featuring high-quality annotations across nine document sources, including academic papers, textbooks, and more challenging cases such as handwritten notes and densely typeset newspapers. OmniDocBench supports flexible, multi-level evaluations--ranging from an end-to-end assessment to the task-specific and attribute--based analysis using 19 layout categories and 15 attribute labels. We conduct a thorough evaluation of both pipeline-based methods and end-to-end vision-language models, revealing their strengths and weaknesses across different document types. OmniDocBench sets a new standard for the fair, diverse, and fine-grained evaluation in document parsing. Dataset and code are available at https://github.com/opendatalab/OmniDocBench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_07626 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations Ouyang, Linke Qu, Yuan Zhou, Hongbin Zhu, Jiawei Zhang, Rui Lin, Qunshu Wang, Bin Zhao, Zhiyuan Jiang, Man Zhao, Xiaomeng Shi, Jin Wu, Fan Chu, Pei Liu, Minghao Li, Zhenxiang Xu, Chao Zhang, Bo Shi, Botian Tu, Zhongying He, Conghui Computer Vision and Pattern Recognition Artificial Intelligence Information Retrieval Document content extraction is a critical task in computer vision, underpinning the data needs of large language models (LLMs) and retrieval-augmented generation (RAG) systems. Despite recent progress, current document parsing methods have not been fairly and comprehensively evaluated due to the narrow coverage of document types and the simplified, unrealistic evaluation procedures in existing benchmarks. To address these gaps, we introduce OmniDocBench, a novel benchmark featuring high-quality annotations across nine document sources, including academic papers, textbooks, and more challenging cases such as handwritten notes and densely typeset newspapers. OmniDocBench supports flexible, multi-level evaluations--ranging from an end-to-end assessment to the task-specific and attribute--based analysis using 19 layout categories and 15 attribute labels. We conduct a thorough evaluation of both pipeline-based methods and end-to-end vision-language models, revealing their strengths and weaknesses across different document types. OmniDocBench sets a new standard for the fair, diverse, and fine-grained evaluation in document parsing. Dataset and code are available at https://github.com/opendatalab/OmniDocBench. |
| title | OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Information Retrieval |
| url | https://arxiv.org/abs/2412.07626 |