AdaParse: An Adaptive Parallel PDF Parsing and Resource Scaling Engine
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916717773455360 |
|---|---|
| author | Siebenschuh, Carlo Hippe, Kyle Gokdemir, Ozan Brace, Alexander Khan, Arham Hossain, Khalid Babuji, Yadu Chia, Nicholas Vishwanath, Venkatram Stevens, Rick Ramanathan, Arvind Foster, Ian Underwood, Robert |
| author_facet | Siebenschuh, Carlo Hippe, Kyle Gokdemir, Ozan Brace, Alexander Khan, Arham Hossain, Khalid Babuji, Yadu Chia, Nicholas Vishwanath, Venkatram Stevens, Rick Ramanathan, Arvind Foster, Ian Underwood, Robert |
| contents | Language models for scientific tasks are trained on text from scientific publications, most distributed as PDFs that require parsing. PDF parsing approaches range from inexpensive heuristics (for simple documents) to computationally intensive ML-driven systems (for complex or degraded ones). The choice of the "best" parser for a particular document depends on its computational cost and the accuracy of its output. To address these issues, we introduce an Adaptive Parallel PDF Parsing and Resource Scaling Engine (AdaParse), a data-driven strategy for assigning an appropriate parser to each document. We enlist scientists to select preferred parser outputs and incorporate this information through direct preference optimization (DPO) into AdaParse, thereby aligning its selection process with human judgment. AdaParse then incorporates hardware requirements and predicted accuracy of each parser to orchestrate computational resources efficiently for large-scale parsing campaigns. We demonstrate that AdaParse, when compared to state-of-the-art parsers, improves throughput by $17\times$ while still achieving comparable accuracy (0.2 percent better) on a benchmark set of 1000 scientific documents. AdaParse's combination of high accuracy and parallel scalability makes it feasible to parse large-scale scientific document corpora to support the development of high-quality, trillion-token-scale text datasets. The implementation is available at https://github.com/7shoe/AdaParse/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_01435 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AdaParse: An Adaptive Parallel PDF Parsing and Resource Scaling Engine Siebenschuh, Carlo Hippe, Kyle Gokdemir, Ozan Brace, Alexander Khan, Arham Hossain, Khalid Babuji, Yadu Chia, Nicholas Vishwanath, Venkatram Stevens, Rick Ramanathan, Arvind Foster, Ian Underwood, Robert Information Retrieval Computation and Language Distributed, Parallel, and Cluster Computing Machine Learning Language models for scientific tasks are trained on text from scientific publications, most distributed as PDFs that require parsing. PDF parsing approaches range from inexpensive heuristics (for simple documents) to computationally intensive ML-driven systems (for complex or degraded ones). The choice of the "best" parser for a particular document depends on its computational cost and the accuracy of its output. To address these issues, we introduce an Adaptive Parallel PDF Parsing and Resource Scaling Engine (AdaParse), a data-driven strategy for assigning an appropriate parser to each document. We enlist scientists to select preferred parser outputs and incorporate this information through direct preference optimization (DPO) into AdaParse, thereby aligning its selection process with human judgment. AdaParse then incorporates hardware requirements and predicted accuracy of each parser to orchestrate computational resources efficiently for large-scale parsing campaigns. We demonstrate that AdaParse, when compared to state-of-the-art parsers, improves throughput by $17\times$ while still achieving comparable accuracy (0.2 percent better) on a benchmark set of 1000 scientific documents. AdaParse's combination of high accuracy and parallel scalability makes it feasible to parse large-scale scientific document corpora to support the development of high-quality, trillion-token-scale text datasets. The implementation is available at https://github.com/7shoe/AdaParse/ |
| title | AdaParse: An Adaptive Parallel PDF Parsing and Resource Scaling Engine |
| topic | Information Retrieval Computation and Language Distributed, Parallel, and Cluster Computing Machine Learning |
| url | https://arxiv.org/abs/2505.01435 |