pdfQA: Diverse, Challenging, and Realistic Question Answering over PDFs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908750099513344 |
|---|---|
| author | Schimanski, Tobias Kolli, Imene Fan, Yu Vaghefi, Ario Saeid Ni, Jingwei Ash, Elliott Leippold, Markus |
| author_facet | Schimanski, Tobias Kolli, Imene Fan, Yu Vaghefi, Ario Saeid Ni, Jingwei Ash, Elliott Leippold, Markus |
| contents | PDFs are the second-most used document type on the internet (after HTML). Yet, existing QA datasets commonly start from text sources or only address specific domains. In this paper, we present pdfQA, a multi-domain 2K human-annotated (real-pdfQA) and 2K synthetic dataset (syn-pdfQA) differentiating QA pairs in ten complexity dimensions (e.g., file type, source modality, source position, answer type). We apply and evaluate quality and difficulty filters on both datasets, obtaining valid and challenging QA pairs. We answer the questions with open-source LLMs, revealing existing challenges that correlate with our complexity dimensions. pdfQA presents a basis for end-to-end QA pipeline evaluation, testing diverse skill sets and local optimizations (e.g., in information retrieval or parsing). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_02285 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | pdfQA: Diverse, Challenging, and Realistic Question Answering over PDFs Schimanski, Tobias Kolli, Imene Fan, Yu Vaghefi, Ario Saeid Ni, Jingwei Ash, Elliott Leippold, Markus Computation and Language Artificial Intelligence PDFs are the second-most used document type on the internet (after HTML). Yet, existing QA datasets commonly start from text sources or only address specific domains. In this paper, we present pdfQA, a multi-domain 2K human-annotated (real-pdfQA) and 2K synthetic dataset (syn-pdfQA) differentiating QA pairs in ten complexity dimensions (e.g., file type, source modality, source position, answer type). We apply and evaluate quality and difficulty filters on both datasets, obtaining valid and challenging QA pairs. We answer the questions with open-source LLMs, revealing existing challenges that correlate with our complexity dimensions. pdfQA presents a basis for end-to-end QA pipeline evaluation, testing diverse skill sets and local optimizations (e.g., in information retrieval or parsing). |
| title | pdfQA: Diverse, Challenging, and Realistic Question Answering over PDFs |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2601.02285 |