Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.18818 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910234394492928 |
|---|---|
| author | Fehlis, Yao Bengfort, Benjamin Si, Zhangzhang Eyorokon, Vahid Roman, Prema Deziel, Patrick Slonaker, Devon Veldman, Steve Johnson, Ben Rigelo, Joyce Wharton, Michael Kramer, Steve |
| author_facet | Fehlis, Yao Bengfort, Benjamin Si, Zhangzhang Eyorokon, Vahid Roman, Prema Deziel, Patrick Slonaker, Devon Veldman, Steve Johnson, Ben Rigelo, Joyce Wharton, Michael Kramer, Steve |
| contents | Academic research tends to focus on new models for document understanding creating a wide gap in the literature between model definition and running models at production scale. To close that gap, we present a microservice architecture that encapsulates pipelines of multiple models for classification, optical character recognition (OCR), and large language model structured field extraction as well as our experience running this pipeline on thousands of multi-page documents per hour. We describe our primary design decisions, including a hybrid classification, separation of GPU-bound inference from CPU-bound orchestration, use of asynchronous processing for the many IO-bound operations in the pipeline, and an independent, horizontal scaling strategy. Using batch profiling, we identified two surprising qualitative findings that shape production deployments: OCR, not language-model parsing, dominates end-to-end latency, and the system saturates at a concurrency determined by shared GPU-inference capacity rather than worker count. Our goal is to provide practitioners with concrete architectural patterns for building document understanding systems that work beyond the benchmark; effectively operationalizing models in production. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_18818 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Operationalizing Document AI: A Microservice Architecture for OCR and LLM Pipelines in Production Fehlis, Yao Bengfort, Benjamin Si, Zhangzhang Eyorokon, Vahid Roman, Prema Deziel, Patrick Slonaker, Devon Veldman, Steve Johnson, Ben Rigelo, Joyce Wharton, Michael Kramer, Steve Artificial Intelligence Machine Learning Software Engineering Academic research tends to focus on new models for document understanding creating a wide gap in the literature between model definition and running models at production scale. To close that gap, we present a microservice architecture that encapsulates pipelines of multiple models for classification, optical character recognition (OCR), and large language model structured field extraction as well as our experience running this pipeline on thousands of multi-page documents per hour. We describe our primary design decisions, including a hybrid classification, separation of GPU-bound inference from CPU-bound orchestration, use of asynchronous processing for the many IO-bound operations in the pipeline, and an independent, horizontal scaling strategy. Using batch profiling, we identified two surprising qualitative findings that shape production deployments: OCR, not language-model parsing, dominates end-to-end latency, and the system saturates at a concurrency determined by shared GPU-inference capacity rather than worker count. Our goal is to provide practitioners with concrete architectural patterns for building document understanding systems that work beyond the benchmark; effectively operationalizing models in production. |
| title | Operationalizing Document AI: A Microservice Architecture for OCR and LLM Pipelines in Production |
| topic | Artificial Intelligence Machine Learning Software Engineering |
| url | https://arxiv.org/abs/2605.18818 |