Multimodal OCR: Parse Anything from Documents
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912973476331520 |
|---|---|
| author | Zheng, Handong Li, Yumeng Zhang, Kaile Xin, Liang Zhao, Guangwei Liu, Hao Chen, Jiayu Lou, Jie Fu, Qi Yang, Rui Jiang, Shuo Luo, Weijian Su, Weijie Zhang, Weijun Zhu, Xingyu Li, Yabin ma, Yiwei Chen, Yu Ji, Yuqiu Yu, Zhaohui Yang, Guang Zhang, Colin Zhang, Lei Liu, Yuliang Bai, Xiang |
| author_facet | Zheng, Handong Li, Yumeng Zhang, Kaile Xin, Liang Zhao, Guangwei Liu, Hao Chen, Jiayu Lou, Jie Fu, Qi Yang, Rui Jiang, Shuo Luo, Weijian Su, Weijie Zhang, Weijun Zhu, Xingyu Li, Yabin ma, Yiwei Chen, Yu Ji, Yuqiu Yu, Zhaohui Yang, Guang Zhang, Colin Zhang, Lei Liu, Yuliang Bai, Xiang |
| contents | We present Multimodal OCR (MOCR), a document parsing paradigm that jointly parses text and graphics into unified textual representations. Unlike conventional OCR systems that focus on text recognition and leave graphical regions as cropped pixels, our method, termed dots.mocr, treats visual elements such as charts, diagrams, tables, and icons as first-class parsing targets, enabling systems to parse documents while preserving semantic relationships across elements. It offers several advantages: (1) it reconstructs both text and graphics as structured outputs, enabling more faithful document reconstruction; (2) it supports end-to-end training over heterogeneous document elements, allowing models to exploit semantic relations between textual and visual components; and (3) it converts previously discarded graphics into reusable code-level supervision, unlocking multimodal supervision embedded in existing documents. To make this paradigm practical at scale, we build a comprehensive data engine from PDFs, rendered webpages, and native SVG assets, and train a compact 3B-parameter model through staged pretraining and supervised fine-tuning. We evaluate dots.mocr from two perspectives: document parsing and structured graphics parsing. On document parsing benchmarks, it ranks second only to Gemini 3 Pro on our OCR Arena Elo leaderboard, surpasses existing open-source document parsing systems, and sets a new state of the art of 83.9 on olmOCR Bench. On structured graphics parsing, our model achieves higher reconstruction quality than Gemini 3 Pro across image-to-SVG benchmarks, demonstrating strong performance on charts, UI layouts, scientific figures, and chemical diagrams. These results show a scalable path toward building large-scale image-to-code corpora for multimodal pretraining. Code and models are publicly available at https://github.com/rednote-hilab/dots.mocr. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_13032 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Multimodal OCR: Parse Anything from Documents Zheng, Handong Li, Yumeng Zhang, Kaile Xin, Liang Zhao, Guangwei Liu, Hao Chen, Jiayu Lou, Jie Fu, Qi Yang, Rui Jiang, Shuo Luo, Weijian Su, Weijie Zhang, Weijun Zhu, Xingyu Li, Yabin ma, Yiwei Chen, Yu Ji, Yuqiu Yu, Zhaohui Yang, Guang Zhang, Colin Zhang, Lei Liu, Yuliang Bai, Xiang Computer Vision and Pattern Recognition We present Multimodal OCR (MOCR), a document parsing paradigm that jointly parses text and graphics into unified textual representations. Unlike conventional OCR systems that focus on text recognition and leave graphical regions as cropped pixels, our method, termed dots.mocr, treats visual elements such as charts, diagrams, tables, and icons as first-class parsing targets, enabling systems to parse documents while preserving semantic relationships across elements. It offers several advantages: (1) it reconstructs both text and graphics as structured outputs, enabling more faithful document reconstruction; (2) it supports end-to-end training over heterogeneous document elements, allowing models to exploit semantic relations between textual and visual components; and (3) it converts previously discarded graphics into reusable code-level supervision, unlocking multimodal supervision embedded in existing documents. To make this paradigm practical at scale, we build a comprehensive data engine from PDFs, rendered webpages, and native SVG assets, and train a compact 3B-parameter model through staged pretraining and supervised fine-tuning. We evaluate dots.mocr from two perspectives: document parsing and structured graphics parsing. On document parsing benchmarks, it ranks second only to Gemini 3 Pro on our OCR Arena Elo leaderboard, surpasses existing open-source document parsing systems, and sets a new state of the art of 83.9 on olmOCR Bench. On structured graphics parsing, our model achieves higher reconstruction quality than Gemini 3 Pro across image-to-SVG benchmarks, demonstrating strong performance on charts, UI layouts, scientific figures, and chemical diagrams. These results show a scalable path toward building large-scale image-to-code corpora for multimodal pretraining. Code and models are publicly available at https://github.com/rednote-hilab/dots.mocr. |
| title | Multimodal OCR: Parse Anything from Documents |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.13032 |