DocLLM: A layout-aware generative language model for multimodal document understanding
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909059018391552 |
|---|---|
| author | Wang, Dongsheng Raman, Natraj Sibue, Mathieu Ma, Zhiqiang Babkin, Petr Kaur, Simerjot Pei, Yulong Nourbakhsh, Armineh Liu, Xiaomo |
| author_facet | Wang, Dongsheng Raman, Natraj Sibue, Mathieu Ma, Zhiqiang Babkin, Petr Kaur, Simerjot Pei, Yulong Nourbakhsh, Armineh Liu, Xiaomo |
| contents | Enterprise documents such as forms, invoices, receipts, reports, contracts, and other similar records, often carry rich semantics at the intersection of textual and spatial modalities. The visual cues offered by their complex layouts play a crucial role in comprehending these documents effectively. In this paper, we present DocLLM, a lightweight extension to traditional large language models (LLMs) for reasoning over visual documents, taking into account both textual semantics and spatial layout. Our model differs from existing multimodal LLMs by avoiding expensive image encoders and focuses exclusively on bounding box information to incorporate the spatial layout structure. Specifically, the cross-alignment between text and spatial modalities is captured by decomposing the attention mechanism in classical transformers to a set of disentangled matrices. Furthermore, we devise a pre-training objective that learns to infill text segments. This approach allows us to address irregular layouts and heterogeneous content frequently encountered in visual documents. The pre-trained model is fine-tuned using a large-scale instruction dataset, covering four core document intelligence tasks. We demonstrate that our solution outperforms SotA LLMs on 14 out of 16 datasets across all tasks, and generalizes well to 4 out of 5 previously unseen datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_00908 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | DocLLM: A layout-aware generative language model for multimodal document understanding Wang, Dongsheng Raman, Natraj Sibue, Mathieu Ma, Zhiqiang Babkin, Petr Kaur, Simerjot Pei, Yulong Nourbakhsh, Armineh Liu, Xiaomo Computation and Language Enterprise documents such as forms, invoices, receipts, reports, contracts, and other similar records, often carry rich semantics at the intersection of textual and spatial modalities. The visual cues offered by their complex layouts play a crucial role in comprehending these documents effectively. In this paper, we present DocLLM, a lightweight extension to traditional large language models (LLMs) for reasoning over visual documents, taking into account both textual semantics and spatial layout. Our model differs from existing multimodal LLMs by avoiding expensive image encoders and focuses exclusively on bounding box information to incorporate the spatial layout structure. Specifically, the cross-alignment between text and spatial modalities is captured by decomposing the attention mechanism in classical transformers to a set of disentangled matrices. Furthermore, we devise a pre-training objective that learns to infill text segments. This approach allows us to address irregular layouts and heterogeneous content frequently encountered in visual documents. The pre-trained model is fine-tuned using a large-scale instruction dataset, covering four core document intelligence tasks. We demonstrate that our solution outperforms SotA LLMs on 14 out of 16 datasets across all tasks, and generalizes well to 4 out of 5 previously unseen datasets. |
| title | DocLLM: A layout-aware generative language model for multimodal document understanding |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2401.00908 |