3MVRD: Multimodal Multi-task Multi-teacher Visually-Rich Form Document Understanding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917733741887488 |
|---|---|
| author | Ding, Yihao Vaiani, Lorenzo Han, Caren Lee, Jean Garza, Paolo Poon, Josiah Cagliero, Luca |
| author_facet | Ding, Yihao Vaiani, Lorenzo Han, Caren Lee, Jean Garza, Paolo Poon, Josiah Cagliero, Luca |
| contents | This paper presents a groundbreaking multimodal, multi-task, multi-teacher joint-grained knowledge distillation model for visually-rich form document understanding. The model is designed to leverage insights from both fine-grained and coarse-grained levels by facilitating a nuanced correlation between token and entity representations, addressing the complexities inherent in form documents. Additionally, we introduce new inter-grained and cross-grained loss functions to further refine diverse multi-teacher knowledge distillation transfer process, presenting distribution gaps and a harmonised understanding of form documents. Through a comprehensive evaluation across publicly available form document understanding datasets, our proposed model consistently outperforms existing baselines, showcasing its efficacy in handling the intricate structures and content of visually complex form documents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_17983 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | 3MVRD: Multimodal Multi-task Multi-teacher Visually-Rich Form Document Understanding Ding, Yihao Vaiani, Lorenzo Han, Caren Lee, Jean Garza, Paolo Poon, Josiah Cagliero, Luca Computation and Language Computer Vision and Pattern Recognition This paper presents a groundbreaking multimodal, multi-task, multi-teacher joint-grained knowledge distillation model for visually-rich form document understanding. The model is designed to leverage insights from both fine-grained and coarse-grained levels by facilitating a nuanced correlation between token and entity representations, addressing the complexities inherent in form documents. Additionally, we introduce new inter-grained and cross-grained loss functions to further refine diverse multi-teacher knowledge distillation transfer process, presenting distribution gaps and a harmonised understanding of form documents. Through a comprehensive evaluation across publicly available form document understanding datasets, our proposed model consistently outperforms existing baselines, showcasing its efficacy in handling the intricate structures and content of visually complex form documents. |
| title | 3MVRD: Multimodal Multi-task Multi-teacher Visually-Rich Form Document Understanding |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2402.17983 |