A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915292174614528 |
|---|---|
| author | Lu, Jinghui Yu, Haiyang Wang, Yanjie Ye, Yongjie Tang, Jingqun Yang, Ziwei Wu, Binghong Liu, Qi Feng, Hao Wang, Han Liu, Hao Huang, Can |
| author_facet | Lu, Jinghui Yu, Haiyang Wang, Yanjie Ye, Yongjie Tang, Jingqun Yang, Ziwei Wu, Binghong Liu, Qi Feng, Hao Wang, Han Liu, Hao Huang, Can |
| contents | Recently, many studies have demonstrated that exclusively incorporating OCR-derived text and spatial layouts with large language models (LLMs) can be highly effective for document understanding tasks. However, existing methods that integrate spatial layouts with text have limitations, such as producing overly long text sequences or failing to fully leverage the autoregressive traits of LLMs. In this work, we introduce Interleaving Layout and Text in a Large Language Model (LayTextLLM)} for document understanding. LayTextLLM projects each bounding box to a single embedding and interleaves it with text, efficiently avoiding long sequence issues while leveraging autoregressive traits of LLMs. LayTextLLM not only streamlines the interaction of layout and textual data but also shows enhanced performance in KIE and VQA. Comprehensive benchmark evaluations reveal significant improvements of LayTextLLM, with a 15.2% increase on KIE tasks and 10.7% on VQA tasks compared to previous SOTA OCR-based LLMs. All resources are available at https://github.com/LayTextLLM/LayTextLLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_01976 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding Lu, Jinghui Yu, Haiyang Wang, Yanjie Ye, Yongjie Tang, Jingqun Yang, Ziwei Wu, Binghong Liu, Qi Feng, Hao Wang, Han Liu, Hao Huang, Can Computation and Language Artificial Intelligence Multimedia Recently, many studies have demonstrated that exclusively incorporating OCR-derived text and spatial layouts with large language models (LLMs) can be highly effective for document understanding tasks. However, existing methods that integrate spatial layouts with text have limitations, such as producing overly long text sequences or failing to fully leverage the autoregressive traits of LLMs. In this work, we introduce Interleaving Layout and Text in a Large Language Model (LayTextLLM)} for document understanding. LayTextLLM projects each bounding box to a single embedding and interleaves it with text, efficiently avoiding long sequence issues while leveraging autoregressive traits of LLMs. LayTextLLM not only streamlines the interaction of layout and textual data but also shows enhanced performance in KIE and VQA. Comprehensive benchmark evaluations reveal significant improvements of LayTextLLM, with a 15.2% increase on KIE tasks and 10.7% on VQA tasks compared to previous SOTA OCR-based LLMs. All resources are available at https://github.com/LayTextLLM/LayTextLLM. |
| title | A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding |
| topic | Computation and Language Artificial Intelligence Multimedia |
| url | https://arxiv.org/abs/2407.01976 |