DocEdit-v2: Document Structure Editing Via Multimodal LLM Grounding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917812427030528 |
|---|---|
| author | Suri, Manan Mathur, Puneet Dernoncourt, Franck Jain, Rajiv Morariu, Vlad I Sawhney, Ramit Nakov, Preslav Manocha, Dinesh |
| author_facet | Suri, Manan Mathur, Puneet Dernoncourt, Franck Jain, Rajiv Morariu, Vlad I Sawhney, Ramit Nakov, Preslav Manocha, Dinesh |
| contents | Document structure editing involves manipulating localized textual, visual, and layout components in document images based on the user's requests. Past works have shown that multimodal grounding of user requests in the document image and identifying the accurate structural components and their associated attributes remain key challenges for this task. To address these, we introduce the DocEdit-v2, a novel framework that performs end-to-end document editing by leveraging Large Multimodal Models (LMMs). It consists of three novel components: (1) Doc2Command, which simultaneously localizes edit regions of interest (RoI) and disambiguates user edit requests into edit commands; (2) LLM-based Command Reformulation prompting to tailor edit commands originally intended for specialized software into edit instructions suitable for generalist LMMs. (3) Moreover, DocEdit-v2 processes these outputs via Large Multimodal Models like GPT-4V and Gemini, to parse the document layout, execute edits on grounded Region of Interest (RoI), and generate the edited document image. Extensive experiments on the DocEdit dataset show that DocEdit-v2 significantly outperforms strong baselines on edit command generation (2-33%), RoI bounding box detection (12-31%), and overall document editing (1-12\%) tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_16472 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DocEdit-v2: Document Structure Editing Via Multimodal LLM Grounding Suri, Manan Mathur, Puneet Dernoncourt, Franck Jain, Rajiv Morariu, Vlad I Sawhney, Ramit Nakov, Preslav Manocha, Dinesh Computation and Language Document structure editing involves manipulating localized textual, visual, and layout components in document images based on the user's requests. Past works have shown that multimodal grounding of user requests in the document image and identifying the accurate structural components and their associated attributes remain key challenges for this task. To address these, we introduce the DocEdit-v2, a novel framework that performs end-to-end document editing by leveraging Large Multimodal Models (LMMs). It consists of three novel components: (1) Doc2Command, which simultaneously localizes edit regions of interest (RoI) and disambiguates user edit requests into edit commands; (2) LLM-based Command Reformulation prompting to tailor edit commands originally intended for specialized software into edit instructions suitable for generalist LMMs. (3) Moreover, DocEdit-v2 processes these outputs via Large Multimodal Models like GPT-4V and Gemini, to parse the document layout, execute edits on grounded Region of Interest (RoI), and generate the edited document image. Extensive experiments on the DocEdit dataset show that DocEdit-v2 significantly outperforms strong baselines on edit command generation (2-33%), RoI bounding box detection (12-31%), and overall document editing (1-12\%) tasks. |
| title | DocEdit-v2: Document Structure Editing Via Multimodal LLM Grounding |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.16472 |