mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Model
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929203512868864 |
|---|---|
| author | Hu, Anwen Shi, Yaya Xu, Haiyang Ye, Jiabo Ye, Qinghao Yan, Ming Li, Chenliang Qian, Qi Zhang, Ji Huang, Fei |
| author_facet | Hu, Anwen Shi, Yaya Xu, Haiyang Ye, Jiabo Ye, Qinghao Yan, Ming Li, Chenliang Qian, Qi Zhang, Ji Huang, Fei |
| contents | Recently, the strong text creation ability of Large Language Models(LLMs) has given rise to many tools for assisting paper reading or even writing. However, the weak diagram analysis abilities of LLMs or Multimodal LLMs greatly limit their application scenarios, especially for scientific academic paper writing. In this work, towards a more versatile copilot for academic paper writing, we mainly focus on strengthening the multi-modal diagram analysis ability of Multimodal LLMs. By parsing Latex source files of high-quality papers, we carefully build a multi-modal diagram understanding dataset M-Paper. By aligning diagrams in the paper with related paragraphs, we construct professional diagram analysis samples for training and evaluation. M-Paper is the first dataset to support joint comprehension of multiple scientific diagrams, including figures and tables in the format of images or Latex codes. Besides, to better align the copilot with the user's intention, we introduce the `outline' as the control signal, which could be directly given by the user or revised based on auto-generated ones. Comprehensive experiments with a state-of-the-art Mumtimodal LLM demonstrate that training on our dataset shows stronger scientific diagram understanding performance, including diagram captioning, diagram analysis, and outline recommendation. The dataset, code, and model are available at https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/PaperOwl. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_18248 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Model Hu, Anwen Shi, Yaya Xu, Haiyang Ye, Jiabo Ye, Qinghao Yan, Ming Li, Chenliang Qian, Qi Zhang, Ji Huang, Fei Multimedia Computation and Language Recently, the strong text creation ability of Large Language Models(LLMs) has given rise to many tools for assisting paper reading or even writing. However, the weak diagram analysis abilities of LLMs or Multimodal LLMs greatly limit their application scenarios, especially for scientific academic paper writing. In this work, towards a more versatile copilot for academic paper writing, we mainly focus on strengthening the multi-modal diagram analysis ability of Multimodal LLMs. By parsing Latex source files of high-quality papers, we carefully build a multi-modal diagram understanding dataset M-Paper. By aligning diagrams in the paper with related paragraphs, we construct professional diagram analysis samples for training and evaluation. M-Paper is the first dataset to support joint comprehension of multiple scientific diagrams, including figures and tables in the format of images or Latex codes. Besides, to better align the copilot with the user's intention, we introduce the `outline' as the control signal, which could be directly given by the user or revised based on auto-generated ones. Comprehensive experiments with a state-of-the-art Mumtimodal LLM demonstrate that training on our dataset shows stronger scientific diagram understanding performance, including diagram captioning, diagram analysis, and outline recommendation. The dataset, code, and model are available at https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/PaperOwl. |
| title | mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Model |
| topic | Multimedia Computation and Language |
| url | https://arxiv.org/abs/2311.18248 |