RxnCaption: Reformulating Reaction Diagram Parsing as Visual Prompt Guided Captioning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909981974986752 |
|---|---|
| author | Song, Jiahe Wang, Chuang Jiang, Bowen Wang, Yinfan Zheng, Hao Wei, Xingjian Liu, Chengjin Nie, Rui Gao, Junyuan Sun, Jiaxing Wang, Yubin Wu, Lijun Huang, Zhenhua Wu, Jiang Yu, Qian He, Conghui |
| author_facet | Song, Jiahe Wang, Chuang Jiang, Bowen Wang, Yinfan Zheng, Hao Wei, Xingjian Liu, Chengjin Nie, Rui Gao, Junyuan Sun, Jiaxing Wang, Yubin Wu, Lijun Huang, Zhenhua Wu, Jiang Yu, Qian He, Conghui |
| contents | Large-scale chemical reaction datasets are crucial for AI research in chemistry. However, existing chemical reaction data often exist as images within papers, making them not machine-readable and unusable for training machine learning models. In response to this challenge, we propose the RxnCaption framework for the task of chemical Reaction Diagram Parsing (RxnDP). Our framework reformulates the traditional coordinate prediction driven parsing process into an image captioning problem, which Large Vision Language Models (LVLMs) handle naturally. We introduce a strategy termed BBox and Index as Visual Prompt (BIVP), which uses our state-of-the-art molecular detector, MolYOLO, to pre-draw molecular bounding boxes and indices directly onto the input image. This turns the downstream parsing into a natural-language description problem. Extensive experiments show that the BIVP strategy significantly improves structural extraction quality while simplifying model design. We further construct the RxnCaption-15k dataset, an order of magnitude larger than prior real-world literature benchmarks, with a balanced test subset across four layout archetypes. Experiments demonstrate that RxnCaption-VL achieves state-of-the-art performance on multiple metrics. We believe our method, dataset, and models will advance structured information extraction from chemical literature and catalyze broader AI applications in chemistry. We will release data, models, and code on GitHub. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_02384 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RxnCaption: Reformulating Reaction Diagram Parsing as Visual Prompt Guided Captioning Song, Jiahe Wang, Chuang Jiang, Bowen Wang, Yinfan Zheng, Hao Wei, Xingjian Liu, Chengjin Nie, Rui Gao, Junyuan Sun, Jiaxing Wang, Yubin Wu, Lijun Huang, Zhenhua Wu, Jiang Yu, Qian He, Conghui Computer Vision and Pattern Recognition Large-scale chemical reaction datasets are crucial for AI research in chemistry. However, existing chemical reaction data often exist as images within papers, making them not machine-readable and unusable for training machine learning models. In response to this challenge, we propose the RxnCaption framework for the task of chemical Reaction Diagram Parsing (RxnDP). Our framework reformulates the traditional coordinate prediction driven parsing process into an image captioning problem, which Large Vision Language Models (LVLMs) handle naturally. We introduce a strategy termed BBox and Index as Visual Prompt (BIVP), which uses our state-of-the-art molecular detector, MolYOLO, to pre-draw molecular bounding boxes and indices directly onto the input image. This turns the downstream parsing into a natural-language description problem. Extensive experiments show that the BIVP strategy significantly improves structural extraction quality while simplifying model design. We further construct the RxnCaption-15k dataset, an order of magnitude larger than prior real-world literature benchmarks, with a balanced test subset across four layout archetypes. Experiments demonstrate that RxnCaption-VL achieves state-of-the-art performance on multiple metrics. We believe our method, dataset, and models will advance structured information extraction from chemical literature and catalyze broader AI applications in chemistry. We will release data, models, and code on GitHub. |
| title | RxnCaption: Reformulating Reaction Diagram Parsing as Visual Prompt Guided Captioning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2511.02384 |