ChemDFM-X: Towards Large Multimodal Model for Chemistry
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912174165721088 |
|---|---|
| author | Zhao, Zihan Chen, Bo Li, Jingpiao Chen, Lu Wen, Liyang Wang, Pengyu Zhu, Zichen Zhang, Danyang Wan, Ziping Li, Yansi Dai, Zhongyang Chen, Xin Yu, Kai |
| author_facet | Zhao, Zihan Chen, Bo Li, Jingpiao Chen, Lu Wen, Liyang Wang, Pengyu Zhu, Zichen Zhang, Danyang Wan, Ziping Li, Yansi Dai, Zhongyang Chen, Xin Yu, Kai |
| contents | Rapid developments of AI tools are expected to offer unprecedented assistance to the research of natural science including chemistry. However, neither existing unimodal task-specific specialist models nor emerging general large multimodal models (LMM) can cover the wide range of chemical data modality and task categories. To address the real demands of chemists, a cross-modal Chemical General Intelligence (CGI) system, which serves as a truly practical and useful research assistant utilizing the great potential of LMMs, is in great need. In this work, we introduce the first Cross-modal Dialogue Foundation Model for Chemistry (ChemDFM-X). Diverse multimodal data are generated from an initial modality by approximate calculations and task-specific model predictions. This strategy creates sufficient chemical training corpora, while significantly reducing excessive expense, resulting in an instruction-tuning dataset containing 7.6M data. After instruction finetuning, ChemDFM-X is evaluated on extensive experiments of different chemical tasks with various data modalities. The results demonstrate the capacity of ChemDFM-X for multimodal and inter-modal knowledge comprehension. ChemDFM-X marks a significant milestone toward aligning all modalities in chemistry, a step closer to CGI. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_13194 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ChemDFM-X: Towards Large Multimodal Model for Chemistry Zhao, Zihan Chen, Bo Li, Jingpiao Chen, Lu Wen, Liyang Wang, Pengyu Zhu, Zichen Zhang, Danyang Wan, Ziping Li, Yansi Dai, Zhongyang Chen, Xin Yu, Kai Machine Learning Computation and Language Multimedia Rapid developments of AI tools are expected to offer unprecedented assistance to the research of natural science including chemistry. However, neither existing unimodal task-specific specialist models nor emerging general large multimodal models (LMM) can cover the wide range of chemical data modality and task categories. To address the real demands of chemists, a cross-modal Chemical General Intelligence (CGI) system, which serves as a truly practical and useful research assistant utilizing the great potential of LMMs, is in great need. In this work, we introduce the first Cross-modal Dialogue Foundation Model for Chemistry (ChemDFM-X). Diverse multimodal data are generated from an initial modality by approximate calculations and task-specific model predictions. This strategy creates sufficient chemical training corpora, while significantly reducing excessive expense, resulting in an instruction-tuning dataset containing 7.6M data. After instruction finetuning, ChemDFM-X is evaluated on extensive experiments of different chemical tasks with various data modalities. The results demonstrate the capacity of ChemDFM-X for multimodal and inter-modal knowledge comprehension. ChemDFM-X marks a significant milestone toward aligning all modalities in chemistry, a step closer to CGI. |
| title | ChemDFM-X: Towards Large Multimodal Model for Chemistry |
| topic | Machine Learning Computation and Language Multimedia |
| url | https://arxiv.org/abs/2409.13194 |