Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.08719 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918089230123008 |
|---|---|
| author | Chai, Linzheng Yang, Jian Liu, Shukai Zhang, Wei Wang, Liran Jin, Ke Sun, Tao Liu, Congnan Zhang, Chenchen Zhu, Hualei Liu, Jiaheng Wu, Xianjie Zhang, Ge Liu, Tianyu Li, Zhoujun |
| author_facet | Chai, Linzheng Yang, Jian Liu, Shukai Zhang, Wei Wang, Liran Jin, Ke Sun, Tao Liu, Congnan Zhang, Chenchen Zhu, Hualei Liu, Jiaheng Wu, Xianjie Zhang, Ge Liu, Tianyu Li, Zhoujun |
| contents | The rapid advancement of Large Language Models (LLMs) has significantly improved code generation, yet most models remain text-only, neglecting crucial visual aids like diagrams and flowcharts used in real-world software development. To bridge this gap, we introduce MM-Coder, a Multilingual Multimodal software developer. MM-Coder integrates visual design inputs-Unified Modeling Language (UML) diagrams and flowcharts (termed Visual Workflow)-with textual instructions to enhance code generation accuracy and architectural alignment. To enable this, we developed MMc-Instruct, a diverse multimodal instruction-tuning dataset including visual-workflow-based code generation, allowing MM-Coder to synthesize textual and graphical information like human developers, distinct from prior work on narrow tasks. Furthermore, we introduce MMEval, a new benchmark for evaluating multimodal code generation, addressing existing text-only limitations. Our evaluations using MMEval highlight significant remaining challenges for models in precise visual information capture, instruction following, and advanced programming knowledge. Our work aims to revolutionize industrial programming by enabling LLMs to interpret and implement complex specifications conveyed through both text and visual designs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_08719 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Multilingual Multimodal Software Developer for Code Generation Chai, Linzheng Yang, Jian Liu, Shukai Zhang, Wei Wang, Liran Jin, Ke Sun, Tao Liu, Congnan Zhang, Chenchen Zhu, Hualei Liu, Jiaheng Wu, Xianjie Zhang, Ge Liu, Tianyu Li, Zhoujun Computation and Language Artificial Intelligence Software Engineering The rapid advancement of Large Language Models (LLMs) has significantly improved code generation, yet most models remain text-only, neglecting crucial visual aids like diagrams and flowcharts used in real-world software development. To bridge this gap, we introduce MM-Coder, a Multilingual Multimodal software developer. MM-Coder integrates visual design inputs-Unified Modeling Language (UML) diagrams and flowcharts (termed Visual Workflow)-with textual instructions to enhance code generation accuracy and architectural alignment. To enable this, we developed MMc-Instruct, a diverse multimodal instruction-tuning dataset including visual-workflow-based code generation, allowing MM-Coder to synthesize textual and graphical information like human developers, distinct from prior work on narrow tasks. Furthermore, we introduce MMEval, a new benchmark for evaluating multimodal code generation, addressing existing text-only limitations. Our evaluations using MMEval highlight significant remaining challenges for models in precise visual information capture, instruction following, and advanced programming knowledge. Our work aims to revolutionize industrial programming by enabling LLMs to interpret and implement complex specifications conveyed through both text and visual designs. |
| title | Multilingual Multimodal Software Developer for Code Generation |
| topic | Computation and Language Artificial Intelligence Software Engineering |
| url | https://arxiv.org/abs/2507.08719 |