ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation?
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929613044711424 |
|---|---|
| author | Zhang, Leixin Eger, Steffen Cheng, Yinjie Zhai, Weihe Belouadi, Jonas Leiter, Christoph Ponzetto, Simone Paolo Moafian, Fahimeh Zhao, Zhixue |
| author_facet | Zhang, Leixin Eger, Steffen Cheng, Yinjie Zhai, Weihe Belouadi, Jonas Leiter, Christoph Ponzetto, Simone Paolo Moafian, Fahimeh Zhao, Zhixue |
| contents | Multimodal large language models (LLMs) have demonstrated impressive capabilities in generating high-quality images from textual instructions. However, their performance in generating scientific images--a critical application for accelerating scientific progress--remains underexplored. In this work, we address this gap by introducing ScImage, a benchmark designed to evaluate the multimodal capabilities of LLMs in generating scientific images from textual descriptions. ScImage assesses three key dimensions of understanding: spatial, numeric, and attribute comprehension, as well as their combinations, focusing on the relationships between scientific objects (e.g., squares, circles). We evaluate five models, GPT-4o, Llama, AutomaTikZ, Dall-E, and StableDiffusion, using two modes of output generation: code-based outputs (Python, TikZ) and direct raster image generation. Additionally, we examine four different input languages: English, German, Farsi, and Chinese. Our evaluation, conducted with 11 scientists across three criteria (correctness, relevance, and scientific accuracy), reveals that while GPT-4o produces outputs of decent quality for simpler prompts involving individual dimensions such as spatial, numeric, or attribute understanding in isolation, all models face challenges in this task, especially for more complex prompts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_02368 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation? Zhang, Leixin Eger, Steffen Cheng, Yinjie Zhai, Weihe Belouadi, Jonas Leiter, Christoph Ponzetto, Simone Paolo Moafian, Fahimeh Zhao, Zhixue Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Multimodal large language models (LLMs) have demonstrated impressive capabilities in generating high-quality images from textual instructions. However, their performance in generating scientific images--a critical application for accelerating scientific progress--remains underexplored. In this work, we address this gap by introducing ScImage, a benchmark designed to evaluate the multimodal capabilities of LLMs in generating scientific images from textual descriptions. ScImage assesses three key dimensions of understanding: spatial, numeric, and attribute comprehension, as well as their combinations, focusing on the relationships between scientific objects (e.g., squares, circles). We evaluate five models, GPT-4o, Llama, AutomaTikZ, Dall-E, and StableDiffusion, using two modes of output generation: code-based outputs (Python, TikZ) and direct raster image generation. Additionally, we examine four different input languages: English, German, Farsi, and Chinese. Our evaluation, conducted with 11 scientists across three criteria (correctness, relevance, and scientific accuracy), reveals that while GPT-4o produces outputs of decent quality for simpler prompts involving individual dimensions such as spatial, numeric, or attribute understanding in isolation, all models face challenges in this task, especially for more complex prompts. |
| title | ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation? |
| topic | Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.02368 |