AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materials
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918529269235712 |
|---|---|
| author | Lv, Taoyuze Chen, Alexander Xie, Fengyu Wu, Chu Meng, Jeffrey Zhou, Dongzhan Wang, Yingheng Hoex, Bram Zhong, Zhicheng Xie, Tong |
| author_facet | Lv, Taoyuze Chen, Alexander Xie, Fengyu Wu, Chu Meng, Jeffrey Zhou, Dongzhan Wang, Yingheng Hoex, Bram Zhong, Zhicheng Xie, Tong |
| contents | Large language models (LLMs) have shown promising potential in scientific research, enabling tasks ranging from knowledge retrieval to property prediction. Existing science benchmarks mainly focus on perceptual or knowledge-based tasks, largely ignoring the modelling tasks, a fundamental starting point for any real scientific research. For materials science, constructing and manipulating atomic structures is one of the most creative and least automated steps. In this work, we introduce AtomWorld, a benchmark designed to evaluate the abilities of LLMs on structure modifications. The benchmark includes ten fundamental actions under four widely used modelling categories, enabling verifiable evaluation metrics. We find that Claude Opus 4.6 generally performs the best. While the success rate decreases markedly with increasing modelling complexity, with particularly low success rates (below 12\% for rotation) for operations involving complex spatial relations. Our results suggest that contemporary LLMs are better suited as copilots for materials structure modelling rather than fully unsupervised autonomous scientific agents. Beyond evaluation, AtomWorld also serves as a testbed and playground for developing future structure-aware models, including reinforcement learning and agentic approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_04704 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materials Lv, Taoyuze Chen, Alexander Xie, Fengyu Wu, Chu Meng, Jeffrey Zhou, Dongzhan Wang, Yingheng Hoex, Bram Zhong, Zhicheng Xie, Tong Materials Science Artificial Intelligence Computation and Language Large language models (LLMs) have shown promising potential in scientific research, enabling tasks ranging from knowledge retrieval to property prediction. Existing science benchmarks mainly focus on perceptual or knowledge-based tasks, largely ignoring the modelling tasks, a fundamental starting point for any real scientific research. For materials science, constructing and manipulating atomic structures is one of the most creative and least automated steps. In this work, we introduce AtomWorld, a benchmark designed to evaluate the abilities of LLMs on structure modifications. The benchmark includes ten fundamental actions under four widely used modelling categories, enabling verifiable evaluation metrics. We find that Claude Opus 4.6 generally performs the best. While the success rate decreases markedly with increasing modelling complexity, with particularly low success rates (below 12\% for rotation) for operations involving complex spatial relations. Our results suggest that contemporary LLMs are better suited as copilots for materials structure modelling rather than fully unsupervised autonomous scientific agents. Beyond evaluation, AtomWorld also serves as a testbed and playground for developing future structure-aware models, including reinforcement learning and agentic approaches. |
| title | AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materials |
| topic | Materials Science Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2510.04704 |