Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.07644 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914598416809984 |
|---|---|
| author | Rodionov, Fedor Eldesokey, Abdelrahman Birsak, Michael Femiani, John Ghanem, Bernard Wonka, Peter |
| author_facet | Rodionov, Fedor Eldesokey, Abdelrahman Birsak, Michael Femiani, John Ghanem, Bernard Wonka, Peter |
| contents | We introduce FloorplanQA, a diagnostic benchmark for evaluating spatial reasoning in large language models (LLMs). FloorplanQA is grounded in structured representations of indoor scenes, such as (e.g., kitchens, living rooms, bedrooms, bathrooms, and others), encoded symbolically in JSON or XML layouts. The benchmark covers core spatial tasks, including distance measurement, visibility, path finding, and object placement within constrained spaces. Our results across a variety of frontier open-source and commercial LLMs reveal that while models may succeed in shallow queries, they often fail to respect physical constraints, preserve spatial coherence, though they remain mostly robust to small spatial perturbations. FloorplanQA uncovers a blind spot in today's LLMs: inconsistent reasoning about indoor layouts. We hope this benchmark inspires new work on language models that can accurately infer and manipulate spatial and geometric properties in practical settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_07644 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FloorplanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representations Rodionov, Fedor Eldesokey, Abdelrahman Birsak, Michael Femiani, John Ghanem, Bernard Wonka, Peter Artificial Intelligence We introduce FloorplanQA, a diagnostic benchmark for evaluating spatial reasoning in large language models (LLMs). FloorplanQA is grounded in structured representations of indoor scenes, such as (e.g., kitchens, living rooms, bedrooms, bathrooms, and others), encoded symbolically in JSON or XML layouts. The benchmark covers core spatial tasks, including distance measurement, visibility, path finding, and object placement within constrained spaces. Our results across a variety of frontier open-source and commercial LLMs reveal that while models may succeed in shallow queries, they often fail to respect physical constraints, preserve spatial coherence, though they remain mostly robust to small spatial perturbations. FloorplanQA uncovers a blind spot in today's LLMs: inconsistent reasoning about indoor layouts. We hope this benchmark inspires new work on language models that can accurately infer and manipulate spatial and geometric properties in practical settings. |
| title | FloorplanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representations |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2507.07644 |