Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.29199 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911556432822272 |
|---|---|
| author | Mankodiya, Harsh Gallik, Chase Galanos, Theodoros Mulyar, Andriy |
| author_facet | Mankodiya, Harsh Gallik, Chase Galanos, Theodoros Mulyar, Andriy |
| contents | The AEC-Bench is a multimodal benchmark for evaluating agentic systems on real-world tasks in the Architecture, Engineering, and Construction (AEC) domain. The benchmark covers tasks requiring drawing understanding, cross-sheet reasoning, and construction project-level coordination. This report describes the benchmark motivation, dataset taxonomy, evaluation protocol, and baseline results across several domain-specific foundation model harnesses. We use AEC-Bench to identify consistent tools and harness design techniques that uniformly improve performance across foundation models in their own base harnesses, such as Claude Code and Codex. We openly release our benchmark dataset, agent harness, and evaluation code for full replicability at https://github.com/nomic-ai/aec-bench under an Apache 2 license. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_29199 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AEC-Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Construction Mankodiya, Harsh Gallik, Chase Galanos, Theodoros Mulyar, Andriy Artificial Intelligence The AEC-Bench is a multimodal benchmark for evaluating agentic systems on real-world tasks in the Architecture, Engineering, and Construction (AEC) domain. The benchmark covers tasks requiring drawing understanding, cross-sheet reasoning, and construction project-level coordination. This report describes the benchmark motivation, dataset taxonomy, evaluation protocol, and baseline results across several domain-specific foundation model harnesses. We use AEC-Bench to identify consistent tools and harness design techniques that uniformly improve performance across foundation models in their own base harnesses, such as Claude Code and Codex. We openly release our benchmark dataset, agent harness, and evaluation code for full replicability at https://github.com/nomic-ai/aec-bench under an Apache 2 license. |
| title | AEC-Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Construction |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2603.29199 |