A LLM Benchmark based on the Minecraft Builder Dialog Agent Task
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914875130773504 |
|---|---|
| author | Madge, Chris Poesio, Massimo |
| author_facet | Madge, Chris Poesio, Massimo |
| contents | In this work we proposing adapting the Minecraft builder task into an LLM benchmark suitable for evaluating LLM ability in spatially orientated tasks, and informing builder agent design. Previous works have proposed corpora with varying complex structures, and human written instructions. We instead attempt to provide a comprehensive synthetic benchmark for testing builder agents over a series of distinct tasks that comprise of common building operations. We believe this approach allows us to probe specific strengths and weaknesses of different agents, and test the ability of LLMs in the challenging area of spatial reasoning and vector based math. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_12734 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A LLM Benchmark based on the Minecraft Builder Dialog Agent Task Madge, Chris Poesio, Massimo Computation and Language In this work we proposing adapting the Minecraft builder task into an LLM benchmark suitable for evaluating LLM ability in spatially orientated tasks, and informing builder agent design. Previous works have proposed corpora with varying complex structures, and human written instructions. We instead attempt to provide a comprehensive synthetic benchmark for testing builder agents over a series of distinct tasks that comprise of common building operations. We believe this approach allows us to probe specific strengths and weaknesses of different agents, and test the ability of LLMs in the challenging area of spatial reasoning and vector based math. |
| title | A LLM Benchmark based on the Minecraft Builder Dialog Agent Task |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.12734 |