Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.09503 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918289141137408 |
|---|---|
| author | Liu, Siyuan Yuan, Hongbang Li, Xinze Zhu, Ziyue Cao, Yixin Jiang, Yu-Gang |
| author_facet | Liu, Siyuan Yuan, Hongbang Li, Xinze Zhu, Ziyue Cao, Yixin Jiang, Yu-Gang |
| contents | Large language model (LLM) agents have demonstrated remarkable capabilities in complex decision-making and tool-use tasks, yet their ability to generalize across varying environments remains a under-examined concern. Current evaluation paradigms predominantly rely on trajectory-based metrics that measure task success, while failing to assess whether agents possess a grounded, transferable model of the environment. To address this gap, we propose Task-to-Quiz (T2Q), a deterministic and automated evaluation paradigm designed to decouple task execution from world-state understanding. We instantiate this paradigm in T2QBench, a suite comprising 30 environments and 1,967 grounded QA pairs across multiple difficulty levels. Our extensive experiments reveal that task success is often a poor proxy for environment understanding, and that current memory machanism can not effectively help agents acquire a grounded model of the environment. These findings identify proactive exploration and fine-grained state representation as primary bottlenecks, offering a robust foundation for developing more generalizable autonomous agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_09503 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | What Do LLM Agents Know About Their World? Task2Quiz: A Paradigm for Studying Environment Understanding Liu, Siyuan Yuan, Hongbang Li, Xinze Zhu, Ziyue Cao, Yixin Jiang, Yu-Gang Artificial Intelligence Large language model (LLM) agents have demonstrated remarkable capabilities in complex decision-making and tool-use tasks, yet their ability to generalize across varying environments remains a under-examined concern. Current evaluation paradigms predominantly rely on trajectory-based metrics that measure task success, while failing to assess whether agents possess a grounded, transferable model of the environment. To address this gap, we propose Task-to-Quiz (T2Q), a deterministic and automated evaluation paradigm designed to decouple task execution from world-state understanding. We instantiate this paradigm in T2QBench, a suite comprising 30 environments and 1,967 grounded QA pairs across multiple difficulty levels. Our extensive experiments reveal that task success is often a poor proxy for environment understanding, and that current memory machanism can not effectively help agents acquire a grounded model of the environment. These findings identify proactive exploration and fine-grained state representation as primary bottlenecks, offering a robust foundation for developing more generalizable autonomous agents. |
| title | What Do LLM Agents Know About Their World? Task2Quiz: A Paradigm for Studying Environment Understanding |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2601.09503 |