Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2606.01725 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916072288944128 |
|---|---|
| author | Kim, Donghwan Singh, Prakhar Min, Younghoon Kim, Jongryool Park, Jongse Maeng, Kiwan |
| author_facet | Kim, Donghwan Singh, Prakhar Min, Younghoon Kim, Jongryool Park, Jongse Maeng, Kiwan |
| contents | Agentic AI completes tasks through iterative planning, tool use, and reasoning based on observed outcomes. Despite its popularity, its system-level behavior remains poorly understood, particularly for complex datasets and agent architectures-owing to highly non-deterministic execution, prohibitive evaluation costs, and limited visibility into proprietary models. This paper presents GAIATrace, the first token-level trace dataset of two state-of-the-art agentic systems (MiroThinker and OWL) running GAIA, a benchmark composed of a heterogeneous mix of general-purpose tasks. Unlike prior trace datasets, GAIATrace captures full reasoning tokens, task-level structures, and activities of every major participating LLMs, enabling in-depth systems research. Complementing the dataset, we present Vidur-Agent, a trace-driven simulator that can replay GAIATrace to perform reproducible, low-cost system evaluation across diverse simulated environments. Using both artifacts, we characterize how modern agentic systems handle general tasks and how various system design choices shape their behavior, yielding several unique findings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_01725 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulation Kim, Donghwan Singh, Prakhar Min, Younghoon Kim, Jongryool Park, Jongse Maeng, Kiwan Artificial Intelligence Machine Learning Agentic AI completes tasks through iterative planning, tool use, and reasoning based on observed outcomes. Despite its popularity, its system-level behavior remains poorly understood, particularly for complex datasets and agent architectures-owing to highly non-deterministic execution, prohibitive evaluation costs, and limited visibility into proprietary models. This paper presents GAIATrace, the first token-level trace dataset of two state-of-the-art agentic systems (MiroThinker and OWL) running GAIA, a benchmark composed of a heterogeneous mix of general-purpose tasks. Unlike prior trace datasets, GAIATrace captures full reasoning tokens, task-level structures, and activities of every major participating LLMs, enabling in-depth systems research. Complementing the dataset, we present Vidur-Agent, a trace-driven simulator that can replay GAIATrace to perform reproducible, low-cost system evaluation across diverse simulated environments. Using both artifacts, we characterize how modern agentic systems handle general tasks and how various system design choices shape their behavior, yielding several unique findings. |
| title | Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulation |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2606.01725 |