Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.05508 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918117390680064 |
|---|---|
| author | Bhonsle, Roshita Dutta, Rishav Vavilapalli, Sneha Seth, Harsh Jaye, Abubakarr Chang, Yapei Rungta, Mukund Boateng, Emmanuel Aboah Hasan, Sadid Nosakhare, Ehi Srinivasan, Soundar |
| author_facet | Bhonsle, Roshita Dutta, Rishav Vavilapalli, Sneha Seth, Harsh Jaye, Abubakarr Chang, Yapei Rungta, Mukund Boateng, Emmanuel Aboah Hasan, Sadid Nosakhare, Ehi Srinivasan, Soundar |
| contents | The increasing adoption of foundation models as agents across diverse domains necessitates a robust evaluation framework. Current methods, such as LLM-as-a-Judge, focus only on final outputs, overlooking the step-by-step reasoning that drives agentic decision-making. Meanwhile, existing Agent-as-a-Judge systems, where one agent evaluates another's task completion, are typically designed for narrow, domain-specific settings. To address this gap, we propose a generalizable, modular framework for evaluating agent task completion independent of the task domain. The framework emulates human-like evaluation by decomposing tasks into sub-tasks and validating each step using available information, such as the agent's output and reasoning. Each module contributes to a specific aspect of the evaluation process, and their outputs are aggregated to produce a final verdict on task completion. We validate our framework by evaluating the Magentic-One Actor Agent on two benchmarks, GAIA and BigCodeBench. Our Judge Agent predicts task success with closer agreement to human evaluations, achieving 4.76% and 10.52% higher alignment accuracy, respectively, compared to the GPT-4o based LLM-as-a-Judge baseline. This demonstrates the potential of our proposed general-purpose evaluation framework. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_05508 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Auto-Eval Judge: Towards a General Agentic Framework for Task Completion Evaluation Bhonsle, Roshita Dutta, Rishav Vavilapalli, Sneha Seth, Harsh Jaye, Abubakarr Chang, Yapei Rungta, Mukund Boateng, Emmanuel Aboah Hasan, Sadid Nosakhare, Ehi Srinivasan, Soundar Artificial Intelligence The increasing adoption of foundation models as agents across diverse domains necessitates a robust evaluation framework. Current methods, such as LLM-as-a-Judge, focus only on final outputs, overlooking the step-by-step reasoning that drives agentic decision-making. Meanwhile, existing Agent-as-a-Judge systems, where one agent evaluates another's task completion, are typically designed for narrow, domain-specific settings. To address this gap, we propose a generalizable, modular framework for evaluating agent task completion independent of the task domain. The framework emulates human-like evaluation by decomposing tasks into sub-tasks and validating each step using available information, such as the agent's output and reasoning. Each module contributes to a specific aspect of the evaluation process, and their outputs are aggregated to produce a final verdict on task completion. We validate our framework by evaluating the Magentic-One Actor Agent on two benchmarks, GAIA and BigCodeBench. Our Judge Agent predicts task success with closer agreement to human evaluations, achieving 4.76% and 10.52% higher alignment accuracy, respectively, compared to the GPT-4o based LLM-as-a-Judge baseline. This demonstrates the potential of our proposed general-purpose evaluation framework. |
| title | Auto-Eval Judge: Towards a General Agentic Framework for Task Completion Evaluation |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2508.05508 |