Cover Image

AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Shi, Wentao, Wang, Yu, Zhao, Yuyang, Chen, Yuxin, Feng, Fuli, Hao, Xueyuan, Su, Xi, Gu, Qi, Su, Hui, Cai, Xunliang, He, Xiangnan
Format:	Preprint
Published:	2026
Subjects:	Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

AppAgent-Pro: A Proactive GUI Agent System for Multidomain Information Integration and User Assistance
by: Zhao, Yuyang, et al.
Published: (2025)

VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications
by: He, Wei, et al.
Published: (2025)

Look Before You Leap: Autonomous Exploration for LLM Agents
by: Ye, Ziang, et al.
Published: (2026)

AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition
by: Wang, Ruipeng, et al.
Published: (2026)

Efficient Multi-Agent System Training with Data Influence-Oriented Tree Search
by: Shi, Wentao, et al.
Published: (2025)

EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulation
by: Liu, Yi, et al.
Published: (2026)

MemOCR: Layout-Aware Visual Memory for Efficient Long-Horizon Reasoning
by: Shi, Yaorui, et al.
Published: (2026)

JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Framework
by: Liu, Fan, et al.
Published: (2024)

TopoCurate:Modeling Interaction Topology for Tool-Use Agent Training
by: Yang, Jinluan, et al.
Published: (2026)

Lower-Left Partial AUC: An Effective and Efficient Optimization Metric for Recommendation
by: Shi, Wentao, et al.
Published: (2024)

Fair Recommendations with Limited Sensitive Attributes: A Distributionally Robust Optimization Approach
by: Shi, Tianhao, et al.
Published: (2024)

Learning to Act under Noise: Enhancing Agent Robustness via Noisy Environments
by: Chen, Yuxin, et al.
Published: (2026)

Generative Multi-Target Cross-Domain Recommendation
by: Jin, Jinqiu, et al.
Published: (2025)

JudgeBench: A Benchmark for Evaluating LLM-based Judges
by: Tan, Sijun, et al.
Published: (2024)

AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment
by: Xiao, Jianfei, et al.
Published: (2026)

ScaleEnv: Scaling Environment Synthesis from Scratch for Generalist Interactive Tool-Use Agent Training
by: Tu, Dunwei, et al.
Published: (2026)

Unraveling the Mystery of Scaling Laws: Part I
by: Su, Hui, et al.
Published: (2024)

Large Language Models are Learnable Planners for Long-Term Recommendation
by: Shi, Wentao, et al.
Published: (2024)

Preliminary Study on Incremental Learning for Large Language Model-based Recommender Systems
by: Shi, Tianhao, et al.
Published: (2023)

Direct Multi-Turn Preference Optimization for Language Agents
by: Shi, Wentao, et al.
Published: (2024)

Be Aware of the Neighborhood Effect: Modeling Selection Bias under Interference
by: Li, Haoxuan, et al.
Published: (2024)

FinResearchBench: A Logic Tree based Agent-as-a-Judge Evaluation Framework for Financial Research Agents
by: Sun, Rui, et al.
Published: (2025)

Proactive Recommendation with Iterative Preference Guidance
by: Bi, Shuxian, et al.
Published: (2024)

Learning to Self-Verify Makes Language Models Better Reasoners
by: Chen, Yuxin, et al.
Published: (2026)

IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation
by: Wen, Bosi, et al.
Published: (2026)

ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution
by: Wang, Yubang, et al.
Published: (2026)

Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation
by: Fu, Lingyue, et al.
Published: (2025)

AgentEscapeBench: Evaluating Out-of-Domain Tool-Grounded Reasoning in LLM Agents
by: Guo, Zhengkang, et al.
Published: (2026)

MTR-Bench: A Comprehensive Benchmark for Multi-Turn Reasoning Evaluation
by: Li, Xiaoyuan, et al.
Published: (2025)

R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
by: Yuan, Tongxin, et al.
Published: (2024)

AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rollouts
by: Fang, Shicheng, et al.
Published: (2026)

CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents
by: Liu, Jiayu, et al.
Published: (2025)

AgentBench: Evaluating LLMs as Agents
by: Liu, Xiao, et al.
Published: (2023)

CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agent
by: Lin, Haojia, et al.
Published: (2025)

CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMs
by: Yao, Zhiyuan, et al.
Published: (2026)

Generative Recommendation: Towards Next-generation Recommender Paradigm
by: Wang, Wenjie, et al.
Published: (2023)

Agentic Feedback Loop Modeling Improves Recommendation and User Simulation
by: Cai, Shihao, et al.
Published: (2024)

TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use
by: He, Pengfei, et al.
Published: (2025)

Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Values
by: Dong, Haonan, et al.
Published: (2026)

PortBench: A Correlation-Aware, Full-Pipeline Benchmark for LLM-Driven Portfolio Management
by: Zhao, Yuxuan, et al.
Published: (2026)