Cover Image

ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Shen, Haiyang, Li, Yue, Meng, Desong, Cai, Dongqi, Qi, Sheng, Zhang, Li, Xu, Mengwei, Ma, Yun
Format:	Preprint
Published:	2024
Subjects:	Software Engineering Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

PandasBench: A Benchmark for the Pandas API
by: Broihier, Alex, et al.
Published: (2025)

ComBench: A Repo-level Real-world Benchmark for Compilation Error Repair
by: Li, Jia, et al.
Published: (2026)

MRG-Bench: Evaluating and Exploring the Requirements of Context for Repository-Level Code Generation
by: Li, Haiyang
Published: (2025)

Decompile-Bench: Million-Scale Binary-Source Function Pairs for Real-World Binary Decompilation
by: Tan, Hanzhuo, et al.
Published: (2025)

Detecting and Mitigating Flakiness in REST API Fuzzing
by: Zhang, Man, et al.
Published: (2026)

Every Software as an Agent: Blueprint and Case Study
by: Xu, Mengwei
Published: (2025)

Identifying and Mitigating API Misuse in Large Language Models
by: Zhuo, Terry Yue, et al.
Published: (2025)

PyBench: Evaluating LLM Agent on various real-world coding tasks
by: Zhang, Yaolun, et al.
Published: (2024)

RealBench: A Repo-Level Code Generation Benchmark Aligned with Real-World Software Development Practices
by: Li, Jia, et al.
Published: (2026)

Your Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models
by: Zhang, Kunpeng, et al.
Published: (2025)

OpenAI for OpenAPI: Automated generation of REST API specification via LLMs
by: Chen, Hao, et al.
Published: (2026)

MioHint: LLM-assisted Mutation for Whitebox API Testing
by: Li, Jia, et al.
Published: (2025)

CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agent
by: Lin, Haojia, et al.
Published: (2025)

DataGovBench: Benchmarking LLM Agents for Real-World Data Governance Workflows
by: Liu, Zhou, et al.
Published: (2025)

GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging
by: Ni, Ziyi, et al.
Published: (2025)

RESTestBench: A Benchmark for Evaluating the Effectiveness of LLM-Generated REST API Test Cases from NL Requirements
by: Kogler, Leon, et al.
Published: (2026)

QuanBench: Benchmarking Quantum Code Generation with Large Language Models
by: Guo, Xiaoyu, et al.
Published: (2025)

AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators
by: Chou, Jason, et al.
Published: (2025)

Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware
by: Chen, Yujia, et al.
Published: (2025)

Leveraging Large Language Models to Improve REST API Testing
by: Kim, Myeongsoo, et al.
Published: (2023)

Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
by: Merrill, Mike A., et al.
Published: (2026)

API-guided Dataset Synthesis to Finetune Large Code Models
by: Li, Zongjie, et al.
Published: (2024)

APIDocBooster: An Extract-Then-Abstract Framework Leveraging Large Language Models for Augmenting API Documentation
by: Yang, Chengran, et al.
Published: (2023)

SpecSyn: LLM-based Synthesis and Refinement of Formal Specifications for Real-world Program Verification
by: Ma, Lezhi, et al.
Published: (2026)

Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation
by: Pysklo, Hubert M., et al.
Published: (2026)

APIDA-Chat: Structured Synthesis of API Search Dialogues to Bootstrap Conversational Agents
by: Eberhart, Zachary, et al.
Published: (2025)

SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications?
by: Tian, Muxin, et al.
Published: (2026)

FireBench: Evaluating Instruction Following in Enterprise and API-Driven LLM Applications
by: Zhang, Yunfan, et al.
Published: (2026)

KAT: Dependency-aware Automated API Testing with Large Language Models
by: Le, Tri, et al.
Published: (2024)

Automated Update of Android Deprecated API Usages with Large Language Models
by: Mahmud, Tarek, et al.
Published: (2024)

Developer Perspectives on REST API Usability: A Study of REST API Guidelines
by: Peldszus, Sven, et al.
Published: (2026)

From Specification to Service: Accelerating API-First Development Using Multi-Agent Systems
by: Chauhan, Saurabh, et al.
Published: (2025)

APIRAT: Integrating Multi-source API Knowledge for Enhanced Code Translation with LLMs
by: Wang, Chaofan, et al.
Published: (2025)

What Makes a Good LLM Agent for Real-world Penetration Testing?
by: Deng, Gelei, et al.
Published: (2026)

PBT-Bench: Benchmarking AI Agents on Property-Based Testing
by: Jing, Lucas, et al.
Published: (2026)

BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software
by: Zhang, Zehua, et al.
Published: (2025)

IDE-Bench: Evaluating Large Language Models as IDE Agents on Real-World Software Engineering Tasks
by: Mateega, Spencer, et al.
Published: (2026)

Live API-Bench: 2500+ Live APIs for Testing Multi-Step Tool Calling
by: Elder, Benjamin, et al.
Published: (2025)

ASAP-Repair: API-Specific Automated Program Repair Based on API Usage Graphs
by: Nielebock, Sebastian, et al.
Published: (2024)

APITestGenie: Generating Web API Tests from Requirements and API Specifications with LLMs
by: Pereira, André, et al.
Published: (2026)