Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Quality

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Koohestani, Roham, de Bekker, Philippe, Koç, Begüm, Izadi, Maliheh
Format:	Preprint
Published:	2025
Subjects:	Software Engineering Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Rethinking IDE Customization for Enhanced HAX: A Hyperdimensional Perspective
by: Koohestani, Roham, et al.
Published: (2025)

Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Tests
by: Deljouyi, Amirhossein, et al.
Published: (2024)

AST-PAC: AST-guided Membership Inference for Code
by: Koohestani, Roham, et al.
Published: (2026)

TriCEGAR: A Trace-Driven Abstraction Mechanism for Agentic AI
by: Koohestani, Roham, et al.
Published: (2026)

HyperSeq: A Hyper-Adaptive Representation for Predictive Sequencing of States
by: Koohestani, Roham, et al.
Published: (2025)

AgentGuard: Runtime Verification of AI Agents
by: Koohestani, Roham
Published: (2025)

Code4MeV2: a Research-oriented Code-completion Platform
by: Koohestani, Roham, et al.
Published: (2025)

Does In-IDE Calibration of Large Language Models work at Scale?
by: Koohestani, Roham, et al.
Published: (2025)

Code Red! On the Harmfulness of Applying Off-the-shelf Large Language Models to Programming Tasks
by: Al-Kaswan, Ali, et al.
Published: (2025)

Prompt-with-Me: in-IDE Structured Prompt Management for LLM-Driven Software Engineering
by: Li, Ziyou, et al.
Published: (2025)

Evaluating Large Language Models for Functional and Maintainable Code in Industrial Settings: A Case Study at ASML
by: Mundhra, Yash, et al.
Published: (2025)

Human-AI Experience in Integrated Development Environments: A Systematic Literature Review
by: Sergeyuk, Agnia, et al.
Published: (2025)

Unified Software Engineering Agent as AI Software Engineer
by: Applis, Leonhard, et al.
Published: (2025)

Traces of Memorisation in Large Language Models for Code
by: Al-Kaswan, Ali, et al.
Published: (2023)

Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs
by: Al-Kaswan, Ali, et al.
Published: (2026)

Towards Comprehensive Benchmarking Infrastructure for LLMs In Software Engineering
by: Rodriguez-Cardenas, Daniel, et al.
Published: (2026)

CodeClash: Benchmarking Goal-Oriented Software Engineering
by: Yang, John, et al.
Published: (2025)

Are Agents Probabilistic Automata? A Trace-Based, Memory-Constrained Theory of Agentic AI
by: Koohestani, Roham, et al.
Published: (2025)

TreeRanker: Fast and Model-agnostic Ranking System for Code Suggestions in IDEs
by: Cipollone, Daniele, et al.
Published: (2025)

A Transformer-Based Approach for Smart Invocation of Automatic Code Completion
by: de Moor, Aral, et al.
Published: (2024)

ATime-Consistent Benchmark for Repository-Level Software Engineering Evaluation
by: Xianpeng, et al.
Published: (2026)

SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback
by: Kumar, Deepak
Published: (2026)

Search-Based Software Engineering and AI Foundation Models: Current Landscape and Future Roadmap
by: Sartaj, Hassan, et al.
Published: (2025)

LoCoBench: A Benchmark for Long-Context Large Language Models in Complex Software Engineering
by: Qiu, Jielin, et al.
Published: (2025)

GBQA: A Game Benchmark for Evaluating LLMs as Quality Assurance Engineers
by: Jiang, Shufan, et al.
Published: (2026)

CodeRepoQA: A Large-scale Benchmark for Software Engineering Question Answering
by: Hu, Ruida, et al.
Published: (2024)

AI-Driven Tools in Modern Software Quality Assurance: An Assessment of Benefits, Challenges, and Future Directions
by: Pysmennyi, Ihor, et al.
Published: (2025)

LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering
by: Qiu, Jielin, et al.
Published: (2025)

Long Code Arena: a Set of Benchmarks for Long-Context Code Models
by: Bogomolov, Egor, et al.
Published: (2024)

EvolveTool-Bench: Evaluating the Quality of LLM-Generated Tool Libraries as Software Artifacts
by: Kaliyev, Alibek T., et al.
Published: (2026)

OmniCode: A Benchmark for Evaluating Software Engineering Agents
by: Sonwane, Atharv, et al.
Published: (2026)

AI-Tutoring in Software Engineering Education
by: Frankford, Eduard, et al.
Published: (2024)

Beyond Retrieval: A Multitask Benchmark and Model for Code Search
by: Xue, Siqiao, et al.
Published: (2026)

ToolScan: A Benchmark for Characterizing Errors in Tool-Use LLMs
by: Kokane, Shirley, et al.
Published: (2024)

Large Language Models for Software Engineering: A Systematic Literature Review
by: Hou, Xinyi, et al.
Published: (2023)

Code Review Agent Benchmark
by: Zhang, Yuntong, et al.
Published: (2026)

Agentic AI Software Engineers: Programming with Trust
by: Roychoudhury, Abhik, et al.
Published: (2025)

In-IDE Human-AI Experience in the Era of Large Language Models; A Literature Review
by: Sergeyuk, Agnia, et al.
Published: (2024)

ToolMisuseBench: An Offline Deterministic Benchmark for Tool Misuse and Recovery in Agentic Systems
by: Sigdel, Akshey, et al.
Published: (2026)

Software Reuse in the Generative AI Era: From Cargo Cult Towards AI Native Software Engineering
by: Mikkonen, Tommi, et al.
Published: (2025)