Cover Image

A Benchmark for Language Models in Real-World System Building

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Jin, Weilin, Zhao, Chenyu, Huang, Zeshun, Zhang, Chaoyun, Lin, Qingwei, Bansal, Chetan, Rajmohan, Saravan, Zhang, Shenglin, Sun, Yongqian, Pei, Dan, Wu, Yifan, Jia, Tong, Li, Ying, Wu, Zhonghai, Ma, Minghua
Format:	Preprint
Published:	2026
Subjects:	Software Engineering
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Can Language Models Go Beyond Coding? Assessing the Capability of Language Models to Build Real-World Systems
by: Zhao, Chenyu, et al.
Published: (2025)

EvidenT: An Evidence-Preserving Framework for Iterative System-Level Package Repair
by: Zhao, Chenyu, et al.
Published: (2026)

Debugging the Debuggers: Failure-Anchored Structured Recovery for Software Engineering Agents
by: Zhao, Chenyu, et al.
Published: (2026)

Large Language Models can Deliver Accurate and Interpretable Time Series Anomaly Detection
by: Liu, Jun, et al.
Published: (2024)

Triage in Software Engineering: A Systematic Review of Research and Practice
by: Zhao, Yongxin, et al.
Published: (2025)

An Empirical Study of Production Incidents in Generative AI Cloud Services
by: Yan, Haoran, et al.
Published: (2025)

Automated Root Causing of Cloud Incidents using In-Context Learning with GPT-4
by: Zhang, Xuchao, et al.
Published: (2024)

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
by: Gupta, Taneesh, et al.
Published: (2025)

REFA: Reference Free Alignment for multi-preference optimization
by: Gupta, Taneesh, et al.
Published: (2024)

Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generation
by: Ding, Ruomeng, et al.
Published: (2023)

From Reasoning to Answer: Empirical, Attention-Based and Mechanistic Insights into Distilled DeepSeek R1 Models
by: Zhang, Jue, et al.
Published: (2025)

Text2Grad: Reinforcement Learning from Natural Language Feedback
by: Wang, Hanyang, et al.
Published: (2025)

API Agents vs. GUI Agents: Divergence and Convergence
by: Zhang, Chaoyun, et al.
Published: (2025)

Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarks
by: Tan, Rongyuan, et al.
Published: (2026)

AutoAdapt: An Automated Domain Adaptation Framework for LLMs
by: Sinha, Sidharth, et al.
Published: (2026)

Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts
by: Gupta, Taneesh, et al.
Published: (2024)

UFO3: Weaving the Digital Agent Galaxy
by: Zhang, Chaoyun, et al.
Published: (2025)

UFO: A UI-Focused Agent for Windows OS Interaction
by: Zhang, Chaoyun, et al.
Published: (2024)

Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttention
by: Liao, Mengqi, et al.
Published: (2026)

WebXSkill: Skill Learning for Autonomous Web Agents
by: Wang, Zhaoyang, et al.
Published: (2026)

Intelligent Monitoring Framework for Cloud Services: A Data-Driven Approach
by: Srinivas, Pooja, et al.
Published: (2024)

AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds
by: Chen, Yinfang, et al.
Published: (2025)

Revisiting VAE for Unsupervised Time Series Anomaly Detection: A Frequency Perspective
by: Wang, Zexin, et al.
Published: (2024)

VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Model
by: Zheng, Jiani, et al.
Published: (2025)

MEETING DELEGATE: Benchmarking LLMs on Attending Meetings on Our Behalf
by: Hu, Lingxiang, et al.
Published: (2025)

GUI-360$^\circ$: A Comprehensive Dataset and Benchmark for Computer-Using Agents
by: Mu, Jian, et al.
Published: (2025)

Large Language Model-Brained GUI Agents: A Survey
by: Zhang, Chaoyun, et al.
Published: (2024)

Building AI Agents for Autonomous Clouds: Challenges and Design Principles
by: Shetty, Manish, et al.
Published: (2024)

Exploring LLM-based Agents for Root Cause Analysis
by: Roy, Devjeet, et al.
Published: (2024)

G-KV: Decoding-Time KV Cache Eviction with Global Attention
by: Liao, Mengqi, et al.
Published: (2025)

CI-Work: Benchmarking Contextual Integrity in Enterprise LLM Agents
by: Fu, Wenjie, et al.
Published: (2026)

AllHands: Ask Me Anything on Large-scale Verbatim Feedback via Large Language Models
by: Zhang, Chaoyun, et al.
Published: (2024)

Attention Enhanced Entity Recommendation for Intelligent Monitoring in Cloud Systems
by: Hussain, Fiza, et al.
Published: (2025)

Dependency Aware Incident Linking in Large Cloud Systems
by: Ghosh, Supriyo, et al.
Published: (2024)

CARMO: Dynamic Criteria Generation for Context-Aware Reward Modelling
by: Gupta, Taneesh, et al.
Published: (2024)

eARCO: Efficient Automated Root Cause Analysis with Prompt Optimization
by: Goel, Drishti, et al.
Published: (2025)

X-lifecycle Learning for Cloud Incident Management using LLMs
by: Goel, Drishti, et al.
Published: (2024)

LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals
by: Sun, Lihao, et al.
Published: (2026)

ExeCoder: Empowering Large Language Models with Executability Representation for Code Translation
by: He, Minghua, et al.
Published: (2025)

COIN: Chance-Constrained Imitation Learning for Uncertainty-aware Adaptive Resource Oversubscription Policy
by: Wang, Lu, et al.
Published: (2024)