A Benchmark for Language Models in Real-World System Building
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915738150764544 |
|---|---|
| author | Jin, Weilin Zhao, Chenyu Huang, Zeshun Zhang, Chaoyun Lin, Qingwei Bansal, Chetan Rajmohan, Saravan Zhang, Shenglin Sun, Yongqian Pei, Dan Wu, Yifan Jia, Tong Li, Ying Wu, Zhonghai Ma, Minghua |
| author_facet | Jin, Weilin Zhao, Chenyu Huang, Zeshun Zhang, Chaoyun Lin, Qingwei Bansal, Chetan Rajmohan, Saravan Zhang, Shenglin Sun, Yongqian Pei, Dan Wu, Yifan Jia, Tong Li, Ying Wu, Zhonghai Ma, Minghua |
| contents | During migration across instruction set architectures (ISAs), software package build repair is a critical task for ensuring the reliability of software deployment and the stability of modern operating systems. While Large Language Models (LLMs) have shown promise in tackling this challenge, prior work has primarily focused on single instruction set architecture (ISA) and homogeneous programming languages. To address this limitation, we introduce a new benchmark designed for software package build repair across diverse architectures and languages. Comprising 268 real-world software package build failures, the benchmark provides a standardized evaluation pipeline. We evaluate six state-of-the-art LLMs on the benchmark, and the results show that cross-ISA software package repair remains difficult and requires further advances. By systematically exposing this challenge, the benchmark establishes a foundation for advancing future methods aimed at improving software portability and bridging architectural gaps. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_12927 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | A Benchmark for Language Models in Real-World System Building Jin, Weilin Zhao, Chenyu Huang, Zeshun Zhang, Chaoyun Lin, Qingwei Bansal, Chetan Rajmohan, Saravan Zhang, Shenglin Sun, Yongqian Pei, Dan Wu, Yifan Jia, Tong Li, Ying Wu, Zhonghai Ma, Minghua Software Engineering During migration across instruction set architectures (ISAs), software package build repair is a critical task for ensuring the reliability of software deployment and the stability of modern operating systems. While Large Language Models (LLMs) have shown promise in tackling this challenge, prior work has primarily focused on single instruction set architecture (ISA) and homogeneous programming languages. To address this limitation, we introduce a new benchmark designed for software package build repair across diverse architectures and languages. Comprising 268 real-world software package build failures, the benchmark provides a standardized evaluation pipeline. We evaluate six state-of-the-art LLMs on the benchmark, and the results show that cross-ISA software package repair remains difficult and requires further advances. By systematically exposing this challenge, the benchmark establishes a foundation for advancing future methods aimed at improving software portability and bridging architectural gaps. |
| title | A Benchmark for Language Models in Real-World System Building |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2601.12927 |