Barbarians at the Gate: How AI is Upending Systems Research
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915544424251392 |
|---|---|
| author | Cheng, Audrey Liu, Shu Pan, Melissa Li, Zhifei Wang, Bowen Krentsel, Alex Xia, Tian Cemri, Mert Park, Jongseok Yang, Shuo Chen, Jeff Agrawal, Lakshya Desai, Aditya Xing, Jiarong Sen, Koushik Zaharia, Matei Stoica, Ion |
| author_facet | Cheng, Audrey Liu, Shu Pan, Melissa Li, Zhifei Wang, Bowen Krentsel, Alex Xia, Tian Cemri, Mert Park, Jongseok Yang, Shuo Chen, Jeff Agrawal, Lakshya Desai, Aditya Xing, Jiarong Sen, Koushik Zaharia, Matei Stoica, Ion |
| contents | Artificial Intelligence (AI) is starting to transform the research process as we know it by automating the discovery of new solutions. Given a task, the typical AI-driven approach is (i) to generate a set of diverse solutions, and then (ii) to verify these solutions and select one that solves the problem. Crucially, this approach assumes the existence of a reliable verifier, i.e., one that can accurately determine whether a solution solves the given problem. We argue that systems research, long focused on designing and evaluating new performance-oriented algorithms, is particularly well-suited for AI-driven solution discovery. This is because system performance problems naturally admit reliable verifiers: solutions are typically implemented in real systems or simulators, and verification reduces to running these software artifacts against predefined workloads and measuring performance. We term this approach as AI-Driven Research for Systems (ADRS), which iteratively generates, evaluates, and refines solutions. Using penEvolve, an existing open-source ADRS instance, we present case studies across diverse domains, including load balancing for multi-region cloud scheduling, Mixture-of-Experts inference, LLM-based SQL queries, and transaction scheduling. In multiple instances, ADRS discovers algorithms that outperform state-of-the-art human designs (e.g., achieving up to 5.0x runtime improvements or 50% cost reductions). We distill best practices for guiding algorithm evolution, from prompt design to evaluator construction, for existing frameworks. We then discuss the broader implications for the systems community: as AI assumes a central role in algorithm design, we argue that human researchers will increasingly focus on problem formulation and strategic guidance. Our results highlight both the disruptive potential and the urgent need to adapt systems research practices in the age of AI. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_06189 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Barbarians at the Gate: How AI is Upending Systems Research Cheng, Audrey Liu, Shu Pan, Melissa Li, Zhifei Wang, Bowen Krentsel, Alex Xia, Tian Cemri, Mert Park, Jongseok Yang, Shuo Chen, Jeff Agrawal, Lakshya Desai, Aditya Xing, Jiarong Sen, Koushik Zaharia, Matei Stoica, Ion Artificial Intelligence Artificial Intelligence (AI) is starting to transform the research process as we know it by automating the discovery of new solutions. Given a task, the typical AI-driven approach is (i) to generate a set of diverse solutions, and then (ii) to verify these solutions and select one that solves the problem. Crucially, this approach assumes the existence of a reliable verifier, i.e., one that can accurately determine whether a solution solves the given problem. We argue that systems research, long focused on designing and evaluating new performance-oriented algorithms, is particularly well-suited for AI-driven solution discovery. This is because system performance problems naturally admit reliable verifiers: solutions are typically implemented in real systems or simulators, and verification reduces to running these software artifacts against predefined workloads and measuring performance. We term this approach as AI-Driven Research for Systems (ADRS), which iteratively generates, evaluates, and refines solutions. Using penEvolve, an existing open-source ADRS instance, we present case studies across diverse domains, including load balancing for multi-region cloud scheduling, Mixture-of-Experts inference, LLM-based SQL queries, and transaction scheduling. In multiple instances, ADRS discovers algorithms that outperform state-of-the-art human designs (e.g., achieving up to 5.0x runtime improvements or 50% cost reductions). We distill best practices for guiding algorithm evolution, from prompt design to evaluator construction, for existing frameworks. We then discuss the broader implications for the systems community: as AI assumes a central role in algorithm design, we argue that human researchers will increasingly focus on problem formulation and strategic guidance. Our results highlight both the disruptive potential and the urgent need to adapt systems research practices in the age of AI. |
| title | Barbarians at the Gate: How AI is Upending Systems Research |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2510.06189 |