Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2401.06786 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916089916555264 |
|---|---|
| author | Xu, Yifei Chen, Yuning Zhang, Xumiao Lin, Xianshang Hu, Pan Ma, Yunfei Lu, Songwu Du, Wan Mao, Zhuoqing Zhai, Ennan Cai, Dennis |
| author_facet | Xu, Yifei Chen, Yuning Zhang, Xumiao Lin, Xianshang Hu, Pan Ma, Yunfei Lu, Songwu Du, Wan Mao, Zhuoqing Zhai, Ennan Cai, Dennis |
| contents | Among the thriving ecosystem of cloud computing and the proliferation of Large Language Model (LLM)-based code generation tools, there is a lack of benchmarking for code generation in cloud-native applications. In response to this need, we present CloudEval-YAML, a practical benchmark for cloud configuration generation. CloudEval-YAML tackles the diversity challenge by focusing on YAML, the de facto standard of numerous cloud-native tools. We develop the CloudEval-YAML benchmark with practicality in mind: the dataset consists of hand-written problems with unit tests targeting practical scenarios. We further enhanced the dataset to meet practical needs by rephrasing questions in a concise, abbreviated, and bilingual manner. The dataset consists of 1011 problems that take more than 1200 human hours to complete. To improve practicality during evaluation, we build a scalable evaluation platform for CloudEval-YAML that achieves a 20 times speedup over a single machine. To the best of our knowledge, the CloudEval-YAML dataset is the first hand-written dataset targeting cloud-native applications. We present an in-depth evaluation of 12 LLMs, leading to a deeper understanding of the problems and LLMs, as well as effective methods to improve task performance and reduce cost. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_06786 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | CloudEval-YAML: A Practical Benchmark for Cloud Configuration Generation Xu, Yifei Chen, Yuning Zhang, Xumiao Lin, Xianshang Hu, Pan Ma, Yunfei Lu, Songwu Du, Wan Mao, Zhuoqing Zhai, Ennan Cai, Dennis Distributed, Parallel, and Cluster Computing Artificial Intelligence Among the thriving ecosystem of cloud computing and the proliferation of Large Language Model (LLM)-based code generation tools, there is a lack of benchmarking for code generation in cloud-native applications. In response to this need, we present CloudEval-YAML, a practical benchmark for cloud configuration generation. CloudEval-YAML tackles the diversity challenge by focusing on YAML, the de facto standard of numerous cloud-native tools. We develop the CloudEval-YAML benchmark with practicality in mind: the dataset consists of hand-written problems with unit tests targeting practical scenarios. We further enhanced the dataset to meet practical needs by rephrasing questions in a concise, abbreviated, and bilingual manner. The dataset consists of 1011 problems that take more than 1200 human hours to complete. To improve practicality during evaluation, we build a scalable evaluation platform for CloudEval-YAML that achieves a 20 times speedup over a single machine. To the best of our knowledge, the CloudEval-YAML dataset is the first hand-written dataset targeting cloud-native applications. We present an in-depth evaluation of 12 LLMs, leading to a deeper understanding of the problems and LLMs, as well as effective methods to improve task performance and reduce cost. |
| title | CloudEval-YAML: A Practical Benchmark for Cloud Configuration Generation |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2401.06786 |