Image de couverture de livre

FullStack Bench: Evaluating LLMs as Full Stack Coders

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Bytedance-Seed-Foundation-Code-Team, :, Cheng, Yao, Chen, Jianfeng, Chen, Jie, Chen, Li, Chen, Liyu, Chen, Wentao, Chen, Zhengyu, Geng, Shijie, Li, Aoyan, Li, Bo, Li, Bowen, Li, Linyi, Liu, Boyi, Liu, Jiaheng, Liu, Kaibo, Liu, Qi, Liu, Shukai, Liu, Siyao, Liu, Tianyi, Liu, Tingkai, Liu, Yongfei, Long, Rui, Mai, Jing, Ning, Guanghan, Peng, Z. Y., Shen, Kai, Su, Jiahao, Su, Jing, Sun, Tao, Sun, Yifan, Tao, Yunzhe, Wang, Guoyin, Wang, Siwei, Wang, Xuwu, Wang, Yite, Wang, Zihan, Xia, Jinxiang, Xiang, Liang, Xiao, Xia, Xiao, Yongsheng, Xi, Chenguang, Xin, Shulin, Xu, Jingjing, Xu, Shikun, Yang, Hongxia, Yang, Jack, Yang, Yingxiang, Yuan, Jianbo, Zhang, Jun, Zhang, Yufeng, Zhang, Yuyu, Zheng, Shen, Zhu, He, Zhu, Ming
Format:	Preprint
Publié:	2024
Sujets:	Artificial Intelligence Software Engineering
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

_version_	1866909606776668160
author	Bytedance-Seed-Foundation-Code-Team : Cheng, Yao Chen, Jianfeng Chen, Jie Chen, Li Chen, Liyu Chen, Wentao Chen, Zhengyu Geng, Shijie Li, Aoyan Li, Bo Li, Bowen Li, Linyi Liu, Boyi Liu, Jiaheng Liu, Kaibo Liu, Qi Liu, Shukai Liu, Siyao Liu, Tianyi Liu, Tingkai Liu, Yongfei Long, Rui Mai, Jing Ning, Guanghan Peng, Z. Y. Shen, Kai Su, Jiahao Su, Jing Sun, Tao Sun, Yifan Tao, Yunzhe Wang, Guoyin Wang, Siwei Wang, Xuwu Wang, Yite Wang, Zihan Xia, Jinxiang Xiang, Liang Xiao, Xia Xiao, Yongsheng Xi, Chenguang Xin, Shulin Xu, Jingjing Xu, Shikun Yang, Hongxia Yang, Jack Yang, Yingxiang Yuan, Jianbo Zhang, Jun Zhang, Yufeng Zhang, Yuyu Zheng, Shen Zhu, He Zhu, Ming
author_facet	Bytedance-Seed-Foundation-Code-Team : Cheng, Yao Chen, Jianfeng Chen, Jie Chen, Li Chen, Liyu Chen, Wentao Chen, Zhengyu Geng, Shijie Li, Aoyan Li, Bo Li, Bowen Li, Linyi Liu, Boyi Liu, Jiaheng Liu, Kaibo Liu, Qi Liu, Shukai Liu, Siyao Liu, Tianyi Liu, Tingkai Liu, Yongfei Long, Rui Mai, Jing Ning, Guanghan Peng, Z. Y. Shen, Kai Su, Jiahao Su, Jing Sun, Tao Sun, Yifan Tao, Yunzhe Wang, Guoyin Wang, Siwei Wang, Xuwu Wang, Yite Wang, Zihan Xia, Jinxiang Xiang, Liang Xiao, Xia Xiao, Yongsheng Xi, Chenguang Xin, Shulin Xu, Jingjing Xu, Shikun Yang, Hongxia Yang, Jack Yang, Yingxiang Yuan, Jianbo Zhang, Jun Zhang, Yufeng Zhang, Yuyu Zheng, Shen Zhu, He Zhu, Ming
contents	As the capabilities of code large language models (LLMs) continue to expand, their applications across diverse code intelligence domains are rapidly increasing. However, most existing datasets only evaluate limited application domains. To address this gap, we have developed a comprehensive code evaluation dataset FullStack Bench focusing on full-stack programming, which encompasses a wide range of application domains (e.g., basic programming, data analysis, software engineering, mathematics, and machine learning). Besides, to assess multilingual programming capabilities, in FullStack Bench, we design real-world instructions and corresponding unit test cases from 16 widely-used programming languages to reflect real-world usage scenarios rather than simple translations. Moreover, we also release an effective code sandbox execution tool (i.e., SandboxFusion) supporting various programming languages and packages to evaluate the performance of our FullStack Bench efficiently. Comprehensive experimental results on our FullStack Bench demonstrate the necessity and effectiveness of our FullStack Bench and SandboxFusion.
format	Preprint
id	arxiv_https___arxiv_org_abs_2412_00535
institution	arXiv
publishDate	2024
record_format	arxiv
spellingShingle	FullStack Bench: Evaluating LLMs as Full Stack Coders Bytedance-Seed-Foundation-Code-Team : Cheng, Yao Chen, Jianfeng Chen, Jie Chen, Li Chen, Liyu Chen, Wentao Chen, Zhengyu Geng, Shijie Li, Aoyan Li, Bo Li, Bowen Li, Linyi Liu, Boyi Liu, Jiaheng Liu, Kaibo Liu, Qi Liu, Shukai Liu, Siyao Liu, Tianyi Liu, Tingkai Liu, Yongfei Long, Rui Mai, Jing Ning, Guanghan Peng, Z. Y. Shen, Kai Su, Jiahao Su, Jing Sun, Tao Sun, Yifan Tao, Yunzhe Wang, Guoyin Wang, Siwei Wang, Xuwu Wang, Yite Wang, Zihan Xia, Jinxiang Xiang, Liang Xiao, Xia Xiao, Yongsheng Xi, Chenguang Xin, Shulin Xu, Jingjing Xu, Shikun Yang, Hongxia Yang, Jack Yang, Yingxiang Yuan, Jianbo Zhang, Jun Zhang, Yufeng Zhang, Yuyu Zheng, Shen Zhu, He Zhu, Ming Artificial Intelligence Software Engineering As the capabilities of code large language models (LLMs) continue to expand, their applications across diverse code intelligence domains are rapidly increasing. However, most existing datasets only evaluate limited application domains. To address this gap, we have developed a comprehensive code evaluation dataset FullStack Bench focusing on full-stack programming, which encompasses a wide range of application domains (e.g., basic programming, data analysis, software engineering, mathematics, and machine learning). Besides, to assess multilingual programming capabilities, in FullStack Bench, we design real-world instructions and corresponding unit test cases from 16 widely-used programming languages to reflect real-world usage scenarios rather than simple translations. Moreover, we also release an effective code sandbox execution tool (i.e., SandboxFusion) supporting various programming languages and packages to evaluate the performance of our FullStack Bench efficiently. Comprehensive experimental results on our FullStack Bench demonstrate the necessity and effectiveness of our FullStack Bench and SandboxFusion.
title	FullStack Bench: Evaluating LLMs as Full Stack Coders
topic	Artificial Intelligence Software Engineering
url	https://arxiv.org/abs/2412.00535