Rethinking Technology Stack Selection with AI Coding Proficiency
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916949003337728 |
|---|---|
| author | Zhang, Xiaoyu Jiang, Weipeng Zhai, Juan Ma, Shiqing Bao, Qingshuang Lin, Chenhao Shen, Chao Li, Tianlin Liu, Yang |
| author_facet | Zhang, Xiaoyu Jiang, Weipeng Zhai, Juan Ma, Shiqing Bao, Qingshuang Lin, Chenhao Shen, Chao Li, Tianlin Liu, Yang |
| contents | Large language models (LLMs) are now an integral part of software development workflows and are reshaping the whole process. Traditional technology stack selection has not caught up. Most of the existing selection methods focus solely on the inherent attributes of the technology, overlooking whether the LLM can effectively leverage the chosen technology. For example, when generating code snippets using popular libraries like Selenium (one of the most widely used test automation tools with over 33k GitHub stars), existing LLMs frequently generate low-quality code snippets (e.g., using deprecated APIs and methods, or containing syntax errors). As such, teams using LLM assistants risk choosing technologies that cannot be used effectively by LLMs, yielding high debugging effort and mounting technical debt. We foresee a practical question in the LLM era, is a technology ready for AI-assisted development? In this paper, we first propose the concept, AI coding proficiency, the degree to which LLMs can utilize a given technology to generate high-quality code snippets. We conduct the first comprehensive empirical study examining AI proficiency across 170 third-party libraries and 61 task scenarios, evaluating six widely used LLMs. Our findings reveal that libraries with similar functionalities can exhibit up to 84% differences in the quality score of LLM-generated code, while different models also exhibit quality gaps among their generation results using the same library. These gaps translate into real engineering costs and can steer developer choices toward a narrow set of libraries with high AI coding proficiency, threatening technological diversity in the ecosystem. We call on the community to integrate AI proficiency assessments into technology selection frameworks and develop mitigation strategies, preserving competitive balance in AI-driven development. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_11132 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Rethinking Technology Stack Selection with AI Coding Proficiency Zhang, Xiaoyu Jiang, Weipeng Zhai, Juan Ma, Shiqing Bao, Qingshuang Lin, Chenhao Shen, Chao Li, Tianlin Liu, Yang Software Engineering Large language models (LLMs) are now an integral part of software development workflows and are reshaping the whole process. Traditional technology stack selection has not caught up. Most of the existing selection methods focus solely on the inherent attributes of the technology, overlooking whether the LLM can effectively leverage the chosen technology. For example, when generating code snippets using popular libraries like Selenium (one of the most widely used test automation tools with over 33k GitHub stars), existing LLMs frequently generate low-quality code snippets (e.g., using deprecated APIs and methods, or containing syntax errors). As such, teams using LLM assistants risk choosing technologies that cannot be used effectively by LLMs, yielding high debugging effort and mounting technical debt. We foresee a practical question in the LLM era, is a technology ready for AI-assisted development? In this paper, we first propose the concept, AI coding proficiency, the degree to which LLMs can utilize a given technology to generate high-quality code snippets. We conduct the first comprehensive empirical study examining AI proficiency across 170 third-party libraries and 61 task scenarios, evaluating six widely used LLMs. Our findings reveal that libraries with similar functionalities can exhibit up to 84% differences in the quality score of LLM-generated code, while different models also exhibit quality gaps among their generation results using the same library. These gaps translate into real engineering costs and can steer developer choices toward a narrow set of libraries with high AI coding proficiency, threatening technological diversity in the ecosystem. We call on the community to integrate AI proficiency assessments into technology selection frameworks and develop mitigation strategies, preserving competitive balance in AI-driven development. |
| title | Rethinking Technology Stack Selection with AI Coding Proficiency |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2509.11132 |