Do Large Code Models Understand Programming Concepts? Counterfactual Analysis for Code Predicates
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909488744759296 |
|---|---|
| author | Hooda, Ashish Christodorescu, Mihai Allamanis, Miltiadis Wilson, Aaron Fawaz, Kassem Jha, Somesh |
| author_facet | Hooda, Ashish Christodorescu, Mihai Allamanis, Miltiadis Wilson, Aaron Fawaz, Kassem Jha, Somesh |
| contents | Large Language Models' success on text generation has also made them better at code generation and coding tasks. While a lot of work has demonstrated their remarkable performance on tasks such as code completion and editing, it is still unclear as to why. We help bridge this gap by exploring to what degree auto-regressive models understand the logical constructs of the underlying programs. We propose Counterfactual Analysis for Programming Concept Predicates (CACP) as a counterfactual testing framework to evaluate whether Large Code Models understand programming concepts. With only black-box access to the model, we use CACP to evaluate ten popular Large Code Models for four different programming concepts. Our findings suggest that current models lack understanding of concepts such as data flow and control flow. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_05980 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Do Large Code Models Understand Programming Concepts? Counterfactual Analysis for Code Predicates Hooda, Ashish Christodorescu, Mihai Allamanis, Miltiadis Wilson, Aaron Fawaz, Kassem Jha, Somesh Software Engineering Artificial Intelligence Machine Learning Programming Languages Large Language Models' success on text generation has also made them better at code generation and coding tasks. While a lot of work has demonstrated their remarkable performance on tasks such as code completion and editing, it is still unclear as to why. We help bridge this gap by exploring to what degree auto-regressive models understand the logical constructs of the underlying programs. We propose Counterfactual Analysis for Programming Concept Predicates (CACP) as a counterfactual testing framework to evaluate whether Large Code Models understand programming concepts. With only black-box access to the model, we use CACP to evaluate ten popular Large Code Models for four different programming concepts. Our findings suggest that current models lack understanding of concepts such as data flow and control flow. |
| title | Do Large Code Models Understand Programming Concepts? Counterfactual Analysis for Code Predicates |
| topic | Software Engineering Artificial Intelligence Machine Learning Programming Languages |
| url | https://arxiv.org/abs/2402.05980 |