Towards Understanding What Code Language Models Learned
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910345948299264 |
|---|---|
| author | Ahmed, Toufique Yu, Dian Huang, Chengxuan Wang, Cathy Devanbu, Prem Sagae, Kenji |
| author_facet | Ahmed, Toufique Yu, Dian Huang, Chengxuan Wang, Cathy Devanbu, Prem Sagae, Kenji |
| contents | Pre-trained language models are effective in a variety of natural language tasks, but it has been argued their capabilities fall short of fully learning meaning or understanding language. To understand the extent to which language models can learn some form of meaning, we investigate their ability to capture semantics of code beyond superficial frequency and co-occurrence. In contrast to previous research on probing models for linguistic features, we study pre-trained models in a setting that allows for objective and straightforward evaluation of a model's ability to learn semantics. In this paper, we examine whether such models capture the semantics of code, which is precisely and formally defined. Through experiments involving the manipulation of code fragments, we show that code pre-trained models of code learn a robust representation of the computational semantics of code that goes beyond superficial features of form alone |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_11943 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Towards Understanding What Code Language Models Learned Ahmed, Toufique Yu, Dian Huang, Chengxuan Wang, Cathy Devanbu, Prem Sagae, Kenji Software Engineering Computation and Language Machine Learning Pre-trained language models are effective in a variety of natural language tasks, but it has been argued their capabilities fall short of fully learning meaning or understanding language. To understand the extent to which language models can learn some form of meaning, we investigate their ability to capture semantics of code beyond superficial frequency and co-occurrence. In contrast to previous research on probing models for linguistic features, we study pre-trained models in a setting that allows for objective and straightforward evaluation of a model's ability to learn semantics. In this paper, we examine whether such models capture the semantics of code, which is precisely and formally defined. Through experiments involving the manipulation of code fragments, we show that code pre-trained models of code learn a robust representation of the computational semantics of code that goes beyond superficial features of form alone |
| title | Towards Understanding What Code Language Models Learned |
| topic | Software Engineering Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2306.11943 |