Exploiting Code Symmetries for Learning Program Semantics
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912018381930496 |
|---|---|
| author | Pei, Kexin Li, Weichen Jin, Qirui Liu, Shuyang Geng, Scott Cavallaro, Lorenzo Yang, Junfeng Jana, Suman |
| author_facet | Pei, Kexin Li, Weichen Jin, Qirui Liu, Shuyang Geng, Scott Cavallaro, Lorenzo Yang, Junfeng Jana, Suman |
| contents | This paper tackles the challenge of teaching code semantics to Large Language Models (LLMs) for program analysis by incorporating code symmetries into the model architecture. We introduce a group-theoretic framework that defines code symmetries as semantics-preserving transformations, where forming a code symmetry group enables precise and efficient reasoning of code semantics. Our solution, SymC, develops a novel variant of self-attention that is provably equivariant to code symmetries from the permutation group defined over the program dependence graph. SymC obtains superior performance on five program analysis tasks, outperforming state-of-the-art code models without any pre-training. Our results suggest that code LLMs that encode the code structural prior via the code symmetry group generalize better and faster. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2308_03312 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Exploiting Code Symmetries for Learning Program Semantics Pei, Kexin Li, Weichen Jin, Qirui Liu, Shuyang Geng, Scott Cavallaro, Lorenzo Yang, Junfeng Jana, Suman Machine Learning Cryptography and Security Programming Languages This paper tackles the challenge of teaching code semantics to Large Language Models (LLMs) for program analysis by incorporating code symmetries into the model architecture. We introduce a group-theoretic framework that defines code symmetries as semantics-preserving transformations, where forming a code symmetry group enables precise and efficient reasoning of code semantics. Our solution, SymC, develops a novel variant of self-attention that is provably equivariant to code symmetries from the permutation group defined over the program dependence graph. SymC obtains superior performance on five program analysis tasks, outperforming state-of-the-art code models without any pre-training. Our results suggest that code LLMs that encode the code structural prior via the code symmetry group generalize better and faster. |
| title | Exploiting Code Symmetries for Learning Program Semantics |
| topic | Machine Learning Cryptography and Security Programming Languages |
| url | https://arxiv.org/abs/2308.03312 |