Nova: Generative Language Models for Assembly Code with Hierarchical Attention and Contrastive Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911258642481152 |
|---|---|
| author | Jiang, Nan Wang, Chengxiao Liu, Kevin Xu, Xiangzhe Tan, Lin Zhang, Xiangyu Babkin, Petr |
| author_facet | Jiang, Nan Wang, Chengxiao Liu, Kevin Xu, Xiangzhe Tan, Lin Zhang, Xiangyu Babkin, Petr |
| contents | Binary code analysis is the foundation of crucial tasks in the security domain; thus building effective binary analysis techniques is more important than ever. Large language models (LLMs) although have brought impressive improvement to source code tasks, do not directly generalize to assembly code due to the unique challenges of assembly: (1) the low information density of assembly and (2) the diverse optimizations in assembly code. To overcome these challenges, this work proposes a hierarchical attention mechanism that builds attention summaries to capture the semantics more effectively and designs contrastive learning objectives to train LLMs to learn assembly optimization. Equipped with these techniques, this work develops Nova, a generative LLM for assembly code. Nova outperforms existing techniques on binary code decompilation by up to 14.84 -- 21.58% (absolute percentage point improvement) higher Pass@1 and Pass@10, and outperforms the latest binary code similarity detection techniques by up to 6.17% Recall@1, showing promising abilities on both assembly generation and understanding tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_13721 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Nova: Generative Language Models for Assembly Code with Hierarchical Attention and Contrastive Learning Jiang, Nan Wang, Chengxiao Liu, Kevin Xu, Xiangzhe Tan, Lin Zhang, Xiangyu Babkin, Petr Software Engineering Artificial Intelligence Binary code analysis is the foundation of crucial tasks in the security domain; thus building effective binary analysis techniques is more important than ever. Large language models (LLMs) although have brought impressive improvement to source code tasks, do not directly generalize to assembly code due to the unique challenges of assembly: (1) the low information density of assembly and (2) the diverse optimizations in assembly code. To overcome these challenges, this work proposes a hierarchical attention mechanism that builds attention summaries to capture the semantics more effectively and designs contrastive learning objectives to train LLMs to learn assembly optimization. Equipped with these techniques, this work develops Nova, a generative LLM for assembly code. Nova outperforms existing techniques on binary code decompilation by up to 14.84 -- 21.58% (absolute percentage point improvement) higher Pass@1 and Pass@10, and outperforms the latest binary code similarity detection techniques by up to 6.17% Recall@1, showing promising abilities on both assembly generation and understanding tasks. |
| title | Nova: Generative Language Models for Assembly Code with Hierarchical Attention and Contrastive Learning |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2311.13721 |