LEANCODE: Understanding Models Better for Code Simplification of Pre-trained Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917248406388736 |
|---|---|
| author | Wang, Yan Ding, Ling Nguyen, Tien N Wang, Shaohua Zheng, Yanan |
| author_facet | Wang, Yan Ding, Ling Nguyen, Tien N Wang, Shaohua Zheng, Yanan |
| contents | Large Language Models for code often entail significant computational complexity, which grows significantly with the length of the input code sequence. We propose LeanCode for code simplification to reduce training and prediction time, leveraging code contexts in utilizing attention scores to represent the tokens' importance. We advocate for the selective removal of tokens based on the average context-aware attention scores rather than average scores across all inputs. LeanCode uses the attention scores of `CLS' tokens within the encoder for classification tasks, such as code search. It also employs the encoder-decoder attention scores to determine token significance for sequence-to-sequence tasks like code summarization. Our evaluation shows LeanCode's superiority over the SOTAs DietCode and Slimcode, with improvements of 60% and 16% for code search, and 29% and 27% for code summarization, respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_14759 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LEANCODE: Understanding Models Better for Code Simplification of Pre-trained Large Language Models Wang, Yan Ding, Ling Nguyen, Tien N Wang, Shaohua Zheng, Yanan Software Engineering Machine Learning Large Language Models for code often entail significant computational complexity, which grows significantly with the length of the input code sequence. We propose LeanCode for code simplification to reduce training and prediction time, leveraging code contexts in utilizing attention scores to represent the tokens' importance. We advocate for the selective removal of tokens based on the average context-aware attention scores rather than average scores across all inputs. LeanCode uses the attention scores of `CLS' tokens within the encoder for classification tasks, such as code search. It also employs the encoder-decoder attention scores to determine token significance for sequence-to-sequence tasks like code summarization. Our evaluation shows LeanCode's superiority over the SOTAs DietCode and Slimcode, with improvements of 60% and 16% for code search, and 29% and 27% for code summarization, respectively. |
| title | LEANCODE: Understanding Models Better for Code Simplification of Pre-trained Large Language Models |
| topic | Software Engineering Machine Learning |
| url | https://arxiv.org/abs/2505.14759 |