LONGCODEU: Benchmarking Long-Context Language Models on Long Code Understanding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910861497466880 |
|---|---|
| author | Li, Jia Guo, Xuyuan Li, Lei Zhang, Kechi Li, Ge Li, Jia Tao, Zhengwei Liu, Fang Tao, Chongyang Zhu, Yuqi Jin, Zhi |
| author_facet | Li, Jia Guo, Xuyuan Li, Lei Zhang, Kechi Li, Ge Li, Jia Tao, Zhengwei Liu, Fang Tao, Chongyang Zhu, Yuqi Jin, Zhi |
| contents | Current advanced long-context language models offer great potential for real-world software engineering applications. However, progress in this critical domain remains hampered by a fundamental limitation: the absence of a rigorous evaluation framework for long code understanding. To gap this obstacle, we propose a long code understanding benchmark LONGCODEU from four aspects (8 tasks) to evaluate LCLMs' long code understanding ability required for practical applications, including code unit perception, intra-code unit understanding, inter-code unit relation understanding, and long code documentation understanding. We evaluate 9 popular LCLMs on LONGCODEU (i.e., 6 general models and 3 code models). Our experimental results reveal key limitations in current LCLMs' capabilities for long code understanding. Particularly, the performance of LCLMs drops dramatically when the long code length is greater than 32K, falling far short of their claimed 128K-1M context windows. In the four aspects, inter-code unit relation understanding is the most challenging for LCLMs. Our study provides valuable insights for optimizing LCLMs and driving advancements in software engineering. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_04359 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LONGCODEU: Benchmarking Long-Context Language Models on Long Code Understanding Li, Jia Guo, Xuyuan Li, Lei Zhang, Kechi Li, Ge Li, Jia Tao, Zhengwei Liu, Fang Tao, Chongyang Zhu, Yuqi Jin, Zhi Software Engineering Current advanced long-context language models offer great potential for real-world software engineering applications. However, progress in this critical domain remains hampered by a fundamental limitation: the absence of a rigorous evaluation framework for long code understanding. To gap this obstacle, we propose a long code understanding benchmark LONGCODEU from four aspects (8 tasks) to evaluate LCLMs' long code understanding ability required for practical applications, including code unit perception, intra-code unit understanding, inter-code unit relation understanding, and long code documentation understanding. We evaluate 9 popular LCLMs on LONGCODEU (i.e., 6 general models and 3 code models). Our experimental results reveal key limitations in current LCLMs' capabilities for long code understanding. Particularly, the performance of LCLMs drops dramatically when the long code length is greater than 32K, falling far short of their claimed 128K-1M context windows. In the four aspects, inter-code unit relation understanding is the most challenging for LCLMs. Our study provides valuable insights for optimizing LCLMs and driving advancements in software engineering. |
| title | LONGCODEU: Benchmarking Long-Context Language Models on Long Code Understanding |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2503.04359 |