LoCoBench: A Benchmark for Long-Context Large Language Models in Complex Software Engineering
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908533352562688 |
|---|---|
| author | Qiu, Jielin Liu, Zuxin Liu, Zhiwei Murthy, Rithesh Zhang, Jianguo Chen, Haolin Wang, Shiyu Zhu, Ming Yang, Liangwei Tan, Juntao Cen, Zhepeng Qian, Cheng Heinecke, Shelby Yao, Weiran Savarese, Silvio Xiong, Caiming Wang, Huan |
| author_facet | Qiu, Jielin Liu, Zuxin Liu, Zhiwei Murthy, Rithesh Zhang, Jianguo Chen, Haolin Wang, Shiyu Zhu, Ming Yang, Liangwei Tan, Juntao Cen, Zhepeng Qian, Cheng Heinecke, Shelby Yao, Weiran Savarese, Silvio Xiong, Caiming Wang, Huan |
| contents | The emergence of long-context language models with context windows extending to millions of tokens has created new opportunities for sophisticated code understanding and software development evaluation. We propose LoCoBench, a comprehensive benchmark specifically designed to evaluate long-context LLMs in realistic, complex software development scenarios. Unlike existing code evaluation benchmarks that focus on single-function completion or short-context tasks, LoCoBench addresses the critical evaluation gap for long-context capabilities that require understanding entire codebases, reasoning across multiple files, and maintaining architectural consistency across large-scale software systems. Our benchmark provides 8,000 evaluation scenarios systematically generated across 10 programming languages, with context lengths spanning 10K to 1M tokens, a 100x variation that enables precise assessment of long-context performance degradation in realistic software development settings. LoCoBench introduces 8 task categories that capture essential long-context capabilities: architectural understanding, cross-file refactoring, multi-session development, bug investigation, feature implementation, code comprehension, integration testing, and security analysis. Through a 5-phase pipeline, we create diverse, high-quality scenarios that challenge LLMs to reason about complex codebases at unprecedented scale. We introduce a comprehensive evaluation framework with 17 metrics across 4 dimensions, including 8 new evaluation metrics, combined in a LoCoBench Score (LCBS). Our evaluation of state-of-the-art long-context models reveals substantial performance gaps, demonstrating that long-context understanding in complex software development represents a significant unsolved challenge that demands more attention. LoCoBench is released at: https://github.com/SalesforceAIResearch/LoCoBench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_09614 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LoCoBench: A Benchmark for Long-Context Large Language Models in Complex Software Engineering Qiu, Jielin Liu, Zuxin Liu, Zhiwei Murthy, Rithesh Zhang, Jianguo Chen, Haolin Wang, Shiyu Zhu, Ming Yang, Liangwei Tan, Juntao Cen, Zhepeng Qian, Cheng Heinecke, Shelby Yao, Weiran Savarese, Silvio Xiong, Caiming Wang, Huan Software Engineering Artificial Intelligence The emergence of long-context language models with context windows extending to millions of tokens has created new opportunities for sophisticated code understanding and software development evaluation. We propose LoCoBench, a comprehensive benchmark specifically designed to evaluate long-context LLMs in realistic, complex software development scenarios. Unlike existing code evaluation benchmarks that focus on single-function completion or short-context tasks, LoCoBench addresses the critical evaluation gap for long-context capabilities that require understanding entire codebases, reasoning across multiple files, and maintaining architectural consistency across large-scale software systems. Our benchmark provides 8,000 evaluation scenarios systematically generated across 10 programming languages, with context lengths spanning 10K to 1M tokens, a 100x variation that enables precise assessment of long-context performance degradation in realistic software development settings. LoCoBench introduces 8 task categories that capture essential long-context capabilities: architectural understanding, cross-file refactoring, multi-session development, bug investigation, feature implementation, code comprehension, integration testing, and security analysis. Through a 5-phase pipeline, we create diverse, high-quality scenarios that challenge LLMs to reason about complex codebases at unprecedented scale. We introduce a comprehensive evaluation framework with 17 metrics across 4 dimensions, including 8 new evaluation metrics, combined in a LoCoBench Score (LCBS). Our evaluation of state-of-the-art long-context models reveals substantial performance gaps, demonstrating that long-context understanding in complex software development represents a significant unsolved challenge that demands more attention. LoCoBench is released at: https://github.com/SalesforceAIResearch/LoCoBench. |
| title | LoCoBench: A Benchmark for Long-Context Large Language Models in Complex Software Engineering |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2509.09614 |