Comparative Analysis of Large Language Models for Context-Aware Code Completion using SAFIM Framework
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910838000975872 |
|---|---|
| author | Zhang, Hang Shen, Yanxin Wang, Lun Shi, Chuanqi Du, Shaoshuai Tao, Yiyi Shen, Yixian |
| author_facet | Zhang, Hang Shen, Yanxin Wang, Lun Shi, Chuanqi Du, Shaoshuai Tao, Yiyi Shen, Yixian |
| contents | The advent of Large Language Models (LLMs) has revolutionized code completion, transforming it into a more intelligent and context-aware feature in modern integrated development environments. These advancements have significantly enhanced developers' ability to write efficient and error-free code. This study evaluates the performance of several chat-based LLMs, including Gemini 1.5 Flash, Gemini 1.5 Pro, GPT-4o, GPT-4o-mini, and GPT-4 Turbo, using the Syntax-Aware Fill-in-the-Middle (SAFIM) dataset. This benchmark is specifically designed to assess models' capabilities in syntax-sensitive code generation. Performance metrics, such as cosine similarity with ground-truth completions and latency, were employed to measure both accuracy and efficiency. The findings reveal substantial differences in the models' code completion abilities, offering valuable insights into their respective strengths and weaknesses. This work provides a comparative analysis that underscores the trade-offs between accuracy and speed, establishing a benchmark for future advancements in LLM-based code completion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_15243 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Comparative Analysis of Large Language Models for Context-Aware Code Completion using SAFIM Framework Zhang, Hang Shen, Yanxin Wang, Lun Shi, Chuanqi Du, Shaoshuai Tao, Yiyi Shen, Yixian Software Engineering Artificial Intelligence The advent of Large Language Models (LLMs) has revolutionized code completion, transforming it into a more intelligent and context-aware feature in modern integrated development environments. These advancements have significantly enhanced developers' ability to write efficient and error-free code. This study evaluates the performance of several chat-based LLMs, including Gemini 1.5 Flash, Gemini 1.5 Pro, GPT-4o, GPT-4o-mini, and GPT-4 Turbo, using the Syntax-Aware Fill-in-the-Middle (SAFIM) dataset. This benchmark is specifically designed to assess models' capabilities in syntax-sensitive code generation. Performance metrics, such as cosine similarity with ground-truth completions and latency, were employed to measure both accuracy and efficiency. The findings reveal substantial differences in the models' code completion abilities, offering valuable insights into their respective strengths and weaknesses. This work provides a comparative analysis that underscores the trade-offs between accuracy and speed, establishing a benchmark for future advancements in LLM-based code completion. |
| title | Comparative Analysis of Large Language Models for Context-Aware Code Completion using SAFIM Framework |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2502.15243 |