Code Generation with Small Language Models: A Codeforces-Based Study
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911164385984512 |
|---|---|
| author | Souza, Débora Gheyi, Rohit Albuquerque, Lucas Soares, Gustavo Ribeiro, Márcio |
| author_facet | Souza, Débora Gheyi, Rohit Albuquerque, Lucas Soares, Gustavo Ribeiro, Márcio |
| contents | Large Language Models (LLMs) demonstrate capabilities in code generation, potentially boosting developer productivity. However, their adoption remains limited by high computational costs, among other factors. Small Language Models (SLMs) present a lightweight alternative. While LLMs have been evaluated on competitive programming tasks, prior work often emphasizes metrics like Elo or pass rates, neglecting failure analysis. The potential of SLMs in this space remains underexplored. In this study, we benchmark three open SLMs - Llama-3.2-3B, Gemma-3-12B, and Phi-4-14B - across 280 Codeforces problems spanning Elo ratings from 800 to 2100 and covering 36 distinct topics. All models were tasked with generating Python solutions. Phi-4-14B achieved the best SLM performance with a pass@3 of 63.6%, nearing o3-mini-high (86.8%). Combining Python and C++ outputs increased Phi-4-14B's pass@6 to 73.6%. A qualitative analysis revealed some failures stemmed from minor implementation issues rather than reasoning flaws. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_07343 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Code Generation with Small Language Models: A Codeforces-Based Study Souza, Débora Gheyi, Rohit Albuquerque, Lucas Soares, Gustavo Ribeiro, Márcio Software Engineering Large Language Models (LLMs) demonstrate capabilities in code generation, potentially boosting developer productivity. However, their adoption remains limited by high computational costs, among other factors. Small Language Models (SLMs) present a lightweight alternative. While LLMs have been evaluated on competitive programming tasks, prior work often emphasizes metrics like Elo or pass rates, neglecting failure analysis. The potential of SLMs in this space remains underexplored. In this study, we benchmark three open SLMs - Llama-3.2-3B, Gemma-3-12B, and Phi-4-14B - across 280 Codeforces problems spanning Elo ratings from 800 to 2100 and covering 36 distinct topics. All models were tasked with generating Python solutions. Phi-4-14B achieved the best SLM performance with a pass@3 of 63.6%, nearing o3-mini-high (86.8%). Combining Python and C++ outputs increased Phi-4-14B's pass@6 to 73.6%. A qualitative analysis revealed some failures stemmed from minor implementation issues rather than reasoning flaws. |
| title | Code Generation with Small Language Models: A Codeforces-Based Study |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2504.07343 |