When Many-Shot Prompting Fails: An Empirical Study of LLM Code Translation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908701156179968 |
|---|---|
| author | Oskooei, Amirkia Rafiei Cosdan, Kaan Baturalp Isiktas, Husamettin Aktas, Mehmet S. |
| author_facet | Oskooei, Amirkia Rafiei Cosdan, Kaan Baturalp Isiktas, Husamettin Aktas, Mehmet S. |
| contents | Large Language Models (LLMs) with vast context windows offer new avenues for in-context learning (ICL), where providing many examples ("many-shot" prompting) is often assumed to enhance performance. We investigate this assumption for the complex task of code translation. Through a large-scale empirical study of over 90,000 translations, we systematically evaluate the impact of scaling in-context examples from zero-shot to many-shot configurations of up to 625 examples, with prompts spanning from approximately 100,000 to 800,000 tokens. Our findings reveal a "many-shot paradox": while static similarity metrics may modestly improve with more examples, functional correctness consistently peaks with few-shot prompting (5-25 examples). Providing substantially more examples often degrades this crucial functional performance. This study highlights that for code translation, the quality of a few well-chosen examples outweighs sheer quantity, challenging the universal efficacy of "more is better" for ICL and underscoring the task-dependent nature of optimal prompting strategies. Our results have significant implications for effectively leveraging LLMs in software engineering. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_16809 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | When Many-Shot Prompting Fails: An Empirical Study of LLM Code Translation Oskooei, Amirkia Rafiei Cosdan, Kaan Baturalp Isiktas, Husamettin Aktas, Mehmet S. Software Engineering Artificial Intelligence Computation and Language Programming Languages 68T50, 68N30, 68W40 I.2.7; D.2.7; I.2.6 Large Language Models (LLMs) with vast context windows offer new avenues for in-context learning (ICL), where providing many examples ("many-shot" prompting) is often assumed to enhance performance. We investigate this assumption for the complex task of code translation. Through a large-scale empirical study of over 90,000 translations, we systematically evaluate the impact of scaling in-context examples from zero-shot to many-shot configurations of up to 625 examples, with prompts spanning from approximately 100,000 to 800,000 tokens. Our findings reveal a "many-shot paradox": while static similarity metrics may modestly improve with more examples, functional correctness consistently peaks with few-shot prompting (5-25 examples). Providing substantially more examples often degrades this crucial functional performance. This study highlights that for code translation, the quality of a few well-chosen examples outweighs sheer quantity, challenging the universal efficacy of "more is better" for ICL and underscoring the task-dependent nature of optimal prompting strategies. Our results have significant implications for effectively leveraging LLMs in software engineering. |
| title | When Many-Shot Prompting Fails: An Empirical Study of LLM Code Translation |
| topic | Software Engineering Artificial Intelligence Computation and Language Programming Languages 68T50, 68N30, 68W40 I.2.7; D.2.7; I.2.6 |
| url | https://arxiv.org/abs/2510.16809 |