On Leakage of Code Generation Evaluation Datasets
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914963243663360 |
|---|---|
| author | Matton, Alexandre Sherborne, Tom Aumiller, Dennis Tommasone, Elena Alizadeh, Milad He, Jingyi Ma, Raymond Voisin, Maxime Gilsenan-McMahon, Ellen Gallé, Matthias |
| author_facet | Matton, Alexandre Sherborne, Tom Aumiller, Dennis Tommasone, Elena Alizadeh, Milad He, Jingyi Ma, Raymond Voisin, Maxime Gilsenan-McMahon, Ellen Gallé, Matthias |
| contents | In this paper, we consider contamination by code generation test sets, in particular in their use in modern large language models. We discuss three possible sources of such contamination and show findings supporting each of them: (i) direct data leakage, (ii) indirect data leakage through the use of synthetic data and (iii) overfitting to evaluation sets during model selection. To address this, we release Less Basic Python Problems (LBPP): an uncontaminated new benchmark of 161 prompts with their associated Python solutions. LBPP is released at https://huggingface.co/datasets/CohereForAI/lbpp . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_07565 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On Leakage of Code Generation Evaluation Datasets Matton, Alexandre Sherborne, Tom Aumiller, Dennis Tommasone, Elena Alizadeh, Milad He, Jingyi Ma, Raymond Voisin, Maxime Gilsenan-McMahon, Ellen Gallé, Matthias Computation and Language In this paper, we consider contamination by code generation test sets, in particular in their use in modern large language models. We discuss three possible sources of such contamination and show findings supporting each of them: (i) direct data leakage, (ii) indirect data leakage through the use of synthetic data and (iii) overfitting to evaluation sets during model selection. To address this, we release Less Basic Python Problems (LBPP): an uncontaminated new benchmark of 161 prompts with their associated Python solutions. LBPP is released at https://huggingface.co/datasets/CohereForAI/lbpp . |
| title | On Leakage of Code Generation Evaluation Datasets |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.07565 |