Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914843427078144 |
|---|---|
| author | Deng, Chunyuan Zhao, Yilun Heng, Yuzhao Li, Yitong Cao, Jiannan Tang, Xiangru Cohan, Arman |
| author_facet | Deng, Chunyuan Zhao, Yilun Heng, Yuzhao Li, Yitong Cao, Jiannan Tang, Xiangru Cohan, Arman |
| contents | Data contamination has garnered increased attention in the era of large language models (LLMs) due to the reliance on extensive internet-derived training corpora. The issue of training corpus overlap with evaluation benchmarks--referred to as contamination--has been the focus of significant recent research. This body of work aims to identify contamination, understand its impacts, and explore mitigation strategies from diverse perspectives. However, comprehensive studies that provide a clear pathway from foundational concepts to advanced insights are lacking in this nascent field. Therefore, we present a comprehensive survey in the field of data contamination, laying out the key issues, methodologies, and findings to date, and highlighting areas in need of further research and development. In particular, we begin by examining the effects of data contamination across various stages and forms. We then provide a detailed analysis of current contamination detection methods, categorizing them to highlight their focus, assumptions, strengths, and limitations. We also discuss mitigation strategies, offering a clear guide for future research. This survey serves as a succinct overview of the most recent advancements in data contamination research, providing a straightforward guide for the benefit of future research endeavors. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14644 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediation Deng, Chunyuan Zhao, Yilun Heng, Yuzhao Li, Yitong Cao, Jiannan Tang, Xiangru Cohan, Arman Computation and Language Data contamination has garnered increased attention in the era of large language models (LLMs) due to the reliance on extensive internet-derived training corpora. The issue of training corpus overlap with evaluation benchmarks--referred to as contamination--has been the focus of significant recent research. This body of work aims to identify contamination, understand its impacts, and explore mitigation strategies from diverse perspectives. However, comprehensive studies that provide a clear pathway from foundational concepts to advanced insights are lacking in this nascent field. Therefore, we present a comprehensive survey in the field of data contamination, laying out the key issues, methodologies, and findings to date, and highlighting areas in need of further research and development. In particular, we begin by examining the effects of data contamination across various stages and forms. We then provide a detailed analysis of current contamination detection methods, categorizing them to highlight their focus, assumptions, strengths, and limitations. We also discuss mitigation strategies, offering a clear guide for future research. This survey serves as a succinct overview of the most recent advancements in data contamination research, providing a straightforward guide for the benefit of future research endeavors. |
| title | Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.14644 |