Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2509.00072 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918498764062720 |
|---|---|
| author | Zhang, Terry Jingchen Dev, Gopal Wang, Ning Obreiter, Max Pandey, Punya Syon Samway, Keenan Jiang, Wenyuan Huang, Yinya Schölkopf, Bernhard Sachan, Mrinmaya Jin, Zhijing |
| author_facet | Zhang, Terry Jingchen Dev, Gopal Wang, Ning Obreiter, Max Pandey, Punya Syon Samway, Keenan Jiang, Wenyuan Huang, Yinya Schölkopf, Bernhard Sachan, Mrinmaya Jin, Zhijing |
| contents | Post-cutoff performance decay of LLMs has been widely interpreted as a temporal signal for benchmark contamination, where public information released before the training cutoff may have been included into training corpora and inflated model performance by memorization. We critically examine this view and demonstrate that this temporal signal is highly sensitive to how benchmark questions are constructed, even if the underlying source material remains invariant. Specifically, we show that LLM-transformed questions can produce remarkably different temporal patterns compared to fill-in-the-blank (cloze) questions directly retrieved from the very same documents. We validate this effect on prior benchmarks that report clear post-cutoff decay (LiveCodeBench), and show that a simple LLM-driven transformation of the same problems can effectively remove the temporal pattern. We further provide a mechanistic understanding of this phenomenon using influence function analysis. Overall, our results suggest that post-cutoff performance decay is a sensitive contamination signal, motivating more robust contamination probes for reliable LLM evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_00072 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Test of Time: Rethinking Temporal Signal of Benchmark Contamination Zhang, Terry Jingchen Dev, Gopal Wang, Ning Obreiter, Max Pandey, Punya Syon Samway, Keenan Jiang, Wenyuan Huang, Yinya Schölkopf, Bernhard Sachan, Mrinmaya Jin, Zhijing Artificial Intelligence Post-cutoff performance decay of LLMs has been widely interpreted as a temporal signal for benchmark contamination, where public information released before the training cutoff may have been included into training corpora and inflated model performance by memorization. We critically examine this view and demonstrate that this temporal signal is highly sensitive to how benchmark questions are constructed, even if the underlying source material remains invariant. Specifically, we show that LLM-transformed questions can produce remarkably different temporal patterns compared to fill-in-the-blank (cloze) questions directly retrieved from the very same documents. We validate this effect on prior benchmarks that report clear post-cutoff decay (LiveCodeBench), and show that a simple LLM-driven transformation of the same problems can effectively remove the temporal pattern. We further provide a mechanistic understanding of this phenomenon using influence function analysis. Overall, our results suggest that post-cutoff performance decay is a sensitive contamination signal, motivating more robust contamination probes for reliable LLM evaluation. |
| title | Test of Time: Rethinking Temporal Signal of Benchmark Contamination |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2509.00072 |