Is Your Benchmark (Still) Useful? Dynamic Benchmarking for Code Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909532379152384 |
|---|---|
| author | Guan, Batu Wu, Xiao Yuan, Yuanyuan Li, Shaohua |
| author_facet | Guan, Batu Wu, Xiao Yuan, Yuanyuan Li, Shaohua |
| contents | In this paper, we tackle a critical challenge in model evaluation: how to keep code benchmarks useful when models might have already seen them during training. We introduce a novel solution, dynamic benchmarking framework, to address this challenge. Given a code understanding or reasoning benchmark, our framework dynamically transforms each input, i.e., programs, with various semantic-preserving mutations to build a syntactically new while semantically identical benchmark. We evaluated ten popular language models on our dynamic benchmarks. Our evaluation reveals several interesting or surprising findings: (1) all models perform significantly worse than before, (2) the ranking between some models shifts dramatically, and (3) our dynamic benchmarks can resist against the data contamination problem. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_06643 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Is Your Benchmark (Still) Useful? Dynamic Benchmarking for Code Language Models Guan, Batu Wu, Xiao Yuan, Yuanyuan Li, Shaohua Software Engineering Computation and Language In this paper, we tackle a critical challenge in model evaluation: how to keep code benchmarks useful when models might have already seen them during training. We introduce a novel solution, dynamic benchmarking framework, to address this challenge. Given a code understanding or reasoning benchmark, our framework dynamically transforms each input, i.e., programs, with various semantic-preserving mutations to build a syntactically new while semantically identical benchmark. We evaluated ten popular language models on our dynamic benchmarks. Our evaluation reveals several interesting or surprising findings: (1) all models perform significantly worse than before, (2) the ranking between some models shifts dramatically, and (3) our dynamic benchmarks can resist against the data contamination problem. |
| title | Is Your Benchmark (Still) Useful? Dynamic Benchmarking for Code Language Models |
| topic | Software Engineering Computation and Language |
| url | https://arxiv.org/abs/2503.06643 |