RADAR: Mechanistic Pathways for Detecting Data Contamination in LLM Evaluation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909833841606656 |
|---|---|
| author | Kattamuri, Ashish Fartale, Harshwardhan Vats, Arpita Raja, Rahul Prasad, Ishita |
| author_facet | Kattamuri, Ashish Fartale, Harshwardhan Vats, Arpita Raja, Rahul Prasad, Ishita |
| contents | Data contamination poses a significant challenge to reliable LLM evaluation, where models may achieve high performance by memorizing training data rather than demonstrating genuine reasoning capabilities. We introduce RADAR (Recall vs. Reasoning Detection through Activation Representation), a novel framework that leverages mechanistic interpretability to detect contamination by distinguishing recall-based from reasoning-based model responses. RADAR extracts 37 features spanning surface-level confidence trajectories and deep mechanistic properties including attention specialization, circuit dynamics, and activation flow patterns. Using an ensemble of classifiers trained on these features, RADAR achieves 93\% accuracy on a diverse evaluation set, with perfect performance on clear cases and 76.7\% accuracy on challenging ambiguous examples. This work demonstrates the potential of mechanistic interpretability for advancing LLM evaluation beyond traditional surface-level metrics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_08931 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RADAR: Mechanistic Pathways for Detecting Data Contamination in LLM Evaluation Kattamuri, Ashish Fartale, Harshwardhan Vats, Arpita Raja, Rahul Prasad, Ishita Artificial Intelligence Machine Learning Data contamination poses a significant challenge to reliable LLM evaluation, where models may achieve high performance by memorizing training data rather than demonstrating genuine reasoning capabilities. We introduce RADAR (Recall vs. Reasoning Detection through Activation Representation), a novel framework that leverages mechanistic interpretability to detect contamination by distinguishing recall-based from reasoning-based model responses. RADAR extracts 37 features spanning surface-level confidence trajectories and deep mechanistic properties including attention specialization, circuit dynamics, and activation flow patterns. Using an ensemble of classifiers trained on these features, RADAR achieves 93\% accuracy on a diverse evaluation set, with perfect performance on clear cases and 76.7\% accuracy on challenging ambiguous examples. This work demonstrates the potential of mechanistic interpretability for advancing LLM evaluation beyond traditional surface-level metrics. |
| title | RADAR: Mechanistic Pathways for Detecting Data Contamination in LLM Evaluation |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2510.08931 |