From Data Leak to Secret Misses: The Impact of Data Leakage on Secret Detection Models
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910006139420672 |
|---|---|
| author | Soltaniani, Farnaz Ghafari, Mohammad |
| author_facet | Soltaniani, Farnaz Ghafari, Mohammad |
| contents | Machine learning models are increasingly used for software security tasks. These models are commonly trained and evaluated on large Internet-derived datasets, which often contain duplicated or highly similar samples. When such samples are split across training and test sets, data leakage may occur, allowing models to memorize patterns instead of learning to generalize. We investigate duplication in a widely used benchmark dataset of hard coded secrets and show how data leakage can substantially inflate the reported performance of AI-based secret detectors, resulting in a misleading picture of their real-world effectiveness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22946 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | From Data Leak to Secret Misses: The Impact of Data Leakage on Secret Detection Models Soltaniani, Farnaz Ghafari, Mohammad Cryptography and Security Artificial Intelligence Machine Learning Machine learning models are increasingly used for software security tasks. These models are commonly trained and evaluated on large Internet-derived datasets, which often contain duplicated or highly similar samples. When such samples are split across training and test sets, data leakage may occur, allowing models to memorize patterns instead of learning to generalize. We investigate duplication in a widely used benchmark dataset of hard coded secrets and show how data leakage can substantially inflate the reported performance of AI-based secret detectors, resulting in a misleading picture of their real-world effectiveness. |
| title | From Data Leak to Secret Misses: The Impact of Data Leakage on Secret Detection Models |
| topic | Cryptography and Security Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2601.22946 |