Automatic Calibration for Membership Inference Attack on Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915275401592832 |
|---|---|
| author | Zade, Saleh Zare Qiang, Yao Zhou, Xiangyu Zhu, Hui Roshani, Mohammad Amin Khanduri, Prashant Zhu, Dongxiao |
| author_facet | Zade, Saleh Zare Qiang, Yao Zhou, Xiangyu Zhu, Hui Roshani, Mohammad Amin Khanduri, Prashant Zhu, Dongxiao |
| contents | Membership Inference Attacks (MIAs) have recently been employed to determine whether a specific text was part of the pre-training data of Large Language Models (LLMs). However, existing methods often misinfer non-members as members, leading to a high false positive rate, or depend on additional reference models for probability calibration, which limits their practicality. To overcome these challenges, we introduce a novel framework called Automatic Calibration Membership Inference Attack (ACMIA), which utilizes a tunable temperature to calibrate output probabilities effectively. This approach is inspired by our theoretical insights into maximum likelihood estimation during the pre-training of LLMs. We introduce ACMIA in three configurations designed to accommodate different levels of model access and increase the probability gap between members and non-members, improving the reliability and robustness of membership inference. Extensive experiments on various open-source LLMs demonstrate that our proposed attack is highly effective, robust, and generalizable, surpassing state-of-the-art baselines across three widely used benchmarks. Our code is available at: \href{https://github.com/Salehzz/ACMIA}{\textcolor{blue}{Github}}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_03392 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Automatic Calibration for Membership Inference Attack on Large Language Models Zade, Saleh Zare Qiang, Yao Zhou, Xiangyu Zhu, Hui Roshani, Mohammad Amin Khanduri, Prashant Zhu, Dongxiao Machine Learning Artificial Intelligence Membership Inference Attacks (MIAs) have recently been employed to determine whether a specific text was part of the pre-training data of Large Language Models (LLMs). However, existing methods often misinfer non-members as members, leading to a high false positive rate, or depend on additional reference models for probability calibration, which limits their practicality. To overcome these challenges, we introduce a novel framework called Automatic Calibration Membership Inference Attack (ACMIA), which utilizes a tunable temperature to calibrate output probabilities effectively. This approach is inspired by our theoretical insights into maximum likelihood estimation during the pre-training of LLMs. We introduce ACMIA in three configurations designed to accommodate different levels of model access and increase the probability gap between members and non-members, improving the reliability and robustness of membership inference. Extensive experiments on various open-source LLMs demonstrate that our proposed attack is highly effective, robust, and generalizable, surpassing state-of-the-art baselines across three widely used benchmarks. Our code is available at: \href{https://github.com/Salehzz/ACMIA}{\textcolor{blue}{Github}}. |
| title | Automatic Calibration for Membership Inference Attack on Large Language Models |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2505.03392 |