The Elusive Pursuit of Reproducing PATE-GAN: Benchmarking, Auditing, Debugging
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912224862273536 |
|---|---|
| author | Ganev, Georgi Annamalai, Meenatchi Sundaram Muthu Selva De Cristofaro, Emiliano |
| author_facet | Ganev, Georgi Annamalai, Meenatchi Sundaram Muthu Selva De Cristofaro, Emiliano |
| contents | Synthetic data created by differentially private (DP) generative models is increasingly used in real-world settings. In this context, PATE-GAN has emerged as one of the most popular algorithms, combining Generative Adversarial Networks (GANs) with the private training approach of PATE (Private Aggregation of Teacher Ensembles).
In this paper, we set out to reproduce the utility evaluation from the original PATE-GAN paper, compare available implementations, and conduct a privacy audit. More precisely, we analyze and benchmark six open-source PATE-GAN implementations, including three by (a subset of) the original authors. First, we shed light on architecture deviations and empirically demonstrate that none reproduce the utility performance reported in the original paper. We then present an in-depth privacy evaluation, which includes DP auditing, and show that all implementations leak more privacy than intended. Furthermore, we uncover 19 privacy violations and 5 other bugs in these six open-source implementations.
Lastly, our codebase is available from: https://github.com/spalabucr/pategan-audit. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_13985 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Elusive Pursuit of Reproducing PATE-GAN: Benchmarking, Auditing, Debugging Ganev, Georgi Annamalai, Meenatchi Sundaram Muthu Selva De Cristofaro, Emiliano Machine Learning Cryptography and Security Synthetic data created by differentially private (DP) generative models is increasingly used in real-world settings. In this context, PATE-GAN has emerged as one of the most popular algorithms, combining Generative Adversarial Networks (GANs) with the private training approach of PATE (Private Aggregation of Teacher Ensembles). In this paper, we set out to reproduce the utility evaluation from the original PATE-GAN paper, compare available implementations, and conduct a privacy audit. More precisely, we analyze and benchmark six open-source PATE-GAN implementations, including three by (a subset of) the original authors. First, we shed light on architecture deviations and empirically demonstrate that none reproduce the utility performance reported in the original paper. We then present an in-depth privacy evaluation, which includes DP auditing, and show that all implementations leak more privacy than intended. Furthermore, we uncover 19 privacy violations and 5 other bugs in these six open-source implementations. Lastly, our codebase is available from: https://github.com/spalabucr/pategan-audit. |
| title | The Elusive Pursuit of Reproducing PATE-GAN: Benchmarking, Auditing, Debugging |
| topic | Machine Learning Cryptography and Security |
| url | https://arxiv.org/abs/2406.13985 |