Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2503.02857 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913164785876992 |
|---|---|
| author | Chandra, Nuria Alina Lee, Hannah Murtfeldt, Ryan Qiu, Lin Karmakar, Arnab Tanumihardja, Emmanuel Farhat, Kevin Caffee, Ben Lee, Changyeon Choi, Jongwook Paik, Sejin Kim, Aerin Etzioni, Oren |
| author_facet | Chandra, Nuria Alina Lee, Hannah Murtfeldt, Ryan Qiu, Lin Karmakar, Arnab Tanumihardja, Emmanuel Farhat, Kevin Caffee, Ben Lee, Changyeon Choi, Jongwook Paik, Sejin Kim, Aerin Etzioni, Oren |
| contents | In the age of increasingly realistic generative AI, robust deepfake detection is essential for mitigating fraud and disinformation. While many deepfake detectors report high accuracy on academic datasets, we show that these academic benchmarks are out of date and not representative of real-world deepfakes. We introduce Deepfake-Eval-2024, a new deepfake detection benchmark consisting of in-the-wild deepfakes collected from social media and deepfake detection platform users in 2024. Deepfake-Eval-2024 consists of 45 hours of videos, 56.5 hours of audio, and 1,975 images, encompassing the latest manipulation technologies. The benchmark contains diverse media content from 88 different websites in 52 different languages. We find that the performance of open-source state-of-the-art deepfake detection models drops precipitously when evaluated on Deepfake-Eval-2024, with AUC decreasing by 50% for video, 48% for audio, and 45% for image models compared to previous benchmarks. We also evaluate commercial deepfake detection models and models finetuned on Deepfake-Eval-2024, and find that they have superior performance to off-the-shelf open-source models, but do not yet reach the accuracy of deepfake forensic analysts. The dataset is available at https://github.com/nuriachandra/Deepfake-Eval-2024. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_02857 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes Circulated in 2024 Chandra, Nuria Alina Lee, Hannah Murtfeldt, Ryan Qiu, Lin Karmakar, Arnab Tanumihardja, Emmanuel Farhat, Kevin Caffee, Ben Lee, Changyeon Choi, Jongwook Paik, Sejin Kim, Aerin Etzioni, Oren Computer Vision and Pattern Recognition Artificial Intelligence Computers and Society In the age of increasingly realistic generative AI, robust deepfake detection is essential for mitigating fraud and disinformation. While many deepfake detectors report high accuracy on academic datasets, we show that these academic benchmarks are out of date and not representative of real-world deepfakes. We introduce Deepfake-Eval-2024, a new deepfake detection benchmark consisting of in-the-wild deepfakes collected from social media and deepfake detection platform users in 2024. Deepfake-Eval-2024 consists of 45 hours of videos, 56.5 hours of audio, and 1,975 images, encompassing the latest manipulation technologies. The benchmark contains diverse media content from 88 different websites in 52 different languages. We find that the performance of open-source state-of-the-art deepfake detection models drops precipitously when evaluated on Deepfake-Eval-2024, with AUC decreasing by 50% for video, 48% for audio, and 45% for image models compared to previous benchmarks. We also evaluate commercial deepfake detection models and models finetuned on Deepfake-Eval-2024, and find that they have superior performance to off-the-shelf open-source models, but do not yet reach the accuracy of deepfake forensic analysts. The dataset is available at https://github.com/nuriachandra/Deepfake-Eval-2024. |
| title | Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes Circulated in 2024 |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computers and Society |
| url | https://arxiv.org/abs/2503.02857 |