Gumbel Rao Monte Carlo based Bi-Modal Neural Architecture Search for Audio-Visual Deepfake Detection

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	PN, Aravinda Reddy, Ramachandra, Raghavendra, Rao, Krothapalli Sreenivasa, Rathod, Pabitra Mitra Vinod
Format:	Preprint
Published:	2024
Subjects:	Cryptography and Security Sound Audio and Speech Processing
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866909342988500992
author	PN, Aravinda Reddy Ramachandra, Raghavendra Rao, Krothapalli Sreenivasa Rathod, Pabitra Mitra Vinod
author_facet	PN, Aravinda Reddy Ramachandra, Raghavendra Rao, Krothapalli Sreenivasa Rathod, Pabitra Mitra Vinod
contents	Deepfakes pose a critical threat to biometric authentication systems by generating highly realistic synthetic media. Existing multimodal deepfake detectors often struggle to adapt to diverse data and rely on simple fusion methods. To address these challenges, we propose Gumbel-Rao Monte Carlo Bi-modal Neural Architecture Search (GRMC-BMNAS), a novel architecture search framework that employs Gumbel-Rao Monte Carlo sampling to optimize multimodal fusion. It refines the Straight through Gumbel Softmax (STGS) method by reducing variance with Rao-Blackwellization, stabilizing network training. Using a two-level search approach, the framework optimizes the network architecture, parameters, and performance. Crucial features are efficiently identified from backbone networks, while within the cell structure, a weighted fusion operation integrates information from various sources. By varying parameters such as temperature and number of Monte carlo samples yields an architecture that maximizes classification performance and better generalisation capability. Experimental results on the FakeAVCeleb and SWAN-DF datasets demonstrate an impressive AUC percentage of 95.4\%, achieved with minimal model parameters.
format	Preprint
id	arxiv_https___arxiv_org_abs_2410_06543
institution	arXiv
publishDate	2024
record_format	arxiv
spellingShingle	Gumbel Rao Monte Carlo based Bi-Modal Neural Architecture Search for Audio-Visual Deepfake Detection PN, Aravinda Reddy Ramachandra, Raghavendra Rao, Krothapalli Sreenivasa Rathod, Pabitra Mitra Vinod Cryptography and Security Sound Audio and Speech Processing Deepfakes pose a critical threat to biometric authentication systems by generating highly realistic synthetic media. Existing multimodal deepfake detectors often struggle to adapt to diverse data and rely on simple fusion methods. To address these challenges, we propose Gumbel-Rao Monte Carlo Bi-modal Neural Architecture Search (GRMC-BMNAS), a novel architecture search framework that employs Gumbel-Rao Monte Carlo sampling to optimize multimodal fusion. It refines the Straight through Gumbel Softmax (STGS) method by reducing variance with Rao-Blackwellization, stabilizing network training. Using a two-level search approach, the framework optimizes the network architecture, parameters, and performance. Crucial features are efficiently identified from backbone networks, while within the cell structure, a weighted fusion operation integrates information from various sources. By varying parameters such as temperature and number of Monte carlo samples yields an architecture that maximizes classification performance and better generalisation capability. Experimental results on the FakeAVCeleb and SWAN-DF datasets demonstrate an impressive AUC percentage of 95.4\%, achieved with minimal model parameters.
title	Gumbel Rao Monte Carlo based Bi-Modal Neural Architecture Search for Audio-Visual Deepfake Detection
topic	Cryptography and Security Sound Audio and Speech Processing
url	https://arxiv.org/abs/2410.06543