Measuring Structural Limitations of Deepfake Media Datasets - A Systematic Review

Watch a 13-minute conference presentation from USENIX Security '24 examining the structural limitations of deepfake media datasets. Explore the first systematic analysis of deepfake media, comparing traditional anomaly detection datasets to characterize metrics, generation techniques, and class distributions. Learn about significant challenges affecting system comparability, including unaddressed class imbalance and limited metric usage. Understand the real-world implications through a case study demonstrating how current best detection systems would perform poorly in practical scenarios like call centers. Discover recommendations for improving dataset construction, implementing better result reporting templates, and advancing the field through model score file releases. Gain insights from University of Florida researchers on enhancing reproducibility and establishing more meaningful comparisons in deepfake detection research.