I recently embarked on a personal side project of attempting to detect whether an image posted online is a deepfake or not. To do so, I created a two-step model inspired by this paper, with an implementation adapted from this paper. Model architecture diagram:
The first part of the model is a Siamese network to detect similarities between real and fake images, and the second is a convolutional neural network with a binary classifier. Believing that I could optimize the model even harder, I used an image preprocessing strategy that involved using OpenCV to draw facial features that help the model discern significant differences between real and deepfake components of the face, such as nose and eyes. Despite the accomplishments of the model described by the latter paper, its architecture is excessively large, and its hyperparameters are not supported by my hardware limitations, so I decided to decrease the batch size to 8 and shift from a "batch all" to a "batch hard" strategy when calculating the triplet loss to make better use of the data provided. Despite this reduction, my execution of the program on Azure Cloud reached a fairly high test accuracy(over 0.7), suggesting that with a hardware that supports a larger batch size and a bigger dataset (like the Facebook Detection Challenge one used by the paper's researchers), similar if not potentially higher (due to the added OpenCV preprocessing step) metrics can be attained.
The file distribution of the project is as follows:
- display.py → used to visualize certain images in the dataset during preprocessing. This allows us to visualize, for example, markings on the nose and eyes among the training images.
- process_data.py → file used to do the preprocessing of the data, which is initially in a folder called "archive" (whose contents are available here: https://www.kaggle.com/datasets/xhlulu/140k-real-and-fake-faces). This file will add facial landmarks to these images and store them in a folder called "data/", following preprocessing steps that resemble those proposed by participants of the Facebook Deepfake Detection Challenge (https://www.kaggle.com/code/robikscube/kaggle-deepfake-detection-introduction)
- functions.py → file containing the functions necessary for main.py to execute properly. These include functions to train, validate, and test the model.
- main.py → main file of the program, which determines when certain steps of the program's execution (such as training and testing) will be run.
- paper.pdf → reference paper for the model's implementation. Describes the model's architecture, most of which was adopted by this implementation. However, hyperparameters such as batch size and learning rate were modified in my implementation. Furthermore, my preprocessing steps differed entirely from the paper at hand and instead involved using OpenCV to draw facial landmarks.
- unzip.py → script used to unzip the contents of the "data/" folder, used for training the model in a cloud computing environment where uploading a large number of files is not recommended.