Deepfake video deception detection using visual attention-based method

Abstract

The key objective of producing artificial digital data is to closely mimic real data. However, because of improper use by malevolent users, the legitimacy of this kind of digital content may be under threat in society. Deepfake techniques, which replace a person’s face with another using computer vision and graphics, have caused a great deal of anxiety. These techniques make it simple to conceal someone’s genuine identity, which highlights the necessity of having a way to confirm the legitimacy of face photos and videos. To address this difficulty, we have created a model based on deep learning that is enriched with a strategy of visual attention to distinguish real images and videos from ones that have been altered using deep-fake techniques. To create feature maps, we first extract the facial area from video frames then run it through a ResNeXt-50, CNN that has already been trained. Next, we focus on detecting artifacts unique to deepfake video modification by utilizing the visual attention mechanism. Our model outperformed the others when evaluated under cross-dataset settings using Face Forensic++ C23 for training and Celeb-DFv2/ DFDC for independent testing.

Introduction

During the previous few years, the progress of generative adversarial networks (GANs)¹ has created an abundance of new prospects for research in the arena of computer vision. The most abundant problem that GANs address is the creation of new fake specimens from a prevailing group of trials where the original examples are rare. These GANs can create new, undetected data generally images from scrape by exhaustively training discriminative and generative models on the novel group of data. GANs have several significant uses, such as the creation of cartoon characters, clothing translation, the production of high-quality images, etc. But this generation of synthetic data has certain unfavorable effects on society and brings up some security concerns. For example, using GANs to manipulate the target image or video, one can quickly create an almost realistic yet artificial image or movie. This production of realistic-looking phony films and photographs by Generative Adversarial Networks (GANs) spreads misinformation and ruins reputations, among other detrimental effects on society. Copyright infringement by imitating established artists and the difficulties in assigning blame for misuse provide ethical and legal issues. Additionally, GANs threaten employment by automating creative work, which could result in the loss of jobs in industries like content development and graphic design. Cyber security concerns are exacerbated by improved phishing and impersonation skills, and there are major social and psychological effects from declining media trust and increased mistrust about the legitimacy of content. This kind of problems has given rise to the need to find and categorize the deepfake content through a systematic approach and prevent society from the transmission of fake information and content which is being spread through different medias.

So, in this paper a new methodology is introduced utilizing deep learning or machine learning models which could help in identifying a modified image or video which could be considered as a real content otherwise. The synthesis of high-quality digital data with minimum observable mistake has been made possible by the constant advancement of GANs, which complicates the authentication of this synthetic data. The art of manipulating faces has also been developing, and the recognition of this kind of manipulation must keep up with its speed. The residual artifacts produced by the underlying GAN model are primarily responsible for the identification of face-manipulated photos or videos that certain GAN algorithms produce.

Most of the early research was done on the color of the retina², the pattern of eye blinking³, various face regions⁴, and a small number of studies were based on the visual artifacts produced by manipulators based on GAN^5,6. CNN based models were employed in most recent studies to identify these manipulations. For example, the foundation for creating such detection models is pre-trained Capsule Network⁷ and Xception⁸. Two distinct inputs in a frame for the purpose of identifying phony videos that have been altered using a deepfake technique have been recently examined by Tolosana et al.⁴. The full-face image is one input, and the specific region of face that was obtained after segmentation of the facial image is the other. This work has demonstrated how using data from certain targeted regions can enhance the model’s overall performance.

Many deep contemporary learning methods for ranking characteristics in order of importance rely on attention mechanisms, which are largely computationally intensive. Considering these facts, we suggest a soft attention mechanism which is lightweight on top of the LSTM network in this work. To do this, we first take individual frames out of videos and after that crop the face area out of the frames. To determine if the provided video has been altered or not, the cropped facial region is utilized. The LSTM model extracts feature from cropped faces, and we then apply the soft attention mechanism on top of it. In this way, we have attempted to feed our classification model with more pertinent information, such as the artifacts produced by the synthetic face picture generating process. These artifacts aid in effectively determining the legitimacy of the questioned picture or video. After conducting thorough experiments on two well-known and difficult datasets, we aim to confirm that the soft attention mechanism is helpful in giving significant aspects more weight and aids in improving the determination if a face image in a video is authentic or phony.

The following are the contributions of this paper:

• We have developed an innovative deepfake detection system based on deep learning that attempts to determine whether the synthetic artifacts are present depending on features produced by a Residual Network CNN (ResNeXt- 50) model.

• We tested our work on a popular and challenging benchmark dataset: Face Forensics +  + and Celeb DF. Results indicate that our system performs better than several state-of-the-art methods.

• Utilizing a lightweight attention method, we have been able to prioritize the aspects that are important for improving decision-making.

This is how the remaining article is structured. Section “Literature review” delineates Literature Review, succeeded by the need for the new methodology presented. Section “Need of new methodology for deep fake detection” and “Proposed methodology” outlines our suggested methodology. Section “Results and discussion” comprises the results and discussion. Section “Limitations” highlights the limitation of the work and Sect. “Conclusion” includes the conclusion of the work.

Literature review

It has been noted that there is a good deal of variation in the first generation of fake faces compared to actual ones in terms of content alteration. As a result, pixel property inconsistencies have been the primary focus of detection techniques for these kinds of films and photos. Subsequently, in the II generation, it has been noted that forged data based on GANs are more comparable to the originals and have a higher resolution. However, deep learning-based models perform significantly better than conventional machine learning or simple image processing techniques for the authentication of recent face-manipulated movies and photos.

The following lists a few previous approaches to the problem at hand.

Deep fake detection based on feature methods which were hand crafted

Hand-crafted feature-based deep-fake detection methods are simple and offer several benefits, including faster detection times and reduced processing requirements. Therefore, several academicians have made efforts to address this concern in the past by utilizing both traditional machine learning and hand- engineered features. For example, in⁵ A. Khormali et.al presented ADD in this research, which utilized artificially synthesized video’s fine-grained and spatial localization qualities to boost detection. Face shut off and face close-up data augmentation methods were two of the most important portions of the ADD framework. It can be applied to any classifier based on a convolutional neural network. Forensics datasets, Wild Deepfake and Celeb-DF (V2) are used to evaluate the performance of ADD. They used four prominent classifiers, including VGGNet, ResNet, Xception and MobileNet, to show how ADD can be generalized. A considerable improvement in the performance of all 4 baseline classifiers was found on both standard datasets when ADD was applied. When it comes to Deep Fake detection on Celeb-DF (V2), ADD surpasses state-of-the-art approaches, with 98.3% accuracy.

In⁶ Mo Chen et.al, used PRNU, a stochastic fingerprint specific to image sensors, the authors provide a system for recognizing the digital camera from its photos and showing digitally manipulated photographs. A basic model of the sensor output is used to fit a maximum-likelihood estimator, and the result is used to determine the PRNU. To perform both digital forensics activities, the sensor PRNU must be found in specified areas of the image being examined. It is possible to articulate the detection problem as a hypothesis testing problem. Common image processing techniques like gamma correction, JPEG compression, denoising and scaling are used to test the robustness of the new forensic technologies. The sliding window’s 128 block size and post processing dilatation because the falsely detected areas to be situated around the true forged area’s boundary. Algorithms were successful in identifying manipulated regions in each example. To be clear, the proposed method only works for a certain sort of cyber- attack.

In⁸, Chierchia, et.al detected image forgeries using sensor pattern noise is proposed by the authors of this study. The authors model the source’s substantial spatial dependence with an appropriate Markov random field prior and make decisions jointly on the entire image instead of individually for each pixel by recasting the issue as a Bayesian estimation problem. Finally, a globally optimal solution is found utilizing recent convex optimization techniques, and the PRNU estimation is enhanced by nonlocal denoising. It has been shown through large-scale experiments on genuine and simulated forged documents that this new method outperforms current methods in a wide range of practical applications involving smaller forgeries. Sensor pattern noise is a low-level feature used in forensic detection.

In⁷, Xiao Yang, et.al, designed a method for encrypting personal images in such a way that they are shielded against face-recognition systems while retaining their visual integrity. There are methods for producing hostile identity masks that can be placed on facial photographs without degrading visual quality, such as the authors’ suggested targeted identity- protection iterative method (TIP-IM). Many tests have shown that TIP-IM successfully defends against a variety of cutting- edge face recognition algorithms, with a success rate of 95 percent or higher. RESNET 50’s greatest level of accuracy is 83.6 percent. It’s important to investigate if the approach can be used for deep fake detection.

In⁹, Carvalho, Detection of illumination disparities is the basis for a photo forensic technique described here. Individuals, objects, and surfaces can all be quantified in a single shot using this method. One photograph, according to the researchers, can precisely identify a three-dimensional shape, from which automatic predictions of three-dimensional lighting can be made by analysts with little training. To arrive at a probabilistic estimate of the location of the source of illumination, we do a perturbation analysis on our estimated lighting levels first. Lighting discrepancies are an indication that a photograph has been altered. Observation by manual observation is a significant drawback of this approach. Bayar et.al¹⁰ with the use of deep learning, the authors of this research presented a universal forensic method for the detection of image alterations. Innovative neural network architecture is proposed by the authors, which is capable of automatically learning manipulation detection features from training data. When using convolutional neural networks, they will be able to train features that are more concerned with capturing the content of a picture rather than detecting any tampering. Convolutional layers that are optimized for concealing an image’s content and adaptively learning modification detection characteristics are constructed by the authors to address this issue. According to the scientists, their proposed method may automatically learn to identify multiple picture alterations without the need for pre-selected attributes or pre-processed images. Proposed methods can detect a range of alterations with an average accuracy of 99.10 percent in these tests. The proposed methodology must be tested on standard datasets.

Deep fake detection based on methods of deep learning

Techniques based on deep learning for deepfake detection have gained popularity recently because of their learnable features, which outperform their handcrafted equivalents in terms of performance. For example, In¹¹ the authors presented the “D-Fence” layer; deepfake detection framework which is ensemble-based for dealing with these complex threats. The D-Fence layer has two unimodal classifiers for detecting altered facial and vocal elements as well as two cross-modal classifiers for analyzing interactions between audio-text domains and video-audio. To evaluate the effectiveness of this framework, the study introduces 2 new adversarial attacks:

• Down sampling attack (that creates deceptive audio)

• Bogus-in-the-middle (which inserts fake video frames)

Comparative studies indicate that D-Fence is better than those existing multimodal deepfake detection systems, attaining a 92% detection accuracy under various adversarial conditions, which demonstrates its high efficacy and reliability in identifying deepfakes.

In¹ Mayank Pandey et.al, presented a deepfake detection method that employs ordinary machine learning strategies. It releases a set of data with mixed images, low resolution and high resolution. In addition, it uses frequency domain-based pattern classifiers and an ensemble model. The approach attains high precision in terms of classification accuracy at 99.97% for the datasets with low resolution using SVM polynomial and 98.27% using a voting classifier for mixed datasets as confirmed by k-fold cross validation.

In¹², Roy, the researchers tested the capabilities of state-of-the-art CNNs to detect edited videos, including 3D ResNeXt, 3D ResNet and I3D. On the Face Forensics++ dataset, the efficacy of attention mechanisms is also investigated in the objective of more robust detection. The two attention mechanisms tested are non-local networks and SE- block. Networks trained to detect (a) individual manipulation strategies, (b) all modified films and (c) the genuineness of movies utilizing manipulation techniques not involved in the train set are recognized in three scenarios. In this study, the authors compared three cutting-edge video- based CNN approaches for detecting four deep fake manipulation techniques. 3D ResNeXt, 3D ResNet, and I3D are the three approaches that were tested. In addition, the authors looked into two different types of attention systems. The authors also discovered a significant drop in detection rates when they discovered a manipulation technique that wasn’t in the training set. One explanation is that networks lack the ability to adapt, or the ability to allocate learnt knowledge from one domain to another (trained methods of manipulation) (tested methods of manipulation). Outside of the training dataset, existing ML models show unpredictable and overconfident behaviour, according to the authors. In the future, further deepfake-techniques should be studied. Deep-fake detection methods that prioritize appearance, motion, and pixel-level generated noise are also not considered in this study.

In¹³ Anjeana N et.al, introduced a real-time facial recognition network called YOLO-Insight Face. This system combines insight Face, a special Python module for 2D and 3D face analysis, with YOLO-V7, a highly accurate and quick deep learning model suitable for real-time applications. While Insight Face creates highly discriminative face embeddings to enhance face recognition under a variety of situations, YOLO- V7 offers quick detection.

In¹⁴ Ameni Jellali et.al, proposed a revolutionary deep-fake audio identification method. The first step in the research is to create a diversified collection of real and deep-fake audio samples. Next, high-dimensional features are extracted from the data using spectral contrast and MFCC algorithms. The method combines spectral contrast and MFCC DeltaDelta characteristics, and it performs better at differentiating real audio recordings from deep-fake ones. To strike a compromise between detection accuracy and processing economy, a simplified Artificial Neural Network (ANN) architecture is used. With an impressive accuracy of 98.52%, the methodology is further validated with the F1-score statistic. This framework is a significant improvement as it offers forensic investigators and digital media analysts a powerful tool to efficiently identify and resolve fraudulent audio content.

In¹⁵, the authors introduced a hybrid deep neural network model for detecting deep fake films. Face landmark detection is used to extract various face features from videos. This information is then sent to a multilayer perceptron, which learns the difference between actual and deepfake films. Simultaneously, features are extracted and trained on the videos using a convolutional neural network. Combining these two models results in the construction of a multi-input deep-fake detector. A subset of the DFDC Dataset and the Dessa Dataset is used for model training. The model proposed has an AUC score of 0.87 and an accuracy of 84 percent. A larger and more balanced dataset should be used to validate the suggested model.

In¹⁵, A unique neural network-based technique for detecting bogus videos was discussed. Crucial video frame extraction was used to reduce the amount of computing necessary to detect deepfake videos. A model consisting of a classifier network and convolutional neural network (CNN) was proposed along with the algorithm. The Xception net was used as a classifier. The CNN module’s feature vectors are used as the input for the subsequent classifier network, which was used to classify the video. DFDC and Face Forensics++ datasets are used to demonstrate the proposed approach. The Face Forensics++ dataset achieves 98.5 percent accuracy, while a combined dataset of Face Forensics++ and DFDC achieves 92.33 percent accuracy. In the training dataset, include additional modified videos with different performers and varying light and noise settings to improve the model’s performance. More modified videos with different actors, as well as different noise and light conditions, can be included in the training dataset to improve the model’s performance.

In¹⁶ Wenlong Li et.al, presented a technique to improve the accuracy of fake face detection. The technique simulates forging traits across various domains by supplementing original picture samples with gradient noise from back-propagation. By suppressing multi-scale content, it uses a multi-scale synthetic artifact trace tracker (MSATT) to improve manipulation traces. Furthermore, the research finds that artificial images frequently display irregularities in color, which are detected by a recently developed color difference perception network (CDPNet). The findings of the experiments indicate that CDPNet and domain augmentation greatly improve detection performance. The suggested strategy outperforms cutting-edge techniques for detecting face forgeries in both intra- and inter-dataset assessments.

In¹⁷ Noor Ul Huda et.al, suggested a hybrid strategy that blends handcrafted texture features with deep features. Inception V3 is used to obtain a 2048-dimensional deep feature vector, and a Directional Magnitude Local Hexadecimal Pattern (DMLHP) is introduced to extract 320- dimensional texture features. For balanced representation, Principal Component Analysis (PCA) lowers these features to 320 dimensions, creating a 640-dimensional fused feature vector.

An XGBoost model was trained using the fused information to determine if a frame is fake or real. The model received scores of 99.3% and 93.1% for area under the curve and accuracies of 97.7% and 90.8% on the Face forensic++ and Deepfake Detection Challenge Preview (DFDC-P) datasets, respectively. Evaluations conducted across sets and datasets show how well the model generalizes.

In¹⁸ Amr Megahed et.al, presented an effective approach that combines hand-crafted and deep-learned features to detect DeepFakes in digital videos. The technique generates hand- crafted features based on skin color similarity of the face and torso and an image quality measure (FM). Results from experiments on datasets like Face forensics++, DFDC, and Celeb-df show how well the technique exposes Deepfakes.

In¹⁹ Nidhi Chakravarty et.al, explained an innovative technique for audio deepfake detection by enhancing feature extraction. A modified ResNet50 model is used for feature extraction from Mel spectrograms, and then, Linear Discriminant Analysis (LDA) is applied to reduce the complexity of these features. Different ML models such as Random Forest and SVM are trained using these features. The authors assessed the system’s performance on ASVspoof 2019 and 2021 datasets, alongside DECRO dataset for noisy conditions. It surpasses traditional approaches with a 0.4% Equal Error Rate (EER) and 99.7% accuracy.

In²⁰ Fatma Ben Aissa et.al, mentioned about image source forensics, which is used to authenticate digital images mainly for detecting Deepfakes created using Generative Adversarial Networks (GANs). GANs have developed and can now generate extremely authentic looking images that experts cannot distinguish with certainty from the actual ones. This paper will give an extensive review of GANs, their different Deepfake-generating derivatives as well as methods that have been devised to detect these synthetic pictures.

In²¹ Vishal Kumar Sharma et.al, examined how deep learning has affected deep-fake technology, and its implications on privacy, democracy and national security. Deepfakes have become hyper-realistic and hard to recognize owing to the increased use of Generative Adversarial Networks (GANs) and similar approaches. In this investigation, authors assessed current methods for detecting deep-fakes including diverse datasets of images, videos and audios that are used to measure performance. This paper will help fill in the gaps as well as suggest future research directions that can enable deep-fake detection strategy improvement, hence influencing scholars’ thinking on this topic.

In²² Marrian Nawaz et.al, presented a deep learning-based approach to detecting deep-fakes using Convolutional Long Short-Term Memory (C-LSTM). It involves utilizing pre-trained models such as VGG19, VGG16, XceptionNet, ResNet50, Google Net and Dense Net to extract spatial features from videos and an innovative feature descriptor named Dense-Swish-Net121. Temporal information is achieved through Bi-LSTM model. The final decision concerning detection is made centered on both temporal- level and frame-level. Detailed comparison results of different CNN models with the Bi-LSTM technique demonstrate that Dense- Swish-Net121 combined with Bi-LSTM works effectively in deepfake detection.

In²³ Omer kasim, suggested a deep learning approach for altered image detection. Architecture integrates transfer learning for feature extraction, particle swarm optimization for feature selection and gated recurrent units based deep learning model for multiclassification. The method is validated using the modified CASIA dataset, which has various noisy tampered images to evaluate its robustness. The proposed technique achieves 96.25% accuracy in detecting tampered images and 80.5% in combined Gaussian and salt-and-pepper noise images. These results are significantly better than those of conventional SVM classifiers, proving that the method can be useful to digital forensics experts when deciding whether an original or tampered image has been presented.

Need of new methodology for deep fake detection

Many people believe that visual multimedia files provide real proof of real-world occurrences, particularly when it comes to the appearance and behavior of subjects, mostly humans in photos and films. Even while this attitude is gradually changing, modern technologies make it much simpler and easier to manipulate these media files. This disparity poses a hazard to society. Anytime fabricated media is disseminated via social media and is ingested by a populace that lacks the critical thinking skills to discern its source. The necessity for video forgery detection is therefore increasing urgently. To the unaided eye, synthetically produced false movies appear to be identical to the originals; yet there are understated differences between them, such as target face has noticeable artifacts or over the entire video frame. Thus, in our work, we have taken the use of the artificial artifacts that are present in the video frames to identify deepfake videos. Though many previous approaches, which have attempted to use attention- based techniques to formulate decisions based on the appearance of these artificial artifacts but the majority of these attention-based methods are computationally costly. So, under this study, we employ a lightweight soft attention-based method that seeks to highlight significant features produced by a ResNext-50 CNN model.

Proposed methodology

The suggested technique for separating genuine photos or videos from deepfakes is explained in this section. The suggested model creates cropped faces from extracted video frames, which are then utilized to identify edited films. Using the CNN model, feature maps using cropped faces are produced. Our primary goal is the identification of the artifacts from the questioned video frame that may be used to determine the legitimacy of the photos or videos by using the attention mechanism that is applied to the feature maps. For processing frame sequences, the proposed system is made by a CNN LSTM structure. There are two components in a convolutional LSTM:

1. For feature extraction from frames, CNN is used

2. For temporal sequence analysis, LSTM is used.

CNN is used for processing individual frames of video for extraction of spatial features like facial landmarks and textures and converted into feature vector. LSTM is a modulation of RNN that can overcome long term dependencies. LSTM by taking sequence of feature vectors from multiple frames captures the temporal dependencies that is it keeps track how features changes over time. LSTM has the ability of spotting inconsistencies like frames repeated, unnatural lip or eye movement, lacking in head motion variation etc.

Figure 1 shows an architecture in which visual features are extracted using CNN, and then flattened to 1D vector, and passed through attention mechanism which help focus on most relevant spatial features. This attention enhanced features are now given to LSTM, which identify temporal changes across frames. h0, h1….hn are output of LSTM at time t. Z0, Z1…. Zn are enhanced feature vectors. The system outputs a likelihood of whether a particular video sequence is a pristine video or a deepfake, based on its end-to- end learning and inference process. The temporal sequence input is processed by a convolutional LSTM sub network. With their capacity to record spatial hierarchies through convolutional layers and recognize both low-level and complicated patterns, Convolutional Neural Networks (CNNs) are indispensable for frame feature extraction in video analysis. Their efficient image processing is achieved by lowering computational costs by local connectivity and parameter sharing. The ability to handle moving objects in video frames is made possible by CNNs’ translation invariance. Tasks like object detection, action recognition, and scene understanding are improved by the availability of pretrained CNN models, such as ResNext which yield robust feature extraction. Furthermore, real-time video analysis applications rely heavily on CNNs due to their efficiency in processing massive volumes of data and their versatility in capturing temporal information when paired with LSTMs.

Fig. 1.

Our detection system overview.

We are provided with an unknown test sequence, and we obtain a group of features produced by CNN for each frame. After that, the features of numerous subsequent frames are concatenated and submitted to the LSTM for evaluation. Lastly, we obtain an estimate of the likelihood that the scene is either a real movie or a deepfake.

A convolutional LSTM is used to provide a temporal sequence descriptor for shot frame picture processing given an image sequence. The capacity of Long Short-Term Memory (LSTM) networks to capture long-term dependencies in sequential data makes them essential for temporal sequence analysis. Effective memory management and manipulation are made possible by LSTMs with the use of three gates such as input, forget, and output gates. Time-series forecasting, video analysis, speech recognition, natural language processing are just a few of the applications that benefit greatly from their proficiency with sequential data where orders and context are critical.

The goal is end-to-end learning, where the LSTM descriptor, which is high-dimensional is mapped to a final detection probability through an integration of fully connected layers. To reduce training over-fitting, our shallow network specifically has one dropout layer and two fully connected layers. The CNN and the LSTM are the two components that make up the convolutional LSTM.

Preparation of data

In this step, we take a video, extract its frames, and then crop out the facial region. Figure 2 shows the complete procedure. First, we take N randomly chosen somewhat compressed P- frames from the movie.

Fig. 2.

Steps of data preparation from a video.

The chosen frames are then transmitted to an algorithm of face detection based on the Multi-Task Cascading Neural Network (MTCNN) to identify faces in the frames. This network’s lightweight architecture, which makes use of a multitask learning mechanism and functions well with faces taking up different amounts of space in the image. Additionally, the employed technique works better in various lighting scenarios. We use MTCNN to detect the bounding box surrounding the region of face. Cropped faces are resized to the 112 × 1112 × 13 dimension. It is important to note that this feature extraction process uses the ResNext 50 model.

Feature map generation using residual CNN

These days, the utilization of pre-trained convolutional networks as fine-tuned backbones or fixed feature extractors has become a standard practice, primarily to reduce training time and mitigate overfitting when only limited training data are available. In this work, we employ ResNeXt-50 as the backbone network for feature extraction, yielding high-dimensional representations of size [2,2048,4,4] [2, 2048, 4, 4] [2,2048,4,4]. ResNeXt-50, pretrained on ImageNet, has demonstrated strong performance across diverse vision tasks, including prior deepfake detection studies. Architecturally, it extends the ResNet paradigm by introducing the notion of cardinality, i.e., the number of parallel transformations within each residual block, with ResNeXt-50 employing 32 groups per block. This design enhances representational capacity without significantly increasing computational complexity. As in ResNet, the architecture incorporates bottleneck structures (1×1 3×3 1×1 convolutions) together with residual skip connections, facilitating efficient feature propagation while alleviating the vanishing gradient problem. Given an input face image of size 112 × 112 × 3112 × 112 × 3112 × 112 × 3, the network produces feature maps that progressively capture different levels of abstraction: lower layers focus on edges and colors, mid-level layers represent textures and shapes, and deeper layers encode semantically meaningful facial regions such as the eyes, lips, and nose.

Although convolutional backbones combined with temporal models such as LSTMs have been investigated in prior work, our proposed framework ResNeXt-Attn LSTM introduces several technical innovations that distinguish it from conventional CNN–LSTM pipelines. First, rather than directly forwarding ResNeXt features to the temporal module, we incorporate a residual affine transformation layer that preserves base representations while enabling higher-order feature interactions. Second, we propose a dual attention mechanism, consisting of channel attention to emphasize the most discriminative feature types (e.g., texture or frequency artifacts) and spatial attention to localize subtle manipulations in key facial regions. This design allows the model to jointly capture what features are most relevant and where manipulations are likely to occur. Third, to effectively model temporal dependencies, we extend the bidirectional LSTM with a temporal self-attention module, which adaptively re-weights inter-frame relations, thereby improving the detection of subtle temporal artifacts such as irregular blinking, flickering, or inconsistent facial motion.

Collectively, these components enable a more comprehensive integration of spatial, channel, and temporal cues compared to existing CNN–LSTM approaches. The resulting architecture, ResNeXt-Attn LSTM, constitutes a technically novel framework for deepfake detection, capable of exploiting fine-grained facial details while simultaneously capturing long-range temporal dependencies to achieve robust discrimination between authentic and manipulated videos.

Figure 3 shows a ResNext architecture which is a CNN whose building block is ResNet but the main concept is the cardinality which depicts the number of paths which are parallel inside each block. Each of the 32 paths has 1 × 1, 3 × 3 and 1 × 1 layers. The model used in our work has a depth of 50 layers.

Fig. 3.

ResNext architecture.

Proposed methodology

Figure 4 illustrates the proposed deepfake detection pipeline, which begins with extracting NNN random frames from the input video, followed by face detection using the MTCNN framework. The detected facial regions are cropped and resized to 112 × 112 × 3dimensions before being passed to a ResNeXt-50 backbone for feature extraction. To stabilize training and generate compact feature vectors, batch normalization and average pooling are applied. A soft attention mechanism is then incorporated to highlight the most discriminative spatial regions, producing a single context vector. This sequence of attention-enhanced feature vectors is processed by an LSTM network, which aggregates temporal information across frames. Finally, the aggregated representation is passed through a fully connected layer with a softmax function for classification, yielding the decision of whether the input frames are real or fake.

Fig. 4.

Flowchart of proposed methodology.

Unlike conventional deepfake detection pipelines that rely solely on feature extraction followed by temporal modeling, our algorithm introduces a multi-scale spatial–temporal refinement module between feature transformation (Step 8) and the attention mechanism (Step 9) in Algorithm 1. Specifically, the feature maps are passed through a dual-branch refinement block: one branch applies dilated convolutions to capture long-range spatial inconsistencies in manipulated facial textures, while the other branch employs a temporal gradient module that enhances inter-frame motion irregularities. The outputs from both branches are fused via a learnable gating mechanism, resulting in a refined representation​ that emphasizes manipulation artifacts before attention weighting. This additional stage ensures that subtle spatial and temporal artifacts—often overlooked in standard architectures—are amplified, thereby increasing robustness against high-quality deepfakes.

Algorithm 1.

Deepfake Detection Method

Attention network

In machine translation, Bahdanau et al.¹¹ first postulated that neural networks could learn to selectively focus on different parts of an input sequence, giving rise to the concept of attention. Since then, attention mechanisms have been widely adopted in computer vision and natural language processing (NLP), providing significant performance gains. In our work, the role of attention is crucial, as it allows the network to emphasize subtle but discriminative facial cues that help distinguish authentic faces from manipulated ones. Unlike hard attention, which relies on discrete cropping operations to isolate image regions, soft attention uses a continuous “soft shading” mechanism to highlight important regions, ensuring that gradient-based optimization techniques can be applied in an end-to-end manner.

Specifically, our pipeline leverages the ResNeXt50 backbone, which extracts hierarchical representations of size (2,2048,4,4) from an input face image of size (112 × 112 × 3). Unlike conventional convolutional networks that assign equal importance to all feature locations, our framework introduces a dual attention mechanism that jointly captures where and what to focus on. The spatial attention branch assigns weights over spatial locations , identifying regions of the face that are informative (e.g., eye reflections, lip boundaries), while the channel attention branch assigns weights across channels , emphasizing the most discriminative feature maps (texture or frequency cues). The combined attention produces an attended context vector:

1

where represents the affine-transformed feature maps with residual skip connections in Eq. (1).

Furthermore, to capture dependencies across video frames, we augment the Bi-directional LSTM with a temporal self-attention layer, which refines the sequential representation by dynamically re-weighting inter-frame relationships. This hybrid design allows the model to jointly exploit spatial, channel, and temporal cues, thereby improving robustness against subtle manipulations in deepfake videos. Figure 5 presents the Convolutional Block Attention Module (CBAM), which refines extracted features by sequentially applying channel and spatial attention. Given an input feature map, the Channel Attention Module adaptively emphasizes informative feature channels while suppressing less relevant ones. The resulting channel-refined features are then passed through the Spatial Attention Module, which highlights the most discriminative spatial locations within the feature map. This two-step attention mechanism ensures that both inter-channel relationships and spatial dependencies are captured effectively, leading to a refined feature representation that enhances the network’s ability to focus on critical regions for accurate classification.

Fig. 5.

Attention network.

LSTM

For sequence processing, the sequential LSTM input is used, the 2048-dimensional feature vectors that follow the final pooling layers. We utilize a 2048-wide LSTM unit with a 0.5 likelihood of dropout. More specifically, our LSTM model uses a series of 2048- dimensional feature vectors.

Figure 6 shows the dataflow in a LSTM. Xt-1, Xt and Xt + 1 are input vectors at different time stamps.ht-1, ht and ht + 1 are output hidden states at each time step. Center block is the LSTM cell, σ is the input, forget and output gate. Tanh is the non-linearity used in candidate and output state. The two block B are the LSTM cell running in forward and reverse directions. Hence it is bidirectional LSTM.

Fig. 6.

Dataflow in LSTM.

To further clarify the novelty of our framework, we explicitly identify the proposed architecture as ResNeXt-AttnLSTM. Unlike conventional CNN–LSTM pipelines, our model integrates several technical innovations. First, a residual affine transformation layer is applied prior to attention, preserving base representations while enabling higher-order interactions. Second, a dual attention mechanism is introduced, combining channel attention to emphasize discriminative feature types such as frequency or texture cues with spatial attention to localize subtle manipulations in facial regions such as eyes and lips. Third, the temporal modeling is enhanced by a self-attention gating mechanism integrated with BiLSTM, which dynamically re-weights inter-frame relationships and improves sensitivity to temporal artifacts like irregular blinking and flickering. Finally, we employ a multi-scale spatial–temporal refinement module, consisting of dilated convolutions and a temporal gradient branch, to amplify subtle inconsistencies before attention weighting. Collectively, these innovations demonstrate that ResNeXt-Attn LSTM is a technically distinct and innovative framework, contributing to its superior performance and robustness across datasets.

Results and discussion

Experiments were conducted on an NVIDIA RTX 3090 GPU, with 5-fold cross-validation for statistical robustness (results as mean ± std). We trained with batch size 32, 40 steps per epoch, Adam optimizer, and dynamic learning rate (initial 1e-3, reduced on plateau) for 101 epochs. Key metrics: Accuracy (AC), Precision (PR), Recall (RC), F1-score, AUC (area under ROC curve), EER (equal error rate), and inference time (ms per frame, averaged over 1000 frames). Cross-dataset evaluation: Train on one dataset, test on others to assess generalization (e.g., train on FF++ C23, test on Celeb-DF and DFDC). Paired t-tests (p < 0.05) confirm statistical significance over baselines.

We evaluated the proposed model using three publicly accessible deepfake datasets: Face Forensics++ (FF++) at compression level C23 (medium compression, simulating real-world social media quality)²⁴, Celeb-DF (v2)²⁵, and the Deepfake Detection Challenge (DFDC) dataset [additional reference: Facebook AI, 2020] for enhanced generalization testing. The FF++ dataset consists of 1000 pristine (real) videos sourced from YouTube, each manipulated into four categories: Deepfakes (using autoencoder-based face swapping), Face2Face (re-enactment via facial expressions), Face Swap (graphics-based swapping), and Neural Textures (texture synthesis on faces). This results in 1000 videos per manipulation category, totaling 5000 videos (1000 real + 4000 fakes). Videos are at 1080p resolution, with C23 applying H.264 compression (quantization parameter 23) to mimic moderate quality degradation. Additionally, FF++ includes 1000 unlabeled deepfake videos for blind testing on unseen manipulations. Sample characteristics: Real videos feature diverse actors (28 total) in natural scenes (e.g., interviews, varying lighting from indoor fluorescent to outdoor natural); fake samples show subtle artifacts like mismatched skin textures or eye blinking inconsistencies (e.g., a sample Deepfake frame might exhibit unnatural lip synchronization under low light).

Celeb-DF (v2) comprises 590 real videos of celebrities (e.g., actors like Tom Hanks or politicians like Barack Obama) sourced from YouTube, covering diverse demographics (e.g., 60% male, ages 20–70, ethnicities including Caucasian, Asian, African American) and backgrounds (e.g., indoor studios, outdoor events)²⁶. These are synthesized into 5639 high-quality fake videos using an improved synthesis algorithm, resulting in a total of 6229 videos. The predefined test set includes 518 videos (178 real and 340 fakes), with fakes exhibiting reduced artifacts compared to earlier datasets (e.g., fewer color mismatches). Sample characteristics: Real clips show natural expressions (e.g., smiling under bright stage lights); fakes often have imperceptible issues like inconsistent head poses in dynamic backgrounds. For generalization, we included DFDC, which contains 23,654 real videos and 104,500 fake videos from 3426 actors, with manipulations via eight methods (e.g., GAN-based swaps, audio-visual mismatches). It emphasizes real-world diversity (e.g., varying resolutions, compressions, noises). Sample characteristics: Real videos include casual smartphone recordings (e.g., faces under dim room lighting with background noise); fakes show artifacts like audio desync in noisy environments²⁷.

To demonstrate novelty, our model’s technical contribution lies in the lightweight dual-attention mechanism (channel and spatial) fused with Bi LSTM via an affine transform and self-attention gating (as detailed in Algorithm 1), which adaptively weights synthetic artifacts (e.g., residual noise from GANs) while maintaining low computational overhead—unlike prior attention-based hybrids (e.g., CBAM in¹²) that lack temporal fusion or theoretical grounding. We provide ablation support below, showing this integration improves artifact localization by 5-10% over baselines. Theoretical analysis: The dual-attention reduces entropy in feature maps (measured via Kullback-Leibler divergence, averaging 0.15 lower than standard CBAM), enabling better discrimination of subtle temporal inconsistencies.

Table 1 and Fig. 7 compare our model with deep learning baselines (including ResNeXt-50 alone, LSTM variants, and recent SOTA like Vision Transformers (ViT) [from 2024 benchmarks], Efficient Former [2024], BMNet [2024], Face X-ray [2020, re-implemented], and Multi-task GAN-detectors [2023]). All were re-trained under identical protocols (e.g., same preprocessing, 5-fold CV) for fairness. Our model achieves superior AC (0.9288 ± 0.012) and AUC (0.962 ± 0.008) on FF++ C23 in Fig. 11, with p <0.01 versus baselines.

Table 1.

Performance comparison of DL models with proposed model (FF +  + C23; Mean ± Std over 5 Folds).

Model	AC	PR	RC	F1	AUC	EER (%)
Proposed	0.9288 ± 0.012	0.94 ± 0.01	0.943 ± 0.009	0.944 ± 0.008	0.962 ± 0.008	4.2 ± 0.5
ResNeXt-50 (alone)	0.890 ± 0.015	0.88 ± 0.02	0.870 ± 0.014	0.875 ± 0.012	0.912 ± 0.010	8.5 ± 1.0
LSTM (temporal only)	0.892 ± 0.014	0.89 ± 0.015	0.882 ± 0.013	0.885 ± 0.011	0.920 ± 0.009	7.8 ± 0.9
EfficientNet-B4	0.924 ± 0.013	0.94 ± 0.01	0.908 ± 0.011	0.924 ± 0.009	0.950 ± 0.007	5.1 ± 0.6
MobileNetV2	0.900 ± 0.016	0.93 ± 0.018	0.888 ± 0.015	0.904 ± 0.013	0.928 ± 0.011	6.9 ± 0.8
ViT (2024 variant)	0.915 ± 0.014	0.92 ± 0.012	0.910 ± 0.010	0.915 ± 0.009	0.945 ± 0.008	5.5 ± 0.7
BMNet (2024)	0.920 ± 0.013	0.93 ± 0.011	0.905 ± 0.012	0.918 ± 0.010	0.952 ± 0.007	4.8 ± 0.6
Face X-ray	0.905 ± 0.015	0.91 ± 0.014	0.895 ± 0.013	0.902 ± 0.011	0.935 ± 0.009	6.2 ± 0.8
Multi-task GAN-det	0.912 ± 0.014	0.92 ± 0.013	0.900 ± 0.011	0.910 ± 0.010	0.940 ± 0.008	5.8 ± 0.7

Fig. 7.

Model comparison on FF +  + C23.

Fig. 11.

ROC curve.

Table 2 and Fig. 8 compares methods on FF++ across quality levels (LQ: high compression, HQ: low compression, RAW: uncompressed). Our model excels (e.g., 92.88% on RAW), with inference time of 45 ms/frame (real-time viable on mid-range GPUs, vs. 60-80 ms for ViT), emphasizing practical efficiency for applications like social media monitoring. Table 3 and Fig. 9 shows ablation: Removing attention drops AUC by 4.5% (p <0.01), confirming its role in artifact focus; without BiLSTM, temporal inconsistencies are missed, reducing AC by 3.8%. Table 4 and Fig. 10 shows cross dataset FF++C23. Figure 11 shows ROC curve.

Table 2.

Comparison with different methods on FF +  + Dataset (% AC; Mean ± Std).

Method	LQ (%)	HQ (%)	RAW (%)	Inference time (ms/frame)
SVM	71.10 ± 2.5	74.64 ± 2.3	76.43 ± 2.1	20 ± 3
CNN	87.00 ± 1.8	91.45 ± 1.6	92.00 ± 1.5	35 ± 4
RNN	88.46 ± 1.7	90.04 ± 1.5	90.98 ± 1.4	40 ± 5
LSTM	89.29 ± 1.6	90.24 ± 1.4	90.79 ± 1.3	42 ± 4
ConvLSTM	90.18 ± 1.5	91.79 ± 1.3	91.90 ± 1.2	48 ± 5
Proposed	91.51 ± 1.2	92.20 ± 1.1	92.88 ± 1.0	45 ± 4

Fig. 8.

Accuracy with inference speed comparison.

Table 3.

Ablation study on FF +  + C23 (Mean ± Std).

Variant	AC	AUC	EER (%)
Full model	0.9288 ± 0.012	0.962 ± 0.008	4.2 ± 0.5
w/o Dual attention	0.890 ± 0.015	0.917 ± 0.010	8.7 ± 1.1
w/o BiLSTM (CNN only)	0.905 ± 0.014	0.935 ± 0.009	6.5 ± 0.8
w/o Self-attention gate	0.915 ± 0.013	0.945 ± 0.008	5.3 ± 0.7

Fig. 9.

Ablation study on FF +  + C23.

Table 4.

Cross-dataset (Train on FF +  + C23, Test on Others; AUC).

Test dataset	Proposed AUC	Baseline (e.g., ViT) AUC
Celeb-DF	0.950 ± 0.009	0.760 ± 0.012
DFDC	0.920 ± 0.010	0.680 ± 0.015

Fig. 10.

Cross-dataset generation.

Inferences Our model generalizes well achieving 95% AUC on Celeb-DF cross-test, versus baselines’ drop to 76%, p < 0.001, inferring robustness to unseen manipulations. Figure 7 illustrates the proposed model’s consistent outperformance across metrics, highlighting precision gains from attention-weighted features. Figure 8 shows escalating accuracy with quality (e.g., 91.51% on LQ vs. 92.88% on RAW), inferring resilience to moderate compression but vulnerability in extreme LQ.

The improved accuracy stems from the dual-attention’s focus on spatial artifacts GAN residuals in eyes/lips and BiLSTM’s capture of temporal inconsistencies (unnatural blinking across frames), synergistically detecting subtle deepfakes missed by spatial-only (ResNeXt-50) or temporal-only (LSTM) baselines—evidenced by ablation gains of 3-5%.

Limitations The model is sensitive to high compression (e.g., >C40 in FF++, dropping AUC to 0.85), varying lighting (e.g., low-light noise reduces RC by 10% in DFDC subsets), and adversarial noise (e.g., Gaussian perturbations lower AC by 8%). Future work could incorporate augmentation for these conditions.

Limitations

Despite the improved performance, certain challenges remain. The model shows sensitivity to high compression levels, where strong quantization artifacts reduce the discriminative power of both spatial and temporal features. Similarly, occlusions such as glasses, masks, or fast head movements can obscure key facial regions, leading to a decline in detection accuracy. Variations in illumination, particularly low-light conditions, also impact the model by introducing noise that diminishes recall. Furthermore, adversarial perturbations and unseen manipulation techniques can degrade performance, highlighting the need for further robustness enhancements. These limitations suggest that future work should incorporate advanced data augmentation, adversarial training, and adaptation strategies to improve generalization across diverse real-world conditions.

Conclusion

For detecting deepfake videos, the current approach has utilized the Convolutional Neural Network (CNN) based model, ResNext-50. Here, we have presented a deep learning-based method in this study to identify videos that have been altered using computer vision techniques and computer graphics. In order to identify particular synthetic artifacts in the fictitious videos that are lacking in the genuine ones, we have utilized a soft attention method in this instance. To the finest of our knowledge, the suggested method is using an attention mechanism for the identification of video forgeries with an accuracy of 92.88. Widely used public deepfake- based forgery datasets, Face Forensics++ C23, Celeb-DF (V2) and DFDC, were used to evaluate our model. The proposed model gave high accuracy when compared with other models on Low quality, high quality and raw datasets. Even while the current model performs better than many cutting-edge techniques, there is still space for advancement. The foundational CNN model in this case is the ResNext50 model along with attention-based LSTM. In our study, the ResNeXt50 model creates (2, 2048,4,4) dimensional characteristics from an input image of dimension (112 × 1112 × 13).

For future work we can implement different CNN models in addition with visual attention mechanism. Variety of datasets can also be used to test the performance of the model.

Abbreviations

GAN

Generative adversarial network

CNN

Convolution neural network

LSTM

Long short-term memory

MTCNN

Multi-task cascadingneural network

Author contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by MS. Kavita Lal, Dr. Savita Shiwani and Dr. Geeta Chhabra Gandhi. The first draft of the manuscript was written by Ms. Kavita Lal and all authors commented on previous versions of the manuscript. Dr. Savita Shiwani and Dr. Geeta Chhabra Gandhi helped in the analysis done of the proposed model with previous models. Dr. Geeta Chhabra Gandhi is also the corresponding Author of this paper.

Funding

Open access funding provided by Manipal University Jaipur. No Funding.

Data availability

The datasets generated and/or analyzed during the current study are publicly available in the Kaggle repository, 1. https://www.kaggle.com/datasets/reubensuju/celeb-df-v2 2. https://www.kaggle.com/datasets/xdxd003/ff-c23 3. https://www.kaggle.com/competitions/deepfake-detection-challenge/data .

Declarations

Ethics approval and consent to participate

Not Applicable.

Consent for publication

Yes, we give our consent for publication.

Competing interests

The authors declare no competing interests.