Dynamic driver drowsiness detection with attention enhanced convolutional neural networks for real time monitoring and road safety applications

Abstract

Driver fatigue remains a major contributor to traffic accidents worldwide, underscoring the need for accurate and timely drowsiness detection systems. This work presents a real-time driver drowsiness detection framework utilizing deep learning-based facial analysis. The approach integrates pre-trained models (VGG19, VGG16, ResNet150, and DenseNet201) and a custom convolutional neural network (CNN) with an attention mechanism to classify drowsiness based on eye and mouth behavior. These models were fine-tuned and evaluated on the NTHU-DDD dataset. Experimental results show that the proposed CNN with attention mechanism achieves the highest performance, with an accuracy of 99.63% and perfect precision, recall, and F1-score. The attention mechanism further improves detection by emphasizing relevant facial regions. This framework demonstrates the potential for deployment in real-world driver monitoring systems and contributes to advancing driver-assistance technologies.

Introduction

Driver fatigue remains a pervasive and critical challenge to road safety, contributing significantly to the occurrence of vehicular collisions and fatalities across the globe. Despite substantial advancements in transportation infrastructure and the development of sophisticated safety technologies, the inability to effectively detect and mitigate driver drowsiness has left this issue largely unresolved. Fatigue impairs cognitive function, reduces attentiveness, and slows reaction times, significantly increasing the risk of accidents^1,2. Professional drivers, particularly those subjected to strenuous schedules and extended working hours, are disproportionately affected, often operating under conditions that exacerbate fatigue-induced risks^3,4.

Recent statistical analyses underscore the gravity of this issue. According to the National Highway Traffic Safety Administration (NHTSA), driver fatigue is responsible for approximately 100,000 collisions annually in the United States, resulting in 71,000 injuries and economic damages exceeding $12.5 billion⁵. Furthermore, the AAA Foundation for Traffic Safety attributes nearly 6,400 annual fatalities in the U.S. to drowsy driving⁶. Between 2017 and 2021, drowsiness accounted for nearly 30,000 fatalities, representing approximately 18% of all traffic-related deaths⁷. Globally, the prevalence of fatigue-related accidents is similarly concerning, with underreporting in official statistics likely masking the true extent of the problem. This alarming reality highlights an urgent need for innovative, scalable solutions capable of addressing the multifaceted risks associated with driver fatigue​.

The severity of the issue extends globally, with alarming statistics reflecting the widespread impact of driver fatigue. Studies estimate that drowsy driving contributes to 25–30% of all road accidents in the United Kingdom and is implicated in an astounding 85% of fatal traffic accidents in France⁸. In addition, between 2006 and 2010, fatigue-related driving was responsible for 18% of road deaths and 17% of fatal motor vehicle accidents globally⁹. However, the actual impact of drowsy driving is likely underestimated due to significant underreporting, a pervasive issue across all regions¹⁰. These figures emphasize the urgent necessity for multifaceted solutions, including public awareness campaigns to educate drivers, stricter legislation to enforce safer driving practices¹¹, and cutting-edge monitoring technologies to address this preventable risk effectively.

While advancements in driver assistance technologies, such as automatic braking, lane departure warnings, and adaptive cruise control¹², have substantially enhanced vehicle safety by reducing collisions and improving control¹³, these systems fall short of addressing driver drowsiness. This critical limitation persists despite drowsiness being a significant contributing factor in numerous road accidents¹⁴. Real-time identification of driver fatigue and timely intervention remain unachieved in many advanced driver-assistance systems (ADAS), underscoring the pressing need for innovative, adaptive solutions¹⁵.

Deep learning emerges as a promising paradigm to address this challenge¹⁶. Unlike traditional machine learning techniques, deep learning models excel at processing and analyzing complex visual data¹⁷, including nuanced facial expressions, eye movements, and other subtle fatigue indicators¹⁸. By employing transfer learning techniques, researchers can fine-tune pre-trained models such as VGG19 and ResNet50V2 on domain-specific datasets to detect early signs of drowsiness in drivers with high accuracy and efficiency^19,20. These models can not only identify fatigue symptoms in real-time but also generate timely alerts to prevent potential disasters. This approach has the potential to outperform traditional methods while offering a scalable, reliable framework to enhance road safety and mitigate the risks associated with drowsy driving.

Driver fatigue is a pervasive and critical issue in road safety, significantly contributing to traffic collisions and fatalities worldwide. While ADAS and semi-autonomous technologies have successfully mitigated various human errors, such as unintended lane departures and sudden breaking, they remain unable to monitor and respond to driver alertness in real time. This shortcoming is particularly alarming, as fatigue-induced impairments in cognitive and motor functions can lead to catastrophic outcomes, especially during prolonged driving or under challenging conditions. Traditional machine learning-based approaches for detecting driver drowsiness are constrained by their reliance on manually crafted features and often fail to generalize effectively to diverse real-world scenarios. Furthermore, these methods lack the precision needed to analyze subtle but critical fatigue indicators, such as eyelid movements and yawning. Addressing these limitations demands innovative, adaptive solutions capable of delivering real-time fatigue detection with high precision and robustness across varied environments and driver behaviors.

The escalating prevalence of fatigue-related traffic accidents and the inadequacy of existing solutions underscore the urgent need for advanced driver-monitoring systems. Recent advancements in artificial intelligence, particularly deep learning, provide an unprecedented opportunity to develop systems capable of accurately detecting drowsiness by analyzing intricate facial features. Integrating attention mechanisms into Convolutional Neural Networks (CNNs) offers a promising approach to improving sensitivity and precision in identifying fatigue-related behaviors^21,22. Moreover, the availability of comprehensive datasets, such as the NTHU-DDD, enables researchers to train models under diverse conditions, ensuring greater adaptability and robustness. This work is driven by the need to bridge the gap between traditional ADAS technologies and intelligent, real-time driver-monitoring solutions, thereby enhancing road safety and contributing to the development of reliable semi-autonomous and autonomous vehicles.

This work introduces a novel framework for addressing the limitations of current ADAS and self-driving technologies, focusing on their inability to detect and respond to driver fatigue. While ADAS features such as adaptive cruise control, automatic braking, and lane departure warnings have significantly improved safety, their lack of real-time drowsiness detection leaves a critical gap. This gap is especially concerning in semi-autonomous systems, where driver engagement is vital during manual control transitions. To address these challenges, this work leverages advanced methodologies, integrating precise facial analysis with deep learning techniques. By utilizing the NTHU-DDD dataset, a robust repository of fatigue-related facial expressions, the proposed approach employs a CNN model enhanced with an attention mechanism. The inclusion of Dlib for accurate detection of facial landmarks, such as eyes and mouth, ensures high precision in identifying drowsiness indicators.

The key contributions of this work are summarized as follows:

• We propose a novel CNN-based driver drowsiness detection framework enhanced with an attention mechanism, capable of focusing on critical facial regions to improve detection accuracy.

• We fine-tune multiple pre-trained deep learning models (VGG19, VGG16, ResNet150, DenseNet201) on the NTHU-DDD dataset to evaluate their performance in recognizing fatigue indicators such as eye closure and yawning.

• The proposed attention-based CNN outperforms existing models, achieving state-of-the-art performance with 99.63% accuracy, and 100% precision, recall, and F1-score.

• Our framework demonstrates robust real-time performance across varied driving conditions, addressing key limitations of current ADAS technologies and enabling practical integration into intelligent driver-monitoring systems.

While numerous studies have applied traditional machine learning and even deep learning models for driver drowsiness detection, several limitations remain. Many existing approaches rely heavily on handcrafted features or rule-based logic, which restricts their adaptability to diverse real-world conditions and often fail to detect subtle indicators such as micro-expressions, partial eye closure, or yawning. Moreover, some deep learning models lack mechanisms to focus on the most informative facial regions, leading to misclassification, especially under varying lighting or occlusion.

In contrast, our proposed framework introduces a CNN integrated with an attention mechanism that dynamically emphasizes key facial regions (such as eyes and mouth), improving detection of fine-grained fatigue cues. Unlike previous models, our system is trained and validated on the NTHU-DDD dataset, which includes a wide range of driving conditions, thereby enhancing robustness. Our method achieves state-of-the-art performance, outperforming both classical and existing deep learning-based systems in accuracy, precision, recall, and F1-score. This demonstrates a meaningful advancement by delivering a scalable, reliable, and highly accurate solution suitable for integration into real-time driver-monitoring systems.

The paper is organized into six sections to provide a comprehensive and systematic presentation of the research. The section “Literature review” presents a detailed literature review, discussing existing methods for driver drowsiness detection and identifying their limitations, which this work seeks to address. The section “Materials and methods” describes the materials and methods employed, including the dataset, preprocessing techniques, and the architecture of the proposed CNN with Attention mechanism, emphasizing the methodological innovations introduced. In the section “Experiments and results”, the experimental setup and results are elaborated, highlighting the performance metrics of various models and demonstrating the effectiveness of the proposed approach. The section “Comparative study” offers a comparative analysis, benchmarking the proposed method against existing approaches and underscoring its superior accuracy, precision, and scalability. Finally, the section “Conclusion, limitations, and future studies” concludes the paper by summarizing the key contributions, discussing the broader implications, and outlining future research directions, such as real-time optimization and the integration of multimodal data for enhanced fatigue detection systems.

Literature review

Because of the shortcomings of conventional monitoring systems, driver drowsiness detection has emerged as a crucial area of study in road safety. More complex methods for identifying weariness have been made possible by recent developments in computer vision, deep learning, and machine learning. Based on feature selection, algorithmic techniques, and practical applicability, this section classifies and critically evaluates earlier research, emphasizing the study’s advantages, disadvantages, and research gaps.

Feature-based approaches

Numerous research has depended on particular physiological or visual cues of sleepiness. For example²³, used dashboard-mounted cameras and CNNs to identify yawning. CNNs were successful in localizing the mouth region, but their generalizability was constrained by their reliance on a single indicator (yawning) without quantitative assessment. In a similar vein²⁴, employed facial landmarks to identify tiredness, reporting an overall accuracy of 83% with accuracies of 88% during the day and 85% during the night. Even with encouraging outcomes, these techniques had trouble capturing a variety of real-world situations where several fatigue signs coexist.

Other research looked at blink patterns and eye movement as important characteristics. The importance of combining spatial and temporal information was highlighted by the hybrid CNN–BiLSTM strategy in²⁵, which increased accuracy from 85% (CNN just) to 94% (CNN + BiLSTM). Similarly²⁶, used C-LSTM for eye movement analysis and obtained 95–97% accuracy; nevertheless, eye-only detection presents difficulties in complicated situations. In addition to visual cues²⁷, presented EEG signals for the identification of tiredness. This approach provided a new neurophysiological viewpoint, but its practical usefulness was limited by its 73.22% accuracy.

Algorithmic innovations

In this field, deep learning architectures have undergone substantial development. On the NTHU-DDD dataset, a system that combined InceptionV3 with LSTM in²⁸ obtained 93.69% training and 81.04% testing accuracy; nevertheless, performance was negatively impacted by its dependence on a small number of facial features. Similar to this²⁹, used several CNN models (AlexNet, VGG-FaceNet, FlowImageNet, and ResNet) to categorise fatigue behaviours. They were able to achieve 85% accuracy at the expense of high computational complexity, which limited their ability to operate in real time.

ResNet and LSTM were merged in hierarchical and hybrid techniques, such the HDDD network in^30,31, to analyse spatial and temporal features simultaneously, with an accuracy of 87.19%. Additionally, in order to assess driving states (normal, fatigued, inebriated, and irresponsible)³², developed emotion analysis with CNNs. This achieved 93% accuracy; however it mostly relied on predefined emotional categories. Some models, however, fared poorly:³³ coupled CNNs and RNNs over 60-second image sequences, but only obtained 60% testing accuracy, indicating difficulties in accurately modelling temporal dependencies.

Real-time and mobile-oriented systems

There is still a significant research gap in practical applications. Using facial landmark detection, the Android-based system in³⁴ demonstrated portability but lacked resilience for a range of situations, achieving 80% accuracy. Concerns regarding real-time applicability in embedded systems are also raised by the computationally demanding frameworks in^29,30.

Comparative analysis and research gaps

Table 1 summarizes the reviewed studies, comparing datasets, approaches, and performance metrics. Across these studies, several limitations such as many approaches rely on single indicators (e.g., yawning, blinking), limiting adaptability to varied conditions.

Table 1.

Comparison of deep learning approaches for driver drowsiness detection.

Work	Approach	Dataset/Method	Accuracy	Strengths	Weaknesses
23	CNN for yawning detection	Dashboard camera, face detection	Not reported	High accuracy potential in yawning detection	Relying solely on yawning may be insufficient
24	CNN for facial landmarks	Facial landmark detection	88% (no glasses), 85% (night), 83% overall	Effective use of facial landmarks	Accuracy capped at 83%; may need more features
28	InceptionV3 + LSTM	NTHU-DDD dataset	93.69% (training), 81.04% (testing), F1: 82	High training accuracy	Reliance on eyes/mouth may limit effectiveness
29	Multiple CNNs (AlexNet, VGG, etc.)	Various facial expressions	85%	Reliable with multiple models	High computational cost, heavy model
33	CNN + RNN	60s sequence of images	65% (training), 60% (testing)	Unique approach	Poor results, low accuracy
34	CNN for facial landmarks	Android device on dashboard	80%	Portable solution (Android)	Accuracy needs improvement
25	CNN + BiLSTM	Eye detection	85% (CNN), 94% (CNN + BiLSTM)	Effective combination of CNN and BiLSTM	Focus on eyes may not be sufficient
30	ResNet + LSTM	Facial and eye detection	87.19%	Combines spatial and temporal information	May be computationally intensive
27	CNN + EEG features	EEG data	73.22%	Uses EEG for neurophysiological data	Low accuracy, can be improved
26	RNN (R-LSTM, C-LSTM) for eye detection	Eye images	82% (R-LSTM), 95–97% (C-LSTM)	High accuracy with C-LSTM	Eye-only detection may not work well in practice
32	CNN + Emotion analysis	Facial recognition, driving patterns	93%	Considers both facial recognition and driving patterns	Requires comprehensive data, may be complex

In addition, multi-model frameworks hinder real-time implementation. Also, several methods lack evaluation on large, diverse datasets, reducing generalizability.

Contribution of the current work

In order to fill these deficiencies, the proposed work uses pre-trained CNN models (VGG19, VGG16, ResNet150, and DenseNet201) that have been fine-tuned and combined with an attention mechanism to dynamically priorities important face regions (mouth, eyes). This method, which has been validated on the varied NTHU-DDD dataset, guarantees high accuracy, real-time efficiency, and robustness, establishing a new standard in driver sleepiness detection research.

Materials and methods

This section provides a comprehensive overview of the materials and methodologies employed in this work to develop an effective driver drowsiness detection system. It begins by detailing the NTHU-DDD dataset, a crucial resource for training and evaluating the deep learning models. The subsequent subsections describe the preprocessing techniques used to prepare the dataset, including resizing, facial feature detection, and normalization. Additionally, the section elaborates on the integration of pretrained models and the development of a custom CNN with an attention mechanism, highlighting their roles in enhancing detection accuracy. Finally, the training process, hyperparameter fine-tuning, and detection outcomes are presented, illustrating the systematic approach undertaken to achieve reliable and robust performance in real-time driver monitoring.

NTHU-DDD dataset

The NTHU-DDD dataset, a comprehensive and enhanced version of the original NTHU-DDD dataset, is an essential resource for advancing research in driver drowsiness detection. Developed by the Computer Vision Lab at National Tsing Hua University, this dataset includes meticulously annotated features that capture key fatigue indicators such as eye state, mouth movements, head positions, and overall drowsiness levels. These detailed annotations facilitate multi-task learning, enabling models to predict multiple behaviors simultaneously, thereby improving their efficiency and mitigating overfitting risks.

One of the most significant strengths of the dataset lies in its ability to reflect real-world driving scenarios. It incorporates diverse conditions, including variations in lighting, head poses, and facial expressions, ensuring robust model training and testing. This diversity equips deep learning algorithms with the capacity to detect drowsiness in dynamic environments, enhancing their applicability in real-time settings.

The dataset’s utility extends to supporting advanced deep learning architectures, such as Inception-ResNet, for efficient spatial feature analysis and accurate prediction of fatigue-related behaviors. Its rich annotations and comprehensive nature make it a foundational tool for developing reliable and scalable drowsiness detection systems. By addressing the critical challenges of generalization and real-time detection, the NTHU-DDD dataset has become instrumental in advancing road safety through cutting-edge driver-monitoring technologies.

Data acquisition and preprocessing

This subsection outlines the key steps involved in preparing the data for use in the driver drowsiness detection system. It begins by describing the data acquisition process, which involves sourcing high-quality, annotated video data from the NTHU-DDD dataset to capture essential fatigue-related behaviors. Following this, the preprocessing techniques applied to the dataset are discussed in detail, including resizing, normalization, and facial feature detection. These steps ensure that the data is standardized and optimized for effective training, enabling the deep learning models to accurately identify subtle and complex drowsiness-related patterns.

Data acquisition

Data acquisition forms the foundation of any deep learning model, as depicted in Fig. 1, and is particularly critical in applications such as driver drowsiness detection, where the quality, diversity, and richness of input data directly impact model performance. This work leverages the NTHU-DDD dataset, a publicly available resource specifically curated for driver fatigue detection research.

Fig. 1.

Workflow diagram for the steps of the proposed work.

The dataset comprises high-resolution videos capturing a wide range of driving scenarios, with drivers recorded in varying states of alertness. These videos are meticulously annotated to highlight key indicators of fatigue, such as drowsiness, yawning, blinking, and other subtle behavioral cues. These detailed labels serve as a vital resource for training robust models capable of accurately identifying fatigue-related behaviors. By providing diverse and well-labeled data, the NTHU-DDD dataset ensures the reliability and scalability of the resulting deep learning models, facilitating effective real-world implementations.

The NTHU-DDD dataset is particularly valuable for this work as it is specifically tailored to driver fatigue detection. It provides not only detailed video footage of drivers but also meticulously annotated data capturing critical elements such as facial expressions and contextual information within the vehicle. These features are essential for identifying subtle signs of fatigue, which are often challenging to detect. By focusing on real-world driving conditions, the dataset ensures that the model learns to recognize drowsiness under diverse scenarios, enhancing its robustness and applicability.

Following data acquisition, a data conversion phase is performed to prepare the videos for processing by deep learning models. In this step, the videos are decomposed into individual frames, with each frame treated as a distinct image. This conversion simplifies the dataset and enables the model to analyze each frame independently while also capturing temporal patterns over time. By examining these frames, the system tracks changes in facial features, such as eyelid movements, facial expressions, and blinking patterns, critical indicators of fatigue. This preprocessing step is fundamental for creating a robust dataset that equips deep learning models to learn various patterns and characteristics associated with drowsiness. Leveraging such a comprehensive dataset ensures that the model is capable of accurately detecting signs of fatigue across a wide range of real-world scenarios.

Data preprocessing

Once the data has been acquired, it undergoes preprocessing to ensure that the input is standardized and optimized for training in deep learning models. The preprocessing stage involves several key operations:

• A.Data resizing

In the preprocessing phase, the images extracted from video frames are resized to a uniform resolution of 128 × 128 pixels. This resizing step ensures consistency across all images in the dataset, which is essential for CNNs that require fixed input dimensions. Without uniform sizing, CNNs would encounter processing inconsistencies during training, potentially leading to inefficiencies and reduced learning capacity.

By standardizing the image dimensions, the model can efficiently process each frame without requiring additional configurations or separate models for varying image sizes. This uniformity simplifies the data pipeline and allows the network to focus on detecting patterns related to driver drowsiness. Furthermore, resizing ensures that computational resources are used effectively, striking a balance between preserving critical image details and reducing the input size for faster processing.

This preprocessing step plays a pivotal role in improving the model’s ability to learn patterns associated with fatigue-related behaviors, such as eyelid movements, yawning, and facial expressions, ensuring reliable and robust performance across the dataset.

• B.Face, eye, and mouth detection with Dlib

To enhance the accuracy of driver drowsiness detection, the system employs the Dlib library for facial landmark detection. Dlib is a robust and widely adopted tool for face detection and analysis, capable of identifying critical facial features such as the eyes, mouth, and overall face shape. These features are instrumental in detecting subtle signs of fatigue, including drooping eyelids, frequent blinking, and yawning. By focusing on these specific facial cues, the system can effectively assess a driver’s state of alertness with greater precision.

Face detection isolates the most relevant region of the image, the driver’s face, where signs of drowsiness are most evident. Within this region, the eyes and mouth serve as primary indicators, enabling the system to monitor critical behaviors such as blinking frequency, squinting, and yawning. These behaviors provide valuable insights into fatigue-related states, ensuring a more comprehensive and nuanced analysis. By leveraging Dlib’s precise facial landmark detection capabilities, the system enhances its ability to track and interpret these key features, significantly improving the reliability and timeliness of drowsiness predictions.

This detailed analysis of facial features not only increases detection accuracy but also ensures prompt and dependable alerts, contributing to enhanced driver safety.

• C.Data rescaling

Once the images are resized and facial features are detected, the pixel values undergo a rescaling process to a range between 0 and 1. This normalization step standardizes the input data, ensuring uniformity across all pixel values. Normalization plays a critical role in enhancing the efficiency and effectiveness of the training process, as neural networks perform optimally when input features are scaled consistently.

By transforming pixel values into a uniform range, the model avoids being disproportionately influenced by extreme values within the dataset. This helps to stabilize the learning process and accelerates model convergence, allowing for faster and more reliable training. Additionally, rescaling ensures that all features contribute equally to the learning process, reducing the risk of numerical instabilities and improving the overall performance of the neural network.

This preprocessing step is crucial for building a robust and efficient driver drowsiness detection system, as it lays the foundation for stable and accurate model performance during training and inference.

Detection: applying pretrained models and fine-tuning

At this stage, the processed data is utilized to train deep learning models for accurate detection of driver drowsiness. The detection framework leverages pretrained models, including VGG19, VGG16, ResNet50, and DenseNet201, as well as custom models developed from scratch. Pretrained models are particularly advantageous, as they are initialized with weights derived from extensive prior training on large-scale datasets, such as ImageNet. This transfer learning approach allows the models to effectively generalize to the driver drowsiness detection task, even with limited labeled data, significantly reducing the time and computational resources required for training.

The pretrained models are fine-tuned on the NTHU-DDD dataset to adapt their parameters to domain-specific features, such as eye movements, yawning, and head posture changes. This fine-tuning process enhances the models’ ability to accurately detect subtle signs of fatigue, improving their robustness in diverse and dynamic driving scenarios. By combining the generalization power of pretrained architectures with the specificity achieved through fine-tuning, this approach ensures high detection accuracy and reliability.

Custom models trained from scratch are also incorporated into the framework, providing an opportunity to design architectures tailored specifically to the nuances of the dataset. These scratch models complement pretrained architectures by allowing for greater flexibility in optimizing for task-specific requirements.

This hybrid approach of leveraging pretrained and scratch models facilitates real-time driver monitoring systems that are both efficient and accurate. The integration of these models enhances safety by enabling prompt detection of drowsiness, thereby mitigating risks and improving overall driving outcomes.

Pretrained models with transfer learning

The adoption of pretrained models such as VGG19, VGG16, ResNet50, and DenseNet201 plays a pivotal role in enhancing the efficiency and accuracy of detecting driver drowsiness. These models, initially trained on large-scale datasets like ImageNet, possess the ability to recognize a wide range of complex features and patterns. By leveraging transfer learning, these pretrained models are fine-tuned on the specific NTHU-DDD dataset, adapting their parameters to identify domain-specific fatigue indicators, including blinking, yawning, and head nodding.

Transfer learning significantly reduces the need for extensive labeled data by utilizing the generalized feature extraction capabilities of pretrained architectures. The fine-tuning process adjusts the models to focus on the nuances of driver fatigue, ensuring accurate detection even in diverse real-world scenarios. This approach not only minimizes computational and data requirements but also accelerates the model’s training process, making it highly suitable for real-time applications.

By integrating pretrained models with the NTHU-DDD dataset, the system effectively combines the advantages of extensive prior training with domain-specific adaptability. This strategy enables robust and reliable detection of drowsiness, ensuring timely alerts to enhance driver safety and reduce the risks associated with fatigue-induced incidents.

• A.VGG16

VGG16 is a widely recognized CNN architecture introduced by the Visual Geometry Group (VGG) at Oxford University in 2014. It comprises sixteen layers, including thirteen convolutional layers and three fully connected layers, with max-pooling layers strategically placed between convolutional layers to downsample the spatial dimensions of the input. What sets VGG16 apart is its simplicity and uniform design, as all convolutional layers employ 3 × 3 filters and max-pooling layers use a consistent 2 × 2 kernel. This straightforward structure makes VGG16 highly implementable while maintaining its robust feature extraction capabilities.

The depth of VGG16 plays a crucial role in its ability to capture hierarchical features. The initial layers detect fundamental patterns, such as edges and textures, while deeper layers capture more complex and abstract features, such as facial expressions. This hierarchical feature extraction is particularly beneficial for driver drowsiness detection, where recognizing subtle facial cues like blinking, eyelid closure, and head nodding is essential. While the computational cost of VGG16 is relatively high due to its depth, its precision and versatility make it an ideal candidate for tasks requiring detailed pattern recognition, such as detecting signs of driver fatigue.

When fine-tuned on domain-specific datasets like NTHU-DDD, VGG16 demonstrates exceptional adaptability in detecting facial features associated with driver fatigue. This fine-tuning allows the model to classify drivers’ states as alert or fatigued accurately, providing critical real-time alerts to enhance road safety and mitigate fatigue-induced risks. The model’s ability to generalize effectively to fatigue-specific patterns ensures its reliability in diverse driving conditions.

• B.VGG19

VGG19, developed by the same team behind VGG16, is an extended version of the VGG16 architecture, consisting of nineteen layers in total. This deeper model includes sixteen convolutional layers and three fully connected layers, with additional convolutional layers allowing VGG19 to capture more complex and nuanced features. While maintaining the core design principles of VGG16, such as the use of 3 × 3 convolutional filters and 2 × 2 max-pooling layers, VGG19’s increased depth makes it more adept at detecting subtle and intricate patterns, which are essential in tasks like driver drowsiness detection.

The deeper architecture of VGG19 enables it to extract hierarchical features at varying levels of abstraction. While VGG16 is proficient in identifying prominent facial expressions, VGG19 refines these capabilities, excelling in detecting subtle cues such as slight blinking, gradual eyelid closures, or minor head nods. These subtle indicators are critical for accurately distinguishing between alert and fatigued drivers, making VGG19 an excellent choice for applications requiring high precision and sensitivity.

However, the enhanced feature extraction capabilities of VGG19 come at the cost of increased computational demands, including higher memory usage and longer processing times. Despite these challenges, VGG19 remains a popular choice in transfer learning due to its pretrained weights from large datasets like ImageNet. Fine-tuning VGG19 on domain-specific datasets such as NTHU-DDD allows the model to adapt to specialized tasks like facial expression recognition and drowsiness detection, ensuring robust and reliable performance in real-time driver monitoring systems. This fine-tuning process enables VGG19 to effectively capture fatigue-related behaviors, such as eye closures and yawning, ultimately improving road safety by delivering timely alerts to mitigate risks.

• C.ResNet50

ResNet (Residual Networks), introduced by Kaiming He and colleagues in 2015, revolutionized deep learning by addressing the challenges of training very deep neural networks. The innovation of residual connections, which enable the network to bypass one or more layers during training, effectively solved the problem of vanishing gradients, a persistent issue in deep architectures. These residual connections facilitate the flow of gradients during backpropagation, allowing networks with hundreds or even thousands of layers to train efficiently without performance degradation. ResNet50, a fifty-layer architecture, has become one of the most widely used models for complex computer vision tasks, such as image classification, object detection, and facial recognition.

A defining feature of ResNet50 is its use of residual connections, which allow the network to skip specific layers if they are deemed redundant during training. This capability ensures stable gradient flow, enabling the network to learn complex features more effectively while maintaining efficient training. The architecture’s depth and ability to extract detailed hierarchical features make it particularly well-suited for applications like driver drowsiness detection, where identifying subtle facial cues such as blinking, eyelid closure, and head nodding is critical. Despite the computational demands of its deeper architecture, ResNet50 consistently outperforms many other models by efficiently capturing and processing intricate patterns.

In the context of driver drowsiness detection, ResNet50’s ability to detect fine-grained facial cues is invaluable. It excels in identifying micro-expressions, such as slight changes in eyelid movements or subtle head nods, which are key indicators of fatigue. These capabilities make it a powerful tool for real-time monitoring systems, where the early detection of fatigue-related behaviors can significantly enhance road safety by preventing accidents. By fine-tuning ResNet50 on specialized datasets like NTHU-DDD, the model adapts to domain-specific features, achieving high accuracy in classifying facial cues related to alertness and fatigue. This adaptability and precision establish ResNet50 as an essential component in the development of robust driver monitoring systems.

• D.DenseNet201

DenseNet (Densely Connected Convolutional Networks), introduced by Gao Huang and colleagues in 2017, represents a significant advancement in deep learning architecture. DenseNet enhances traditional CNNs by introducing dense connections, where each layer receives input from all preceding layers. This approach contrasts with standard CNNs, where layers typically rely only on direct connections from the immediately prior layer. By enabling extensive feature reuse, DenseNet achieves more efficient learning and requires fewer parameters to deliver performance that matches or surpasses traditional models. DenseNet201, a specific variant of this architecture, consists of 201 layers and showcases the effectiveness of densely connected networks in handling complex tasks.

A defining characteristic of DenseNet201 is its dense connectivity. In this architecture, each layer benefits from the features learned by all earlier layers, fostering a richer representation of the data. This connectivity enables the network to capture intricate patterns, making it particularly well-suited for applications like facial expression recognition in driver drowsiness detection. For instance, dense connections allow the model to effectively learn subtle facial cues such as blinking patterns, yawning, and slight head movements, essential indicators of fatigue. This feature reuse mechanism not only enhances learning but also reduces the number of parameters required for training, making DenseNet201 more efficient in both training and optimization compared to traditional deep networks.

DenseNet201 also addresses the common problem of vanishing gradients in deep networks, similar to ResNet. By ensuring a strong flow of gradients throughout the network, DenseNet supports effective backpropagation, even in very deep architectures. This capability is critical for training on large datasets, as it prevents stagnation during learning and improves model performance over time. Furthermore, the feature reuse inherent in DenseNet201 enables it to achieve superior accuracy with fewer parameters, making it an ideal choice for real-time systems like driver drowsiness detection. Its computational efficiency ensures that DenseNet201 can be deployed in resource-constrained environments without compromising on speed or precision, making it a valuable tool for enhancing road safety.

Fine-tuning for hyperparameters

Fine-tuning is a pivotal step in adapting pre-trained models to domain-specific tasks, such as driver drowsiness detection. It involves adjusting key hyperparameters to optimize the model’s performance and ensure efficient learning on the target dataset. By tailoring these parameters, the model can better adapt to the unique characteristics of datasets like NTHU-DDD, which emphasize fatigue-related facial cues. This process not only improves the model’s accuracy but also enhances its generalization to unseen data, ensuring robust performance across diverse conditions.

One of the most critical hyperparameters fine-tuned during this process is the learning rate (), which determines the step size for updating the model’s weights during training. The weight update at each iteration can be mathematically expressed as:

1

where represents the weight at iteration  t, is the learning rate, and is the gradient of the loss function  L with respect to ​. Selecting an optimal learning rate is crucial for achieving efficient training. A learning rate that is too high may cause instability or divergence, as the weight updates overshoot the optimal solution. Conversely, a learning rate that is too low can result in slow convergence, utilizing the training data and increasing computational cost.

In the context of driver drowsiness detection, fine-tuning also involves adjusting other hyperparameters, such as batch size, dropout rate, and the number of epochs, to balance computational efficiency and model performance. By systematically optimizing these parameters, the fine-tuning process ensures that the model is well-suited for detecting subtle fatigue-related patterns, ultimately improving its reliability in real-world scenarios.

Another key hyperparameter in fine-tuning is the batch size, which determines the number of samples processed before the model updates its weights. Smaller batch sizes result in more frequent weight updates, introducing noise into the gradient descent process but often leading to better generalization. On the other hand, larger batch sizes provide smoother and more stable updates but demand greater computational resources and carry the risk of converging to sharp minima, which may negatively affect the model’s generalization ability.

The number of epochs is another critical hyperparameter. An epoch represents one complete pass through the training dataset. If the model is trained with too few epochs, it may underfit, failing to learn sufficient patterns from the data. Conversely, excessive epochs can lead to overfitting, where the model memorizes the training data at the expense of its ability to perform well on unseen data. Striking the right balance in the number of epochs is essential to ensure the model learns effectively without sacrificing its ability to generalize.

Fine-tuning involves systematically adjusting these hyperparameters to balance learning efficiency, model accuracy, and computational cost. For instance, in driver drowsiness detection tasks, models like ResNet50 or DenseNet201 may require different learning rates, batch sizes, or epoch numbers depending on the complexity of the dataset and the subtlety of the drowsiness-related facial features being analyzed. This process ensures that pretrained models are optimally adapted to their specific applications, achieving robust and reliable performance in real-world scenarios, such as real-time driver monitoring systems.

Scratch model using CNN with attention

The integration of an attention mechanism into a scratch model built on a CNN architecture significantly enhances the model’s ability to detect subtle yet critical features associated with driver drowsiness. While CNNs are highly effective at learning spatial hierarchies of features from images, their focus is typically distributed across the entire image, which can dilute attention to the most relevant regions. By incorporating an attention mechanism, the model dynamically prioritizes specific regions of interest, such as the eyes and mouth, that are essential indicators of fatigue. This targeted approach ensures the model accurately identifies drowsiness-related cues, including blinking frequency, eyelid drooping, and yawning, which might otherwise go unnoticed.

Mathematically, an attention mechanism is represented as a weighted focus on specific features extracted by the CNN. Let  X denote the input features extracted by the CNN, and let ​ represent the weights corresponding to the importance of each feature ​. The attention-enhanced output  Y is computed as:

2

Here, the weights ​ are dynamically learned by the model during training, based on the relevance of each feature to the task of detecting drowsiness. Regions with higher relevance, such as those associated with eyelid movements or mouth positions, receive greater attention scores, allowing the model to focus its computational resources on the most critical aspects of the image.

This attention-driven refinement not only improves the model’s accuracy but also enhances its interpretability by highlighting the regions contributing most to its predictions. The combination of CNN’s robust spatial feature extraction with the precision of attention mechanisms makes the scratch model particularly effective for real-time driver monitoring systems, ensuring timely and accurate detection of fatigue-related behaviors.

Without an attention mechanism, CNNs may broadly generalize features across the entire facial region, potentially overlooking critical micro-expressions that are essential for differentiating between an alert and drowsy driver. Incorporating an attention mechanism allows the model to dynamically adjust its focus based on the visual cues present in each image. For instance, when fatigue-related signs such as frequent blinking or drooping eyelids are detected in the eyes, the attention mechanism prioritizes these regions, leading to more precise and sensitive predictions.

As illustrated in Fig. 2, the proposed model is a custom-designed CNN integrated with an attention mechanism using Squeeze-and-Excitation (SE) blocks. This architecture enhances feature selection by dynamically reweighting feature channels, enabling the network to concentrate on the most relevant aspects of the input data. The inclusion of SE blocks significantly improves classification accuracy, making the model more effective at identifying subtle indicators of fatigue.

Fig. 2.

Scratch model using CNN with attention.

The architecture consists of three primary convolutional blocks, each followed by an SE block, culminating in fully connected layers and a SoftMax output layer for classification. These convolutional blocks form the backbone of the network, processing input data sequentially to extract spatial and hierarchical features. The first convolutional block employs 32 filters, while the second and third blocks progressively increase the filter count to 64 and 128, respectively, to capture increasingly complex patterns. Batch normalization follows each convolutional layer to stabilize and accelerate the training process, ensuring consistent weight updates.

Non-linearity is introduced through the Rectified Linear Unit (ReLU) activation function, enhancing the network’s ability to model complex relationships in the data. Max pooling layers, represented as downscaling operations in Fig. 4, are incorporated to reduce the spatial dimensions of feature maps. This reduces computational complexity while preserving essential information, optimizing the network for real-time applications. Together, these components create a robust model capable of accurately detecting drowsiness-related behaviors, enhancing driver safety through timely and reliable predictions.

Fig. 4.

VGG19 lose curve.

Squeeze-and-Excitation (SE) blocks are strategically incorporated after each convolutional block to enhance the network’s ability to prioritize significant features. The SE mechanism operates in three stages:

1. Squeeze Phase: Global average pooling is applied to condense the spatial dimensions of the feature maps into channel-wise descriptors, summarizing the global distribution of information for each channel.

2. Excitation Phase: A lightweight fully connected network computes importance weights for each channel. This step dynamically learns which channels contribute most to the task at hand, such as detecting drowsiness-related features.

3. Reweighting Phase: Element-wise multiplication is used to apply the computed weights to the feature maps, amplifying relevant features while suppressing less significant ones. This mechanism strengthens the model’s focus on key regions of the input image, such as the eyes and mouth, improving its ability to detect subtle fatigue-related cues.

These SE blocks form the backbone of the attention mechanism, dynamically adjusting the network’s focus to maximize its predictive performance. The final stage of the architecture processes the extracted features globally using fully connected layers. After the feature maps are flattened into a one-dimensional vector, the data is passed through a dense layer with 512 neurons. To prevent overfitting, regularization techniques such as L2 regularization and dropout are applied during this stage. The network’s output layer uses a SoftMax activation function, producing probabilities for the four target classes (e.g., alert, drowsy, partially drowsy, and fatigued), as illustrated in the output section.

For training and optimization, the Adam optimizer is employed with a learning rate of 0.0001, ensuring adaptive adjustments to the model’s weights for efficient convergence. The binary cross-entropy loss function is used to compute the error, with accuracy serving as the primary evaluation metric during training. These strategies ensure that the model learns effectively while minimizing the risk of overfitting, ultimately achieving reliable performance in real-world applications.

The integration of attention mechanisms significantly reduces the likelihood of misclassifications, especially in scenarios where drowsiness indicators are subtle or fleeting. By dynamically emphasizing the most informative regions of the image, such as the eyes and mouth, the model minimizes the influence of irrelevant features and noise. This targeted focus enhances both the accuracy and robustness of the system, ensuring reliable detection of fatigue-related behaviors. These qualities make attention-enhanced CNNs particularly well-suited for real-time applications, such as driver-monitoring systems, where precision and efficiency are critical to maintaining safety and preventing accidents.

Detection outcomes

The deep learning models classify the driver’s state into two primary categories: Drowsy and Not Drowsy. This classification is derived from the analysis of various facial features and behaviors indicative of fatigue, enabling the system to provide accurate and actionable insights for driver safety.

Drowsy class

The Drowsy category encompasses specific subcategories representing distinct signs of driver fatigue:

• Sleepy Combination: Reflects a general sense of tiredness, characterized by subtle yet observable fatigue indicators.

• Blink with Nodding Head: Captures frequent and involuntary head nodding, often accompanied by blinking patterns, both of which are strongly associated with drowsiness.

• Yawning: A well-established and prominent indicator of fatigue, frequently observed in drowsy drivers.

By segmenting the drowsy state into these subcategories, the system ensures a more granular analysis, allowing for a deeper understanding of fatigue-related behaviors.

Not drowsy class

The Not Drowsy category signifies the absence of detectable signs of fatigue or drowsiness. Drivers classified under this category are considered alert and attentive, with no observable indicators of impaired focus or fatigue-related behaviors.

This binary classification framework enables the system to reliably distinguish between alert and fatigued states, offering critical insights for real-time driver-monitoring applications. By identifying subtle and momentary signs of drowsiness, the system supports proactive interventions, enhancing road safety and reducing the risk of fatigue-related accidents.

Experiments and results

This section presents a detailed account of the experimental design, implementation, and evaluation conducted in this work. It begins by outlining the experimental setup and hyperparameter configurations, highlighting the hardware and software used to optimize the performance of both the proposed CNN with Attention Mechanism and pretrained models. Subsequent subsections provide a comprehensive analysis of the results, including accuracy, loss, and other performance metrics, supported by visualizations such as accuracy curves, loss curves, confusion matrices, and ROC curves. The findings demonstrate the effectiveness of the proposed approach in outperforming existing methodologies, establishing a new benchmark in driver drowsiness detection.

Experimental setup

The model was evaluated on the NTHU-DDD dataset, which includes video recordings of drivers under various conditions (day/night, with/without glasses, etc.). We extracted individual frames, resized them to 224 × 224 pixels, and split the dataset into training (70%), validation (15%), and test (15%) sets. The experiments were conducted using an NVIDIA RTX 3090 GPU with 24 GB of VRAM, using the Adam optimizer, a learning rate of 0.0001, and a batch size of 32 over 50 epochs.

Evaluation metrics

To evaluate the performance of the proposed model, we adopted four widely used metrics in computer vision and machine learning research: accuracy, precision, recall, and F1-score. These metrics provide a comprehensive view of both the overall classification capability and the balance between correctly identifying drowsy and non-drowsy instances.

Analysis results

The Proposed CNN with Attention Mechanism demonstrates substantial advancements over pretrained models and prior research, as summarized in Table 2. This table provides a comprehensive comparison of performance metrics, including accuracy, loss, precision, recall, and F1-score, which collectively highlight the superior performance of the proposed approach.

Table 2.

Performance of the proposed CNN with attention mechanism and pretrained models in driver drowsiness detection.

Model	Resolution	Accuracy (%)	Loss	Precision (%)	Recall (%)	F1-score (%)
VGG19 With Fine Tunning	(128,128,3)	98.15	0.05	97	98	98
VGG16 With Fine Tunning		98.60	0.03	98	99	98
ResNet150 With Fine Tunning		99.30	0.05	99	99	99
DenseNet201 With Fine Tunning		96.80	0.07	96	97	97
Proposed CNN + Attention		99.63	0.03	100	100	100

The proposed model achieves remarkable accuracy and robust generalization, outperforming pretrained models such as VGG19, VGG16, ResNet50, and DenseNet201. The integration of the attention mechanism allows the model to focus dynamically on key facial features, such as the eyes and mouth, enabling precise identification of fatigue-related behaviors like blinking and yawning. These results underscore the effectiveness of the custom architecture in handling subtle drowsiness indicators while maintaining computational efficiency.

By achieving higher precision, recall, and F1-scores compared to pretrained models, the proposed CNN with Attention Mechanism demonstrates its ability to minimize false positives and false negatives, ensuring reliable classification of drowsy and alert states. These improvements are critical for real-world applications, such as real-time driver monitoring systems, where both accuracy and speed are paramount to enhancing road safety.

Table 2 provides a clear illustration of the proposed model’s superior performance compared to pretrained architectures like VGG16, VGG19, ResNet50, and DenseNet201. The metrics in the table emphasize the critical role of attention mechanisms in enhancing deep learning models for real-time driver drowsiness detection. By delivering near-perfect results across all evaluation criteria, the proposed model sets a new benchmark for improving road safety through advanced driver monitoring systems.

The proposed model achieves an impressive accuracy of 99.63%, significantly surpassing pretrained architectures such as ResNet50 (99.30%), VGG16 (98.60%), and DenseNet201 (96.80%). This exceptional performance underscores the transformative impact of integrating an attention mechanism into the CNN architecture, allowing the model to dynamically focus on critical facial regions, such as the eyes and mouth, key indicators of fatigue.

In terms of loss, the proposed model achieves the lowest value of 0.03, outperforming ResNet50 (0.05) and DenseNet201 (0.07). This lower loss value highlights the proposed model’s superior optimization and generalization capabilities, reducing the risk of overfitting and ensuring reliable performance across diverse datasets and conditions.

Moreover, the proposed model achieves perfect scores of 100% for precision, recall, and F1-score, demonstrating its unparalleled ability to detect all instances of drowsiness while avoiding false classifications. In contrast, pretrained models like VGG16 and ResNet50, while exhibiting strong performance with metrics in the 98–99% range, fall short of the flawless detection achieved by the proposed model.

The key advantage of the proposed model lies in its attention mechanism, which dynamically prioritizes specific regions of interest. While pretrained models rely on static feature extraction, which can result in the misclassification of subtle drowsiness cues, the proposed model excels by emphasizing facial regions that exhibit fatigue-related behaviors, such as blinking, yawning, or eye closure. This dynamic focus enables the proposed model to achieve robust and efficient detection, as reflected across all performance metrics.

VGG19 results

The VGG19 model demonstrated reliable performance, achieving an accuracy of 98.15% and a loss value of 0.05. The accuracy curve (Fig. 3) indicates a stable learning process, with consistent improvements observed across the training epochs. Similarly, the loss curve (Fig. 4) exhibits minimal fluctuations, highlighting effective optimization and a reduced risk of overfitting.

Fig. 3.

VGG19 accuracy curve.

The confusion matrix (Fig. 5) provides detailed insights into the model’s classification capabilities. While VGG19 accurately detects the majority of drowsiness instances, occasional misclassifications occur in borderline cases. These cases often involve subtle indicators of fatigue, such as minor blinking or yawning, which may overlap with non-drowsy behaviors, reflecting the inherent challenges of distinguishing such nuanced states.

Fig. 5.

VGG19 confusion matrix.

The ROC curve (Fig. 6) further illustrates the model’s performance, capturing the trade-off between sensitivity and specificity. VGG19 demonstrates robust capability in distinguishing between drowsy and non-drowsy states, with a high area under the curve (AUC). However, the model leaves room for improvement in refining its sensitivity to subtle fatigue-related cues, which could further enhance its applicability in real-world driver monitoring systems.

Fig. 6.

VGG19 ROC curve.

VGG16 results

The VGG16 model demonstrates superior performance compared to VGG19 in several key metrics, achieving an accuracy of 98.60% and a lower loss value of 0.03. The accuracy curve (Fig. 7) reflects more rapid convergence, highlighting improved training efficiency and faster stabilization during the learning process. Similarly, the loss curve (Fig. 8) exhibits smoother training dynamics, further emphasizing the model’s optimization stability across all epochs.

Fig. 7.

VGG16 accuracy curve.

Fig. 8.

VGG16 loss curve.

The confusion matrix (Fig. 9) reveals VGG16’s enhanced classification capabilities, showing a notable reduction in both false positives and false negatives. This improved performance underscores the model’s ability to detect fatigue-related features, such as blinking and yawning, with greater precision. These enhancements make VGG16 a more reliable choice for distinguishing between drowsy and non-drowsy states.

Fig. 9.

VGG16 confusion matrix.

The ROC curve (Fig. 10) further illustrates VGG16’s ability to balance sensitivity and specificity, achieving a high AUC. This balanced performance ensures reliable detection of drowsiness indicators, making VGG16 particularly suitable for real-world applications in driver monitoring systems. Its refined capability to accurately classify subtle fatigue-related behaviors positions it as a robust model for enhancing road safety.

Fig. 10.

VGG16 ROC curve.

ResNet50 results

ResNet50 emerged as one of the most effective pretrained models, achieving a high accuracy of 99.30%, with precision, recall, and F1-scores all at 99%. The accuracy curve (Fig. 11) reflects rapid and steady convergence, showcasing the efficiency of its residual connections. These connections address the vanishing gradient problem, enabling deeper feature extraction and improved learning dynamics. Similarly, the loss curve (Fig. 12) confirms the model’s stability, demonstrating a clear trajectory toward minimal loss values throughout the training process.

Fig. 11.

ResNet50 accuracy curve.

Fig. 12.

ResNet50 loss curve.

The confusion matrix (Fig. 13) highlights ResNet50’s minimal misclassification rates, underscoring its capability to accurately differentiate between drowsy and non-drowsy states. This strong performance is attributed to the model’s ability to capture subtle features, such as micro-expressions and slight head movements, which are critical for fatigue detection.

Fig. 13.

ResNet50 confusion matrix.

The ROC curve (Fig. 14) further emphasizes ResNet50’s exceptional balance between sensitivity and specificity, achieving a high AUC. This balance ensures reliable detection of nuanced fatigue indicators, making ResNet50 a robust option for driver monitoring systems. Its ability to handle complex patterns with efficiency positions, it is a strong contender among pretrained models for real-world applications in enhancing road safety.

Fig. 14.

ResNet50 ROC curve.

DenseNet201 results

DenseNet201, despite its innovative use of dense connections to facilitate feature reuse and reduce parameter redundancy, achieved slightly lower performance compared to other models. The model attained an accuracy of 96.80% and a loss value of 0.07. The accuracy curve (Fig. 15) indicates slower convergence, suggesting challenges in optimizing its extensive 201-layer architecture for the specific task of drowsiness detection. Similarly, the loss curve (Fig. 16) exhibits higher variability, pointing to difficulties in achieving stable training dynamics.

Fig. 15.

DenseNet201 accuracy curve.

Fig. 16.

DenseNet201 loss curve.

The confusion matrix (Fig. 17) highlights higher misclassification rates, particularly in detecting borderline drowsy cases. Subtle fatigue indicators, such as transient blinking or brief yawning, proved more challenging for DenseNet201 to classify correctly. This limitation suggests that the dense connection strategy, while beneficial in other contexts, may have led to redundancy or less focused feature extraction in this specific application.

Fig. 17.

DenseNet201 confusion matrix.

The ROC curve (Fig. 18), though adequate, underscores these challenges, reflecting DenseNet201’s relative difficulty in maintaining an optimal balance between sensitivity and specificity. While the model demonstrates reasonable performance, its struggles with nuanced fatigue detection highlight areas for improvement. Despite these limitations, DenseNet201 remains a valuable model for applications where dense feature reuse and generalization are prioritized, although it may require further fine-tuning for tasks involving subtle, transient visual cues.

Fig. 18.

DenseNet201 ROC curve.

Results of the proposed CNN with attention mechanism

The Proposed CNN with Attention Mechanism achieved outstanding performance, with an accuracy of 99.63% and perfect scores of 100% for precision, recall, and F1-score. The accuracy curve (Fig. 19) illustrates rapid and stable convergence, driven by the dynamic focus enabled by the attention mechanism. This mechanism enhances the model’s ability to prioritize key facial regions, such as the eyes and mouth, ensuring precise detection of fatigue-related behaviors.

Fig. 19.

Proposed CNN with attention accuracy curve.

The loss curve (Fig. 20) demonstrates minimal values across all epochs, reflecting efficient optimization and robust training dynamics. The model’s ability to minimize loss consistently highlights the effectiveness of its architecture in avoiding overfitting while generalizing well to unseen data.

Fig. 20.

Proposed CNN with attention loss curve.

The confusion matrix (Fig. 21) reveals flawless classification performance, with no false positives or false negatives. This result underscores the critical advantage of the attention mechanism in dynamically focusing on regions most indicative of fatigue, such as subtle blinking, yawning, or eyelid closure. By doing so, the model accurately differentiates between drowsy and non-drowsy states, even in challenging borderline cases.

Fig. 21.

Proposed CNN with attention confusion matrix.

The ROC curve (Fig. 22) confirms the model’s unparalleled capability in distinguishing between classes, achieving near-perfect sensitivity and specificity. This exceptional balance ensures that the model reliably detects all instances of drowsiness while maintaining minimal errors. The proposed CNN with Attention Mechanism sets a new standard in driver monitoring systems, demonstrating the transformative impact of incorporating attention mechanisms into deep learning models for real-time applications aimed at enhancing road safety.

Fig. 22.

Proposed CNN with attention ROC curve.

Comparative study

Driver drowsiness detection has been a central area of focus in recent research, with numerous methodologies proposed to enhance detection accuracy and address the limitations of traditional systems. Table 3 provides a summary of key contributions and performance metrics from several prominent studies in this domain, highlighting the variations in datasets, methodologies, and evaluation criteria. These comparisons underscore the strengths and limitations of each approach, offering valuable insights into the advancements and challenges in the field.

Table 3.

Comparison of results with earlier studies.

References	Dataset	Accuracy (%)	Precision (%)	Recall (%)	F1-score (%)
35	NTHU-DDD	98.26%	99.45%	98.10%	-
29		85%.	86%	82%	84%
36		98.91%	-	-	-
37		97.31%	96.82%	98.35%	97.58%
38		99%	99%	99%	99%
Proposed CNN with attention		99.63%	100%	100%	100%

One notable study by³⁵ employed a Multi-Scale Convolutional Neural Network (MCNN) framework in conjunction with the Flamingo Search Algorithm (FSA) for feature optimization. To preprocess video frames, the study used advanced wavelet transforms, enabling the extraction of multi-scale features critical for identifying fatigue-related behaviors. This approach achieved an accuracy of 98.26% on the NTHU-DDD dataset, demonstrating its effectiveness in detecting driver drowsiness. However, the absence of metrics such as precision, recall, and F1-score limited the evaluation of the model’s robustness and its ability to balance sensitivity and specificity. While the study’s high accuracy is commendable, the lack of comprehensive performance metrics underscores the importance of multi-dimensional evaluations in assessing the practical applicability of drowsiness detection systems.

In²⁹, an ensemble system combining multiple deep learning models, AlexNet, VGG-FaceNet, FlowImageNet, and ResNet, was proposed. This innovative approach integrated diverse features, including facial expressions, hand gestures, and behavioral traits, aiming to enhance the model’s robustness. However, despite its complexity, the system achieved an accuracy of only 85%, indicating that the integration of multiple features might have introduced challenges in optimization, potentially hindering performance. This result highlights the trade-off between architectural complexity and practical effectiveness.

In³⁶, a Multi-Task CNN was introduced to analyze eye closure and yawning frequency, classifying driver fatigue into three distinct levels. This model achieved a notable accuracy of 98.91%, effectively leveraging both eye and mouth features for classification. However, the study did not report key metrics such as precision, recall, or F1-score, which are essential for a comprehensive evaluation of the model’s reliability, especially in real-world scenarios with diverse and subtle fatigue indicators.

A hybrid CNN-LSTM model was presented in³⁷, combining the feature extraction capabilities of convolutional layers with the sequential learning power of LSTM networks. This hybrid approach achieved a validation accuracy of 97.31% on the NTHU dataset, demonstrating its effectiveness in capturing temporal dependencies critical for detecting driver drowsiness. However, its slightly lower performance compared to other methods suggests room for optimization, particularly in refining its temporal modeling or improving its handling of subtle fatigue-related behaviors.

In^38,39, the focus was on a non-invasive, real-time detection system that analyzed visual features such as eye aspect ratio, mouth aspect ratio, and head pose. The system employed traditional machine learning classifiers, including random forest, sequential neural networks, and support vector machines (SVMs), achieving an impressive accuracy of 99%. Its simplicity and suitability for real-time applications make it a strong candidate for practical implementation. However, the reliance on traditional classifiers may pose scalability challenges when handling complex or large-scale datasets, as these methods lack the dynamic adaptability of modern deep learning architectures.

In contrast, the Proposed CNN with Attention Mechanism leverages state-of-the-art deep learning techniques, integrating attention mechanisms to dynamically prioritize critical facial features, such as the eyes and mouth. This approach, combined with the NTHU-DDD dataset, produced exceptional results, achieving an accuracy of 99.63% with perfect precision, recall, and F1-score (100%). The attention mechanism significantly enhances the model’s ability to detect subtle and transient fatigue indicators, such as blinking, yawning, and eyelid closure, ensuring robustness and reliability in real-world scenarios.

The advanced architecture of the proposed model, supported by its ultramodern metrics, demonstrates clear advantages over traditional methods. By focusing dynamically on fatigue-related features, the proposed model outperforms conventional approaches in terms of both accuracy and robustness. Its ability to adaptively prioritize regions of interest makes it a more scalable and effective solution for real-time drowsiness detection, positioning it as a benchmark for next-generation driver monitoring systems.

Table 3 provides a comparative summary of the performance metrics and methodologies of various driver drowsiness detection approaches. While studies such as^35,38 demonstrated strong accuracy, they fell short in offering comprehensive evaluations across multiple metrics or ensuring scalability for broader applications. In contrast, the Proposed CNN with Attention Mechanism achieves the highest metrics across all evaluation parameters, including perfect precision, recall, and F1-scores. By integrating attention mechanisms with transfer learning on pretrained architectures such as VGG19 and ResNet150, the proposed model bridges these gaps. Its robust and scalable design positions it as a leading solution for future applications in semi-autonomous and autonomous vehicles, ensuring reliable and efficient real-time drowsiness detection.

Conclusion, limitations, and future studies

In conclusion, the proposed CNN with an attention mechanism establishes a new benchmark in driver drowsiness detection, achieving exceptional performance across all evaluation metrics, including accuracy, precision, recall, and F1-score. The integration of the attention mechanism enables the model to dynamically focus on critical facial regions, such as the eyes and mouth, enhancing its ability to detect subtle fatigue indicators, including blinking, yawning, and eyelid closure. This capability makes the model not only highly efficient but also robust, paving the way for reliable and scalable real-time driver monitoring systems.

Despite these achievements, this work is not without its limitations. The model’s performance heavily relies on the NTHU-DDD dataset, which, while comprehensive, may not fully capture the diversity of real-world driving conditions, such as variations in lighting, weather, and cultural differences in facial features and behaviors. Additionally, the computational requirements for the attention mechanism and deep learning architecture may pose challenges for deployment in resource-constrained environments, such as edge devices in vehicles.

• Limitations

  • The NTHU-DDD dataset may not encompass the full spectrum of driving conditions, limiting the model’s adaptability to extreme lighting conditions or varied demographic and cultural contexts.
  • The attention mechanism increases the computational complexity, potentially limiting its deployment in real-time systems with constrained resources.
  • The reliance on visual features alone may limit the detection of non-visual drowsiness indicators, such as posture or physiological signals.

• Future studies

  • Future research should focus on expanding the dataset to include diverse driving conditions, demographics, and cultural variations to improve the model’s generalization capabilities.
  • Efforts should be directed towards optimizing the model for real-time applications, ensuring minimal latency and compatibility with resource-constrained edge devices.
  • Incorporating multimodal data sources, such as physiological signals (e.g., heart rate, EEG), voice analysis, or vehicle dynamics, can enhance the model’s accuracy and reliability in detecting drowsiness.
  • Future work could explore testing the model in simulated and real-world driving environments to assess its performance under varied and unpredictable conditions.
  • Investigating methods to improve the interpretability of attention mechanisms can make the system more transparent and trustworthy for end-users and stakeholders.

Consequently, this research not only advances the field of deep learning-based driver monitoring systems but also contributes significantly to improving road safety. By addressing driver fatigue, one of the leading causes of vehicular accidents, the proposed work lays a strong foundation for the development of next-generation intelligent transportation systems and semi-autonomous vehicles.

Author contributions

Samy Abd El-Nabi conceived the study, designed the methodology, and contributed to the data analysis. Ahmed F. Ibrahim was responsible for the implementation of the deep learning models and result validation. Hosny H. Abo Emira handled dataset preprocessing and statistical evaluation. Samah Alshathri contributed to the literature review and drafted the initial version of the manuscript. El-Sayed M. El-Rabaie reviewed the experimental setup and assisted in interpreting the findings. Ahmed Emam reviewed the experimental setup and assisted in interpreting the findings. Walid El-Shafai supervised the project, revised the manuscript critically, and approved the final version for submission. All authors are contributing and accepting to submit and publish the current work.

Funding

This work is supported by Princess Nourah bint Abdulrahman University Researchers Supporting Project number (PNURSP2026R197), Princess Nourah bint Abdulrahman University, Riyadh, Saudi Arabia.

Data availability

The datasets used and analysed during the current study are publicly available in the Kaggle repository at https://www.kaggle.com/datasets/banudeep/nthuddd2.

Declarations

Competing interests

The authors declare no competing interests.