An integrative review of cognitive workload assessment for safety management

Abstract

Background

Accurate measurement of cognitive workload is crucial for ensuring system performance and reliability. Currently, cognitive workload has been measured by questionnaire, task performance and physiological parameters. With the rapid development of wearable devices and data mining, more and more researchers applied these technologies to achieve an objective and real-time assessment of cognitive workload, so a review with the updated research on its assessment is in great need.

Objective

To depict a comprehensive view of cognitive workload assessment in safety management, and offer advice on future assessment for nurse managers.

Methods

This study was conducted following the integrative review method of Whittemore and Knafl. A systematic search of cognitive workload assessment in safety management was conducted across PubMed, Web of Science, Scopus and EBSCO from the inception to January 1st 2025. Data from each article were extracted and summarized in accordance with research question.

Results

146 articles were included, most of them came from high-risk industries, only 4 were conducted by nurse researchers. NASA-TLX is the most widely used questionnaire. EEG, EOG and ECG were the top 3 preferred signals for classification. Multi-signals can facilitate better classification performance. Among all the classification method, SVM, KNN and composite classifier were more preferred.

Conclusion

Measuring cognitive workload by physiological parameters through machine learning can facilitate objective and real-time assessment, but feature artifacts and classification efficiency are two major concerns. Most subjects in included studies were male. Given the uniqueness of nursing work, though this review can offer enlightenment on future cognitive workload assessment, more researches should be done to establish adaptable cognitive workload assessment method.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12912-025-03987-w.

Introduction

Cognitive workload is a multidimensional construct proposed at the end of the 1990s, derived from the cognitive workload theory influenced by evolutionary psychology, and refers to the cognitive capacity to perform a specific task. According to cognitive workload theory, humans have a limited capacity to process information [1, 2]. Cognitive workload consists three components: intrinsic load, extrinsic load and germane load. The intrinsic load is determined by the inherent complexity of the task, the extrinsic load is influenced by the manner of information presentation, and the germane load refers to the mental resources devoted to acquiring and automating schemata in long-term memory.

Since its initial formulation by Sweller (1988), the construct of cognitive workload has generated substantial scholarly interest. Early research primarily examined its implications for instructional design and training paradigms, with the increasing accidents in nuclear power plant, aviation and traffic industry, more and more researchers focus cognitive workload assessment in high-risk industry. In the past, limited automation imposed significant cognitive demands on human operators, requiring them to execute numerous manual operations, process complex system information and maintain continuous situational monitoring, thus increasing the likelihood of human error and threatening the reliability of the system. However, with the rapid development of technology and increasing levels of automation, many people, particularly in the automotive industry, are concerned that this could also reduce situational awareness, leading to under-utilisation and increased risks [3].

Given the critical role of cognitive workload in safety management, extensive research has been conducted to develop reliable assessment methodologies. Current measurement approaches can be categorized into three primary types: (1) subjective instruments, (2) performance-based assessments, and (3) physiological parameter evaluations. Subjective instruments are those developed on the basis of cognitive workload theory, such as the NASA-TLX, SWAT, Modified Cooper-Harper, the PAAS and MDT-CL [4–8]. Of all the subjective instruments, the most widely used is the NASA-TLX, which was developed by the NASA Ames Research Center to provide a subjective assessment of the workload of operators working with various human-machine interface systems. It consists six dimensions: mental workload, physical workload, time demands, performance, effort and frustration [5]. It also serves as a validated benchmark in many studies to classify cognitive workload using physiological parameters [9, 10]. However, considering recall bias, inability to offer real time assessment and its susceptibility to people’s previous state or external stressors, more and more researchers and risk managers begin to seek other assessment methods.

Task performance is also regarded as a reliable method of assessing cognitive workload. Excellent performance on the task may indicate sufficient cognitive resources, while poorer performance may reflect fewer cognitive resources [2]. While some studies determine the level of cognitive workload by assessing performance on the primary task, others measure cognitive workload by setting a sub-task alongside the primary task. Performance on the sub-task may reflect the subject’s spare cognitive resources [11]. These methods are widely used in the aviation and automotive industries. However, these methods, especially those involving assessment of sub-task performance, can be disruptive and even dangerous to the daily work of faculty members, limiting their usefulness.

To overcome the limitations of subjective instruments and performance measurements, the assessment of physiological parameters offers promising solutions. As increased cognitive workload affects the autonomic and parasympathetic nervous systems and their effectors, including the brain, circulatory system, eyes, skin and respiratory system, researchers can measure and classify cognitive workload based on certain physiological parameters. Furthermore, the rapid development of non-invasive wearable devices and data mining technology makes it feasible to collect and efficiently process real-time cognitive workload data [12, 13].

However, as cognitive workload can be influenced by task demands, individual characteristics and the environment, we should be cautious about applying cognitive workload assessment methods from high-risk industries to healthcare [14, 15]. In nursing care, nurses interact with information in multiple ways and all cognitive activity is patient-centred, whereas faculty in high-risk industries interact with information primarily through machine interfaces and are information-centred instead [16]. Nursing is a complex process involving intricate interactions between patients, nurses, emerging technologies, advanced equipment and dynamic micro- and macro-environments. Among these, nurses play a critical role in ensuring patient safety, so an accurate and efficient method of assessing cognitive workload is quite important to ensure better patient safety. Thus, our research questions are: (1) What are the cognitive workload assessment methods frequently used in safety management? (2) What are the recommendations for cognitive workload assessment in nursing management in the future?

Methods

This review was conducted in accordance with the procedures recommended by Whitmore and Knalf for integrative reviews [17].

Search strategy

We systematically searched Pubmed, Web of Science, EBSCO and Scopus from the inception of the database to 1st January 2025. The search terms were developed and refined through a pilot search, which included three blocks: (1) cognitive workload, cognitive load, cognitive cost, cognitive demand, mental load, mental workload, mental strain, mental effort and mental demand; (2) risk, safety; (3) measur*, asses* and evaluat*. As the research objective was to summarise assessment methods for cognitive workload and provide insights for safety management in nursing, we did not limit the industry. We combined the search terms using Boolean logic and used sensitive search filters to refine the literature. An example of the search strategy used in PubMed is presented in Appendix 1.

Inclusion and exclusion criteria

Inclusion criteria were: (1) peer-reviewed journal articles on cognitive workload assessment in safety management; (2) the definition of cognitive workload is compatible with the definition of cognitive load theory; (3) written in English; (4) subject should be human. Conferences, editorials, letters, review, dissertation or articles with an incompatible definition were excluded. All searched literature was uploaded into Endnote 20.0 and reviewed by 2 independent reviewers. Any discrepancies were resolved through discussion among the group of authors.

Data extraction

Data were extracted by two reviewers independently using a standardized data extraction table. The data extracted general information of literature (author, year, country or region, subject, sample size and primary research result), signals, classify methods and their performance.

Results

Search outcome

This review included 146 articles from various industries (see Fig. 1) with a substantial agreement between 2 independent reviewers (interrater reliability index = 0.88) [18].

Fig. 1.

Diagram for literature search and selection

Time trend of publication

The publications towards cognitive workload assessment in safety management experienced a clear increase since 2003 (see Fig. 2). From this figure we can divide the timeline into two phases, initial development (1994–2003) and rapid development (2003- ). During the initial development phase, cognitive workload assessment was mainly conducted through subjective instrument and single physiological signals. Since 2003, assessing cognitive workload through multiple physiological parameters has become the mainstream, with more researchers devoted in comparing and selecting of signals and cognitive workload classifying algorithm.

Fig. 2.

Time trend of included publication for cognitive workload assessment

Application of physiological signals for cognitive workload assessment

Table 1 listed the frequently-used physiological signals for cognitive workload assessment in included studies, the abbreviations and full names in this table can be seen in Appendix 2.

Table 1.

Physiological signals for cognitive workload assessment in safety management (N = 70)

Year	Author	ECG	EEG	EOG	fNIRS	RESP	EDA	Voice	EMG	PPG
1998	Hankins & Wilson	*	*	*
2007	Or & Duffy						*
2012	Muth et al.					*
2013	Kitamura et al.	*					*
2014	Gentili et al.	*	*
2014	Hogervorst, Brouwer & van Erp	*	*	*		*	*
2014	Wanyan, Zhuang & Zhang	*	*	*
2014	Yin & Zhang		*
2015	Dimitriadis et al.		*
2015	Matthews et al.	*	*	*	*
2016	Horat et al.		*
2016	Grassmann et al.			*
2017	Barua, Ahmed & Begum		*
2017	Chen, Taylor and Comu		*
2017	Di Stasi et al.			*
2017	Ghaderyan & Abbasi	*
2017	Heine et al.		*
2017	Pépin et al.	*
2017	So et al.		*
2017	Szulewski et al.			*
2018	Blanco et al.		*
2018	Hefron et al.		*
2018	Zhang et al.			*
2018	Jimenez et al.			*
2018	Yang et al.			*
2018	Zhao, Liu & Shi	*			*
2019	Dias et al.	*
2019	Li et al.				*
2019	Morales et al.		*
2019	Tiwari et al.	*
2019	Yan, Wei &Tran			*
2020	Becerra-Sánchez, Reyes & Guerrero-Ibañez		*
2020	Das, Maiti & Krishna			*
2020	Iqbal, Srinivasan & Srinivasan		*
2020	Johannessen et al.	*		*			*
2020	Shafiei et al.		*
2020	Shao et al.	*
2021	Ravi et al.				*
2021	Zhu et al.		*
2022	Bhavsar			*
2022	Cardone et al.	*			*
2022	Lagomarsino et al.	*					*
2022	Li, Liu & Li	*	*
2022	Liu et al.	*
2022	Nilsson et al.	*	*	*
2022	Oppelt et al.	*		*		*	*		*
2022	Raufi & Longo		*
2023	Alyan et al.		*
2023	Beh, Wu & Wu									*
2023	Caiazzo et al.		*
2023	Liu, Gao & Wu		*
2023	Mastropietro et al.		*
2023	Wascher et al.		*
2023	Yang et al.							*
2024	Aksu, Cakit & Cakit		*	*
2024	Alharasees &Kale			*		*
2024	Angkan et al.	*	*	*			*
2024	Das & Maiti			*
2024	Ghasimi & Shamekhi		*		*
2024	Gogna, Tiwari & Singla		*
2024	Hao et al.	*		*						*
2024	Hernández-sabaté et al.			*
2024	Huang et al.	*	*	*			*
2024	Lee et al.	*	*	*			*
2024	Li et al.		*
2024	Lisanne et al.			*
2024	Mark et al.	*	*	*	*				*
2024	Pemmada, Nayak & Routray	*							*
2024	Xu et al.		*
2024	Shafiei, Shadpour & Mohler		*	*
Total count		25	37	27	7	4	9	1	3	2

Classification methods and their performance

Below are the details of the studies that applied machine learning algorithms for cognitive workload classification, including the classification methods and the best performance (see Table 2). The abbreviations and full names in this table can be seen in Appendix 2.

Table 2.

Classification method by machine learning and its performance(N = 21)

Year	Author	Classification method	Performance
2014	Yin & Zhang	KNN & SVM	Average detection rate = 0.717
2015	Dimitriadis et al.	KNN	ACC = 0.96
2017	So et al.	SVM	ACC = 0.75
2018	Blanco et al.	KNN, NB, DT, SVM, LDA	ACC = 0.9017(LDA)
2018	Zhao, Liu & Shi	SVM	ACC (average) = 0.722
2019	Tiwari et al.	SVM	ACC = 0.8438
2019	Yan, Wei & Tran	ANN	R2 = 0.918
2020	Becerra-Sánchez, Reyes & Guerrero-Ibañez	SVM-RBF, SVM-L, KNN, RiL	ACC = 0.9042(SVM-RBF)
2020	Shafiei et al.	DNN	ACC (low) = 0.93 ACC (intermediate) = 0.89 ACC (high) = 0.91
2020	Shao et al.	SVM, KNN, GB, LDA, NB, DT	ACC = 0.9877(KNN)
2021	Zhu et al.	SVM, DT	ACC = 0.869(SVM)
2022	Cardone et al.	SVM	AUC = 0.85
2022	Oppelt et al.	XGBoost	AUC = 0.94
2022	Raufi & Longo	LR, SVM, DT	ACC = 0.845(SVM)
2023	Liu, Gao & Wu	SVM-L, SVM-R, SVM-P, KNN, RF & BNN	ACC = 0.8557(KNN)
2023	Yang et al.	SCNN-TransE	ACC = 0.9748
2024	Hao et al.	RFS and 1D-convolutional neural network	ACC = 0.9067
2024	Lee et al.	Function-on-function linear regression	ACC = 0.90
2024	Li et al.	CNN, CNN-LSTM, Transformer, Swim Transformer, MST-Net	ACC (local dataset) = 0.891(MST-Net) ACC (COG-BCI dataset) = 0.8391(MST-Net)
2024	Shafiei, Shadpour & Mohler	eXtreme Gradient Boosting	R2(0.81 ~ 0.83)
2024	Xu et al.	BRS-based framework	ACC ≧ 0.7

Note:R² (Coefficient of determination) is an important index to evaluate the goodness of fit for computational models. It ranges from 0 to 1, the closer the R² value is to 1, the better goodness of fit. AUC (Area under curve) and ACC (Accuracy) are another important indexes to evaluate the predict ability of the model, they range from 0 to 1 as well, the closer the AUC and ACC value is to 1, the better the performance of the classifier

Subjective instruments for cognitive workload assessment

In our review, the majority of studies used the NASA-TLX to measure cognitive workload in their subjects. Furthermore, the NASA-TLX consistently served as the gold standard in studies that employed various parameters to evaluate cognitive workload via machine learning algorithms. Furthermore, some studies have conducted cultural adaptations of the NASA-TLX or adapted existing subjective instruments for use in specific industries, such as for drivers, physicians, and nurses [8, 19]. One study developed a new instrument with only one item for ease of use [20]. And the frequently-used subjective instruments in the included studies are listed as follows (see Table 3):

Table 3.

Subjective instrument of cognitive workload assessment

Instrument	Dimension	Number of items
NASA Task Load Index	Mental demand, Physical demand, Temporal demand, Effort, Performance and Frustration	Six
Subjective Load Assessment Method	Time load, Mental effort load, Psychological stress load	Nine
Cognitive Load Component Survey	Mental load, Mental effort	Eight
VACP	Visual, Auditory, Cognitive, Psychomotor	Thirty-three
The Rating Scale Mental Effort	Mental effort	One
PAAS scale	Mental effort	One
Persian Driving Activity Load Index	Effort of attention, Visual demand, Auditory demand, Time demand, Interference, Situational stress	Six
Modified Cooper–Harper	Mental effort	One
MDT-CL	Intrinsic cognitive workload, Extraneous workload, germane workload	Ten

Discussion

As cognitive workload is critical for safety management and patient safety remains a major concern in the healthcare industry, we conducted an integrative review of the literature on cognitive workload assessment in safety management. Our aim was to compare existing approaches and provide nursing organisations with advice on cognitive workload assessment and management in the future. Of the 146 articles included in our review, only four were conducted by nurse researchers. Currently, cognitive workload assessment through multiple physiological parameters is favoured, with various machine learning algorithms applied in feature selection and classification. While most studies achieved good performance, there are still some concerns.

Cognitive workload theory has been validated but the mechanism remains unknown

Through our review, the theoretical hypothesis of Sweller’s cognitive workload theory, which suggest that performance will be compromised by under- or over-load, have been validated across many industries, evidenced by lower task accuracy and longer response times [21, 22].These negative effects were thought to be caused by the dynamic change between the sympathetic and parasympathetic nervous systems, which can be detected by visual-, cardio-, neuro- and performance parameters [13]. Despite significant efforts to achieve a more objective and accurate classification of cognitive workload, the underlying neurophysiological mechanisms remain largely unknown.

EEG signals for cognitive workload classification and performance

Because of its ability to directly detect changes in brain activity in real time, the electroencephalography (EEG) sensor has been widely used to analyse cognitive workload. However, EEG signals can easily be affected by artefacts from the environment, muscle activity, eye movements, line noise and heartbeats. This can compromise signal quality and threaten classification performance [23, 24]. In recent years, an increasing number of researchers have addressed this issue by using various methods to extract high-quality neural signals [25–28]. For instance, Havugimana, Moinudin and Yeasin employed generative and discriminative deep neural networks to develop interpretable and parameter-optimised models with 94% accuracy [23].

Studies have shown that a decrease in the power of the δ and α frequency bands in the temporal lobe reflects high awareness, deep thinking and increased workload execution [29]. One study compared the γ and β bands with the α and θ bands and concluded that the former achieved higher classification accuracy [30]. In addition, researchers have begun to explore the brain mechanism of cognitive workload. One study found that the prefrontal, cerebellar, frontal, and parietal areas contribute most to the prediction of cognitive workload [23].

For classification, researchers always applied multiple machine learning algorithms in classification and compared their performance (see Table 2). Among the included studies, the accuracy of cognitive workload is above 0.80 via EEG signal only (0.70 to 0.96). The selected EEG features, their implications and relationship with cognitive workload are summarized in the following table (Table 4).

Table 4.

EEG features for cognitive workload classification

Feature	Implication	Relationship with CL
δ band	Sensitive to mental effort and attention	-
α band	Sensitive to arousal levels	-
β band	Sensitive to conscious focus, stress, alertness	+
γ band	Sensitive to perception and memory	+
θ band	Sensitive to mental effort	+

ECG signals for cognitive workload classification and performance

ECG signals are also widely used to assess cognitive workload because they can be obtained from consumer ECG devices without the inconvenience of wearing multiple sensors. Of all the ECG signals, HRV, which reflects variation in the time interval between heartbeats, is the most commonly used ECG feature [3, 31, 32]. Although most studies achieved good performance using HRV, some studies did not, resulting in contradictory findings regarding its use in assessing cognitive workload. Among the included studies, the accuracy of assessing cognitive workload via EEG signal alone was above 0.80 (ranging from 0.8438 to 0.9877). The ECG signals among included studies are summarized as follows (Table 5).

Table 5.

ECG signals for cognitive workload classification

ECG signals	Features
Time domain	SDNN, RMSSD, PNN50, NN50, HRVTi
Frequency domain	Very low frequency band Low frequency band High frequency band Total power spectrum

EOG signals for cognitive workload classification and performance

Eye-tracking features such as saccades, fixations, blinks and pupil diameter, can provide valuable information about where attention is focused during a task, so EOG signals have been widely used to classify cognitive workload [33]. Researches have shown that pupil size and blink rate increase with greater cognitive workload [21, 34], and that fixation, the average blink interval and the amplitude of saccades increase as cognitive workload increases [35, 36].

However, researchers pointed out that EOG features can easily be affected by light and people’s motor behaviour. This reduces their accuracy in cognitive workload classification, so it is important to clean the data before classification. Pillar and Balakumar proposed using a Kalman filter to remove high-frequency noise caused by sudden changes in ambient light, head or body movement, and measurement noise [37]. Wang et al. applied Independent Component Analysis to the raw EOG data to produce a higher-quality dataset [38]. Among the included studies, only one applied the EOG signal alone and stated that eye responses are sensitive to MWL, with a determination coefficient of 0.918 for classifying performance in the testing data [39]. The selected EOG features, their implications and relationship with cognitive workload are summarized in the following table(Table 6).

Table 6.

EOG features for cognitive workload classification

Features	Implication	Relationship with CL
Count of fixation	The number of fixations on specific an area of interest(AOIs) or the whole stimulus	+
Fixation rate	Eye movements that stabilize the retina over a stationary object of interest	-
Fixation duration	Duration of each individual fixation	+
Saccadic amplitude	The angular distance that the eye moves during a saccadic eye movement	+
Dwell duration	Duration of all visits within an AOI	-
Pupilometry	Changes in diameter of the pupil	+
Blink interval	The duration that participants can keep their eyes open before blinking	+
Blink rate	The number of blinks per minute	+
Gaze time	The summed duration of all fixations falling within a defined region, until the eye exits that region	-

Performance-based assessment and its performance

According to Sweller, the greater the cognitive workload, the fewer spare cognitive resources are available, meaning that primary and sub-task performance can be used to assess cognitive workload. However, sub-task performance assessment is difficult to apply in real work settings as it may compromise the primary task and pose a risk to workers and clients [40, 41]. The most commonly used task paradigms for assessing secondary task performance among the included articles are the n-back task and mental arithmetic.

Challenges in cognitive workload assessment for nursing organization

This review provides an overview of cognitive workload assessment in various industries. The methodologies recommended in high-risk industries could certainly shed light on cognitive workload assessment for nursing managers, helping them to improve patient safety management. However, nurse managers still face challenges in applying these methods to nursing. Firstly, we need to be aware that the existing cognitive workload measuring method in high-risk industries may not be suitable for nursing work. Generally, two major paradigms have been used to develop cognitive workload assessment methods: theoretical derivation, which is based on classical cognitive workload; and data-driven, which relies heavily on the nature of work in a specific industry. For example, when adopting a data-driven approach to developing a cognitive workload assessment method, we can look to other industries for their approach but the differences in work between nursing and other industries should definitely be taken into account. Secondly when we considering physiological signals for cognitive workload assessment, we need to decide which one is more suitable based on tasks’ characteristic. From Table 2 we can assume that nurses’ cognitive workload assessment should apply β and γ band, while for drivers α and θ band is more frequently applied.Thirdly, staff in the aviation and nuclear industries rarely move during work, and the lighting in their workplace is almost constant, so there is less artefact or noise when assessing cognitive workload using EEG, ECG or EOG. However, nurses’ work involves motor behaviour and the light changes dynamically according to the work environment. Therefore, caution should be exercised when assessing cognitive workload, and great attention should be paid to feature selection and data cleaning. Lastly, our review included 4142 subjects, the majority of whom were male (see Appendix 3). However, most nursing staff are female, so this should also be considered when applying existing measurement methods.

Implications and recommendations for nursing management

This review can contribute to future nursing work in the following ways. Firstly, nurse managers can select the most appropriate subjective instruments, physiological signals, machine learning algorithms and data cleaning methods based on this review. This will enable them to develop a more accurate and real-time cognitive workload assessment method, which can be embedded into the hospital management system to facilitate better quality and safety management. Secondly, an accurate, real-time cognitive workload assessment can provide guidance on improving the interface design between nurses and machines during clinical work, thereby reducing extrinsic cognitive workload. This is important since human-machine interactions have increased sharply in recent years. Finally, an accurate, real-time cognitive workload assessment can serve as an important evaluation tool for training programmes.

Strength and limitations

This review is a pioneering article to summarize and compare methodological research on cognitive workload assessment across different industries. We not only summarized the commonly used instrument, task performance and physiological parameters, but also concluded the classification algorithms and their classification performance. But due to language limitation, we only included articles published in English, so we may omit relevant articles published in other languages. Besides, the included studies were all about methodologies and lack of specific tool to evaluate the research quality, but we were strict in literature selection thus to ensure the high quality of included article. Finally, due to the great heterogeneity of assessment features and classification method between different industries, we cannot integrate these research results and come to a consensus on the feature selection and classification method, but our research can indeed serve as a valuable reference for future cognitive workload assessment and management for nursing managers.

Conclusion

Ensuring appropriate levels of cognitive workload is critical for patient safety, as either over- or under-load can lead to increased human error and adverse events. Throughout this review, we observed an increase in research on cognitive workload assessment, and most of the research on cognitive workload classification relied heavily on wearable devices and machine learning algorithms. In addition, researchers have demonstrated that cognitive workload classification based on multi-source data performs better than that based on single signals [42–44]. Through literature review, we have concluded the assessment methods for cognitive workload, stressed that when bringing existing cognitive workload assessment methodology in high-risk industry to nursing work, the feature selection and classification algorithms should be tailored to the nursing work. Finally, more research should be done in the future to explore the neurophysiological mechanism, as the classification model established by machine learning is always difficult to interpret and generalise.

Supplementary Information

Below is the link to the electronic supplementary material.

Author contributions

F.T.T. and W.X. contributed to the design and draft of this review; F.T.T. and Q.T.H. contributed to the literature search and selection; H.L.J. and P.X. contributed to data extraction. All authors reviewed the manuscript.

Funding

This review was supported by National Natural Science Foundation of China (grant number:72471006,72071004).

Data availability

All data generated or analysed during this study are included in this published article.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.