Abstract
Background
The workload in health care is high; physicians and nurses report high stress levels due to a demanding environment where they often have to perform multiple tasks simultaneously. As a result, mental health issues among health care professionals (HCPs) are on the rise and the prevalence of errors in their daily tasks could increase. Processes of demographic change are partly responsible for even higher stress levels among HCPs. The digitization of patient care is intended to counteract these processes. However, it remains unclear whether these health information systems (HIS) and digital health technologies (DHT) support the HCPs and relieve stress, or if they represent a further burden. The mental construct that describes this burden of technologies is mental workload (MWL). Work in the clinic can be viewed as working in safety-critical environments. Particularly in this sensitive setting, the measurement methods of MWL are relevant, mainly due to their strongly differing levels of intrusiveness and sensitivity. The method of eye tracking could be a useful way to measure MWL directly in the field.
Objective
The systematic review aims to address the following questions: (1) In which manner do DHT contribute to the overall MWL of HCPs? (2) Can we observe a direct or indirect effect of DHT on MWL? (3) Which aspects or factors of DHT contribute to an increase in MWL? (4) Which methods/assessments are applied to measure MWL related to HIS/DHT? (5) What role does eye tracking/pupillometry play in the context of measuring MWL? (6) Which outcomes are being assessed via eye tracking?
Methods
Following the PRISMA (Preferred Reporting Items for Systematic Review and Meta-Analysis) statement, we will conduct a systematic review. Based on the research questions, we define keywords that we then combine in search terms. The review follows the following steps: literature search, article selection, data extraction, risk of bias assessment, data analysis, and data synthesis.
Results
We expect results as well as a finalization of the review in the summer of 2021.
Conclusions
This review will evaluate the impact of DHT on the MWL of HCPs. In addition, assessment methods of MWL in the context of digital technologies will be systematically analyzed.
Trial Registration
PROSPERO (International Prospective Register of Systematic Reviews) CRD42021233271; https://www.crd.york.ac.uk/PROSPERO/display_record.php?ID=CRD42021233271
International Registered Report Identifier (IRRID)
DERR1-10.2196/29126
Keywords: mental workload, cognitive load, assessment, healthcare workers, health information system, digital health technology, health care professionals, stress, eyetracking
Introduction
Background
The workload for health care workers has remained high for many years [1,2]. Several factors contribute to this trend and result in different effects for employees and the health care system [3]. Factors that promote a high workload include understaffing, long working hours [4], and information overload [5]. Work-related stress has become one of the main challenges in the health care sector [6] and has different impacts on employees. Nurses in particular report high levels of work-related stress that can lead to negative physical and psychological effects for them as well as for their patients [7]. Nurses describe themselves as feeling empty and report depressive symptoms [8,9]. In Germany, health care professionals (HCPs) have an above-average number of sick days compared to workers in other sectors; overall, there was a 29% increase in sick days between 2004 and 2018 [10]. In addition to musculoskeletal disease diagnoses, which account for the majority of sick leaves, absences due to mental illness are increasing significantly [11].
Partly responsible for the workload-promoting factors described above are the consequences of demographic changes that have led to an increase in the number of multimorbid older adult patients and a decline in the number of nursing staff. The transformation process of digitization in health care is a chance to counteract this change and its consequences. However, in Germany in particular, the process is proceeding very slowly; Germany is ranked 19th of 27 countries in Bertelsmann’s Digital Health Index [12]. The application of digital health technology (DHT) is an important factor of this digitization process. DHT in the context of this work means technologies that are directly linked to outpatient and inpatient care and are applied by nurses or physicians. DHT includes hospital information systems (HIS), medical devices, and other digital applications that support patient care from the perspective of HCPs.
In addition to the positive effects of the use of DHT, there is also evidence to suggest that the use of DHT causes an extra load. This may be due to a lack of usability and user involvement as well as poor implementation processes [13,14].
Poor usability and other factors rooted in technologies can cause a high mental workload (MWL) [14]. High workloads can result in a more error-prone performance—even for experts—induced by difficulties in decision-making processes [15].
Working with patients can be considered a safety-critical environment. This means that many tasks, varying in complexity, occur within limited time windows.
In this context, decisions must be made all the time and are supported by different systems (eg, HIS) through the structured and standardized presentation of information. The interaction between users and systems is complex and interdependent, which makes it difficult to predict the effects of the systems on the users [16].
High workload or overload caused by several factors (including technology) can have a severe impact. Aside from the negative impact on patient care due to a potential increase in errors, overload can also have a negative impact on the health of HCPs, potentially resulting in technostress, mental health issues (eg, depression, burnout), and decreased job satisfaction. These are only a few of the potential negative effects of overload [17]. There is growing evidence that DHT are contributing to increasing mental health problems (eg, burnout) among health care workers [18,19].
In order to identify possible causes of mental health problems in physicians and nurses (eg, emerging burnout [20]), the investigation of MWL in different situations is a possible approach.
Mental Workload
MWL can be defined using different approaches and is usually influenced by different and multiple factors. It is multidimensional, multifaceted, and one of the most important variables to understand and predict human performance.
The possible definitional approaches of workload can be derived from two different perspectives: (1) MWL as an external variable referring to task requirements (ie, the amount of work and the number of tasks to be completed in a limited time [task load]) and (2) interaction between task and human resources resulting in a subjective psychological experience [21,22].
Summarizing different approaches, we can define MWL as the amount of attentional resources that are required to perform a task mediated by task demands and experience [15,23,24]. Following this definition, the state of overload is reached when the task demands are too high while the user's resources are limited. In contrast to this is the condition of underload, which occurs when the task requirements are too low while resources are sufficient. In both cases, the result is poorer performance [25]. Mental states such as a high workload or underload play a critical role in the occurrence of errors as well as preventable adverse events [26]. Regardless of how competent and/or experienced an HCP is, this type of mental state can lead to a higher frequency of errors.
Assessment of Mental Workload
MWL assessments were first developed and applied in other safety-critical environments such as aviation/aerospace and nuclear power plants. Safety-critical environments have similar conditions (already described). Due to these similar conditions, workload assessment could also be a useful approach in the clinical setting.
MWL can be assessed using different techniques. A distinction between analytical and empirical methods may be drawn. Analytical methods tend to be used in system development, while empirical methods are employed when workload is to be measured directly in the executing system or in the simulation [21].
Analytical assessment methods are simulation models, expert opinions, or task analyses. Empirical methods are distinguished into three different categories: performance measures, subjective methods, and physiological techniques [15]. Performance measures refer to the measures of the primary and secondary task.
Depending on the situation and the underlying question, one or more of these techniques are appropriate to apply. Several factors should be considered when selecting assessments, including sensitivity, diagnostic accuracy, intrusiveness, validity, reliability, simplicity of use, and user acceptance [27].
Objectives
DHT may contribute to the heavy workload in health care. MWL can best reflect the workload caused by technology. In addition to the existence of some methodological issues (eg, assessing MWL in the field), there are also some knowledge gaps concerning MWL caused by DHT.
The planned systematic review intends to identify the impact of DHT, particularly HIS, on the MWL of health care workers. In addition, the review will aim to assess what methods are currently being used in health care to measure MWL relating to DHT. In particular, the application of eye tracking or pupillometry as an assessment method will be investigated.
Research Questions
The review will seek to answer the following research questions:
-
1. In which manner do DHT contribute to the overall MWL of health care workers?
1.1. Can we observe a direct or indirect effect of DHT on MWL?
1.2. Which aspects or factors of DHT contribute to an increase in MWL?
-
2. Which methods/assessments are applied to measure MWL related to HIS/DHT?
2.1. What role does eye tracking/pupillometry play in the context of measuring MWL?
2.2. Which outcomes are being assessed via eye tracking?
Methods
Study Registration
The protocol is registered in the International Prospective Register of Systematic Reviews (PROSPERO; CRD42021233271). This protocol follows the PRISMA-P (Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols) 2015 guidelines [28].
Eligibility Criteria
We define the inclusion criteria for this systematic review according to the PICO framework [29] and the research questions. Inclusion criteria relate to the study population (P), intervention (I), outcome (O) of the study, and study setting (C). In addition to these criteria, we include studies by study design as detailed below.
Study Design
All types of study designs reporting original primary data as well as systematic reviews that align with our other inclusion criteria will be included. We will exclude commentaries, letters, guidelines, and narrative reviews.
Study Participants
We focus on HCPs who work with HIS or DHT and who are directly engaged in patient care. These can be nurses, physicians, radiology assistants, or other clinicians. It is essential that the participants are supported by the HIS/DHT in their daily work with patients. We exclude studies that focus on patients who use digital technologies.
Intervention
We include studies that investigate the effects that HIS/DHT have on workers’ MWL. The focus lies on the evaluation of whether there is a direct or indirect effect of DHT on workers' MWL. Since the second research question concerns the extent to which eye tracking is commonly used as a measurement method, we focus on the inclusion of studies that apply eye tracking. We exclude studies that investigate related constructs such as technostress.
Study Setting
We include all studies that take place in inpatient or outpatient care. We exclude studies that focus on the measurement of MWL in other contexts (eg, aviation).
Information Sources
The following databases were systematically searched between February 28 and March 15, 2021, using defined keywords (and synonyms) like “mental workload,” “health information system,” “assessment,” “health care professionals,” and “eye tracking” that resulted in specified search strings: MEDLINE (PubMed), Web of Science, Academic Search Premier and CINAHL (both EBSCO), and PsycINFO. Additionally, we will search for relevant research in the reference sections of included studies as well as those of relevant recently published reviews. Following PRISMA-P [28], we organized the search terms by database and question in a separate document (Multimedia Appendix 1).
Search Strategy
The search strategy includes four categories, each represented by keywords and synonyms: technologies used (eg, HIS), population (eg, health care professionals), methods (eg, assessment), and MWL. In addition, eye tracking will be added for questions 2.1 and 2.2. The terms are linked by the Boolean operators AND or OR.
We restrict our search to articles published in the period between 2000 and 2021. This search time frame was chosen because it documents the development of the current generation of prehospital communication technology, such as telemedicine and electronic patient care reports [30]. The literature search is limited to articles written in English or German since both reviewers have a sufficiently high level of fluency in these languages.
Study Records
Data Management
Citavi is used for literature handling (ie, import and further screening). The Rayaan web-based screening tool is used to perform abstract screening and full-text analysis in a structured way. In this context, the inclusion and exclusion criteria are also provided; they will be the basis for the abovementioned analysis process. The included articles will be then imported to a Microsoft Excel (Microsoft Corp) spreadsheet.
Selection Process
The selection process will be performed by two reviewers (LK and BB; if a consensus cannot be reached, ML and RR will serve as additional reviewers) according to the PRISMA guidelines and will be displayed in a flowchart. First, both reviewers will assess the studies regarding the inclusion and exclusion criteria for abstract screening. In the next step, the full texts of the resulting studies will again be assessed independently. Finally, we will search the references of the papers for further potentially eligible studies. In case of disagreements in any of the phases, a discussion between the two reviewers (LK and BB) based on the inclusion criteria will be attempted first. If the discussion is inconclusive, a third reviewer (ML or RR) will be involved.
Data Collection Process
For data extraction, an Excel spreadsheet based on the outcomes of the review will be used. To ensure uniformity across reviewers, we will conduct a pretest standardization exercise before starting the data extraction process. Each reviewer will extract the themes of interest to an Excel spreadsheet. The extracted data items are presented below.
Data Items
LL and BB will read the full texts and extract information concerning identified and relevant aspects of the studies. We will differentiate between main study characteristics, measurements and outcomes, and relevant findings and recommendations. The aspects are aggregated in Table 1, Table 2, and Table 3.
Table 1.
Systematic analyses of the main study characteristics.
| Theme | Indicator |
| Objectives |
|
| Assessments |
|
| Quality criteria of applied assessments |
|
| Outcomes |
|
| Type of digital health technology |
|
Table 2.
Systematic analyses of measurements and outcomes (study characteristics).
| Theme | Indicator |
| Study identification |
|
| Setting of target |
|
| Study design |
|
| Sample characteristics |
|
| Population type |
|
Table 3.
Systematic analyses of the main findings.
| Theme | Indicator |
| Overall workload level |
|
| Mental workload related to digital health technologies |
|
| Factors of digital health technologies contributing to mental workload |
|
| Eye tracking |
|
| Outcomes measured by eye tracking |
|
In addition to the descriptive presentation of study characteristics and findings, we are aiming to extract factors or aspects of DHT that contribute to an increasing MWL. Furthermore, we would like to extract how the included studies assess workload and in which settings eye tracking is used with regard to specific outcomes. Based on the extraction, we would like to develop an overview of the methods that can be used to measure MWL caused by DHT and provide meaningful and valid data.
The methods, settings, and outcomes will be organized into logical categories that are rated by the reviewers. The typical categories of methods referring to MWL assessments are analytical or empirical techniques. Typical categories for settings are laboratory or field. Categories referring to assessed outcomes have to be defined during the reviewing process. In each category, we will extract how often an indicator for a category was applied (category percentage, ie, method applied/n studies) and how often combinations of specific indicators were used (total percentage, eg, method A with setting B and outcome C; combination applied/N studies). A typical indicator for the category empirical technique would be a questionnaire. If an indicator was identified, the reviewers fill the row with a 1; if no indicator was identified (eg, if the method was not applied), the table is filled with a 0. An example is displayed in Figure 1.
Figure 1.
Example of systematic tabulation of methods, setting, outcomes, and combined investigation procedures.
Outcomes
The primary outcome of the first research question is to explore the correlations between DHT and the MWL of HCPs. The secondary outcome is to investigate the type of effect (direct/indirect) DHT has on the MWL of HCPs as well as the aspects of DHT that contribute to MWL.
The primary outcome of the second research question is the exploration of the best method to determine this relationship. Particular attention will be given to the role of eye tracking technology, which will be included as a secondary outcome.
Risk of Bias in Individual Studies
For the review, two authors will independently rate the methodological quality of the identified studies using the Joanna Briggs Institute Critical Appraisal Tool [31]. An initial screening of studies that could be included indicates a small proportion of studies with an experimental design and adequately defined criteria for conducting the study and analyzing the data. Disagreements will be resolved via discussion (LK and BB) or by a third reviewer (ML or RR), if necessary.
Data Analysis and Synthesis
After screening the search results, we do not expect to be able to conduct a meta-analysis. A first look revealed that comparing the study designs and effect measures of studies will be difficult. This may be explained by the explorative character of the review and the potentially low level of evidence, especially regarding eye tracking. Instead, we will perform a descriptive analysis to summarize the data, starting with a comparison of evaluation methods (qualitative, quantitative, or mixed methods) and survey methods. To do this, we will first compare the studies in terms of the evaluation methods used (qualitative, quantitative, mixed methods), followed by a comparison of survey methods.
For data synthesis, we use two nonquantitative approaches: tabulation and a narrative approach. Table 1 and Table 2 describe the tabular synthesis of potential findings.
In a first step, all main characteristics of each study will be extracted (ie, study design, setting of target population, sample size, age, sex, population type). Studies that do not report those main characteristics and those with a sample size under 20 participants will be excluded. We will analyze studies regarding objectives, outcomes, and assessments, as well as type of DHT. Data on overall MWL in studies, MWL levels related to DHT, quality criteria of assessments, applied eye tracking, and outcomes assessed via eye tracking will be extracted.
All included studies are evaluated with regard to their risk of bias. A textual narrative synthesis of all included studies will be made and the comparable findings will be synthesized. Additionally, a descriptive analysis of eye tracking measures is planned.
Results
As the systematic review is currently ongoing, no results are available as of yet. The preliminary searches have been completed and the piloting of the study selection process as well as the formal screening against eligibility criteria has started. We are currently analyzing the data and expect to complete the review in summer 2021.
Discussion
The aim of the review is to show which methods are currently used to measure MWL in health care and the impact of such technologies on the workload of HCPs. Additionally, the role of eye tracking should be evaluated.
In the discussion section of the review, we will discuss the results and the methodological quality of the findings, strengths and weaknesses of the review (limitations), and research gaps and opportunities for future research.
Abbreviations
- DHT
digital health technology
- HCP
health care professional
- HIS
health information system
- MWL
mental workload
- PRISMA-P
Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols
Appendix
Search Strings: Performed searches sorted by question and database.
Footnotes
Authors' Contributions: LK conceived the study and wrote the paper. LK drafted the topic of the study and provided oversight for editing of the protocol. BB, ML, and RR revised the protocol. LK and BB are currently screening the literature. All authors approved this version to be published and agreed to be accountable for all aspects of the work, ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
Conflicts of Interest: None declared.
References
- 1.Körber Michael, Schmid K, Drexler H, Kiesel J. [Subjective Workload, Job Satisfaction, and Work-Life-Balance of Physicians and Nurses in a Municipal Hospital in a Rural Area Compared to an Urban University Hospital] Gesundheitswesen. 2018 May;80(5):444–452. doi: 10.1055/s-0042-121596. [DOI] [PubMed] [Google Scholar]
- 2.Portoghese I, Galletta M, Coppola RC, Finco G, Campagna M. Burnout and workload among health care workers: the moderating role of job control. Saf Health Work. 2014 Sep;5(3):152–7. doi: 10.1016/j.shaw.2014.05.004. https://linkinghub.elsevier.com/retrieve/pii/S2093-7911(14)00041-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Dugani S, Afari H, Hirschhorn LR, Ratcliffe H, Veillard J, Martin G, Lagomarsino G, Basu L, Bitton A. Prevalence and factors associated with burnout among frontline primary health care providers in low- and middle-income countries: A systematic review. Gates Open Res. 2018;2:4. doi: 10.12688/gatesopenres.12779.3. https://gatesopenresearch.org/articles/10.12688/gatesopenres.12779.3/doi. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Dall'Ora C, Griffiths P, Ball J, Simon M, Aiken LH. Association of 12 h shifts and nurses' job satisfaction, burnout and intention to leave: findings from a cross-sectional study of 12 European countries. BMJ Open. 2015 Aug 23;5(9):e008331. doi: 10.1136/bmjopen-2015-008331. https://bmjopen.bmj.com/lookup/pmidlookup?view=long&pmid=26359284. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Hall A, Walton G. Information overload within the health care system: a literature review. Health Info Libr J. 2004 Jun;21(2):102–8. doi: 10.1111/j.1471-1842.2004.00506.x. doi: 10.1111/j.1471-1842.2004.00506.x. [DOI] [PubMed] [Google Scholar]
- 6.Mark G, Smith AP. Occupational stress, job characteristics, coping, and the mental health of nurses. Br J Health Psychol. 2012 Sep;17(3):505–21. doi: 10.1111/j.2044-8287.2011.02051.x. [DOI] [PubMed] [Google Scholar]
- 7.Baye Y, Demeke T, Birhan N, Semahegn A, Birhanu S. Nurses' work-related stress and associated factors in governmental hospitals in Harar, Eastern Ethiopia: A cross-sectional study. PLoS One. 2020;15(8):e0236782. doi: 10.1371/journal.pone.0236782. https://dx.plos.org/10.1371/journal.pone.0236782. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Molina-Praena J, Ramirez-Baena L, Gómez-Urquiza JL, Cañadas GR, De la Fuente EI, Cañadas-De la Fuente GA. Levels of Burnout and Risk Factors in Medical Area Nurses: A Meta-Analytic Study. Int J Environ Res Public Health. 2018 Dec 10;15(12):1–11. doi: 10.3390/ijerph15122800. https://www.mdpi.com/resolver?pii=ijerph15122800. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Sturm H, Rieger MA, Martus P, Ueding E, Wagner A, Holderried M, Maschmann J, WorkSafeMed Consortium Do perceived working conditions and patient safety culture correlate with objective workload and patient outcomes: A cross-sectional explorative study from a German university hospital. PLoS One. 2019;14(1):e0209487. doi: 10.1371/journal.pone.0209487. https://dx.plos.org/10.1371/journal.pone.0209487. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Grobe T, Steinmann S. Gesundheitsreport 2019: Pflegefall Pflegebranche? So geht's Deutschlands Pflegekräften. Techniker Krankenkasse; 2019. [2021-05-05]. https://www.tk.de/resource/blob/2066542/2690efe8e801ae831e65fd251cc77223/gesundheitsreport-2019-data.pdf. [Google Scholar]
- 11.Marschall J, Hildrebrandt S. Gesundheitsreport 2020: Stress in der modernen Arbeitswelt; Sonderanalyse: Digitalisierung und Homeoffice in der Corona-Krise. DAK-Gesundheit; 2020. [2021-05-05]. https://www.dak.de/dak/gesundheit/gesundheitsreport-2020-stress-in-der-modernen-arbeitswelt-2365966.html#/ [Google Scholar]
- 12.Kostera T, Thranberend T. Spotlight Gesundheit: #SmartHealthSystems: Digitalisierung braucht effektive Strategien, politische Führung und eine koordinierende Institution. Bertelsmann Stiftung; 2018. [2021-05-05]. https://www.bertelsmann-stiftung.de/fileadmin/files/BSt/Publikationen/GrauePublikationen/VV_SG_SHS_dt.pdf. [Google Scholar]
- 13.Trayambak T, Singh AL, Singh IL. Information technology-induced stress and human performance: A critical review. Journal of the Indian Academy of Applied Psychology. 2008;2008(34):A. [Google Scholar]
- 14.Dang Y, Zhang Y, Brown SA, Chen H. Examining the impacts of mental workload and task-technology fit on user acceptance of the social media search system. Inf Syst Front. 2018 Sep 8;22(3):697–718. doi: 10.1007/s10796-018-9879-y. [DOI] [Google Scholar]
- 15.Proctor RW, Zandt T. Human factors in simple and complex systems (second edition) Boca Raton, FL: CRC Press; 2008. pp. 248–259. [Google Scholar]
- 16.Smith K. Observations and issues in the application of cognitive workload modelling for decision making in complex time-critical environments. Human Mental Workload: Models and Applications; International Symposium on Human Mental Workload: Models and Applications; June 28-30, 2017; Dublin, Ireland. 2017. Jun 04, pp. 77–89. [DOI] [Google Scholar]
- 17.López-Núñez MI, Rubio-Valdehita S, Diaz-Ramiro EM, Aparicio-García ME. Psychological Capital, Workload, and Burnout: What’s New? The Impact of Personal Accomplishment to Promote Sustainable Working Conditions. Sustainability. 2020 Oct 01;12(19):8124. doi: 10.3390/su12198124. [DOI] [Google Scholar]
- 18.Khairat S, Coleman C, Ottmar P, Jayachander DI, Bice T, Carson SS. Association of Electronic Health Record Use With Physician Fatigue and Efficiency. JAMA Netw Open. 2020 Jul 01;3(6):e207385. doi: 10.1001/jamanetworkopen.2020.7385. https://jamanetwork.com/journals/jamanetworkopen/fullarticle/10.1001/jamanetworkopen.2020.7385. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Melnick ER, Dyrbye LN, Sinsky CA, Trockel M, West CP, Nedelec L, Tutty MA, Shanafelt T. The Association Between Perceived Electronic Health Record Usability and Professional Burnout Among US Physicians. Mayo Clin Proc. 2020 Mar;95(3):476–487. doi: 10.1016/j.mayocp.2019.09.024. https://linkinghub.elsevier.com/retrieve/pii/S0025-6196(19)30836-5. [DOI] [PubMed] [Google Scholar]
- 20.Heinke W, Dunkel P, Brähler E, Nübling M, Riedel-Heller S, Kaisers UX. [Burnout in anesthesiology and intensive care: is there a problem in Germany?] Anaesthesist. 2011 Dec;60(12):1109–18. doi: 10.1007/s00101-011-1947-3. [DOI] [PubMed] [Google Scholar]
- 21.Lysaght R, Hill S, Dick A, Plamondon B, Linton P, United States Army Research Institute for the Behavioral and Social Sciences - ARI . Operator workload: comprehensive review and evaluation of operator workload methodologies. Springfield, VA: NTIS (Reprod); 1989. [Google Scholar]
- 22.Wieland R. Kognition, Emotion und psychische Beanspruchung: Theoretische und empirische Studien zu informationsverarbeitenden Tätigkeiten. Zugl.: Wuppertal, Univ., Habil-Schrift. Göttingen: Hogrefe; 1990. [Google Scholar]
- 23.Eggemeier T, Wilson G, Kramer A, Damos D. Workload assessment in multi-task environments. In: Damos DL, editor. Multiple-task performance. Boca Raton, FL: CRC Press; 1991. [Google Scholar]
- 24.Gopher D, Donchin E. Handbook of perception and human performance, Vol. 2: Cognitive processes and performance. Oxford, England: John Wiley & Sons; 1986. Workload: An examination of the concept; pp. 41–49. [Google Scholar]
- 25.Yerkes RD. The relation of strength of stimulus to rapidity of habit-formation. Journal of comparative neurology and psychology. 1908;18(5):459–482. doi: 10.1002/cne.920180503. [DOI] [Google Scholar]
- 26.Zenati MA, Kennedy-Metz L, Dias RD. Cognitive Engineering to Improve Patient Safety and Outcomes in Cardiothoracic Surgery. Semin Thorac Cardiovasc Surg. 2020;32(1):1–7. doi: 10.1053/j.semtcvs.2019.10.011. http://europepmc.org/abstract/MED/31629782. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Tsang P, Wilson G, Salvendy G. Handbook of human factors and ergonomics. Hoboken, NJ: John Wiley & Sons; 1997. Mental workload. [Google Scholar]
- 28.Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA, PRISMA- P. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015 Jan;4:1. doi: 10.1186/2046-4053-4-1. http://www.systematicreviewsjournal.com/content/4//1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Sackett DL, Richardson WS, Rosenberg W, Haynes RB. Clinical Chemistry. New York, NY: Churchill Livingstone; 1997. Evidence-Based Medicine—How to Practice and Teach EBM. [Google Scholar]
- 30.Rogers H, Madathil KC, Agnisarman S, Narasimha S, Ashok A, Nair A, Welch BM, McElligott JT. A Systematic Review of the Implementation Challenges of Telemedicine Systems in Ambulances. Telemed J E Health. 2017 Sep;23(9):707–717. doi: 10.1089/tmj.2016.0248. [DOI] [PubMed] [Google Scholar]
- 31.Tufanaru C, Munn Z, Aromataris E, Campbell J, Hopp L. Chapter 3: Systematic review of effectiveness. JBI Manual for Evidence Synthesis. [2021-05-05]. https://wiki.jbi.global/display/MANUAL/Chapter+3%3A+Systematic+reviews+of+effectiveness.
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Search Strings: Performed searches sorted by question and database.