Skip to main content
BMJ Open Quality logoLink to BMJ Open Quality
. 2021 Feb 26;10(1):e001145. doi: 10.1136/bmjoq-2020-001145

Human factors in escalating acute ward care: a qualitative evidence synthesis

Jody Ede 1,2,, Tatjana Petrinic 3, Verity Westgate 4, Julie Darbyshire 4, Ruth Endacott 2,5, Peter J Watkinson 4
PMCID: PMC7919590  PMID: 33637554

Abstract

Background

Identifying how human factors affect clinical staff recognition and managment of the deteriorating ward patient may inform process improvements. We systematically reviewed the literature to identify (1) how human factors affect ward care escalation (2) gaps in the current literature and (3) critique literature methodologies.

Methods

We undertook a Qualitative Evidence Synthesis of care escalation studies. We searched MEDLINE, EMBASE and CINHAL from inception to September 2019. We used the Critical Appraisal Skills Programme and the Grading of Recommendations Assessment-Development and Evaluation and Confidence in Evidence from Reviews of Qualitative Research tool to assess study quality.

Results

Our search identified 24 studies meeting the inclusion criteria. Confidence in findings was moderate (20 studies) to high (4 studies). In 16 studies, the ability to recognise changes in the patient’s condition (soft signals), including skin colour/temperature, respiratory pattern, blood loss, personality change, patient complaint and fatigue, improved the ability to escalate patients. Soft signals were detected through patient assessment (looking/listening/feeling) and not Early Warning Scores (eight studies). In contrast, 13 studies found a high workload and low staffing levels reduced staff’s ability to detect patient deterioration and escalate care. In eight studies quantifiable deterioration evidence (Early Warning Scores) facilitated escalation communication, particularly when referrer/referee were unfamiliar. Conversely, escalating concerning non-triggering patients was challenging but achieved by some clinical staff (three studies). Team decision making facilitated the clinical escalation (six studies).

Conclusions

Early Warning Scores have clinical benefits but can sometimes impede escalation in patients not meeting the threshold. Staff use other factors (soft signals) not captured in Early Warning Scores to escalate care. The literature supports strategies that improve the escalation process such as good patient assessment skills.

PROSPERO registration number

CRD42018104745.

Keywords: continuous quality improvement, critical care, decision making, human factors, medical education

Introduction

Failure to rescue

‘Failure to rescue’ (FTR), defined as mortality following complications during a hospital admission,1 is common.2 At least 11 000 hospital patients each year suffer preventable deaths3 though other sources believe this number to be higher.4 It is also recognised that patients who die following a cardiac arrest are likely to have preceding warning signs that are not adequately managed.5 Though differences between hospital complication rates are small, patients can be three times more likely to die from complications depending on which hospital they are in.2 Poor surveillance of these patients can be linked to inadequate monitoring of abnormal vital signs, poor fluid balance management or diagnostic errors.3 6 Reports to the National Reporting and Learning System deomstrate that 7% were related to a failure to act or recognise patient deterioration.7

Escalation of care

Avoiding FTR requires successful escalation of care8 whereby patients’ deteriorations are detected, communicated and acted on.8 9 Escalation interventions focus primarily on specialist clinical teams such as Critical Care Outreach or Rapid Response Teams (RRT).10 These teams aim to target improvements to the initial detection and ward management of patient deterioration.11 Other interventions target communication breakdowns.12

Human factors (HF) identified to positively or negatively affect care escalation include situational awareness, team working, communication, safety culture, workload, clinical experience, negative emotions and leadership.6 9 13–15 However, research has historically focused on outcomes.8 The aims of this qualitative evidence synthesis (QES) are to identify (1) how HF affect ward care escalation (2) gaps in the current literature and (3) critique literature methodologies.

Methods

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guideline was adhered to16 (see online supplemental file 1). We undertook a QES of the literature exploring escalation of care. The research question was developed by the two authors (JE and VW) using the Population, Interest and Context framework.17 A full protocol has been published in a peer-reviewed journal.18

Supplementary data

bmjoq-2020-001145supp001.pdf (97.4KB, pdf)

The search strategy was assisted by a specialist librarian (TP). Searches were performed on three databases, MEDLINE, EMBASE and CINHAL. Dates searched were from database inception to September 2019. Medical Subject Headings terms were used and searched as free text (full search strategy is included in online supplemental file 2). Reference lists of all eligible studies were also checked, and incidental references included from these.

Supplementary data

bmjoq-2020-001145supp002.pdf (31.5KB, pdf)

Eligibility criteria

This evidence synthesis includes qualitative studies reporting primary data. No limits on publication date or country were applied. We included studies that explored how HF affect FTR and care escalation from staff, patients or relative’s perspective. Qualitative methods include (but are not limited to) ethnography, interviews, focus groups and HF methods. We defined HF as any human interaction affecting teamwork, tasks, equipment, workspace, culture or organisation.19 Data analysis included, but has not been not limited to, thematic analysis, grounded theory and discourse analysis.

Inclusion

  • Qualitative studies reporting primary data.

  • Qualitative studies exploring how HF affect escalation of care of the in-hospital patient population.

  • Studies employing qualitative data collection methods, for example, semistructured interviews, focus groups or observations.

  • Observational studies relating to FTR or care escalation.

  • Adult population.

Exclusion

  • Systematic or literature reviews.

  • Correspondence and short communications.

  • Simulation studies.

  • Studies written in any language other than English.

  • Studies in the emergency department and maternity.

Eligible studies were entered into Covidence systematic review software (Veritas Health Innovation, Melbourne, Australia. Available at www.covidence.org) and deduplicated. Study screening and selection was undertaken by two reviewers independently. The titles and abstracts were screened against the eligibility criteria. Disagreements between reviewers were resolved by third person mediation. Reasons for excluding studies were noted.

Quality assessment and confidence in synthesised findings (Critical Appraisal Skills Programme and Grading of Recommendations Assessment, Development and Evaluation-CERQual)

Two researchers (JE and VW) reached a consensus regarding which study quality assessment tools to use during the review. Two different quality assessments were conducted on all studies by both researchers. The Critical Appraisal Skills Programme (CASP) qualitative checklist was used to assess papers for credibility, confirmability, dependability and transferability.20 This comprehensive framework tool is commonly used in qualitative study assessment.21 22

We assessed confidence in synthesised findings using the Grading of Recommendations Assessment-Development and Evaluation and Confidence in Evidence from Reviews of Qualitative Research (GRADE-CERQual) Criteria and associated guidance publications.23–27 The four-stage assessment (methodological limitations, coherence, adequacy of data and relevance) examines each synthesised finding for confidence by critiquing contributing study rigour.28 The output of this evaluation is a Summary of Qualitative Findings table detailing themes and papers contributing to this theme. This table promotes transparency in the synthesis methods. Themes from the data analysis are presented in order of highest to lowest confidence according to the GRADE-CERQual assessment.

Analysis

We undertook a thematic synthesis22 using Thomas and Harden’s framework to map how HF affect escalation of care.29 This is a three-stage process. Initially, study findings are coded, these codes are then categorised into descriptive themes and finally these descriptive themes are categorised into analytical themes.30 Stage 1 involves line-by-line coding of data, where each sentence is allocated a code. Stage 2 involves categorising each coded sentence into descriptive, broader themes. The final stage involves generating analytical themes, or ‘going beyond’ the findings of the initial study, which relate to the fixed or emerging research question (see table 1 for definitions of analytical themes). This framework supports data extraction from anywhere within the paper and is not confined to the results alone.

Table 1.

Definitions of analytical themes

Analytical theme Definitions and references
Information packaging The use of quantifiable evidence of deterioration (such as vital signs) to initiate escalation of care.15 33 34 40 41
Flattened hierarchy Escalation of care can be initiated from any staff member to any staff member.15 31 35 41–45
Situational awareness The comprehension of clinical elements and projection of their status in the future.73
Team functioning Fragmented team-working with sequential rather than concurrent task completion and poor relationships.35 36 41–45 47 48
Soft signals of deterioration Non-numerical deterioration cues attained from observation rather than instrumentation.15 31 35–37 41 49 50 54 55
Decision making Clinical reasoning surrounding detection, communication and management of escalation of care.
Clinical experience As staff became familiar with deteriorating patients, they were better able to detect and predict impending illness.15 36 38 42 43 46 49 50 54 55
Clinical assessment Involves staff looking, listening and feeling the patient to identify respiratory, skin, neurological or physiological abnormalities.15 49 51 55 57

Data extraction tools were developed and piloted before the review took place to ensure consistency of data extraction. Study data were entered into an Excel spreadsheet (Windows, 2019. Microsoft Office) and study themes were analysed using NVivo software (NVivo qualitative data analysis Software; QSR International, V.10, 2014).

Patient and public involvement

Patient representative (TD) reviewed the original published protocol and aims of the review were discussed and deemed of patient importance.

Results

The search identified 2404 papers which met the initial search criteria (refer to online supplemental file 3 for PRISMA diagram). After duplicates were removed, 1651 articles were screened. 1627 were excluded based on methodology, subject of interest or incorrect population. This resulted in 24 papers meeting the inclusion criteria and being reviewed in full. A full description of synthesised study characteristics are presented in table 2.

Table 2.

Synthesised studies summary table

Study ID Study design Name of journal Methods Sample size (N/hours) Population Data collection date Data analysis method
Andrews33 2005 Qualitative Design Issues and Innovation in Nursing Practice Interviews Observations 44
Not avail
Nurses, doctors, AHPs and CSWs 2002 Grounded Theory
Astroth45 2013 Qualitative Design Journal of Clinical Nursing Interviews 15 Nurses Not avail Concept Analysis
Braaten51 2015 Descriptive Qualitative American Journal of Nursing Interviews 12 Nurses 2012 Content Analysis
Brady42 2014 Qualitative Design BMJ Quality and Safety Focus Groups 31 Nurses, respiratory therapists, physicians 2009 Constant comparison
Bunkenbor55 2013 Descriptive Qualitative Journal of Advanced Nursing Interviews
Observations
13, 70 hours Nurses 2009 Content Analysis
Burns46 2017 Qualitative Design Journal of Advanced Nursing Interviews 25 Nurses 2015 Thematic Analysis
Chua49 2013 Qualitative Exploratory Descriptive International Nursing Review Interviews 15 Nurses 2011 Content Analysis
Chua52 2019 Qualitative Exploratory Descriptive International Journal of Nursing Studies Interviews 22 Nurses 2016–17 Thematic Analysis
Currey32 2017 Descriptive exploratory Australian Critical Care Survey 207 Nurses, Doctors, Care Support Workers, 2014 Content Analysis
Donohue15 2010 Qualitative Design Intensive and Critical Care Nursing Interviews 9 Nurses 2006 Thematic Analysis
Elmufudi43 2017 Qualitative Design American Journal of Medical Quality Interviews 40 Doctors 2014 Thematic Analysis
Foley56 2018 Qualitative Design Journal of Clinical Nursing Interviews
Observations
8 Nurses, CSWs Systematic Text Condensation
Hart40 2016 Descriptive Qualitative Journal of Clinical Nursing Interviews 28 Nurses 2015 Constant Comparison
Mohammmed Iddrisu54 2018 Qualitative Design Journal of Clinical Nursing Focus Groups 14 Nurses 2014 Thematic Analysis
Jamesl31 2010 Qualitative Design Journal of Nursing Management Survey 131 Care Support Workers Not avail Content Analysis
Johnston8 2015 Qualitative Design Annals of Surgery Observations 42 hours Surgical wards 2013 Grounded theory
King 201972 Qualitative Design Health Expectations Focus Groups 26 Patients and families 2014–2017 Thematic Analysis
Searle Leach38 2010 Qualitative Design Quality and Safety Healthcare Interviews 50 Nurses Not avail Grounded Theory
Mackintosh34 2012 Comparative case study Postgraduate Medicine Journal Interviews Observations 35
150 hours
Doctors, nurses, critical care nurses, 2009 Thematic Analysis
Mackintosh41 2014 Qualitative Design Social Science and Medicine Interviews Observations 35
180 hours
Doctors, nurses, critical care nurses, 2009 Thematic Analysis
Massey44 2014 Qualitative Design Australian Critical Care Interviews 15 Nurses 2011 Thematic Analysis
Martland47 2016 Qualitative Design Australian Health Review Focus Groups 43 Doctors and nurses 2007 Grounded Theory
Peebles48 2012 Service evaluation Resuscitation Observationsnotes review 17 care episodes RRT episodes Not avail Thematic Analysis
Petersen50 2017 Qualitative Design BioMedical Central Focus Groups 18 Nurse Not avail Content Analysis

RRT, rapid response team.

Supplementary data

bmjoq-2020-001145supp003.pdf (83.5KB, pdf)

Quality assessment results

Critical Appraisal Skills Programme

Studies were assessed to be of moderate to high quality and no studies were excluded based on this assessment (table 3). Two studies31 32 used surveys to understand nurses’ perceptions of caring for deteriorating patients and were scored poorly for choice of methodology. These studies were still included as open ended free-text questions were used and it was felt that this could still contribute to answering the research question, while acknowledging data from these studies are unlikely to be rich and is therefore a limitation of the method.

Table 3.

CASP quality assessment for synthesised studies

Study ID Was there a clear statement of the aims of the research? Was the research design appropriate to address the aims of the research? Was the recruitment strategy appropriate to the aims of the research? Was the data collected in a way that addresses the research issue? Was the data analysis sufficiently rigorous? Is there a clear statement of findings? Is a qualitative methodology appropriate? How valuable is the research? Have ethical issues been taken into consideration? Has the relationship between researcher and participants been adequately considered?
Andrews33 2005 Low Low Unclear Low Low Unclear Low Low Low high
Astroth45 2013 Low Low Low Low Low Low Low Low Low high
Braaten51 2015 Low Low Low Low Low Low Low Low Low high
Brady42 2014 Low Low Low Low Low Unclear Low Low Low high
Bunkenborg55 2013 Low Unclear High Low Unclear Low Low Low Low high
Burns46 2017 Low Unclear Unclear Unclear Low Unclear Low Low Low high
Chua49 2013 Low Low Low Low Low Low Low Low Unclear high
Chua52 2019 Low Low Low Low Low Low Low Low Low low
Currey32 2017 Low Low Low Unclear Unclear Unclear Low Low Low high
Donohue15 2010 Low Low Low Low Low Low Low Low Unclear Unclear
Elmufdi43 2017 Low Low Low Low Low High Low Unclear Low high
Foley56 2018 Low Low Low Low Low Low Low Low Low high
Hart40 2016 Low Low Low Low Low Low Low Low Low low
Iddrisu54 2018 Low Low Unclear Low Low Low Low Low Low high
James31 2010 Low Low Low Low Low Low Low Low Low high
Johnston8 2015 Low Unclear Low Low Low Low Low Low Low high
King 201972 Low Low Low Low Low Low Low Low Low High
Leach38 2010 Low Low Unclear Low Low Low Low Low Low high
Mackintosh34 2012 Low Low Low Low Low Low Low Low Low high
Mackintosh41 2014 Low Low Low Low Low Low Low High Low high
Martland47 2016 Low Low Unclear Low High Low Low Low Low high
Massey44 2014 Low Low Low Low Unclear Low Low Low Low low
Peebles48 2012 Low Low Low Low Unclear Low Low Low Unclear high
Petersen50 2017 Low Low High High Low Low Low Low Low high

CASP, Critical Appraisal Skills Programme.

Grading of Recommendations Assessment, Development and Evaluation-CERQual

Following the CASP assessment all studies were evaluated against the GRADE-CERQual criteria. A Summary of Qualitative Findings table (table 4) is presented which promotes transparency in this synthesis’ findings and methods. The table includes documented rationale for grading judgements.

Table 4.

Confidence in synthesised findings using the GRADE-CERQual framework

Summary of review finding Studies contributing to the finding Methodological limitations Coherence Adequacy Relevance CERQual confidence assessment Explanation of the CERQual evidence
Information packaging (using quantifiable evidence of patient deterioration) affected perceived communication credibility 15 33–39 Low concerns regarding study methodology Low concerns about coherence Low concerns about adequacy Low concerns about relevance High Confidence All studies, demonstrated good methodology, data were considered moderately thick with high numbers of participants and methods, a high no of studies contributed to review finding,
Flattened hierarchy and were organisational components affecting escalation of care 15 31 32 34 35 41–46 Low concerns regarding methodology Low concerns about coherence Low concerns about adequacy Low concerns about relevance High Confidence One study with minor concerns regarding methodology (survey), high no of studies contributing to finding, data were considered moderately thick with high numbers of participants and methods
Workload and staffing were factors considered by clinical staff to affect their Situational awareness of patient deterioration. 15 32 34–36 41 42 45 47–52 Minor concerns regarding methodology Low concerns about coherence Low concerns about adequacy Low concerns about relevance High confidence Two studies with minor methodological concerns with one study where using a survey, and another study using participants for a focus group put forward by head nurse, high no of studies contributing to review finding, rich data sources and multiple methods of data collection, data were considered moderately thick with high numbers of participants and methods
Team functioning caused problems or facilitated care during escalation 15 32 33 35 36 41–45 47–51 Minor concerns regarding methodology Low concerns about coherence Low concerns about adequacy Low concerns about relevance High confidence Two studies with methodological concerns, one study where using a survey, and another study using participants for a focus group put forward by head nurse, all other studies demonstrate good methodology, high no of studies contributing to review finding, data were considered moderately thick with high numbers of participants and methods
Soft signal of patient deterioration used by clinical staff indicating a patient’s worsening condition, not adequately represented in Early Warning Score 15 31 33 35 36 38 41 42 44 46 49–51 54–56 Moderate concerns regarding methodology Low concerns about coherence Moderate concerns about adequacy Low concerns about relevance Moderate Confidence Three studies had methodological concerns. One utilising a survey methodology with open ended-questions, the other was being observed by the implementer of the local Medical Emergency Team (MET), the last one using participants for a focus group put forward by head nurse, large no of studies contributing to synthesis finding,
Clinician confidence affected decision making during escalation of care 31–36 38 43–45 47 49–52 55 Moderate concerns about methodology Low concerns about coherence Moderate concerns about adequacy Low concerns about relevance Moderate confidence Four studies had methodological concerns, two studies utilised a survey methodology with open ended-questions, the other study had a focus group where participants were selected by head nurse, the other had observation completed by the implementer of the local RRT, data were considered moderately thick with high numbers of participants and different methods, large no of studies contributing to synthesis finding
Clinical Assessment skills relating to patient assessment and staff experience positively or negatively affected deterioration detection by clinical staff 15 35 36 38 42 43 46 49–51 54 55 Moderate concerns regarding methodology Low concerns about coherence Moderate concerns about adequacy Low concerns about relevance Moderate Confidence Two studies with methodological concerns. One had observations completed by the implementer of the local RRT, the other study had a focus group where participants were selected by head nurse, data were considered moderately thick

GRADE, Grading of Recommendations Assessment, Development and Evaluation; RRT, rapid response ream.

Theme results

Themes presented are ranked from the highest to lowest confidence in synthesised findings. Data extracted mostly related to organisational and patient assessment factors affecting escalation of care. Organisational factors could be classified into Information Packaging and Communication Credibility, Flattened Hierarchy, Workload, Staffing and Situational Awareness and Team Functioning. We found patient assessment Themes of ‘Soft Signals of Patient Deterioration’ and Early Warning Scores (EWS), Decision Making and Clinical Assessment Skills and Experience.

Information packaging and communication credibility (high GRADE-CERQual evidence)

Eight studies identified that information packaging during escalation of care was a facilitator to success.15 33–39 Packaging involved using quantifiable evidence of patient deterioration such as vital signs15 33 34 40 41 to initiate escalation of care. This removed ambiguity,33 provided numerical evidence of deterioration15 and was a common language34 for clinical staff. This was particularly important when staff were unfamiliar to each other.34 Conversely, staff felt communication credibility was questioned when referrals were made using non-medical language33 or delivered in an unsystematic way.34 This made an escalation referral difficult to understand and prioritise, with medics often having to question further to gain more information to facilitate decision making.33

Flattened hierarchy (high GRADE-CERQual evidence)

A common organisational facilitator to escalation of care was a flattened hierarchy meaning that escalation is accepted from anyone to anyone.15 31 35 41–45 This created a confidence in staff to raise concerns regarding a patient’s clinical condition, opening channels of communication. Staff also felt that electronic vital signs systems increased the accountability of patient illness32 35 42 46 with acutely unwell patients being everyone’s responsibility. However, it was also acknowledged identifying who is accountable for an unwell patient was sometime a challenging. Synthesised studies demonstrated instances of lack of deteriorating patient ownership32 or passing on of the problem34 by clinical staff to another team or colleague.

Workload, staffing and situational awareness (high GRADE-CERQual evidence)

Several studies described resources as a significant factor affecting care escalation. Three studies identified a lack of skilled staff as limiting the ability to escalate the deteriorating patient.15 35 47 During high workload or low staffing periods staff felt their awareness of patient deterioration reduced due to sensory overload and suboptimal monitoring due to competing demands.15 32 34 36 41 42 45 47–52 Staff believed continuity of care improved situational awareness.36 41 42 49 51 Staff felt that a benchmark ‘baseline’, meant they could identify any significant changes to patient illness. It was not uncommon for staff to employ workarounds during the periods of system pressure such as escalating to the RRT. This was done (rightly or wrongly) to supplement care escalation when medical support was scarce.15 47

Team functioning (high GRADE-CERQual evidence)

Seven studies found poor team relationships were a barrier to escalating patient care, resulting in significant delays.15 32 33 35 42 45 50 Poor team working was presented as tasks being done sequentially rather than concurrently, or where there was a lack of role definition.35 36 41–45 47 48 Staff believed a lack of understanding of team roles and the care individuals could provide contributed to uncertainty about to whom patients should be escalated.35 41 44 45 Poor team functioning meant staff felt deterred from escalating care due to negative emotions such as fear of reprimand, fear of being wrong, intimidation and retribution.32 35 42 45 49–51 53 Escalation to outside resources, such as the RRT, was sometimes perceived to be negative15 43 45 with staff reporting that they preferred to cope with a patient deterioration.

Soft signals of patient deterioration and EWS (moderate GRADE-CERQual evidence)

Staff at times overruled the EWS derived escalation pathways using other patient related factors in their decision-making process when considering escalation.15 31 33 35 36 38 41 42 44 46 49–51 54–56 They identified factors additional to standard EWS variables which caused them concern about a patient’s condition (see online supplemental file 4). These patient factors or ‘soft signals of deterioration’, were (from most to least common finding in studies); pale skin,15 31 41 49 50 respiratory pattern (as distinct from respiratory rate),15 35 37 blood loss,36 49 personality change,36 49 patient complaint,50 54 skin temperature55 and patient fatigue (observed or reported).15 Nine studies found patient assessment was integral to detecting the ‘soft signals of deterioration’ including the early signs of worsening illness before a triggering EWS was evident.15 35 36 38 39 46 49 50 55 Two papers described how staff felt that EWS protocols could place barriers to escalation when patients did not meet the trigger threshold but nurses felt they required an increase in care surveillance.33 50 In some instances staff felt they had to wait for a deterioration to occur before being able to escalate33 but in others they continued to escalate despite normal EWS.46 49 51

Supplementary data

bmjoq-2020-001145supp004.pdf (258.5KB, pdf)

Decision making (moderate GRADE-CERQual evidence)

Escalation decision making involved clinical reasoning surrounding the detection, communication and management of escalation of care. Seven papers found that clinician confidence is a facilitator to decision making during patient deterioration management.31–33 35 36 44 50 Confidence can be derived from staff providing peer support to one another, training or education level. Shared team decisions were sometimes an escalation facilitator.38 44 45 47 49 51 However, a lack of consensus in decision making particularly for end of life care,32 was seen as problematic34 43 49 55 often leading to deviation from guidelines or escalation protocols. Lack of consistency in decisions meant escalation of care demonstrated response variability,34 leading to differing and unpredictable priorities.47 There was also evidence of clinicians assuming physiology changes were not significant and waiting for confirmation of deterioration before responding meaningfully.43

Clinical assessment skills and clinical experience (moderate GRADE-CERQual evidence)

Clinical assessment involved looking, listening and touching the patient to identify respiratory, skin, neurological or physiological abnormalities.15 49 51 55 57 The ability to clinically assess patients well enabled staff to make better escalation of care decisions,35 36 38 43 49–51 55 particularly as the ability to detect ‘soft signs’ was seen as key. Conversely, undertaking a poor patient clinical assessment posed barriers to illness detection.15 Many studies found that as staff gained experience of deteriorating patients they were better able to detect and predict impending illness.15 36 38 42 43 46 49 50 54 55

Discussion

We identified 24 qualitative studies of moderate to high methodological quality that identify how HF affect escalation of care. Our evidence synthesis has contributed to escalation of care literature and themes derived from analysis are pertinent to clinical practice.

The studies within this synthesis demonstrated that EWS provide staff with a tool that facilitates communication of concerns and assists workload prioritisation.33 Studies reported successful escalation of care was best facilitated when a patient’s deterioration packaged neatly with quantifiable evidence. However, some staff in the synthesised studies felt able to escalate non-triggering patients requiring medical attention, although this process was acknowledged to be challenging. It was also suggested that some staff can anticipate clinical deterioration before a triggering EWS58 and that there are soft signals (fatigue, skin temp/colour, patient complaint, personality change, blood loss, respiratory pattern), of deterioration recognised by nurses but are not adequately captured by EWS in their current format. Many studies also found that as staff gained experience of deteriorating patients, they were better able to predict deterioration patterns and anticipate problems. It seems that the EWS alone may not maximise improvements to patient outcomes.59 60 Evidence suggests that organisations should facilitate good patient assessment, as this was key to detecting soft signals that would otherwise go undetected through an alerting system. Research should also aim to identify how clinical staff anticipate problems in certain patient groups and how they recognise and respond to these to ultimately create safety.61 It is evident that the literature does not fully report good escalation catches62 such as rescued non-triggering sick patients. This event is in effect invisible and not measured in current healthcare evaluation systems or metrics. Incorporating this tacit knowledge into education programmes or simulation training and scenarios, may be a feasible strategy to improve care escalation.

A flattened hierarchy63 was implemented in the aviation industry when it was discovered that a number of flight incidents may have been avoided had the copilot been empowered to challenge the pilot.64 Synthesised studies identified that a Flattened Hierarchy was felt by healthcare staff to be a positive strategy for escalating care of deteriorating patients (escalation can be initiated from anyone). However, the effectiveness of a flattened hierarchy may be influenced by poor team functioning. Poor team working was a common barrier to escalation of care identified in this evidence synthesis. This finding is corroborated by a retrospective case records review of preventable hospital deaths3 and a literature review on FTR following surgery.65 In both publications, the authors isolated several contributory factors such as poor team communication, leadership and decision making. Without adequate team communication, the benefits of a flattened hierarchy and team decision making may be lost. If organisations wish to implement a flattened hierarchy escalation system this must also be complemented with an emphasis on non-technical skills and training66 before evidence of full patient benefit.

A clinically significant theme to emerge from the synthesised findings was that the greater the workload, the less staff felt they were able to keep track of patient illness or monitor their patients. This sometimes resulted in staff undermonitoring their patients causing some triggering patient deterioration to go unnoticed.50 This finding is supported by a recent study demonstrating that lower numbers of registered nurses led to a higher rate of missed vital signs observation.67 Organisations could focus on reducing workload, (an unlikely solution), or improving vital signs monitoring processes. A recent option is utilising wearable continuous monitoring that may reduce the nursing workload spent performing regular vital signs observation rounds.68

Other significant clinical effects of high workloads may be a reduction in staff ability to detect deterioration in patients who are not triggering, losing the human safety net for false negative (non-triggering) patients. When mental capacity is limited with reduced team resources, this will directly affect an individual’s situational awareness of the environment as mental resources reduce as cognitive demands increase.69 A recent study found the risk of death increased by 3% for every day a patient experienced nurse staffing levels below ward mean.70 Poor situational awareness, reduced ability to detect soft signals of deterioration and undermonitoring may explain these results. Conversely, staff described improved situational awareness when there was continuity to their patient care. This was felt to facilitate staff in detecting often nuanced clinical changes or soft signals and also bridged the care elements through a patients illness.71 A strong local emphasis on nursing continuity should be encouraged as the evidence suggests that this may improve detection of deterioration and care escalation.

Our study has some limitations. Synthesised studies were assessed for their methodological robustness using the GRADE-CERqual and CASP guidelines. This enabled us to present themes with the highest confidence of good quality evidence first, but results may be limited by the data quality or analysis. within the studies themselves. Publication bis may also affect results that were included. Broadly, studies were methodologically sound but consistently failed to explore the relationship of the researcher to the participants, or this was not explicitly documented. There was also only one study identified that used patients and relatives as study participants.72

Conclusion

This evidence synthesis has identified HF that affect escalation of care. EWS have clinical benefits but can sometime impede escalation in patients not meeting the escalation threshold. Staff use other factors (soft signals) not captured in EWS to escalate care. The literature supports strategies that improve the escalation process such as good patient assessment skills. An organisational emphasis on non-technical skills and team cohesion should be synonymous with a flattened hierarchy to enable effective care escalation.

Footnotes

Twitter: @jodyedeOx

Contributors: All authors have substantially contributed to this work and have agreed the final manuscript. JE designed and led the project. TP provided specialist librarian knowledge and assisted in conducting literature searches. JD, VW, RE and PJW have substantially contributed to this work, provided expertise and have agreed the final manuscript.

Funding: JE, Clinical Doctoral Research Fellow, NIHR (NIHR300509) is funded by Health Education England (HEE) / National Institute for Health Research (NIHR) for this research project. PJW is funded by the National Institute of Health Research (NIHR), UK and the NIHR Biomedical Research Centre, Oxford, during the conduct of the study. JD and VW are funded by Health Innovation Challenge Fund (HICF-R9-524 and WT-103703/Z/14/Z), a parallel funding partnership between the Department of Health and Wellcome Trust.

Disclaimer: The views expressed in this publication are those of the author(s) and not necessarily those of the NIHR, University of Plymouth, NHS or the UK Department of Health and Social Care.

Competing interests: PJW report significant grants from the National Institute of Health Research (NIHR), UK and the NIHR Biomedical Research Centre, Oxford, during the conduct of the study. PJW report modest grants and personal fees from Sensyne Health, outside the submitted work. PJW holds shares in the company.

Patient consent for publication: Not required.

Ethics approval: We did not require ethical approval as no primary data were collected.

Provenance and peer review: Not commissioned; externally peer reviewed.

Data availability statement: Data are available upon request.

Supplemental material: This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

References

  • 1.Ghaferi AA, Osborne NH, Birkmeyer JD, et al. Hospital characteristics associated with failure to rescue from complications after pancreatectomy. J Am Coll Surg 2010;211:325–30. 10.1016/j.jamcollsurg.2010.04.025 [DOI] [PubMed] [Google Scholar]
  • 2.Ghaferi AA, Birkmeyer JD, Dimick JB. Hospital volume and failure to rescue with high-risk surgery. Med Care 2011;49:1076–81. 10.1097/MLR.0b013e3182329b97 [DOI] [PubMed] [Google Scholar]
  • 3.Hogan H, Healey F, Neale G, et al. Preventable deaths due to problems in care in English acute hospitals: a retrospective case record review study. BMJ Qual Saf 2012;21:737–45. 10.1136/bmjqs-2011-001159 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Donaldson L. An organisation with a memory. Clin Med 2002;2:452–7. 10.7861/clinmedicine.2-5-452 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.NCEPOD . Themes and recommendations common to all Hospital Specialities, 2018. [Google Scholar]
  • 6.Ede J, Jeffs E, Vollam S, et al. A qualitative exploration of escalation of care in the acute ward setting. Nurs Crit Care 2020;25:171–8. 10.1111/nicc.12479 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.NRLS national patient safety incident reports: commentary 2018.
  • 8.Johnston MJ, Arora S, King D, Bouras G, et al. A systematic review to identify the factors that affect failure to rescue and escalation of care in surgery. Surgery 2015;157:752–63. 10.1016/j.surg.2014.10.017 [DOI] [PubMed] [Google Scholar]
  • 9.Findlay G, Shotton H, Mason K. Time to intervene? Natl Confid Enq into Patient Oucome Death 2012:1–125. [Google Scholar]
  • 10.NHS Improvement . The adult patient who is deteriorating: sharing learning from literature, incident reports and root cause analysis investigations, 2016: 23. [Google Scholar]
  • 11.Rowan K, Adam S, Ball C. Evaluation of outreach services in critical care, 2004: 399. [Google Scholar]
  • 12.Johnston M, Arora S, Anderson O, et al. Escalation of care in surgery: a systematic risk assessment to prevent avoidable harm in hospitalized patients. Ann Surg 2015;261:831–8. 10.1097/SLA.0000000000000762 [DOI] [PubMed] [Google Scholar]
  • 13.Mochizuki K, Shintani R, Mori K, et al. Importance of respiratory rate for the prediction of clinical deterioration after emergency department discharge: a single-center, case-control study. Acute Med Surg 2017;4:1–7. 10.1002/ams2.252 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Shearer B, Marshall S, Buist MD, et al. What stops Hospital clinical staff from following protocols? an analysis of the incidence and factors behind the failure of bedside clinical staff to activate the rapid response system in a multi-campus Australian metropolitan healthcare service. BMJ Qual Saf 2012;21:569-75. 10.1136/bmjqs-2011-000692 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Donohue LA, Endacott R. Track, trigger and teamwork: communication of deterioration in acute medical and surgical wards. Intensive Crit Care Nurs 2010;26:10–17. 10.1016/j.iccn.2009.10.006 [DOI] [PubMed] [Google Scholar]
  • 16.Shamseer L, Moher D, Clarke M. PRISMA-P (Preferred Reporting Items for Systematic review and Meta-Analysis Protocols) 2015 checklist : recommended items to address in a systematic review protocol *. BMJ 2015;349:g7647. [DOI] [PubMed] [Google Scholar]
  • 17.Stern C, Jordan Z, Mcarthur A. Systematic reviews, step by step developing the review question and inclusion criteria, 2014: 114. [DOI] [PubMed] [Google Scholar]
  • 18.Ede J, Westgate V, Petrinic T, et al. How human factors affect escalation of care: a protocol for a qualitative evidence synthesis of studies. BMJ Open 2019;9:e025969. 10.1136/bmjopen-2018-025969 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Carayon P, Schoofs Hundt A, Karsh B-T, et al. Work system design for patient safety: the SEIPS model. Qual Saf Health Care 2006;15 Suppl 1:50–8. 10.1136/qshc.2005.015842 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Lincoln YS, Guba EG. Naturalistic inquiry. Sage Publications, 1985. [Google Scholar]
  • 21.Greenhalgh T. How to read the basics of evidence based medicine, 2001: 315. [Google Scholar]
  • 22.Tacconelli E. Systematic reviews: CRD’s guidance for undertaking reviews in health care, 2010: 10. [Google Scholar]
  • 23.Lewin S, Booth A, Glenton C, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implement Sci 2018;13:2. 10.1186/s13012-017-0688-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Colvin CJ, Garside R, Wainwright M, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 4: how to assess coherence. Implement Sci 2018;13:13. 10.1186/s13012-017-0691-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Booth A, Rashidian A, Noyes J. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 3: how to assess methodological limitations. Implement Sci 2018;13. 10.1186/s13012-017-0690-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Noyes J, Booth A, Lewin S, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 6: how to assess relevance of the data. Implement Sci 2018;13:4. 10.1186/s13012-017-0693-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Glenton C, Carlsen B, Lewin S, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 5: how to assess adequacy of data. Implement Sci 2018;13:14. 10.1186/s13012-017-0692-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Lewin S, Bohren M, Rashidian A, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 2: how to make an overall CERQual assessment of confidence and create a summary of qualitative findings table. Implement Sci 2018;13:10. 10.1186/s13012-017-0689-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Booth AA, Noyes J, Flemming K. Guidance on choosing qualitative evidence synthesis methods for use in health technology assessments of complex interventions, 2016. Available: Http//WwwIntegrate-HtaEu/Downloads/
  • 30.Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol 2008;9. 10.1186/1471-2288-9-59 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.James J, Butler-Williams C, Hunt J, et al. Vital signs for vital people: an exploratory study into the role of the healthcare assistant in recognising, recording and responding to the acutely ill patient in the general ward setting. J Nurs Manag 2010;18:548–55. 10.1111/j.1365-2834.2010.01086.x [DOI] [PubMed] [Google Scholar]
  • 32.Currey J, Allen J, Jones D. Critical care clinician perceptions of factors leading to medical emergency team review. Aust Crit Care 2018;31:87–92. 10.1016/j.aucc.2017.03.003 [DOI] [PubMed] [Google Scholar]
  • 33.Andrews T, Waterman H. Packaging: a grounded theory of how to report physiological deterioration effectively. J Adv Nurs 2005;52:473–81. 10.1111/j.1365-2648.2005.03615.x [DOI] [PubMed] [Google Scholar]
  • 34.Mackintosh N, Rainey H, Sandall J. Understanding how rapid response systems may improve safety for the acutely ill patient: learning from the frontline. BMJ Qual Saf 2012;21:135–44. 10.1136/bmjqs-2011-000147 [DOI] [PubMed] [Google Scholar]
  • 35.Johnston M, Arora S, King D, et al. Escalation of care and failure to rescue: a multicenter, multiprofessional qualitative study. Surgery 2014;155:989–94. 10.1016/j.surg.2014.01.016 [DOI] [PubMed] [Google Scholar]
  • 36.Hart PL, Spiva L, Dolly L, et al. Medical-Surgical nurses' experiences as first responders during deterioration events: a qualitative study. J Clin Nurs 2016;25:3241–51. 10.1111/jocn.13357 [DOI] [PubMed] [Google Scholar]
  • 37.Massey D, Chaboyer W, Anderson V. What factors influence ward nurses' recognition of and response to patient deterioration? an integrative review of the literature. Nurs Open 2017;4:6–23. 10.1002/nop2.53 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Searle Leach L, Mayo A. How RNs rescue patients: a qualitative study of RNs’ perceived involvement in rapid response teams. Qual Saf Heal Care 2010;19. 10.1136/qshc.2008.030494 [DOI] [PubMed] [Google Scholar]
  • 39.Mackintosh N, Humphrey C, Sandall J. The habitus of 'rescue' and its significance for implementation of rapid response systems in acute health care. Soc Sci Med 2014;120:233–42. 10.1016/j.socscimed.2014.09.033 [DOI] [PubMed] [Google Scholar]
  • 40.Hart LJ, Carr C, Fullerton JT. Task analysis as a resource for strengthening health systems. J Midwifery Womens Health 2016;61:257–62. 10.1111/jmwh.12410 [DOI] [PubMed] [Google Scholar]
  • 41.Mackintosh N, Watson K, Rance S, et al. Value of a modified early obstetric warning system (MEOWS) in managing maternal complications in the peripartum period: an ethnographic study. BMJ Qual Saf 2014;23:26–34. 10.1136/bmjqs-2012-001781 [DOI] [PubMed] [Google Scholar]
  • 42.Brady PW, Goldenhar LM. A qualitative study examining the influences on situation awareness and the identification, mitigation and escalation of recognised patient risk. BMJ Qual Saf 2014;23:153–61. 10.1136/bmjqs-2012-001747 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Elmufdi FS, Burton SL, Sahni N, et al. Clinical and sociocultural factors associated with failure to Escalate care of deteriorating patients. Am J Med Qual 2018;33:391-396. 10.1177/1062860617748739 [DOI] [PubMed] [Google Scholar]
  • 44.Massey D, Chaboyer W, Aitken L. Nurses' perceptions of accessing a medical emergency team: a qualitative study. Aust Crit Care 2014;27:133–8. 10.1016/j.aucc.2013.11.001 [DOI] [PubMed] [Google Scholar]
  • 45.Astroth KS, Woith WM, Stapleton SJ, et al. Qualitative exploration of nurses' decisions to activate rapid response teams. J Clin Nurs 2013;22:2876–82. 10.1111/jocn.12067 [DOI] [PubMed] [Google Scholar]
  • 46.Burns KA, Reber T, Theodore K, et al. Enhanced early warning system impact on nursing practice: a phenomenological study. J Adv Nurs 2018;74:1150–6. 10.1111/jan.13517 [DOI] [PubMed] [Google Scholar]
  • 47.Martland J, Chamberlain D, Hutton A, et al. Communication and general concern criterion prior to activation of the rapid response team: a grounded theory. Aust Health Rev 2016;40:477–83. 10.1071/AH15123 [DOI] [PubMed] [Google Scholar]
  • 48.Peebles E, Subbe CP, Hughes P, et al. Timing and teamwork--an observational pilot study of patients referred to a Rapid Response Team with the aim of identifying factors amenable to re-design of a Rapid Response System. Resuscitation 2012;83:782–7. 10.1016/j.resuscitation.2011.12.019 [DOI] [PubMed] [Google Scholar]
  • 49.Chua WL, Mackey S, Ng EKC, et al. Front line nurses' experiences with deteriorating ward patients: a qualitative study. Int Nurs Rev 2013;60:501–9. 10.1111/inr.12061 [DOI] [PubMed] [Google Scholar]
  • 50.Petersen JA, Rasmussen LS, Rydahl-Hansen S. Barriers and facilitating factors related to use of early warning score among acute care nurses: a qualitative study. BMC Emerg Med 2017;17:36. 10.1186/s12873-017-0147-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51.Braaten JS. Ce: original research: hospital system barriers to rapid response team activation: a cognitive work analysis. Am J Nurs 2015;115:22–32. 10.1097/01.NAJ.0000460672.74447.4a [DOI] [PubMed] [Google Scholar]
  • 52.Chua WL, Legido-Quigley H, Ng PY, et al. Seeing the whole picture in enrolled and registered nurses' experiences in recognizing clinical deterioration in general ward patients: a qualitative study. Int J Nurs Stud 2019;95:56–64. 10.1016/j.ijnurstu.2019.04.012 [DOI] [PubMed] [Google Scholar]
  • 53.Massey CN, Feig EH, Duque-Serrano L, et al. Well-Being interventions for individuals with diabetes: a systematic review. Diabetes Res Clin Pract 2019;147:118–33. 10.1016/j.diabres.2018.11.014 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 54.Mohammmed Iddrisu S, Hutchinson AF, Sungkar Y, et al. Nurses' role in recognising and responding to clinical deterioration in surgical patients. J Clin Nurs 2018;27:1920–30. 10.1111/jocn.14331 [DOI] [PubMed] [Google Scholar]
  • 55.Bunkenborg G, Samuelson K, Åkeson J, et al. Impact of professionalism in nursing on in-hospital bedside monitoring practice. J Adv Nurs 2013;69:1466–77. 10.1111/jan.12003 [DOI] [PubMed] [Google Scholar]
  • 56.Foley C, Dowling M. How do nurses use the early warning score in their practice? A case study from an acute medical unit. J Clin Nurs 2019;28:1183–92. 10.1111/jocn.14713 [DOI] [PubMed] [Google Scholar]
  • 57.Huang H, Chen L, Sanberg PR. Clinical achievements, obstacles, falsehoods, and future directions of cell-based neurorestoratology. Cell Transplant 2012;21 Suppl 1:3–11. 10.3727/096368912X633725 [DOI] [PubMed] [Google Scholar]
  • 58.Clifton DA, Clifton L, Sandu D-M, et al. ‘Errors’ and omissions in paper-based early warning scores: the association with changes in vital signs—a database analysis. BMJ Open 2015;5:e007376–7. 10.1136/bmjopen-2014-007376 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 59.Kim WY, Shin YJ, Lee JM, et al. Modified early warning score changes prior to cardiac arrest in general wards. PLoS One 2015;10:e0130523–11. 10.1371/journal.pone.0130523 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 60.McGaughey J, O'Halloran P, Porter S, et al. Early warning systems and rapid response to the deteriorating patient in hospital: a realist evaluation. J Adv Nurs 2017;73:3119–32. 10.1111/jan.13367 [DOI] [PubMed] [Google Scholar]
  • 61.Vincent C. The essentials of patient safety the essentials of patient safety adapted from patient safety. 2nd Edn, 2001. [Google Scholar]
  • 62.Wallace SC, Mamrol C, Finley E. Promote a culture of safety with good catch reports patient safety analyst data analyst Pennsylvania patient safety authority corresponding author. 14, 2017. [Google Scholar]
  • 63.HSE . High reliability organisations high reliability organisations a review of the literature, 2011. [Google Scholar]
  • 64.Green B, Oeppen RS, Smith DW, et al. Challenging hierarchy in healthcare teams - ways to flatten gradients to improve teamwork and patient care. Br J Oral Maxillofac Surg 2017;55:449–53. 10.1016/j.bjoms.2017.02.010 [DOI] [PubMed] [Google Scholar]
  • 65.Ghaferi AA, Dimick JB. Importance of teamwork, communication and culture on failure-to-rescue in the elderly. Br J Surg 2016;103:e47–51. 10.1002/bjs.10031 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 66.Yule S, Flin R, Paterson-Brown S, et al. Non-Technical skills for surgeons in the operating room: a review of the literature. Surgery 2006;139:140–9. 10.1016/j.surg.2005.06.017 [DOI] [PubMed] [Google Scholar]
  • 67.Redfern OC, Griffiths P, Maruotti A, et al. The association between nurse staffing levels and the timeliness of vital signs monitoring: a retrospective observational study in the UK. BMJ Open 2019;9:e032157. 10.1136/bmjopen-2019-032157 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 68.Areia C, Young L, Vollam S, et al. Wearability testing of ambulatory vital sign monitoring devices: prospective observational cohort study. JMIR Mhealth Uhealth 2020;8:e20214. 10.2196/20214 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 69.Macdonald JSP, Lavie N. Visual perceptual load induces inattentional deafness. Atten Percept Psychophys 2011;73:1780–9. 10.3758/s13414-011-0144-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 70.Griffiths P, Maruotti A, Recio Saucedo A, et al. Nurse staffing, nursing assistants and hospital mortality: retrospective longitudinal cohort study. BMJ Qual Saf 2019;28:609–17. 10.1136/bmjqs-2018-008043 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 71.Haggerty JL, Reid RJ, Freeman GK, et al. Continuity of care: a multidisciplinary review. BMJ 2003;327:1219–21. 10.1136/bmj.327.7425.1219 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 72.King L, Peacock G, Crotty M, et al. Consumers' perspectives on their involvement in recognizing and responding to patient deterioration-Developing a model for consumer reporting. Health Expect 2019;22:385–95. 10.1111/hex.12858 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 73.Endsley MR. Design and evaluation for situation awareness enhancement. Proc Hum Factors Soc Annu Meet 1988;32:97–101. 10.1177/154193128803200221 [DOI] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary data

bmjoq-2020-001145supp001.pdf (97.4KB, pdf)

Supplementary data

bmjoq-2020-001145supp002.pdf (31.5KB, pdf)

Supplementary data

bmjoq-2020-001145supp003.pdf (83.5KB, pdf)

Supplementary data

bmjoq-2020-001145supp004.pdf (258.5KB, pdf)


Articles from BMJ Open Quality are provided here courtesy of BMJ Publishing Group

RESOURCES