Predicting team-performance and leadership in emergency situations by observing standardised operational procedures: a prospective single-blind simulator-based trial

Abstract

Background

Performance of interdisciplinary teams and their leaders is crucial in acute medical care and can be monitored by observing specific events. Standardised operational procedures (SOP) are easily observable, whereas the unpredictability of medical emergencies makes performance monitoring in these situations difficult. The aim of this study was therefore to assess whether performance in emergency situations can be predicted by performance observed during an SOP.

Methods

30 intensive care unit teams composed of one staff physician (leader), one resident and three nurses performed a simulated scenario of an elective electrical cardioversion (SOP) followed by a cardiac arrest (emergency). Video recordings obtained during simulations were used for data analysis. The primary outcome was the correlation between performance scores of electrical cardioversion and performance during cardiopulmonary resuscitation (hands-on time, time to first defibrillation).

Results

None of the cardioversion performance scores significantly correlated with resuscitation performance. Leadership scores during electrical cardioversion correlated positively with leadership scores during cardiopulmonary resuscitation (r=0.365, p=0.047). Moreover, there was a positive correlation of leaders being hands-off during both electrical cardioversion and cardiopulmonary resuscitation (r=0.645, p<0.0001).

Conclusions

Team performance in SOP carried no predictive value for emergency situations. Observing teams in easily observable SOP is therefore no suitable substitute for monitoring the performance in medical emergencies. There was a between-situation consistency for specific elements of leadership.

Introduction

Teamwork is important in high-risk domains such as acute medical care.^1–3 This is especially true for the management of unexpected events and emergencies, which are typically handled by interdisciplinary teams. Moreover, the performance of team leaders is regarded as a key factor for team performance.^4–6 Thus, monitoring and promoting the quality of teamwork and team leaders must be considered as important managerial tasks.

Observing and monitoring teams in emergency situations is inherently difficult. By contrast, observing and monitoring is feasible during standardised operational procedures (SOP). If team performance and leadership during SOPs were found to correspond to team performance and leadership during emergencies, monitoring of SOPs could serve as a suitable surrogate of the institutional quality of teamwork and team leaders. There are, however, no data from medicine or any other high-risk domain on the predictive power of the performance from teams and their leaders during an SOP for team performance during a crisis situation.

Team adaptability is a core aspect of team efficiency in general^7–11 and especially in emergency teams.¹² Extrapolating data from group research^8–14 leads to two alternative hypotheses: (1) team competences are mainly generic¹³ so that teams that are more efficient in a routine situation will also be more efficient in an emergency situation¹⁴ or (2) team performance is mainly situation-specific,^15–17 so that little, if any, correlation should be found between routine and emergency situations. Likewise, quality of leadership might predominantly result from a generic individual competency or, alternatively, is largely situation-dependent.^{5 18–20}

In sum, little is known about how the performance of teams and team leaders in one type of situation relates to their performance in another situation. Accordingly, the purpose of the current study was to investigate whether the quality of performance of medical teams and their leaders in a routine SOP task (electrical cardioversion) predicts the quality of performance in a subsequent emergency situation (cardiopulmonary resuscitation).

Methods

Setting

This is a prospective single-blind (ie, participants were not aware of the purpose of the study) simulator-based study. The study was approved by the local ethics committee, and all participants gave written informed consent.

Participants

The study was performed during simulator-based training sessions that are regularly offered to all healthcare workers of an intensive care unit (ICU). Thirty teams participated, composed of three nurses, one resident and one senior physician (leader) each. All participants participated only once. All participating nurses had a diploma in intensive care nursing or were in their last year of formation. All participating senior physicians were board certified specialists in internal medicine and/or intensive care. At the time of the study, all participants were working in the same large ICU.

Simulator 

A commercially available patient simulator with the possibility of remote control of vital parameters was used (SimMan, Leardal, Norway). A loudspeaker in the head of the manikin allowed verbal interaction with the participants by broadcasting the voice of a confederate. The simulator was placed in a room designed to resemble as closely as possible the working conditions of the ICU. Using a checklist, participants were instructed on technicalities and equipment available.

Scenario:

Participants were confronted with two situations for the same patient: (1) an elective ambulatory electrical cardioversion according to institutional guidelines (SOP), which was immediately followed by (2) a cardiac arrest due to ventricular fibrillation requiring cardiopulmonary resuscitation. During the study period, electrical cardioversions were routinely performed (2–5 cases each week) on the ICU where the participants were recruited. The incidence of cardiac arrests both within and outside the ICU with participation of staff from the ICU was 1–3 events per week. Simulator-based training in resuscitation was part of the mandatory continuous education for all healthcare workers of the ICU.

Participants were informed that the simulation assumes that they were, during their present shift, responsible for the care of the simulated patient admitted to their ICU for a planned ambulatory electrical cardioversion of his atrial fibrillation. At the start of the simulation, participants were not aware of the later development into an emergency situation.

The ‘patient’ (simulator) was placed in a bed and connected to a monitor that displayed a continuous ECG, continuous pulse-oximetry and the non-invasive blood pressure at selected time intervals. A cannula was placed in a peripheral vein to allow for intravenous administration of drugs. At the beginning of the simulation, the participants received a written patient record encompassing a 12-lead ECG confirming the presence of atrial fibrillation; a booklet confirming adequate anticoagulation during the last 4 weeks; and a consent form for the intervention signed by the patient. In addition, the participants received all forms and documents usually used in real cases.

Regardless of team performance during cardioversion, the patients’ atrial fibrillation converted into a regular sinus rhythm after the first electrical shock administered, thereby completing the cardioversion situation. Immediately after successful cardioversion, a transition phase started designed to link the two scenario parts of scientific interest (ie, electrical cardioversion and cardiac arrest) in a medically plausible and authentic way: following cardioversion, the patient’s heart rate gradually slowed to prompt the teams to take countermeasures by injecting atropine, a drug that fastens the heart rate and is considered the standard measure in that situation. Atropine resulted in a steadily increase of the heart rate over 1 min. Thereafter, a supraventricular tachycardia appeared which after 1 min degenerated in pulseless ventricular tachycardia. The onset of ventricular tachycardia marked the end of the transition phase and the start of the cardiac arrest situation.

In the cardiac arrest situation, teams had to perform cardiopulmonary resuscitation. Regardless of the measures taken by the team, ventricular fibrillation and hence the cardiac arrest lasted for a minimum of 3 min. Thereafter, the return of normal sinus rhythm could be achieved by defibrillation. A video-assisted debriefing concluded the training session.

Data analysis

Simulations were video-recorded. Experienced and trained observers made verbatim transcripts of all utterances and noted all activities of all team members

Performance markers

We could not identify suitable existing tools to rate team performances in the situations under investigation. Accordingly, an interdisciplinary team of psychologists, intensive care physicians and intensive care nurses developed performance markers based on the available medical evidence for cardioversion,^21–23 existing guidelines for cardiopulmonary resuscitation^{24 25} and recommendations for the development of team and leadership performance markers.^{26 27}

The technical performance markers used are listed in tables 1 and 2. During the cardiac arrest situation, the presence and type of life support was noted for each second. Hands-on time was defined as presence of cardiac support (ie, cardiac massage or defibrillation) and expressed as percentage of the complete time interval available for cardiac support. For each second team leaders were rated as ‘hands-on’, defined as any manual patient-related activity (eg, examination, injecting drugs or defibrillation) or ‘hands-off’, defined as no manual patient-related activity.

Table 1.

Technical performance markers of ECV

Performance marker	Reason to include	Performance
		N	% yes
ECV preparation phase
Check if correct patient	Patient’s safety issue	30/30	100
Check if consent available	Legal issue	19/30	63
Check anticoagulation*	Patient’s safety issue	30/30	100
Check if arrhythmia still present	Patient’s safety issue	5/30	17
Ask for weight	To calculate drug dose	23/30	77
Confirm fastening*	Patient’s safety issue	15/30	50
Ask about dental prosthesis	Patient’s safety issue	20/30	67
Ask about allergies	Patient’s safety issue	3/30	10
Ask about other diseases	Patient’s safety issue	7/30	23
Ask about current medication	Good clinical practice	4/30	13
Ask about problems in previous anaesthesia	Patient’s safety issue	5/30	17
Check laboratory values	Patient’s safety issue	5/30	17
Check if defibrillator is functional	Good clinical practice	10/30	33
Check airway equipment	Patient’s safety issue	17/30	57
Check drugs	Patient’s safety issue	9/30	30
Monitor: measure blood pressure in appropriate intervals	Good clinical practice	30/30	100
Monitor: switch on sound to make oxygen saturation audible	Patient’s safety issue	19/30	63
ECV execution phase
Inform patient prior to put oxygen mask on his face*	Good clinical practice	30/30	100
Pre-oxygenation via face mask*	Patient’s safety issue	28/30	93
Inform patient of injection of drug 1 (opioid)*	Good clinical practice	21/30	70
Inform patient about possible side-effect of drug 1	Side-effect: dizziness	13/30	43
Inform patient of injection of drug 2 (propofol)*	Good clinical practice	20/30	67
Inform patient about effect of drug 2	Will induce anaesthesia	20/30	67
Inform patient about possible side-effect of drug 2	Side-effect: burning sensation during injection	3/30	10
Check sufficient depth of anaesthesia prior to shock	Good clinical practice	30/30	100
Set adequate energy level of defibrillator*	Good clinical practice	30/30	100
Defibrillator in synchronisation mode*	Patient’s safety issue	30/30	100
Ensure all team members hands-off prior to shock*	Safety issue	30/30	100

For all items: Yes=1 point, No=0 points. Items belonging to the ECV preparation phase were summarised in the ECV preparation score; items belonging to the ECV execution phase were summarised in the ECV execution score; the ECV global technical score is the sum of preparation and execution score.

*Items considered as essential and included in the ESSENTIALS of ECV score.

ECV, electrical cardioversion.

Table 2.

Technical performance markers and calculation of the global technical performance score of cardiopulmonary resuscitation

Performance marker	Performance of 30 teams
	N	%
Correct execution of cardiac massage-to-ventilation cycles*	25/30	83
Counting massage strokes aloud*	22/30	73
Correct number of cycles between subsequent defibrillations†	16/30	53
Counting cycles aloud*	2/30	7
Chest compression rate≥100/min*	24/30	80
Adequate defibrillation energy*	30/30	100
Correct sequence of defibrillation*	26/30	87
Epinephrine given in correct dose*	30/30	100

	Mean	SD
Hands-on time Score (% of available time)‡	7.5	1.2
Time to first defibrillation Score§	1.5	1.4
Time to start cardiac massage after first defibrillation Score§	3.2	1.4

Items were included in the global technical performance score of cardiopulmonary resuscitation as follows:

*Yes=1 point, No=0 points.

†Number of cycles within ±20% of recommended number were coded as correct.

‡1 point for each 10%.

§5 points if <5 s; 4 points if >5 and<10; 3 points if >10 and<15; 2 points if >15 and<20; 1 point if >20 and<25; 0 points if >25.

Behavioural performance markers were used to assess the quality of team performance and the quality of leadership of the senior physician; it encompassed explicit within-team task-distribution (1 point per task distributed); explicit decision on what activity to do next (eg, ‘we have to defibrillate’; 1 point per decision); explicit decisions on how to perform an activity (eg, ‘we defibrillate with 200 J’; 1 point per decision); statements to make the team aware of the current situation (eg, ‘we are now ready to induce anaesthesia’; 1 point per statement and situation). If the same decision or statement relating to the same activity at the same time was repeated, the point was attributed only once.

Using the performance markers, the following scores of technical, team and leadership performance were calculated:

• Electrical cardioversion preparation score=sum of all points achieved by all team members for technical performance markers during the preparation phase for cardioversion.

• Electrical cardioversion execution score=sum of all points achieved by all team members for technical performance markers during the execution phase of cardioversion.

• Electrical cardioversion global technical score=sum of the scores of the preparation and execution phase.

• Electrical cardioversion team score=sum of all points achieved by all team members for behavioural performance markers during the whole electrical cardioversion.

• Electrical cardioversion overall performance score=sum of the global technical score and the team score of electrical cardioversion.

• Essentials of electrical cardioversion score=sum of points achieved for performance markers agreed to be essential by an expert panel.

• Electrical cardioversion leadership score=sum of all points achieved by the team leader for behavioural performance markers during the whole electrical cardioversion.

• Cardiopulmonary resuscitation global technical score.

• Cardiopulmonary resuscitation team score=sum of all points achieved by all team members for behavioural performance markers during the cardiopulmonary resuscitation.

• Cardiopulmonary resuscitation overall performance score=sum of the global technical score and the team score of cardiopulmonary resuscitation.

• Cardiopulmonary resuscitation leadership score=sum of all points achieved by the team leader for behavioural performance markers during the whole cardiopulmonary resuscitation.

Coding

Two experienced raters independently coded all video-recordings and Cohen’s kappa was used to assess inter-rater reliability. If the coders differed less than 5 s for timing of events, their ratings were considered to agree and the shorter timing was used for further analysis. Differences of 5 s and more for timings of events and all differences regarding coding were solved by jointly regarding the video-recordings.

Statistical analysis

Statistical analysis was performed using SPSS V.22. Performance markers of cardioversions were correlated with those of resuscitation. Positive correlations would indicate that teams performing well in the cardioversion also performed well in the cardiac arrest situation, except for time to first defibrillation, for which a low value indicates high performance. Primary outcome was the correlation of hands-on time and time to first defibrillation, two key parameters of cardiopulmonary resuscitation with proven effect on outcome in patients with cardiac arrest, and performance markers of electrical cardioversion. Secondary outcomes were all other correlations across the two subscenarios.

Results

Thirty teams were recruited. All 30 teams performed the scenario as intended, and no protocol violation occurred. The video material obtained allowed for reliable coding in all teams, so that all 30 teams were included in the study.

Cohen’s kappa was 0.90 for coding of the cardioversion situation and 0.75 for coding of the cardiac arrest situation.

Primary outcome

Hands-on time was 75%±11% (mean ±SD; range 49%–95%), the time to first defibrillation 20±8 s (range 5–41 s) and the time to cardiac massage after the first defibrillation 12±7 s (range 4–31 s). There was considerable between-team variability in the performance scores obtained (tables 1–3; figures 1 and 2). There was no significant correlation of any performance score derived from the electrical cardioversion situation with parameters highly relevant for patient outcome (ie, time to first defibrillation and hands-on time) in the resuscitation situation (table 4; figure 1).

Table 3.

Performance scores of 30 teams during ECV and CPR

	Maximum score achievable	Mean	SD	Median	Range
ECV Preparation Score	17	6.0	2.7	6	1–14
ECV Execution Score	11	7.7	1.5	8	5–10
ECV Global Tech Score	28	14.2	3.6	14	9–23
ECV Team Score	11	5.9	1.3	6	2–8
ECV Overall Performance Score	39	21.9	4.2	21	14–34
Essentials of ECV Score	8	6.7	0.9	7	5–8
ECV Leadership Score	11	3.7	1.9	4	0–7
CPR Global Tech Score	28	18.0	2.9	19	12–24
CPR Team Score	13	5.6	0.6	6	5–7
CPR Overall Performance Score	41	25.7	3.2	26	19–31
CPR Leadership Score	13	4.4	1.5	5	1–7

CPR, cardiopulmonary resuscitation; ECV, electrical cardioversion.

Figure 1.

Scatterplot of the electrical cardioversion overall performance score and the cardiopulmonary resuscitation overall performance score in 30 medical teams. R=0.06, p=0.74.

Figure 2.

Scatterplot of the leadership score during electrical cardioversion and the leadership score during cardiopulmonary resuscitation in 30 medical teams.

Table 4.

Pearson correlation coefficients of ECV performance scores with numeric performance markers of CPR

Performance standard situation	Performance emergency resuscitation
	Hands-on time	Time to first defibrillation	Time to massage after first defibrillation
ECV Preparation Score	−0.061	0.045	−0.089
ECV Execution Score	−0.093	−0.110	−0.022
ECV Global Tech Score	−0.084	−0.012	−0.076
ECV Team Score	−0.039	−0.142	0.208
ECV Overall Performance Score	−0.088	−0.066	0.015
ECV Leadership Score	0.352	−0.234	−0.229

None of the correlations reached statistical significance.

ECV, electrical  cardioversion.

Secondary outcomes regarding team performance

There was no significant correlation of any performance score derived from the cardioversion situation with its corresponding score derived from the resuscitation situation (table 5).

Table 5.

Pearson correlation coefficients of performance scores of cardioversion with performance scores of CPR

Performance standard situation	Performance emergency resuscitation
	CPR Global Tech Score	CPR Team Score	CPR Overall Performance Score
ECV Preparation Score	−0.062	0.300	0.054
ECV Execution Score	−0.042	0.217	0.041
ECV Global Tech Score	−0.064	0.316	0.058
ECV Team Score	−0.029	0.149	0.028
ECV Overall Performance Score	−0.066	0.330	0.060

In bold: correlation coefficients of corresponding scores; none of the correlations reached statistical significance.

CPR, cardiopulmonary resuscitation; ECV, electrical cardioversion.

Secondary outcomes regarding leadership

 There was a significant positive correlation between the leadership scores obtained during electrical cardioversion and cardiopulmonary resuscitation (figure 2, table 6). Further analysis revealed that this correlation was entirely due to a positive correlation of statements making the team aware of the current situation while the correlation of the remaining items, relating to leaders’ decisions, was not significant.

Table 6.

Pearson correlation coefficients of leadership performance scores during ECV with same score in CPR

	Correlation coefficient	P values
Hands-off leadership: ECV with CPR	0.645	<0.0001
Leadership score: ECV with CPR	0.365	0.047
Components of the leadership score
Task distribution: ECV with CPR	−0.058	0.759
Decisions (what to do and how to do it): ECV with CPR	0.064	0.737
Making one’s team aware of current situation: ECV with CPR	0.367	0.046

CPR, cardiopulmonary resuscitation; ECV, electrical cardioversion.

There was a significant positive correlation between the leader being hands-off during both electrical cardioversion and cardiopulmonary resuscitation (table 6). Twenty-two out of 30 leaders were hands-off during electrical cardioversion. Seventeen of these 22 leaders also stayed hands-off during resuscitation, the remaining 5 engaging in cardiac massage, (3) defibrillation (1)¹ and both cardiac massage and defibrillation (1).¹ All eight leaders being hands-on during electrical cardioversion remained hands-on during cardiopulmonary resuscitation.

During cardiopulmonary resuscitation, hands-off leadership style was positively correlated with the leadership score (r=0.432, p=0.017) but not with technical performance or team performance. During electrical cardioversion, hands-off leadership style was positively correlated with global technical performance (r=0.374, p=0.042) and a shorter preparation (r=0.428, p=0.018) and execution phase (r=0.452, p=0.012).

Further findings

Within cardioversion, performance during the preparation phase was positively correlated with performance during the execution phase (r=0.428, p=0.02), constituting a positive correlation between two subtasks. Within cardiac arrest, the time to start massage after the first defibrillation was negatively correlated with hands-on time (r=−0.56, p<0.01); as time to start should be as short as possible but hands-on-time as long as possible, this correlation also constitutes a positive within-task association.

Discussion

The present study compared the performance of interdisciplinary teams and their leaders across two different situations. There was no correlation between team performance in a SOP and team performance in an emergency situation. There was a between-procedure consistency in specific elements of leadership only but not in leadership behaviour overall.

We are not aware of real-world data examining the predictive value of team performance or leadership, both in general and within medicine. In keeping with our results, Brannick et al reported that military air crews in simulated missions are not very consistent across two similar scenarios.²⁸

An important managerial implication of our findings regards the quality of acute medical care, which is predominantly provided by interdisciplinary teams. Our data indicate that monitoring an institution’s performance in easily observable SOP carries no predictive value for the same institution’s performance in emergencies. Thus, alternative ways of quality assurance of interdisciplinary teams and their leaders have to be investigated.

Our findings have implications for the understanding of teamwork and leadership and especially the relevance of generic skills or traits. Team theory acknowledges that tasks are important,7 9 13 29 30 but also emphasises that team skills are to some degree generic.^8–14 Our empirical data demonstrate the importance of task-related aspects. This does not preclude that some team skills are generic. However, one cannot assume that people appropriately apply such skills as required in a particular situation. Future research should focus on specific aspects of team performance and their generalisability.3 9 31 32

Leadership theory traditionally was characterised by an emphasis on leader-traits; however, it has become quite clear that leadership is very situation-specific.¹⁸ Moreover, leadership traits have been proposed not to represent specific behaviours, but rather the ability to recognise relevant situational cues and to adapt accordingly.¹⁹ The present empirical study demonstrates what might be called ‘fragmented consistency’ in that consistency occurred in two specific elements of leadership behaviour, (1) hands-off leadership style and (2) making one’s team aware of the situation, but not in leadership behaviour overall. Recognising cues that indicate changing situations is central for adaptation.^33–35 Perceiving changes in situations is likely to be easier if the team leader is not involved in overt action, especially in stressful and hectic situations.³⁶ Leaders being hands-off were, however, useful for team performance in electrical cardioversion only, but not for cardiopulmonary resuscitation, for which it is specifically recommended.³⁷ Moreover, the specific consistencies in leadership behaviour did not induce consistency in team performance across the two situations suggesting that orienting statements and hands-off style are only a part of the behaviour necessary for high team performance.

Our findings have implications for training of interdisciplinary teams and their leaders. Given the dominant role of contextual factors, training should cover a variety of situations and conditions and include a focus on team processes, especially adaptive team skills.12 29 38 39 Though such training is more complex and time consuming than traditional training, it may result in more generalisable or even generic skills.^{4 33 38}

Strengths of the present study include participants who were both adequately trained and regularly confronted with the standardised procedure and the emergency situation; this is important as a high correlation of team performance across two tasks can only be expected if the tasks are similar, or teams have been adequately trained in both tasks.^{12 40 41} The number of teams included in the present study and the between-team variation for both situations were sufficiently large to find medium size effects. Strengths of medical simulations include identical conditions for all participants and the possibility to investigate issues that for a variety of medical, ethical and practical reasons cannot be studied in real cases. Cardioversion, the SOP in the present study, fortunately proceeds uneventfully in the majority of occasions, and severe complications are rare.^{23 42 43} Thus, a prohibitively large number of cardioversions would have to be observed in real life to answer the research question.

Limitations of the present study include the restriction to one standardised procedure and one emergency situation. Thus, our results are not necessarily generalisable to other medical situations or situations outside of the medical field.

In conclusion, the present study demonstrates that the quality of team performance assessed during a SOP carries no predictive value for the quality of team performance during a subsequent emergency situation. By contrast, specific elements of leadership behaviour correlated between the two tasks. Further research is necessary to better understand the joint influence of task and team factors on the functioning of medical emergency teams.