The Cognitive Basis of Effective Team Performance: Features of Failure and Success in Simulated Cardiac Resuscitation

Abstract

Despite a body of research on teams in other fields relatively little is known about measuring teamwork in healthcare. The aim of this study is to characterize the qualitative dimensions of team performance during cardiac resuscitation that results in good and bad outcomes. We studied each team’s adherence to Advanced Cardiac Life Support (ACLS) protocol for ventricular fibrillation/tachycardia and identified team behaviors during simulated critical events that affected their performance. The process was captured by a developed task checklist and a validated team work coding system. Results suggest that deviation from the sequence suggested by the ACLS protocol had no impact on the outcome as the successful team deviated more from this sequence than the unsuccessful team. It isn’t the deviation from the protocol per se that appears to be important, but how the leadership flexibly adapts to the situational changes with deviations is the crucial factor in team competency.

Introduction

Cardiovascular disease is the leading cause of death in the United States, with sudden cardiac death accounting for the majority of fatalities [1]. Providing clinical interventions for the urgent treatment of cardiac arrest is a team endeavor and several healthcare workers need to co-ordinate their activities to ensure optimal team performance [2]. Healthcare has adapted training programs from other fields such as the Crew Resource Management (CRM) from aviation [3]. However direct adoption without understanding what comprises effective team work in healthcare may prove to be insufficient to identify specific instructional strategies for influencing team processes and outcomes. In this paper we describe development and use of tools to evaluate team work as it is directly elicited by the task of simulated cardiac resuscitation. This knowledge can help determine team training needs to develop training strategies based on empirical support.

Background

It has been demonstrated in general that physicians are relatively non-compliant with the prescribed ACLS guidelines when managing cardiac arrest [4]. Within dynamic and complex environments such as critical care where decision making is based on incomplete or conflicting information, effective team performance can promote safety and reduce errors [5]. There is also evidence that suboptimal team performance can have harmful effects. In the context of ACLS training, absence of leadership and explicit task distribution among the resuscitation team members is associated with poor ACLS performance [2]. Research on surgical team behaviors and patient outcomes has also shown that when teams exhibited infrequent team behaviors, patients were more likely to experience death or major complication [6]. Most of these studies have adapted metrics from other fields which do not specify how a construct assessing team performance is assessed in the context of medical teams. In this study we develop metrics of team performance tailored to the assessment of medical teams and present a qualitative analysis of team performance in successful and unsuccessful teams during the process of simulated cardiac resuscitation. Qualitative studies of this nature support in-depth analysis of small numbers of subjects and new phenomenon [7, 8] during emergency medical team performance.

Methods

Data Sample

We studied a scenario based training design for Advanced Cardiac Life Support teams in the simulation of ventricular fibrillation/tachycardia (VF/VT). A pilot video of an expert team in a simulation of pulseless tachycardia was obtained from one of the hospitals in Phoenix. This data was used to develop a coding framework to assess team work and a checklist to assess adherence to protocol. We then selected two team training videos from another hospital in phoenix to study adherence to protocol and teamwork using the developed measurement tools. These teams were selected based on their outcome in the training event, one showing a good outcome (Team A) and the other displaying bad outcome (Team B). Outcome in this context is defined as the ability of team members to stabilize the patient’s condition after a cardiac arrest.

Development of Measurement tools

We developed measurement tools in order to capture two dimensions of team performance: (1) Team work, a measure of the extent to which each team exhibits behaviors considered to reflect effective team performance and (2) Task work, a measure of the accuracy with which the team adheres to the ACLS guideline.

Team work Coding System

An iterative approach was used in the development of our coding system. Version1: A first version of the coding framework was developed using both referential and empirical strategies for category development. The referential strategy utilized existing research findings or theoretical arguments to justify, through corroboration, the use of particular categories [9]. The empirical approach [9] to verification was accomplished by identifying themes in the pilot data and examining the coverage and distinctiveness reflected by the categories. To abstract themes from the video we first transcribed all the interactions. Then we studied each statement to identify differences, similarities and transition of meaning from the preceding or following statements. Two coders coded on the pilot video using the coding scheme. Version 2 (Table 1): Based on observations in the real world and the results obtained using the first version we developed a second version for refinement. New codes such as team composition, role clarification, positioning of team members, establishing mutual support were added based on field observations in the real world. Names of categories such as those under situation awareness were revised as the researchers became familiar with the setting and the phenomenon under study. Categories which belong together were grouped together such that there is better organization of the codes identified in the first version. The second version thus comprised of four main constructs to assess performance: Team organization, Situation awareness, Communication and Team leadership. Each construct has subcategories. Some subcategories are further divided to additional categories as indicated by the number of items included in table 1. This was done to capture the range of team behaviors observed during resuscitation. Team organization refers to the way team is structured reflecting team membership, role clarification, positioning, availability of resources, and organization of interventions to function effectively as a team. Situation awareness [10] is the perception and comprehension of all the activities in the environment. Communication [11] refers to verbal exchange of information that relates to the assessment and treatment of the patient. Team Leadership [12] is the ability of the leader to coordinate the activities of other team members and make critical decisions. The emphasis was on identifying behaviors within each of these constructs which affect team performance.

Table 1:

Developed Team work Coding System

I. TEAM ORGANIZATION	III. COMMUNICATION
1. Team Composition (4 items included)	1. Uncertainty
2. Role Clarification (3 items included)	2. Clarification
3. Positioning of team members (3 items included)	3. Confirmation
4. Availability of Resources (2 items included)	4. Request
5. Organization of Interventions (3 items included)	5. Acknowledgment
II. SITUATION AWARENESS	6. Incomplete sentences
1. Identify life threatening condition/Problem recognition	7. Seek suggestions
2. Provide patient status	8. Provide suggestions
3. Status review and summary of the patient	9. Incorrect Exchange of Information
4. Attempts to obtain information about the patient	10. Non-task related statements
5. Conveying task plans for team members benefit	IV. TEAM LEADERSHIP
6. Provide task status	1. Task planning and decision-making (2 items included)
7. Monitoring  (5 items included)	2. Response Sequencing (5 items included)
	3. Establishing Mutual Support (3 items included)

Codes are clearly defined with conditions regarding when it can be applied with examples to elaborate. This was done in an effort to make each code mutually exclusive and ensure that coders could interpret codes unambiguously.

Adherence to Protocol Checklist

The checklist was developed based on three measures. 1. Accuracy with which a procedure was performed. 2. Timeliness of activities in accordance to the protocol and 3. Orderliness of the sequence of activities performed in accordance to the protocol. It comprises of two sections: The first section details the intervention steps for VF/VT scenario. An excerpt of the steps assessed is illustrated under steps in intervention in table 2. The second section includes assessment of skills during the performance of the tasks such as CPR, airway intervention, breathing and circulation assessment, defibrillation and administration of medications [13, 14]. An excerpt of some of the skills assessed during CPR is presented in table 2. Thus a critical feature of this checklist is that it allows raters to not only assess the presence or absence of tasks used in the algorithm but also enables judgments about the quality of task performance.

Table 2 :

An excerpt of items from the two sections comprising the Adherence to Protocol Checklist

Steps in Intervention	Skills Assessment CPR
Call for help Lowering the bed. pull the pillow off the bed Assess for airway CPR When more than one member present ensure the backboard is placed underneath Continue CPR Attach the AED (Automated External Defibrillator Cheek Rhythm If shockable then give one shock Manual biphasic: 120-200J OR Monophasic360J	Minimal interruptions of chest compressions Chest compressions should be interrupted for ventilation (unless an advanced airway is placed), rhythm checks, actual shock delivery CPR is given for 2 mins after the shock and followed by 2 ventilations (30:2)ratio

From a measurement perspective narrowing the focus in terms of what is measured is essential to maintaining reliable observations [15]. Each task is scored in terms of its criticality and this was achieved using expert opinion. We asked two experts to use a 3 point scale ranging from 1 to 3 wherein 3 indicated (“most important”) 2 indicated (“important but not critical”) and 1 indicated (“minor importance”) to indicate their judgment for each item in the checklist. We obtained a simple agreement of 73%, between experts. The disagreements were mostly of degree as one of the raters did not rate any item as being of minor importance. The maximum of the two expert ratings was calculated to identify criticality of each item in the checklist.

Data Recording

Software Development

We developed a windows application to embed the team performance coding system (Figure 1). The tool was developed in Visual C# and .Net Framework 3.5. Each construct in the coding system is placed in separate tabs for easy navigation.

Figure 1:

Software interface to code team work

Each code is represented as a checkbox to record team interactions. Error checking was built-in to ensure selection of one code at a time. The video files used as data were accessed within the application using the functionality of the windows media player. The time stamp for each interaction is recorded when a code is selected and this is based on the time displayed in the video file. This enabled us to capture exactly when a code was recorded during the event. The result is stored as a HTML report showing the team members involved in an interaction, codes assigned, time of occurrence and an automatic calculation of the frequency and percentage of each code during the entire event.

Team work Assessment

Two coders were trained using the coding manual and examples from pilot data to build a shared mental model about each code defined in the tool. Coders independently coded each interaction using the transcriptions and observing the video tape. An interaction was defined as a verbal or non-verbal exchange of information. The teamwork coding system was used to map the interactions between team members to the codes developed for each of the four constructs used to understand team performance. Since the coders had prior access to the interaction transcriptions they could anticipate interaction during video viewing and code recording. This we believe would increase the reliability for coding events. Also it enables a detailed analysis of the aspects of team performance which is otherwise lost with measures requiring a global assessment (an overall rating) of the construct. Cohen’s kappa was used to calculate inter-rater reliability between the coders for teamwork assessed using the teamwork coding system. The two coders had an excellent inter-rater reliability of 0.84 for team A and 0.94 for team B.

Adherence to Protocol Assessment

A checkbox is provided in the checklist for the coders to indicate whether or not the team performed all the tasks indicated.

Data Analysis

The total number of checked items in the first section (steps in intervention) of the adherence to protocol checklist was used to capture the deviations in the sequence of activities during the intervention for VF/VT and implications of these deviations in terms of task criticality. The total number of checked items in the second section (skill assessment) was used to understand competency based measures such as timeliness and accuracy with which the tasks were performed within the team. Frequency of each code using the team work coding system was used to identify the different aspects of team performance observed in the two teams. We also analyzed the pattern of interaction in the team to gain further insights into team work.

Results and Discussion

Our results using the task checklist indicate that outcome was affected by the number of critical tasks performed by the team but was unaffected by deviation from the sequence of activities recommended in the ACLS protocol. Team A (Successful Team) performed greater number of tasks i.e. 26 of 36 considered critical in the task checklist than Team B (Unsuccessful Team) who completed 22 of the 36 tasks. Results also indicate that Team A deviated more from the sequence of activities in the protocol than Team B. The successful team administered medications such as epinephrine very early in the intervention even before defibrillating the mannequin upon diagnosis. They placed greater emphasis on rhythm and pulse checks which occurred more frequently than what was prescribed in the protocol. However they had minimal interruptions in chest compressions after procedures such as defibrillations and rhythm checks and accurately identified the arrhythmia. Team B on the other hand adhered to the protocol judiciously until an error was made in the critical task of identifying arrhythmia, resulting in a wrong medication being considered for the intervention. They performed relatively poorly with respect to the accuracy and timeliness with which a procedure was performed. They had difficulty identifying and reporting the pulse. They also failed to adhere to the prescribed recommendation for pulse check to take less than 10secs which resulted in their interrupting chest compressions for a longer period of time. The results thus show that though team A deviated from the sequence of activities in the protocol, they exhibited greater skill in the intervention process and performed greater number of critical tasks than Team B.

Results obtained from the team work coding system indicate that interactions in both teams mainly comprised of attempts to obtain patient information (AO-PI), providing patient status (PPS), provide task status (PTS), reminders (R), clarifications (CL), confirmations (CO), non-leader providing suggestions (NL-PS) for intervention, and leader assigning tasks to other members of the team (L-AT) The most frequent and strikingly observed behavior in Team A as seen in figure 2 comprised of NL-PS for intervention which are mainly statements of priority and task assignment made by members of the team other than the leader. This indicates that team members other than the leader actively participated in providing suggestion regarding how to proceed with the intervention. The most frequently observed behavior in Team B (figure 2) comprised of CO which were statements of verification made by team members mainly to attempts to obtain information about the patient, clarifications, task checks, uncertainty, suggestions sought, time tracking. Qualitative analysis of data indicates that these are statements of confirmation made mainly to the leader.

Figure 2:

Frequency of team behaviors in the successful and the unsuccessful teams.

Key: Attempts to Obtain Patient Information (AO-PI), Providing Patient Status (PPS), Provide Task Status (PTS), Reminders(R), Clarifications (CL), Confirmations (CO), Non-leader Providing Suggestions (NL-PS) for intervention, Leader assigning tasks to other members of the team (L-AT)

Results thus suggest that in spite of deviations in the protocol a flexible approach to leadership may be more effective than the traditional approach used by unsuccessful team in which leadership was vested in a single individual. The unsuccessful team complied with the assigned tasks and failed to detect error in the identification of rhythm by the leader. Thus integrating individual team member knowledge into collective knowledge was found to be critical in maintaining situation awareness about the patient condition and identifying goals in stabilizing the patient.

Limitations

A limitation of our study was that we were unable to obtain videos providing intervention for a single diagnosis of VF/ VT. One of our videos was a combination of two scenarios comprising of ventricular tachycardia and profound bradycardia. Hence we assessed adherence to protocol for the former diagnosis until the rhythm changed to the latter. This study also includes a small number of subjects only. While the labor-intensive nature of exploratory qualitative analysis precludes large sample sizes, the trends identified can serve as focal points for future studies with larger subject pools.

Conclusion and Future Work

We developed measurement tools to understand and assess team performance during cardiac resuscitation. Our results suggest that outcome was affected by the number of critical tasks performed however adherence to the exact sequence of the ACLS protocol was not characteristic of the successful team. In addition flexible leadership which encouraged contributions and suggestions served to establish greater situation awareness and maintain a two way flow of communication in the successful team. Further work will assess the extent to which these results generalize providing an empirical basis for the development of team training interventions customized for critical care.