Assessing teamwork in medical education and practice: Relating behavioural teamwork ratings and clinical performance

Abstract

Background:

Problems with communication and team coordination are frequently linked to adverse events in medicine. However, there is little experimental evidence to support a relationship between observer ratings of teamwork skills and objective measures of clinical performance.

Aim:

Our main objective was to test the hypothesis that observer ratings of team skill will correlate with objective measures of clinical performance.

Methods:

Nine teams of medical students were videotaped performing two types of teamwork tasks: (1) low fidelity classroom-based patient assessment and (2) high fidelity simulated emergent care. Observers used a behaviourally anchored rating scale to rate each individual on skills representative of assertiveness, decision-making, situation assessment, leadership, and communication. A checklist-based measure was used to assess clinical team performance.

Results:

Moderate to high inter-observer correlations and moderate correlations between cases established the validity of a behaviourally anchored team skill rating tool for simulated emergent care. There was moderate to high correlation between observer ratings of team skill and checklist-based measures of team performance for the simulated emergent care cases (r = 0.65, p = 0.06 and r = 0.97, p < 0.0001).

Conclusions:

These results provide prospective evidence of a positive relationship between observer ratings of team skills and clinical team performance in a simulated dynamic health care task.

Introduction

Successful delivery of health care relies on effective team coordination and communication. JCAHO statistics indicate that communication errors are the most frequently cited root cause of sentinel events (Joint Commision 2006). Observation of teams in dynamic environments such as the operating room suggest that as many as 30% of the communications transmitted are sub-standard or contain a particular type of communication failure (Helmreich & Davies 1996; Lingard et al. 2004). Even though physician–nurse communications represent a small percentage (2%) of clinical activities in the ICU, these communications were cited in 37% of error reports (Donchin et al. 1995). Considering these data in light of the statistics on preventable adverse events cited in the 2000 IOM report (Kohn et al. 2000), it is clear that effective training in team coordination and communication has the potential to substantially decrease the rate of medical errors and deaths due to preventable adverse events. As a result, there has been a recent push toward greater education of teamwork skills throughout the health care system, including early medical education (Gaba et al. 2001; Morey et al. 2002; Leonard et al. 2004).

Although there is clear evidence to support the importance of team coordination skills in health care, there are few validated tools to assess whether or not individuals possess these skills. This is partly because it is difficult to identify objective indicators of team performance in clinical work environments. There have been a number of recent publications describing methods for assessing teamwork skills (Fletcher et al. 2003; Healey et al. 2004; Thomas et al. 2004). These methods generally involve the use of observer-based rating scales, which often are specific to the health care task at hand. However, empirical evidence establishing a relationship between observer rating of team skills and clinical team performance measures or patient outcomes is limited (see Baker et al. (2003) for a review). One possible exception to this is a study by Morey et al. (2002). They found that hospital emergency departments that received emergency team coordination training and implemented formal teamwork structures and processes revealed concordant improvements in behaviourally anchored ratings of teamwork and clinical error rate. Establishing a relationship between ratings of team skills and clinical team performance is important for two reasons. First, it is important to empirically show (beyond retrospective qualitative analyses of adverse events) that teamwork skills impact clinical performance, and ultimately, patient outcomes. Second, it is important to establish the validity of using behaviourally anchored team skill rating scales for assessing individuals’ team skills and for assessing the effectiveness of training programs and interventions.

Dickinson & McIntyre (1997) proposed a conceptual framework of behaviourally anchored rating scales (BARS) for the assessment of team skills. This framework proposes development of observation (or event) scales that indicate specific behaviours associated with high or low performance along several dimensions of teamwork. These scales are used to record specific instances of these behaviours as a basis for a final team rating on each teamwork dimension. Morey et al. (2002) used a form of behaviourally anchored rating in their assessment of the MedTeams project. For our project, we chose to evaluate a rating scale that assessed general team coordination skills that may be applied in a wide variety of health care environments. The teamwork dimensions selected for this study were similar to those used by Brannick et al. (1995) for assessing military aircrews and include a subset of those defined for the aircrew observation and evaluation scale (Bowers et al. 1993). These skills are similar to those required of health care teams (Baker et al. 2005). They included: (1) assertiveness, (2) decision-making, (3) situation assessment, (4) leadership, and (5) communication.

Some research supports the reliability of checklist-based measures over observer ratings (Morgan et al. 2001; Han et al. 2006). Research involving simulation-based tasks has established that checklists focused on observable clinical behaviour such as technical skills or clinical decisions result in high inter-rater agreement (Gaba et al. 1998; Morgan et al. 2001; Murray et al. 2004; Adler et al. 2007). Thus, for the purposes of this study, we did not assess inter-rater agreement or otherwise validate the clinical performance checklist measures used, which were based on discrete, easily observed clinical tasks (e.g., were specific tests ordered, were specific drugs given).

Aims

Our primary objective was to test the hypothesis that behaviourally anchored observer ratings of teamwork skills would correlate with objective measures of team performance. As it is common in medical education to implement both educational interventions and assessments using classroom-based tasks, we sought to determine whether ratings of team skills would transfer from a low fidelity classroom-based task to a more dynamic high fidelity emergent patient care task. We also sought to validate behaviourally anchored teamwork rating scales with respect to (1) inter-observer agreement and (2) consistency across two cases within the two different types of health care tasks.

Methods

Following IRB approval, 35 first-year medical students were recruited and consented to participate in two different types of educational tasks on two different occasions. They were assigned to teams of four (or three, as required to due to absences) for each session. Team assignments were quasi-random, based on availability and scheduling.

Eight teams of four and one team of three first performed a patient assessment task in a classroom-based environment. Teams were given written and verbal instructions on the task. For the task, each of the four team members was given different details related to a patient’s presenting signs and symptoms. The team members were asked to share their information (but not to allow others to read their written information) and to: (1) generate a list of the patient’s symptoms, signs, and abnormalities, (2) generate a list of the patient’s problems (requiring diagnostic or management plans or interfering with quality of life), and (3) select one of the patient’s problems, analyse that problem for potential diagnoses, and generate an action plan. Each team performed two of these cases. Case 1 was a 29 year-old man presenting with acute chest pain (possibly acute myocardial infarction (MI) but more likely causes of the chest pain include acute pericarditis, reflux peptic esophagitis, costochondritis, and acute gallbladder disease). Case 2 was a 50 year-old woman presenting with chronic, recurring abdominal pain (pancreatitis). The task definition and cases selected were based on problem-based teaching that is common in medical education (Beck 1981). Students had access to the internet, were allowed 25 minutes to complete each case, and were videotaped.

In part two of the study, 33 of the original 35 participants were assigned to new teams (six teams of four and three teams of three) and performed two high fidelity simulated emergent patient assessment and care tasks. Students were given instructions to work as a team to assess the patient and treat any emergent conditions. A Medical Education Technologies, Inc. Human Patient Simulator was used for these sessions. A practicing anaesthesiologist was present and was the voice of the patient in the two cases. The two cases presented were (1) a 28 year-old female presenting with abdominal discomfort (acute appendicitis), and (2) a 70 year-old female presenting with shortness of breath (acute MI with an absence of chest pain). Cases were selected to require interaction between team members and to be solvable, yet challenging, for first year medical students. Students had access to the internet, were allowed 25 minutes to complete each case, and were videotaped.

Two behavioural scientists were trained to use a behaviourally anchored team skill rating scale and rated each participant in each of the four cases. The rating scale required rating on five dimensions of teamwork: assertiveness, decision-making, leadership, communication, and situation assessment. Although the raters had no formal medical training, both had a firm understanding of each of the clinical cases, related tasks, and the expected responses. The observers made their ratings using a video-based method of evaluating team skill behaviours. Each of the two observers independently watched video recordings of the team cases and documented, by pressing buttons in a video-based coding tool (Sportstec™ StudioCode), occurrences of specific positive and negative behaviours exhibited by each team member associated with each teamwork dimension (generated from behaviours presented in Table 1). Using the counts of these behaviours as guidance, ratings were made on a 5-point labelled scale for each dimension (5 = complete skill, 4 = very much skill, 3 = adequate skill, 2 = some skill, 1 = hardly any skill) (see Table 2). A total teamwork skill score was calculated for each individual by summing each of the five teamwork dimension ratings.

Table 1.

Definition of teamwork dimensions and behaviours.

Appropriate assertiveness

Assertiveness refers to the willingness to make decisions, demonstrate initiative, and maintain one’s position until convinced otherwise by the facts. Appropriate assertiveness also takes into account the adaptability of the individual with respect to the input of others and the willingness to give up a position when one’s basis for assertion is weak.

Behaviours that suggest assertiveness include:

• Confronting ambiguities and conflicts

• Asking questions when uncertain

• Maintaining a position when challenged (and appropriate)

• Making suggestions

• Stating an opinion on decisions, procedures, or strategies

• Adaptable when one’s own position is proved to be weak

Decision-making

Includes identifying possible solutions to problems, evaluating the consequences of each alternative, selecting the best alternative, and gathering information needed prior to arriving at a decision.

Behaviours that suggest decision-making skill include:

• Communicates possible solutions

• Gathers information to evaluate solutions

• Communicates consequences of alternatives

• Cross-checks information sources

• Selects the best alternative

• Development of plans

• Implements the decisions that were made

Situation assessment

Situation assessment refers to the verbalization of information related to the perception of elements in the environment, the comprehension of their meaning in terms of task goals, and the projection of their status in the near future. Situation assessment verbalizations may serve to promote shared situation awareness between team members.

Behaviours that suggest situation awareness include:

• Situation assessment updates in which team members communicate the current state of the system

• Identification of problem situations and recognizing the need for action

• Exchange of information for the prevention of errors

• Noting deviations in SA between team members

• Demonstrated awareness (e.g., via verbal communication) of the ongoing mission status and the overall goal

• Integration of information from multiple sources

• Accurately prioritizing information and actions

Leadership

• Team leadership involves providing direction, structure, and support for other team members. It does not necessarily refer to a single individual with formal authority over others. Team leadership can be shown by several team members.

• Behaviours that suggest leadership skill include:

• Explains to other team members exactly what is needed from them during the task

• Listens to the concerns of other team members

• Provides statements of team direction, strategy, or priorities for the task

• Sets goals for the team and orients the team toward those goals

• Provides feedback to other team members regarding his/her performance

Communication

Involves the exchange of information between two or more team members in the prescribed manner and by using proper terminology. One purpose of communication is to clarify or acknowledge the receipt of information.

Behaviours that suggest communication skill include:

• Verifies information prior to taking an action

• Acknowledges and repeats messages to ensure understanding

• Uses accurate terminology

• Makes concise statements with little extraneous information

• Establishes and uses conventional or standard speech (e.g., acronyms/shortcuts)

• Provides unsolicited responses (gives more detail than was asked, when appropriate)

Table 2.

Sample teamwork dimension rating scale.

Situation assessment
Situation assessment refers to the verbalization of information related to the perception of elements in the environment, the comprehension of their meaning in terms of task goals, and the projection of their status in the near future. Situation assessment verbalizations may serve to promote shared situation awareness between team members.
Rate the two-member team by circling the number which most closely represents the skill presented by the team in the dimension of situation assessment:
Complete skill in situation assessment	5	Team members frequently communicate situation assessment information. Team members immediately identify problem situations and recognize the need for actions. Team members notice differences in situation awareness from the other member and correct each other if needed. Team members actions and communications help to further an awareness of the ongoing mission status and ultimate goal.
Very much skill in situation assessment	4
Adequate skill in situation assessment	3	Team members sometimes communicate situation assessment information. Team members sometimes identify problem situations and recognize the need for actions. Team members sometimes notice differences in situation awareness from the other team member and sometimes correct each other. Team members actions and communications usually help to further an awareness of the ongoing mission status and ultimate goal.
Some skill in situation assessment	2
Hardly any skill in situation assessment	1	Team members rarely communicate situation assessment information. Team members usually miss problem situations and fail to recognize the need for actions. Team members rarely notice differences in situation assessment from the other team member and rarely correct each other. Team members actions and communications do not further an awareness of the ongoing mission status and ultimate goal.

Checklist-based measures of clinical team performance were collected for both task types. Team performance in the classroom-based task was measured by counting the number of correct symptoms, signs, abnormalities, and patient-problems that teams documented. For the simulation task, a checklist of key actions associated with patient assessment, differential diagnosis, and treatment was created for each case (see Table 3). Team performance was measured by a total count of the number of relevant checklist items completed. Checklist items were counted during live performance and later verified from video recordings. For both task types, the clinical team performance measures resulted in a single team-based score for each case.

Table 3.

Performance checklists for simulated emergent cases.

28 year-old female presenting with abdominal discomfort (acute appendicitis)	70 year-old female presenting with shortness of breath (acute MI with an absence of chest pain)
__1. Greet patient	__1. Greet patient
__2. Introduction	__2. Introduction
__3. Airway, breathing, circulation – place oxygen, assess breathing	__3. Airway, breathing, circulation – place oxygen, assess breathing
Routine Monitors	Routine Monitors
__4. Blood pressure	__4. Blood pressure
__5. Temperature	__5. Pulse oximeter
__6. Pulse oximeter	__6. Temperature
Focused history	__7. Pulses
__7. Onset	Focused history
__8. Timing	__8. Onset
__9. Relieving factors	__9. Timing
__10. Meds	__10. Relieving factors
__11. Allergies	__11. Meds
__12. Other co-morbidities	__12. Allergies
__13. Sexual history	__13. History of heart problems
__14. Menstrual cycle	__14. Family history
__15. Eating, vomiting, BMs	__15. Other co-morbidities
__16. Travel	Description of pain/shortness of breath
__17. Alcohol/drugs	__16. Temporal
Description of pain	__17. Provocation
__18. Quality	__18. Palliation
__19. Severity	__19. Associated symptoms
__20. Radiation	__20. Risk factors
__21. Temporal	Formulate differential
__22. Provocation	__21. Life threatening acute coronary
__23. Palliation	__22. MI
Formulate differential	__23. Unstable angina
__24. Appendicitis	__24. Coronary artery disease
__25. Ectopic pregnancy	__25. Stable angina
__26. Urinary tract infection	__26. Pulmonary embolism
__27. Other ––––	__27. COPD
__28. Other ––––	__28. Renal failure
Physical exam	__29. Anaphylactic reaction
__29. Palpation of abdomen	__30. Airway obstruction/tumor
__30. Palpation of left lower quadrant for Rovsing’s sign	__31. Pneumonia
__31. Extension of right hip and/or rotation of right hip	__32. Other ___
__32. Looking for guarding	__33. Other ___
__33. Looking for rebound tenderness	Physical exam
__34. Pulses	__34. R/O chest tenderness
__35. Auscultate (listen) chest (heart, lungs, bowel)	__35. R/O tension pneumothorax – breath sounds/chest cavity sounds
__36. Mouth	__36. Auscultate (listen) chest (heart, lungs, bowel)
__37. Pupils	__37. Airway exam
Order labs/tests	__38. Color/cyanosis
__38. CBC (complete blood count)	__39. Edema
__39. Urinalysis	__40. Mottling
__40. Pelvic culture	__41. Palpate legs (clots/warmth)
__41. bHCG (pregnancy, urine or blood)	__42. Jugular vein
__42. CT scan (preferred – give 2), ultrasound, or plain film	__43. Other ____
__43. Chem 7	Labs & Tests
__44. Other relevant (Liver Panel, blood gas, occult blood)	__44. EKG (electrocardiogram)
Interpret labs	__45. Chest X-ray (CXR)
__45. Appendicitis	__46. Cardiac enzymes (troponin, CK-MB)
Treatment	__47. Arterial blood gas (ABG)
__46. Fluid for hydration	__48. CBC
__47. Narcotic for pain	__49. Chem7
__48. Surgical consult	__50. Glucose finger stick
__49. Antibiotics (preferred cefazolin and metronidazole – 2 for right drugs)	__51. Echocardiogram
	Treatment
	__52. Elevated positioning
	__53. Aspirin
	__54. Sublingual nitroglycerin or nitroglycerin
	__55. Beta Blockers
	__56. Morphine
	__57. Diuretic
	__58. Cardiology consult (separate from catheterization lab/thrombolytic treatment)
	__59. Cardiology consult (for purposes of catheterization lab/thrombolytic treatment)
	__60. Recognizes hypotension response to nitroglycerin
	__61. Treat hypotension response
	__62. ABCs
	__63. Discontinue nitroglycerin
	__64. Support blood pressure
	__65. Tracheal Intubation
	__66. Other relevant drugs
	Negative or potentially detrimental actions (subtract from score)
	__67. Fine needle aspiration to remove fluid from lungs
	__68. Insulin

Inter-rater agreement was assessed for each of the four cases using Pearson correlations between observer ratings. Consistency of individual team skill across the two cases within each task type was assessed with Pearson correlations. Transfer of individual team skill ratings from class-room-based tasks to high fidelity simulation tasks was assessed with Pearson correlations of observer ratings between task types. To test our primary hypothesis that observer ratings of individual team skill will correlate with objective measures of team performance, Pearson correlations were conducted between an average team skill rating for each team (an average of the two observer ratings averaged across the three or four team members on each team) and the total team performance score for each of the four cases.

All participants experienced the same task types and case presentations in the same order. Because we were not comparing an individual’s performance across tasks, but rather assessing the association between two concurrently measured variables, we did not control for possible order or learning effects.

With respect to the selection of individual versus team measures of performance, we chose to conduct individual-based rather than team-based ratings of team skills because an important feature of a validated measure of team skill is that it should be able to distinguish the team skills of an individual working within a team. For the team performance measures, we chose team based measures of performance because clinical task performance ultimately depends on the contributions of the entire team such that it is difficult to identify objective measures of team performance that separate contributions at the individual level. Correlations between individual skill ratings and clinical team performance were conducted based on averages of skill ratings for the individuals involved.

Results

Average teamwork skill ratings and standard deviations for each of the two cases in the two task types is shown in Table 4. In general, individuals were rated higher in teamwork skills in the simulated emergent care task than in the classroom-based patient assessment task.

Table 4.

Average teamwork skill score and relevant standard deviations.

	Average teamwork skill score*	Standard deviation* (of individuals’ skill scores)	Average standard deviation within teams*,†	Average standard deviation between teams*,‡
Classroom-based patient assessment task
Case 1, chest pain	17.10	1.89	1.63	0.73
Case 2, abdominal pain	16.80	2.62	1.84	1.64
High fidelity simulated emergent
care task Case 1, abdominal pain	18.00	2.93	2.31	1.65
Case 2, shortness of breath	18.79	2.75	2.54	1.12

^*All measures are based on the average of both observers’ ratings.

^†Standard deviation of the 3 or 4 individuals in a team, averaged across 9 teams.

^‡Standard deviation of the 9 team’s rating scores (team rating score is the average of the team members’ scores).

There was moderate inter-observer agreement between observers in the classroom task: r = 0.47, p < 0.01 for case 1 and r = 0.58, p < 0.01 for case 2. There was slightly higher agreement in the high fidelity simulation task: r = 0.58, p < 0.01 for case 1 and r = 0.73, p < 0.01 for case 2. With respect to consistency of team skill ratings across cases and task types, there was some evidence of consistency within task type but no evidence of consistency between task types. Observer 1 ratings revealed moderate correlations between the two classroom based cases: r = 0.56, p < 0.001, and between the two simulation cases: r = 0.55, p < 0.001. Observer 2 ratings revealed significant correlations between the two simulation cases: r = 0.45, p < 0.001.

For the classroom task, there was no evidence of a relationship between behaviourally anchored skill ratings and objective team performance measures. Inspection of the data (see Figure 1 and Table 4) reveals that, for the first classroom case, there was very little variability in the averaged team skill ratings between teams. In the high fidelity simulated emergency task, there was evidence of a positive relationship between behaviourally anchored skill ratings and objective performance measures. There was a trend toward moderate correlation between team skill ratings and performance scores in the first high fidelity simulation case: r = 0.65, p = 0.06. There was strong correlation in the second high fidelity simulation case: r = 0.97, p < 0.001 (see Figure 1).

Figure 1.

Correlation between observer ratings of team skill and team performance score in simulated emergency tasks.

Conclusions

These results provide some support for the validity and reliability of observer-based rating of teamwork skill when provided with a structured tool for basing ratings on specific observable teamwork behaviours. The results of the study did not support transfer of team skill ratings in a classroom-based patient assessment task to more dynamic and realistic health care team tasks. Reliable ratings of team skill may require a dynamic interactive context, such as was presented through high fidelity patient simulation in this case. These types of tasks may promote a wider variety of team skill behaviours that observers can use as a basis for rating. In addition, the scale used was focused toward team skill behaviours that are important in dynamic interactive tasks. Different team skill behaviours may be important in other types of team tasks such as the artificial classroom task we presented.

Most importantly, for a simulated emergent patient care task, these results establish a positive quantitative relationship between observer ratings of team skills and a measure of clinical team performance. In one scenario this effect presented as a trend, which may be explained by the relatively small sample size with respect to number of teams and the somewhat homogeneous participant population (all first year medical students). Despite this limitation, the effect was both strong and significant in the second scenario. These results further emphasize the importance of training and supporting effective teamwork behaviours in clinical work environments (Morey et al. 2002, Baker et al. 2005, Leonard et al. 2004). These results provide support beyond retrospective analyses to indicate that enhanced team skills are associated with better clinical performance, which should then lead to better patient outcomes. Specifically, for teams of first year medical students, higher ratings on team skills were associated with more comprehensive and appropriate clinical care of a simulated patient presenting with emergent symptoms of acute MI.

One potential limitation of the study design was the decision to rate individuals’ team skills and the somewhat arbitrary decision to average the ratings scores of individuals to attain a team score that could then be correlated with the team-based clinical performance checklist. Is it reasonable to assume that each individual, independent of their teamwork skills contributes in a similar way toward the overall teamwork skills of the team? How is an individuals’ team skill rating likely to be influenced by other members of the team? Raters indicated that they did not have difficulty assigning specific teamwork behaviours to individuals within the team. For example, it is easily noted which individual, within the team, is calling out specific dynamic data or summarizing information that was received. In addition, the variability in scores between individuals did not suggest that there were students with either very good or very poor teamwork skills that may influence the overall working of the team to a great extent. One means of controlling for the influence of teammates in future work would be to assess individuals in an environment where team members are confederates or ‘standardized’, as in Moorthy et al. (2005). However, for transfer to assessment in realistic work environments, the question remains as to whether or not the teamwork skills of individuals can be (or should be) separated from the skills of the team as a whole.

Recent research in teamwork assessment (Fletcher et al. 2003; Thomas et al. 2004; Moorthy et al. 2005; Malec et al. 2007; Morgan et al. 2007) suggests contradictory results with respect to the validity and reliability of behaviourally anchored rating scales. The results of this effort are similar to the findings of Morgan et al. (2007) and provide limited support for a general rating scale that involves rating of specific teamwork dimensions based on observed behaviours. Additional work, perhaps incorporating techniques such as factor analyses to better identify teamwork skills to be rated, may be required to improve the reliability, validity, and feasibility of these types of rating scales. Another approach is for the rating scale to score directly from behaviours rather than on higher-level constructs believed to be linked to those skills. Malec et al. (2007) present an example of this, applied to team-level ratings, which involves three-point scoring of the frequency or quality of specific teamwork behaviours. In addition, specialty and/or case-specific assessments based on observation of specific behaviours or key clinical or teamwork actions may be necessary (Flin & Maran 2004; Morgan et al. 2007). Future efforts to refine assessment of team skills in health care may benefit from a focus on identifying observable teamwork behaviours that are associated with enhanced clinical performance.

Practice points.

• Effective teamwork is critical to safe clinical care. Researchers have preliminarily validated observer rating scales for assessing team skills.

• This research provides evidence of limited validity of a general behaviourally anchored rating scale to assess team performance in health care tasks.

• This research provides evidence of a positive quantitative relationship between observer ratings of team skills and a checklist-based measure of clinical performance in simulated emergent care tasks.

• The results emphasize the importance of task design and case selection for assessing team skills using these scales.

• Future research may (1) focus on identification of factors in scenario design or clinical practice that elicit critical teamwork behaviours and (2) establish reliable observable team skill behaviours that predict clinical performance.