Do technical skills correlate with non-technical skills in crisis resource management: a simulation study

Abstract

Background

Both technical skills (TS) and non-technical skills (NTS) are key to ensuring patient safety in acute care practice and effective crisis management. These skills are often taught and assessed separately. We hypothesized that TS and NTS are not independent of each other, and we aimed to evaluate the relationship between TS and NTS during a simulated intraoperative crisis scenario.

Methods

This study was a retrospective analysis of performances from a previously published work. After institutional ethics approval, 50 anaesthesiology residents managed a simulated crisis scenario of an intraoperative cardiac arrest secondary to a malignant arrhythmia. We used a modified Delphi approach to design a TS checklist, specific for the management of a malignant arrhythmia requiring defibrillation. All scenarios were recorded. Each performance was analysed by four independent experts. For each performance, two experts independently rated the technical performance using the TS checklist, and two other experts independently rated NTS using the Anaesthetists' Non-Technical Skills score.

Results

TS and NTS were significantly correlated to each other (r=0.45, P<0.05).

Conclusions

During a simulated 5 min resuscitation requiring crisis resource management, our results indicate that TS and NTS are related to one another. This research provides the basis for future studies evaluating the nature of this relationship, the influence of NTS training on the performance of TS, and to determine whether NTS are generic and transferrable between crises that require different TS.

Editor's key points.

• There is growing awareness of the importance of non-technical skills (NTS) in clinical practice.

• The authors correlated different elements of NTS with the technical skills (TS) in a simulated scenario.

• A significant correlation was found between the two.

• Importantly, the study highlights the possibility that training in NTS may enhance the performance of TS.

Crisis resource management (CRM) requires two sets of skills: both technical skills (TS) and non-technical skills (NTS), or behavioural performance. TS have been defined as the ‘adequacy of the actions taken from a medical and technical perspective’, and NTS have been defined as ‘decision-making and team interaction processes used during the team's management of a situation’.¹ Both attributes are essential for patient safety in acute care settings, including intensive care, emergency, and operative environments.

High-fidelity simulation is commonly used in healthcare training and increasingly suggested for the assessment of competence of healthcare providers.^2,3 TS and NTS are generally taught and assessed as separate attributes of performance.^4–7 When assessing practice, separate independent scales are often used to measure TS and NTS. Task-specific checklists are commonly used for the assessment of TS^8–11. Rating tools such as the Anaesthetists' Non-Technical Skills (ANTS) scale or the Ottawa global rating scale have been validated for the assessment of behavioural performance.^12–14 Most research focuses on one of these aspects of performance independent of the other.^15–18

To date, some investigators have studied the performance of both TS and NTS in elective situations to determine a possible influence of one on the other; only a limited number have investigated both of these skills in crises situations.^6,17,19–21 Powers and colleagues¹⁷ studied surgeons during crises situations, but did not directly analyse the relationship between TS and NTS. Moorthy and colleagues²⁰ found a low correlation between TS and NTS for surgeons, but assessment tools included numerous skills not directly related to crisis management, such as gloving and briefing.

There is a lack of fundamental understanding as to whether good TS performance is accompanied by good NTS performance, or whether training of NTS has an impact on the performance of TS during a crisis. Understanding the relationship between these two attributes in such situations could have broad implications for clinical performance training, assessment, and research.

This study aims to determine the relationship between TS and NTS during a simulated intraoperative crisis scenario. We hypothesize that TS and NTS are correlated and are not independent of each other during crisis situations.

Methods

Simulation setting and scenario

The present study used pre-test videos from a previous study conducted by our group and reported elsewhere.²² We obtained additional and specific ethics approval (St Michael's Hospital, Toronto, Ontario, Canada) for a retrospective analysis of the videos allowing us to address the new research question. Pre-test videos were used to avoid any influence of the intervention from our previous study on the results of this study. The videotaped performances included anaesthesiology residents in postgraduate years 2 through 5, responding to a crisis situation. Subjects were excluded if they had not participated in an Advanced Cardiac Life Support certification in the last 5 yr. No resident performances were excluded from the study.

After an orientation with the Laerdal SimMan^® high-fidelity simulator mannequin and monitors (Laerdal Medical Canada Ltd, Toronto, Canada), the Datex^® anaesthesia machine (Datex Corporation, St Laurent, Quebec, Canada), and the mock operating theatre environment, subjects completed a questionnaire collecting patient characteristic data and quantifying previous simulator experience. Participants were then presented a standardized intraoperative cardiac arrest secondary to a malignant arrhythmia: either ventricular fibrillation or pulseless ventricular tachycardia. The two scenarios were piloted to ensure a similar level of difficulty. Each scenario lasted exactly 5 min, regardless of management. We chose short cardiac arrest scenarios because these types of cases can be considered as a critical situation archetype. Many standardized procedures and actions are required, and all ANTS can be assessed in a short period of time. To prevent any sequencing effect of the previous study, the order of presentation of the two scenarios was randomized for each participant; therefore, both scenarios were equally administered as the pre-test.

An actor playing the scripted role of the circulating nurse called the subject in for help once a patient's monitor began alarming after the attending anaesthesiologist had left the operating theatre for another emergency. The only available information for managing the arrest was a patient chart indicating that the patient was an elderly patient with known stable angina who had had an uneventful induction of general anaesthesia for knee arthroplasty. A second actor was also present, playing the scripted role of the surgeon.

Data collection

All simulation sessions were video-recorded. Four independent assessors with expertise in simulation, CRM principles, and clinical practice were recruited. For each performance, two assessors independently rated the TS performance, while the other two independently rated NTS performance.

The TS checklist was designed using iterative modified Delphi approach with 10 experts in advanced cardiac life support management.²³ Like other researchers, we used a judgement quantification process to avoid chance agreement among the 10 experts, requiring 80% agreement on all items. Consensus on all items of the checklists and their sequence was reached after three rounds of the Delphi process (Appendix 1).

The ANTS system is a valid and reliable behavioural marker system that assesses ANTS.²⁴ ANTS has four main skill categories: situation awareness, team working, decision-making, and task management. Each category is further subdivided into a number of elements, and for each element, a number of behavioural descriptors for good and poor performance are described.

The two raters assigned to assess NTS were trained by reviewing the background ANTS literature, which included the User Manual.²⁵ They also practiced the application of ANTS scoring to banked videotaped performances of simulated crises similar to the scenarios used in this study. The other two raters were assigned to assess TS, and rehearsed scoring using the TS checklist on the same bank of sample videos with similar simulated crises scenarios. Training was complete when paired raters were both familiar with, and in agreement with, the application of the rating scales.

The checklist score ranged from a minimum of 0 to a maximum of 15. Each item of the TS checklist had to be performed adequately, in appropriate sequence, and in a commonly accepted timeframe in order to be considered correctly performed. For instance, only when the correct energy level was used, the item ‘first shock given’ was considered to be performed correctly. The ANTS was scored at the level of the four main categories, with 4 as the highest score and 1 as the lowest possible score. Half marks were allowed to functionally represent a seven-point Likert scale (Appendix 2).

Outcome measurements

Our primary outcome was the measurement of the relationship between overall ANTS score and TS performance. Our secondary outcomes were (i) the measurement of the relationship between ANTS categories and TS performance and (ii) the measurement of the relationship within PGY levels of the ANTS categories and TS performance.

Statistical analysis

Statistical analysis of data was performed using SPSS 16.0 software (Chicago, IL, USA). The inter-rater reliability was measured using an intra-class correlation coefficient (ICC) for both sets of expert raters.²⁶

Spearman's rank correlation coefficient was used to measure the relationship between ANTS scores and TS checklist scores.

For all tests, a P-value of <0.05 was considered significant (two-sided).

Results

Characteristics

Fifty participants completed the study (anaesthesia residents in postgraduate years 2–5 from the University of Toronto). Table 1 summarizes patient characteristics.

Table 1.

Patient characteristics (n=50)

Gender (M/F)	25/25
No. of postgraduate training year (postgraduate training year 2/3/4/5)	13/14/14/9
No. of previous simulation trainings as trainee (range)	7 (0–10)
No. of previous simulation trainings as instructor (range)	0 (0–3)

Inter-rater reliability

Inter-rater reliability was excellent for the TS checklist (ICC=0.92, P<0.001), and good for the total ANTS score level (ICC=0.76, P<0.001) (Fig. 1).

Fig 1.

TS and NTS scores.

Relationship between TS and NTS

Technical performance, as measured by the TS checklist score, and NTS, as measured by the total ANTS score, reached a correlation of 0.45 (P<0.05). The relationship between ANTS categories and the TS checklist score had statistically significant correlations, with r ranging from 0.31 to 0.45 (Table 2).

Table 2.

Spearman's correlation between ANTS subgroups and checklist scores. **P<0.01; *P<0.05

	Situation awareness	Team working	Decision-making	Task management	ANTStotal
Team working	0.61**
Decision-making	0.68**	0.69**
Task management	0.64**	0.57**	0.72**
Checklist total	0.31*	0.31*	0.45**	0.31*	0.45**

Discussion

The results of this study suggest that TS and NTS performance during crisis management in an intraoperative simulated scenario are related. Specifically, residents who demonstrated good performance for one domain (TS or NTS) tended to show similar levels of performance for the other. The opposite was also true, poor performance for one domain (TS or NTS) was often accompanied by poor performance for the other. From our results, performance in one domain explains as much as 23% of the variance of the other.

Our study supports findings of other research. McCulloch and colleagues¹⁹ investigated the influence of NTS on surgical outcomes and technical performance. Their results showed an improvement of TS performance through NTS training by reducing procedural errors. Most notably, improving situation awareness led to a reduced procedural error rate. Another study, involving teams during elective laparoscopic cholecystectomies, demonstrated that technical errors were negatively correlated with the surgeons' situational awareness scores.²¹ Similar to our findings, both studies demonstrate a relationship between TS and NTS. Other studies have assessed TS and NTS during simulated surgical procedures, but did not investigate the relationship between the different types of skills.^6,9,17,20 While previous research used elective and non-urgent surgical situations, or focused on the performance of the surgeon, our study adds to the previous literature by specifically exploring the relationship between TS and NTS during crisis situations in anaesthesia.

Cognitive load theory may provide an explanation for the relationship observed between TS and NTS. Sweller²⁷ describes cognitive load theory as a model of human cognitive architecture made up of a limited capacity working memory system and unlimited long-term memory. This theory has been developed substantially since its first delineation and has since been applied to the field of medical education.^28–30 A recent review by van Merriënboer and Sweller³⁰ provides an overview of design and strategies related to cognitive load theory in healthcare professional education. While long-term memory has unlimited capacity, working memory can only process a limited amount of novel information at any one time.^30,31 Strategies are often implemented to effectively accommodate the limited capacity system of working memory. The development of expertise includes the use of schemas, developed from learned previous education, experiences, and/or tasks. Ideally, these schemas are developed in practice, and then serve as effort reduction strategies aimed at aligning task demands with working memory capacities. This may then reduce cognitive load and create ‘space’ for additional cognitive demands.

The case used in this study (intraoperative cardiac arrest) is a volatile situation with high cognitive load due to the number of cognitive processes that need to take place, the number of behaviours that need to be demonstrated, the challenges of multitasking, and the constraint of time. Pre-existing automated schemas (or expertise) in one domain (i.e. TS) may have reduced individual cognitive load, thereby freeing cognitive resources for the other domain (i.e. NTS). This could explain why participants with good TS demonstrated better non-technical performance and vice versa.

Access to long-term memory schemas developed from earlier experiences lowers cognitive load for working memory, and therefore allows for greater mental resources for new incoming information. The working memory of a resident can be affected by the intrinsic nature of the task (e.g. intraoperative cardiac arrest). The number of task elements (i.e. incoming information) must be coordinated in working memory to result in a successful performance. Therefore, a resident who managed the scenario in possession of good TS in advanced cardiac life support (i.e. knowledge of course of action, handling of defibrillator, options of medication) may have required fewer mental resources, freeing some of the working memory for non-technical elements. The quantity of individual ‘chunks’ residents can simultaneously manage may depend on the individual level of expertise for a particular problem. This is consistent with the literature, which suggests expertise may have an effect on cognitive load due to an improved ability to retrieve information from long-term memory more rapidly.³²

This study has several limitations. First, the skills were assessed during a specific anaesthesia cardiac arrest scenario. Although it has been shown that NTS seem to be transferable between acute crisis management scenarios, this might not be the case for TS, and therefore, the results may not be generalizable to other populations or situations.³³ Despite the well-structured and algorithm-driven scenario used in this study, it is possible that not all NTS can be assessed during a short scenario.

Secondly, the checklist might not have been able to purely capture TS, and instead, might have measured behavioural aspects. We have tried to limit any bias by recruiting independent raters for rating TS and NTS: the two raters who used the ANTS were different from the two who had used the TS checklist. However, the ‘call for help’ item is present in both the Checklist and in the ANTS scale, therefore, potentially creating a bias in the results.

Finally, the nature of the relationship between TS and NTS was beyond the scope of this study. The aim of the study was to explore the broad link between performance entities expressed by TS and NTS, using the correlation between both. Therefore, further studies investigating the direction of association and nature of the relationship between TS and NTS are needed.

Conclusion

This study showed that TS and NTS are associated and are not independent from each other during intraoperative crisis management. The nature of this relationship has yet to be addressed. This research provides basis for future studies evaluating the influence of NTS training on the performance of TS, and whether NTS are generic and transferrable between crises that require different TS.

Declaration of interest

None declared.

Funding

This work was supported, in part, by a research grant from the Network of Excellence in Simulation for Clinical Teaching & Learning, Toronto, Canada. The University of Ottawa Author Fund supported the Open Access publishing.

Appendix 1: Technical checklist

Technical task	Yes=1; no=0
Call for help
Pulse checked
CPR performed
100% oxygen (ventilator adjustment or Ambu bag with O2)
Pads put on
Checks rhythm (visually—i.e. look at or verbalizes the rhythm)
First shock given
CPR is resumed immediately (without any form of delay—i.e. without rhythm check and/or other task or order)
Checks rhythm before second shock
Continuous CPR while charging
Second shock given
CPR is resumed immediately (without any form of delay—i.e. without rhythm check and/or other task or order)
Epinephrine 1 mg or vasopressin 40 units i.v. is given (correct dose requested)
Considers amiodarone 300 mg i.v.
Looks at chart for or speaks out a possible cause

Appendix 2: ANTS scale

Subscores	Elements
Task management	Planning and preparing
	Prioritizing
	Providing and maintaining standards
	Identifying and utilizing resources
Team working	Coordinating activities with team
	Exchanging information
	Using authority and assertiveness
	Assessing capabilities
	Supporting others
Situation awareness	Gathering information
	Recognizing and understanding
	Anticipating
Decision-making	Identifying options
	Balancing risks and selecting options
	Reevaluating
Rating options	Descriptor
4—Good	Performance was of a consistently high standard, enhancing patient safety; it could be used as a positive example for others
3—Acceptable	Performance was of a satisfactory standard but could be improved
2—Marginal	Performance indicated cause for concern, considerable improvement is needed
1—Poor	Performance endangered or potentially endangered patient safety, serious remediation is required