Measuring non-technical skills of anaesthesiologists in the operating room: a systematic review of assessment tools and their measurement properties

Abstract

Background

Non-technical skills, such as communication or leadership, are integral to clinical competence in anaesthesia. There is a need for valid and reliable tools to measure anaesthetists' non-technical performance for both initial and continuing professional development. This systematic review aims to summarise the measurement properties of existing assessment tools to determine which tool is most robust.

Methods

Embase (via OVID), Medline and Medline in Process (via OVID), and reference lists of included studies and previously published relevant systematic reviews were searched (through August 2017). Quantitative studies investigating the measurement properties of tools used to assess anaesthetists' intraoperative non-technical skills, either in a clinical or simulated environment, were included. Pairs of independent reviewers determined eligibility and extracted data. Risk of bias was assessed using the COSMIN checklist.

Results

The search yielded 978 studies, of which 14 studies describing seven tools met the inclusion criteria. Of these, 12 involved simulated crisis settings only. The measurement properties of the Anaesthetists' Non-Technical Skills (ANTS) tool were most commonly assessed (n=9 studies), with studies of two types of validity (content, concurrent) and two types of reliability (internal consistency, interrater). Most of these studies, however, were at serious risk of bias.

Conclusions

Though there are seven tools for assessing the non-technical skills of anaesthetists, only ANTS has been extensively investigated with regard to its measurement properties. ANTS appears to have acceptable validity and reliability for assessing non-technical skills of anaesthetists in both simulated and clinical settings. Future research should consider additional clinical contexts and types of measurement properties.

Editor's key points.

• •The authors reviewed 14 studies describing 7 tools used to assess anaesthetists' intraoperative non-technical skills.
• •The Anaesthetists' Non-Technical Skills (ANTS) tool was most commonly assessed. Most studies were at risk of bias and featured simulated settings only.
• •Although ANTS appears to be an acceptable assessment tool, more research is needed to determine its properties, validity and reliability across clinical contexts.

Non-technical skills such as communication, leadership, or situation awareness, are integral to clinical competence in anaesthesia.1, 2, 3 Although technical skills are important for effective performance in the operating room (OR), non-technical skills are consistently responsible for a large proportion of intraoperative errors, adverse patient outcomes, and even mortality.4, 5, 6, 7, 8, 9, 10

In recent years, greater accountability of professional performance for patient safety has been required from anaesthesiologists.¹¹ Competency-based medical education has also emerged for initial and continuing professional development,11, 12 leading to more frequent assessment of professional practice. Both US and Canadian medical education organisations identify interpersonal skills, communication skills, leadership, collaboration, situation awareness, and professionalism as core competencies for anaesthesiologists.13, 14 Researchers have investigated the effect of interventions to improve anaesthesia non-technical skills, such as coder readers, or cognitive aid or mental practice and simulation interventions.15, 16, 17 In this context, measuring non-technical performance in the operating room (OR) is paramount to ensuring the provision of safe, high-quality intraoperative care.

In order to comprehensively assess clinical performance of anaesthesiologists for non-technical skills, reliable and valid tools must be available. Although several tools have been developed for this purpose,18, 19, 20 it is currently unclear which assessment tools are the most robust (i.e. are able to effectively measure non-technical skills performance under various conditions according to studied characteristics). A variety of tools are being used in a range of settings, limiting the meaningful conclusions that can be drawn about anaesthesiologists' non-technical skills, their impact on patient outcomes, and their amenability to change. Whereas existing systematic reviews have summarised non-technical skills assessment tools for surgeons,²¹ syntheses of anaesthesiologist-specific assessment tools are absent from the literature.

Through identifying the most robust tool(s) for assessing these skills, the anaesthesia community will be able to standardise future research on evaluation of interventions to improve anaesthesiologists' clinical performance. This will ultimately foster evidence-based education and facilitate accurate, generalisable, and effective performance-tracking mechanisms.

This systematic review aims to summarise and compare the tools used to measure the intraoperative non-technical performance of anaesthesiologists according to their measurement properties.

Methods

The protocol for this review was published on the University of Ottawa's research repository.²² The review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA).²³

Eligibility criteria

Studies were included if they examined the measurement properties (i.e. validity and reliability) of tools specifically intended to assess the non-technical skills of anaesthesiologists (either trainee or graduated). Non-technical skills included were specified by the Royal College of Physicians and Surgeons of Canada and the US Accreditation Council for Graduate Medical Education.13, 14 These included interpersonal and communication skills, leadership, collaboration, situation awareness, and professionalism. Tools for assessing anaesthesia assistants, nurse anaesthetists, and whole interprofessional teams were not included given specific tasks and working conditions differ between professions, which can influence non-technical skills. Although there are certainly generic non-technical skills ‘which are applicable to a wide range of dynamic working environments, the emphasis and behavioural markers in a behavioural rating system will vary depending on the role under investigation.’²⁴ Hence, specific non-technical skills assessment tools have been developed for each profession.25, 26, 27 Tools with technical skills items were also excluded. Tools could be evaluated within a clinical or simulation intraoperative environment.

Our focus of interest for this systematic review was the intraoperative period, defined as the time from when the patient was physically in the operating room or anaesthetics room, where the anaesthesiologist performed procedures or administers medication to the patient, until the time the patient left the OR. This excluded both the pre- and postoperative periods or remote out-of-OR settings (e.g. recovery room, endoscopy suite, interventional radiology). Studies were excluded if measurement assessment was not the primary outcome or if they evaluated anaesthesiologists' performance but not the assessment tool itself. Specifically, studies could be of any design but had to include a quantitative analysis of measurement properties or qualitative assessment of forms of validity (e.g. Delphi consensus for content validity). Tools also had to be developed for objective assessment of skills rather than subjective (i.e. self-reported) assessment because evidence shows that healthcare professionals self-assess poorly.²⁸

Search strategy and information sources

Literature searches were conducted by an experienced librarian (AD) collaborating closely with the team of investigators. The following databases were searched from inception to January 18, 2017: Medline and Medline in Process (via OVID), PsycINFO, CINAHL, Embase (via OVID), and ERIC (see Appendix 1 for search strategies). The Medline search strategy was peer-reviewed by a second information specialist using the PRESS tool.²⁹ Adjustments were made to the search for each database to optimise search results. Language restrictions were not imposed. Studies published in languages other than English and French were translated using Google Translate.³⁰ We also searched reference lists of previously published systematic reviews and of included articles for additional relevant references.

Study selection

Two independent reviewers (LM, HL) screened titles and abstracts for eligibility, followed by the full-texts of articles identified as ‘included’ or ‘unclear’ after consensus (see Appendix 2 for screening tools). Disagreements at each level of screening were resolved by consensus discussion or assistance from another reviewer (NE) if needed.

Data items and abstraction

Data extraction was conducted by one reviewer (HL) using an electronic data collection form (see Appendix 3) for all included articles. Extracted information was verified by a second reviewer (LM). A group of anaesthetists reviewed the final list of included tools to determine accuracy and completeness. The data extraction form collected general article information (e.g. year and study location), characteristics of learners (e.g. trainee status), tool design (e.g. name and number of items), and measurement outcomes (e.g. properties assessed and validation values).

Study quality

The Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) Risk of Bias Checklist³¹ was used by reviewers (LM, HL, NE) in duplicate to assess the methodological quality of included studies, with disagreements resolved through consensus or a third reviewer as required. The COSMIN Risk of Bias checklist assesses the methodological quality of studies on measurement properties. The checklist uses a 4-point rating system applied to each study. If one study examines multiple measurement properties, ‘each assessment of a measurement property is considered to be a separate study.’³² The overall rating for each property is determined using ‘the worst score counts’ principle (i.e. taking the lowest rating).³²

Synthesis

A narrative summary of the types of reliability and validity, measurement coefficients, validation context (i.e. simulation or clinical), and risk of bias was completed for each included assessment tool.

Results

Study selection

The literature search yielded 978 publications. After removal of duplicates, 716 studies proceeded to title and abstract screening, of which 698 were excluded. Full-text review resulted in exclusion of another 4 articles, leaving a final total of 14 included articles in this systematic review (Fig. 1).

Fig 1.

PRISMA flow diagram. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

Study characteristics and synthesis

Seven tools for assessing anaesthesiologists' non-technical skills were identified (Table 1), across 14 included studies measuring the measurement properties of these tools. All but one study¹⁹ were conducted in high-income countries (summarised in Table 2; details in Appendix 4). The most common countries involved were the United States [number of studies, n=4 (29%)] and Canada [n=3 (21%)]. Non-technical skills assessed were relatively common across tools and included communication, leadership, situation awareness, decision-making, task management, team-working, resource utilisation, coordination, and interpersonal skills (Table 1). Reliability and validity of the tools were assessed most often in crisis resource management situations [n=9 (64%)], involving scenarios such as emergency Caesarean delivery and difficult airway management (Table 2). Studies primarily involved a simulation setting [n=12 (86%)] and anaesthesia trainees only [n=9 (64%)].

Table 1.

Anaesthesiologists' non-technical assessment tools

Assessment tool name	Validation studies (first author, year)	Non-technical skills assessed	Scoring system	Basis of rating system
Anaesthetists' Non-technical Skills (ANTS)	Balki, 2014 Cole, 2017 Fletcher, 2003 Gjeraa, 2015 Jepsen, 2016 Jirativanont, 2017 Morgan, 2011 Mudumbai, 2012 Yee, 2005	Task management Team working Situation awareness Decision making	1 (poor) to 4 (good)	Observed behaviour as it relates to patient safety
Ottawa Global Rating Scale (GRS)	Jirativanont, 2017	Leadership Problem solving Situational awareness Resource utilisation Communication	1 (novice) to 7 (superior)	Quality of behaviour as it relates to crisis resource management
Structured coding framework	Weller, 2014	Communication Decision-making	Number of times event occurred	Frequency of behaviour
Structured rating form	Weller, 2008	Communication Teamwork	1 (fail) to 5 (outstanding)	Medical management of the case
Unnamed tool40	Blum, 2014	Communication Leadership Coordination Situation awareness Interpersonal skills	1 (low) to 7 (high)	Ability to manage patient's conditions
BARS tool	Watkins, 2017	Communication Situation awareness Decision-making Task management Teamwork	1 (poor) to 9 (excellent)	Frequency and consistency of behaviour
Workplace-based assessment tool	Crossingham, 2012	Communication Situation awareness Decision-making Teamwork	1 (poor) to 4 (good/excellent)	Behavioural anchors related to performance

Table 2.

Summary of studies assessing the measurement properties of non-technical skills for anaesthesiologists (n=14). *One study included both simulated and clinical settings. Nb. Full study details available in Appendix 4

Characteristic		n studies	Tool(s) studied
Country	Canada	3	•ANTS
	Denmark	2	•ANTS
	New Zealand	2	•Structured coding framework •Structured rating form
	Thailand	1	•ANTS •Ottawa GRS
	UK	2	•ANTS •Workplace-based assessment tool
	USA	4	•ANTS •Unnamed tool •BARS tool
Setting*	Clinical	3	•ANTS •Structured coding framework •Workplace-based assessment tool
	Simulation	12	•ANTS •BARS tool •Structured coding framework •Structured rating form •Unnamed tool40
Team	Staff only	4	•ANTS •Ottawa GRS •Structured coding framework
	Trainees only	9	•ANTS •BARS tool •Structured rating form •Unnamed tool40 •Workplace-based assessment tool
	Staff and trainees	1	•ANTS
Clinical scenario	Anaesthesia emergency in unnamed context (e.g. difficult airway management)	4	•ANTS •BARS tool •Structured rating form
	Obstetrics	2	•ANTS •Ottawa GRS
	Intraoperative period (various or unnamed types of surgeries)	7	•ANTS •Structured coding framework •Unnamed tool40 •Workplace-based assessment tool
	Not reported	1	•ANTS

Measurement properties of assessment tools

Details of measurement evidence for each tool are provided in Appendix 5. Properties are narratively summarised below (Table 3).

Table 3.

Measurement properties examined for tools assessing anaesthesiologists' non-technical skills. Nb. Full study details available in Appendix 5

Measurement property examined	Assessment tools
	Anaesthetists' non-technical skills (ANTS)	Ottawa Global Rating Scale (GRS)	Structured coding framework	Structured rating form	Unnamed tool, Blum 2014	BARS tool	Workplace-based assessment tool
Content validity	✓	✓	✓		✓	✓
Construct validity					✓
Concurrent validity	✓	✓
Convergent validity
Predictive validity
Internal consistency	✓	✓
Interrater reliability	✓	✓		✓	✓	✓	✓
Test–retest reliability

ANTS was the tool with the most studies (n=9) assessing its measurement properties. Properties investigated included: content validity (number of studies, n=2),19, 33 concurrent validity (n=1),¹⁹ internal consistency (n=3),33, 34, 35 and interrater reliability (n=9).19, 33, 34, 35, 36, 37, 38, 39 Investigation of ANTS′ measurement properties primarily involved simulation (n=8) and crisis situations (n=6). Types of surgery involved in the studies included obstetrics/gynaecology (n=2); general, urology and orthopaedic (n=1); general, oncological (n=1), general surgery (n=1), and neurosurgery and vascular surgery (n=1). Three studies did not report the specific clinical scenarios. Like all tools, studies of ANTS were conducted in high-income countries only, with the exception of one study.

Though the Ottawa GRS¹⁹ and an Unnamed Tool from Blum and colleagues⁴⁰ were investigated in only one study each, four and three categories of measurement properties, respectively, were examined. Specifically, content validity, concurrent validity, internal consistency, and interrater reliability were assessed for the Ottawa GRS¹⁹ and content validity, construct validity, and interrater reliability were examined for the Unnamed tool.⁴⁰ The Ottawa GRS study involved a simulated emergency induction whereas the Unnamed tool study involved simulated routine and crisis situations in otolaryngology and urology.

Measurement properties of the BARS tool were investigated by one study,⁴¹ assessing both content validity and interrater reliability. This study involved simulation of hypoxemia, ventricular fibrillation, and supraventricular tachycardia. The Structured Coding Framework study⁴² involved routine and crisis situations in simulated and clinical general surgery and investigated content validity only. Inter-rater reliability was examined for the Structured Rating Form (n=1)⁴³ and the Workplace-based assessment tool (n=1).⁴⁴ The Structured Rating Form study involved a simulated anaesthesia emergency whereas the Workplace-based assessment tool involved routine and crisis situations in clinical obstetrics/gynaecology and orthopaedic surgery.

COSMIN Risk of Bias Assessment

Results from the COSMIN Risk of Bias Assessment are shown in Table 4. Most studies of measurement properties were at ‘very serious’ risk of bias [n=9 (64%)]. Only the studies that assessed the construct validity of the Unnamed tool⁴⁰ and the concurrent validity of ANTS and the Ottawa GRS were found to be at ‘no risk’. For two tools (Structured rating form; Workplace-based assessment tool), risk of bias for inter-rater reliability studies was ‘extremely serious’. For one tool (ANTS), risk of bias was found to be ‘serious’ for content validity and internal consistency.

Table 4.

COSMIN Risk of bias assessment for studies of the measurement properties of anaesthetists' intraoperative non-technical skills assessment tools

Assessment tool	Study of validity or reliability	COSMIN Risk of bias Assessment
		No risk: There are multiple studies of at least adequate quality, or there is one study of very good quality available	Serious risk: there are multiple studies of doubtful quality available, or there is only one study of adequate quality	Very serious risk: There are multiple studies of inadequate quality, or there is only one study of doubtful quality available	Extremely serious: there is only one study of inadequate quality available
Anaesthetists' Non-Technical Skills (ANTS)	Content validity		X
	Concurrent validity	X
	Internal consistency		X
	Inter-rater reliability			X
Ottawa Global Rating Scale (GRS)	Content validity			X
	Concurrent validity	X
	Internal consistency			X
	Inter-rater reliability			X
Structured coding framework	Content validity			X
Structured rating form	Inter-rater reliability				X
Unnamed tool, Blum 2014	Content validity			X
	Construct validity	X
	Inter-rater reliability			X
BARS tool	Content validity			X
	Inter-rater reliability			X
Workplace-based assessment tool	Inter-rater reliability				X

Discussion

This systematic review identified seven tools for assessing anaesthesiologists' non-technical skills. The reliability and validity of these tools were investigated across 14 included studies, which mostly involved crisis resource management situations. The non-technical skills assessed by each tool were relatively similar (e.g. communication, leadership, situation awareness).

It is important to have a robust tool for continued non-technical skill assessment in everyday practice and there appear to be more options for assessment tools in anaesthesia than in surgery45, 46 or nursing.⁴⁷ Clinicians, researchers and educators seeking to evaluate anaesthetists' non-technical skills may use the findings of our systematic review to make an informed decision on what assessment tool to choose. Depending on their specific goal, they may prioritise one type of reliability or validity to another to choose the assessment tool that meet their needs best. For example, ANTS may be a good choice for clinical assessment and feedback, which are most valid and valuable when they occur directly in the workplace, as it is one of the only tools to be investigated in a clinical setting.⁴⁸ However, we need to keep in mind that most studies investigating measurement properties were conducted in a simulated setting or observers were physically present in the OR. Direct observation in the OR has traditionally faced a variety of challenges (e.g. Hawthorne effect, limited number of observations), but is becoming more feasible with the development of new audio–video recording systems using machine learning.⁴⁹ Future studies of ANTS and other tools could benefit from this technology, as simulation and crisis situations were the most common contexts for all tools across all included studies. These systems could also help to alleviate the resource-intensive nature of most assessment tools (e.g. extensive rater training),⁵⁰ which may be a limiting factor in tool selection.

The vast majority of studies that assessed ANTS and the other identified tools were found to be at risk of bias. This is comparable to other studies investigating assessment tools for surgeons' non-technical skills.45, 46 In this review, bias was most often found because of the limited description of the methodology used or the failure to perform statistical tests recommended by the COSMIN Checklist.³² It would be useful for future studies of tools for assessing anaesthesiologists' non-technical skills to follow the COSMIN guidelines.

The measurement properties examined by the included studies were also limited, with most studies looking at content validity, inter-rater reliability, or both. Concurrent validity and internal consistency were examined for only two of the seven tools (ANTS; Ottawa GRS), construct validity was examined for only one tool (Blum), and test–retest reliability was not examined for any tool. Future studies may wish to diversify the measurement properties examined in order to better determine the internal and external validity of assessment tools and their consistency and ability to accurately measure non-technical skills performance. As per this review, ANTS has the greatest number of measurement properties studied (four), and may be the most promising tool to use moving forward.

Though all the tools identified by this review assessed similar non-technical skills, it is noteworthy that none explicitly included communication with the patient. Tools only assessed communication with other OR team members. Yet, communication with the patient is essential for patient comfort, satisfaction, and safety.⁵¹ To promote patient-centred anaesthetic care and improve the patient experience with anaesthesia, it may be critical for anaesthetists to account for the non-technical skills required to interact directly with patients, either before induction or after emergence and when providing neuraxial anaesthesia. Recently, a tool to measure surgeon's non-technical skills from the patient perspective [Patients' Evaluation of Non-technical Skills (PENTS)] was developed.⁵² Future work in anaesthesia may focus on incorporating communication with the patient and other patient-centred skills into existing assessment tools or developing a new tool similar to PENTS and subsequently testing reliability and validity in clinical settings.

This review faces several limitations. First, we were only able to provide a narrative summary of available assessment tools given variability in the validation context of these tools and the types of validity and reliability assessed. Second, most studies included in this review were at high risk of bias, which means any evidence for the reliability and validity of a particular tool should be interpreted with caution.

Conclusion

Although there are seven tools for assessing the non-technical skills of anaesthetists, only ANTS has been extensively investigated with regard to its measurement properties. ANTS appears to have acceptable validity and reliability for assessing non-technical skills of anaesthetists in both simulated and clinical settings. Future research should examine its measurement properties in various clinical contexts and across more types of validity and reliability. Continued study of other assessment tools may also be warranted to establish reliability and validity across a variety of situations.

Authors' contributions

All authors made substantial contribution to conception and design, acquisition of data, or analysis and interpretation of data; drafting the article or revising it critically for important intellectual content; final approval of the version to be published. Agreement to be accountable for all aspects of the work thereby ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Editorial decision: 12 July 2018

Handling editor: J.G. Hardman

Declaration of interests

The authors have no competing interests to declare.

Funding

Dr Boet was supported by the Ottawa Hospital Anaesthesia Alternate Funds Association. DSR software was funded by the Department of Anaesthesiology and Pain Medicine, Ottawa Hospital.

Appendix A. Supplementary data

The following are the supplementary data related to this article: