Abstract
Objectives
To comprehensively synthesise evidence regarding the validity and reliability of the Anaesthetists’ Non-Technical Skills (ANTS) behavioural marker system and its application as a tool for the training and assessment of non-technical skills to improve patient safety.
Design
Systematic review.
Data sources
We employed a citation search strategy. The Scopus and Web of Science databases were searched for articles published from 2002 to May 2022.
Eligibility criteria
English-language publications that applied the ANTS system in a meaningful way, including its use to guide data collection, analysis and reporting.
Data extraction and synthesis
Study screening, data extraction and quality assessment were performed by two independent reviewers. We appraised the quality of included studies using the Joanna Briggs Institute Critical Appraisal Checklists. A framework analysis approach was used to summarise and synthesise the included articles.
Results
54 studies were identified. The ANTS system was applied across a wide variety of study objectives, settings and units of analysis. The methods used in these studies varied and included quantitative (n=42), mixed (n=8) and qualitative (n=4) approaches. Most studies (n=47) used the ANTS system to guide data collection. The most commonly reported reliability statistic was inter-rater reliability (n=35). Validity evidence was reported in 51 (94%) studies. The qualitative application outcomes of the ANTS system provided a reference for the analysis and generation of new theories across disciplines.
Conclusion
Our results suggest that the ANTS system has been used in a wide range of studies. It is an effective tool for assessing non-technical skills. Investigating the methods by which the ANTS system can be evaluated and implemented for training within clinical environments is anticipated to significantly enhance ongoing enhancements in staff performance and patient safety.
PROSPERO registration number
CRD42022297773.
Keywords: ANAESTHETICS, Human resource management, Risk management, Quality in health care, Risk Factors, Systematic Review
STRENGTHS AND LIMITATIONS OF THIS STUDY.
To the best of our knowledge, no previous systematic review has illustrated the validity and reliability of the Anaesthetists’ Non-Technical Skills (ANTS) behavioural marker system and its application as a tool for the training and assessment of non-technical skills to improve patient safety.
We used the Standards for Educational and Psychological Testing framework with various study designs to evaluate and analyse the reliability and validity of the ANTS behavioural marker system.
Non-English-language studies were not searched, which may have led to publication bias.
In addition, our study did not include other research outputs such as unpublished papers, grey literature or conference abstracts.
Introduction
The incidence of adverse medical events is high. It has been estimated that adverse events from unsafe care may be among the top 10 causes of death and disability, resulting in 64 million disability-adjusted life years lost worldwide each year.1 A previous systematic review and meta-analysis found that approximately 1 in 20 patients experience preventable harm in medical care.2 Mazer and Nabhan assessed preventable medical errors as the third leading cause of death in the USA.3 The economic burden caused by adverse events, particularly preventable ones, is substantial.4 Recent evidence suggests that 15% of total hospital expenditures and activities in Organization of Economic Co-operation and Development countries are a direct result of adverse events, with the total cost of injuries in these countries alone amounting to trillions of dollars per year.5 The annual cost associated with medical errors is estimated at US$42 billion worldwide.6
The core causes of adverse surgical events are systems failures.7 Deficiencies in non-technical skills (NTS) at the individual level increased chance of failure and reduced chance of recovery.8 Kirschbaum et al reported that over 25% of operating room communication failures result in procedural errors.9 Engel et al found that team members with limited perioperative information-sharing doubled the risk of surgical complications compared with those frequently sharing information.10 The non-technical performance failures were frequently a contributor to adverse event causation.
Safety is inextricably linked to continued global healthcare improvements, as well as strengthening surgical and anaesthesia care.11 Good health and well-being is one of the pillars of the United Nations Sustainable Development Goals also launched in 2015 (ie, SDG3).12 At least 9 of the 13 SDG3 targets are directly or indirectly addressed by improving anaesthesia and surgical safety standards and practices worldwide.12 13 Modern anaesthetists are part of a complex, dynamic system involving not only the patient, but the surgeon, other coworkers, an array of equipment and a complex infrastructure.14 Surveys have shown that NTS contribute to 80% of adverse events, such as inadequate monitoring, poor communication, and not double-checking machines or drugs.15 Therefore, NTS evaluation and training of anaesthetists are essential to the safety of high-risk anaesthesia.16 Notably, medical education institutions in both the USA and Canada consider NTS, such as situation awareness and interpersonal, communication, leadership and collaboration skills, as core competencies for anaesthetists.17 This has led to the development of training programmes for NTS in anaesthesiology to improve patient safety.
Fletcher et al at the University of Aberdeen (UK) developed and tested the Anaesthetists’ Non-Technical Skills (ANTS) behavioural marker system in 2002, which can be used for training and evaluating anaesthetists’ performance.18 The ANTS behavioural marker system is a framework that includes four skill categories necessary for good anaesthetic practice, including task management, teamwork, situation awareness and decision-making, with 15 elements and examples of good and bad behaviours for each element.19 20 The ANTS system uses a set of 4-point Likert scales to rate observed behaviours in relation to these elements and categories in addition to providing space for open comments.19 20 Researchers have investigated the positive effects of interventions such as cognitive aids, mental practice and simulation interventions on improving ANTS.21 22
The ANTS behavioural marker system has been widely applied as an assessment tool; its use has substantially increased since its publication, but systematic reviews have not been conducted to investigate its development and applications.8 Therefore, we performed a systematic review aimed at synthesising evidence on the development and applications of the ANTS behavioural marker system as a tool for NTS training and assessment to improve patient safety. The specific research objectives of this systematic review were as follows:
Objective 1: to determine the types of studies that have developed and applied the ANTS behavioural marker system.
Objective 2: to determine how the reliability and validity of the ANTS behavioural marker system are assessed.
Objective 3: to determine how the ANTS behavioural marker system has been applied, including the extent of its application.
Objective 4: to determine the contribution of the ANTS behavioural marker system to improve patient safety.
Methods
This systematic review adheres to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) reporting guidelines (online supplemental file 1).23 The protocol for this review was registered in the International Prospective Register of Systematic Reviews (PROSPERO ID: CRD42022297773) and was published in an open-access journal.24
bmjopen-2023-075019supp001.pdf (101.5KB, pdf)
Search strategy
We employed a citation search strategy.25 The cited articles used for our search were the original publications of the ANTS behavioural marker system by Fletcher et al.18–20 Two citation index databases, Web of Science and Scopus, were searched. These databases were chosen because they provide the most comprehensive databases of articles that can be tracked using citations and allow for reference searches.26 Other databases, such as Google Scholar, may provide wider coverage of certain types of publications (international, non-English journals, conference proceedings), but these publication types were excluded from our review.27 Moreover, Web of Science and Scopus yield more consistent and accurate results than other databases that may provide wider coverage (eg, Google Scholar).28 The reference lists of relevant systematic reviews were also screened.
In Scopus, the search string was REF (‘The role of non-technical skills in anaesthesia: a review of current literature’ AND ‘Anaesthetists’ Non-Technical Skills (ANTS): evaluation of a behavioural marker system’ AND ‘Rating non-technical skills: developing a behavioural marker system for use in anaesthesia’). In Web of Science, the search string was TITLE (‘The role of non-technical skills in anaesthesia: a review of current literature’ AND ‘Anaesthetists’ Non-Technical Skills (ANTS): evaluation of a behavioural marker system’ AND ‘Rating non-technical skills: developing a behavioural marker system for use in anaesthesia’). The full texts of all resulting articles were downloaded for review.
Eligibility criteria
We included all English-language empirical studies published in peer-reviewed journals between 2002, when the ANTS system was first published by Fletcher et al,18 and May 2022 which met the following criteria: (1) focused on anaesthetists and other healthcare professions or disciplines; (2) used the ANTS behavioural marker system to guide data collection, measurement, coding, analysis or reporting; (3) reported ANTS-related outcomes (eg, ANTS scores as a tool for NTS training and assessment, completion of the checklist, frequency of adverse events and frequency of errors to improve patient safety).
We excluded protocols, editorials and articles that cited the ANTS but did not report the development and applications of the ANTS behavioural marker system; for example, studies in which the ANTS behavioural marker system was cited in the introduction section to acknowledge the importance of human factors.
Study selection process
Two independent authors (JK and CY) selected the studies. Following exclusion of duplicates, the two authors screened the methods and results to determine whether the full-text manuscripts required further review. We also screened the references of published reviews for additional relevant studies. Titles or abstracts were not initially screened because this review was based on citations of the ANTS behavioural marker system, which was not always mentioned in those sections. Studies identified for full-text screening were screened for evidence of the meaningful use (guiding data collection, measurement, coding, analysis or reporting) of the ANTS system. Disagreements regarding study eligibility were discussed by the three study authors (JK, CY and JH) until a consensus was reached.
Data extraction and analysis
A standardised data extraction form was used by two independent authors (JK and CY) to extract information in duplicate, with disagreements resolved via consensus or by a third author (JH), as needed. We used a framework analysis approach to guide the summary and synthesis of the included articles.29 Data abstraction and analysis were accomplished in the following five phases.
The first step was the familiarisation phase. We reviewed the included studies to familiarise ourselves with the literature. Second, we developed a thematic data extraction framework based on our four research objectives to identify and extract content from the included studies, which served as the code used to identify and extract passages from the articles. These codes comprised the columns in the abstraction matrix. The final study characteristic codes according to the study objectives are presented in table 1. The reliability and validity data were extracted according to the Standards for Educational and Psychological Testing (the Standards) framework applicable to various study designs. Reliability evaluation includes stability, internal consistency and inter-rater reliability. Validity evaluation includes content, response processes and internal structure validity.30 In the indexing and charting phase, two authors (JK and CY) extracted text selections from the included studies into a summary matrix. In the final mapping and interpretation phase, the content from the abstraction matrix was analysed by JK, JH and FZ to develop overarching themes for each code. Themes were discussed among all coauthors until a consensus was reached.
Table 1.
Codes for analysis included in the research objectives
| Research objective | Codes for analysis |
| Objective 1: to determine the types of studies that have developed and applied the ANTS behavioural marker system. | General study characteristics, including:
|
| Objective 2: to determine how the reliability and validity of the ANTS behavioural marker system are assessed. | According to the Standards for Educational and Psychological Testing framework for data extraction, including:
|
| Objective 3: to determine how the ANTS behavioural marker system has been applied, including the extent of its application. | Extent of the ANTS behavioural marker system application, including:
|
| Objective 4: to determine the contribution of the ANTS behavioural marker system to improve patient safety. |
|
ANTS, Anaesthetists’ Non-Technical Skills.
Quality assessment
Two authors (JK and CY) qualitatively assessed all studies using the Joanna Briggs Institute Critical Appraisal Checklists. The overall appraisal of the 54 studies led to the decision to include all in the analysis.
Patient and public involvement
There was no patient and public involvement in this research.
Results
Our search strategy identified 2072 articles. After the removal of 1088 duplicate articles, 984 unique articles remained for the methods and results review. Of these, 523 articles were excluded because they were editorials, study protocols, etc; a further 401 articles were excluded because they did not use the ANTS behavioural marker system to guide data collection, measurement, coding, analysis or reporting; and 1 article was excluded because it was not an empirical study. After excluding five articles that were unavailable, the remaining 54 articles were included in the final analysis. The PRISMA flow diagram of the selection procedure is shown in figure 1.
Figure 1.
Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram. 1ANTS, Anaesthetists’ Non-Technical Skills.
Objective 1: to determine the types of studies that have developed and applied the ANTS behavioural marker system
The characteristics of each study including the research objective, country, setting, scenario, unit of analysis, study design and methods are provided in online supplemental file 2.
bmjopen-2023-075019supp002.pdf (159KB, pdf)
Research objective
The general research objective of most studies (n=47) was to investigate the participants’ performance during different interventions, including two studies exploring the relationship between technical skills (TS) and NTS. Five additional studies aimed to explore NTS in other relevant healthcare populations, including organ procurement in the intensive care unit (ICU),31 nurse anaesthetists,32 ambulance teams,33 low-income country clinicians34 and medical students.35 The remaining two studies measured the usability20 of the ANTS behavioural marker system and of a training test for raters36 using the ANTS system.
Country, setting and scenario
Most studies were conducted in high-income countries, including European and American countries (n=46) and Australia (n=3). Low-income geographical areas included Rwanda (n=3) and Mozambique (n=1). Thailand was the only Asian country included in these studies.
Simulation was the most common setting (n=41). Among them, 38 studies were conducted in simulation centres, focusing on anaesthesia crisis scenarios in the operating room. The other three simulated screen-based delivery room crisis scenarios, including neonatal resuscitation and postpartum haemorrhage. The clinical settings included crisis scenarios in the operating room (n=6), ICU (n=2), emergency department (n=2) and delivery room (n=1). One Canadian study combined simulated and clinical settings of anaesthesia crisis scenarios in the operating room.37 Notably, one study investigated the NTS involved in the surgical and anaesthesiology curriculum in a medical school setting.35
Unit of analysis
The units of analysis were most often the individual (n=40), including anaesthesia residents (n=17), anaesthetists (n=14), other medical specialists (n=4), midwives (n=4) and nurse anaesthetists (n=7). Organisational analysis units (n=13) comprised clinicians of different specialties (n=6) or clinicians and nurses (n=7). The combined individual and organisational units of analysis were used to evaluate the role of organ procurement NTS in the ICU (n=1).31
Study design and methods
Most studies (n=42) employed a quantitative design, including quasi-experimental studies (n=20), randomised controlled trials (n=18), cohort studies (n=3) and case series (n=1). In all quantitative studies, the ANTS behavioural marker system was used as an evaluation tool for quantitative measurements. Eight studies had mixed designs. Four studies used qualitative designs, including participant observation (n=2), ethnography (n=1) and focus group interviews (n=1).
Objective 2: to determine how the reliability and validity of the ANTS behavioural marker system are assessed
We used the Standards to extract and analyse data regarding the reliability and validity data of the ANTS behavioural marker system.30 More detailed reliability and validity data are provided in online supplemental file 3.
bmjopen-2023-075019supp003.pdf (113.1KB, pdf)
Reliability
Reliability was reported in 37 (69%) studies. Inter-rater reliability was the most commonly reported reliability statistic (n=35, 65%). The intraclass correlation coefficient (n=22), Fleiss’ kappa (n=10), Kendall’s w (n=2) and within-group agreement (n=1) were used to comprehensively evaluate the multilevel ANTS behavioural marker system, including total, category or element-level scores. In 26 of 35 (74%) studies, raters were trained to perform professional ANTS scoring as emphasised in the ANTS handbook.38 In total, 24 of 35 (69%) of the raters achieved a substantial agreement, with inter-rater reliability scores greater than 0.61. Moreover, 9 of 35 (26%) of the raters had inter-rater reliability scores greater than 0.8 indicating excellent agreement.
Three studies reported a measure of internal consistency (Cronbach’s alpha). Where Cronbach’s alpha was reported, it exceeded the accepted standard (>0.70) for scales. Stability reliability was statistically analysed in only one study using Pearson’s r coefficient, with results greater than 0.7 among ANTS subscales.39 The strong correlation between individual subscales of the ANTS behavioural marker system indicated a high interdependency of these items for overall improvement.
Validity
In the Standards, validity is a unitary construct in which multiple evidence sources contribute to construct validity. Validity evaluation includes content, response processes and internal structure validity evidence sources.30 In our included studies, no single study reported all four validity evidence sources outlined in the Standards. Validity evidence was reported in 51 (94%) studies; three studies reported no evidence source. Studies were graded according to the reported number of validity measures as level one, two or three (three, two or one measure, respectively).
Studies reporting three sources of validity evidence (level one)
Five studies were graded as level one. All five studies displayed content, response processes and relationships with the validity of other variables.
Studies reporting two sources of validity evidence (level two)
18 studies were graded as level two. The most commonly reported evidence sources were response processes (n=18), followed by content (n=13) and relationships with other variables (n=5).
Studies reporting one source of validity evidence (level three)
The majority (n=28) of the identified studies fell into level three. The most commonly reported evidence sources were content (n=11) and response processes (n=11), followed by relationships with other variables (n=3).
Additional properties
As part of our validity assessment, we paid special attention to how the ANTS behavioural marker system ‘functioned’. All level one (n=5) and level two (n=18) studies and 39% (n=11) of level three studies reported ‘response processes’ validity evidence. Participants’ debriefing was the most common evidence of response processes (n=23), followed by pilot tests (n=9) and 4-point Likert scale adjustments of the ANTS behavioural marker system, such as a 7-point scale (n=4) and an 8-point scale (n=1). All level one (n=5) and most level two (n=13 of 18) studies reported ‘content’ validity evidence on published literature. Moreover, 25% (n=13 of 51) reported ‘relationships with other variables’ validity evidence, with results often closely related to literature hypotheses.
Coherence of the ANTS behavioural marker system regarding terminology and language
In general, the authors offered few reflections on the terminology and construct coherence of the ANTS system. The original study of the ANTS system by Fletcher et al in 2003 to evaluate its usability mentioned that 98% of participants thought that it was meaningful to use category and element labels and that 96% thought that the descriptions of the ANTS system were clear.19 In a study to evaluate the teamwork and communication ability of a role group in the operating room team, questionnaires were collected on the feasibility of the ANTS system.40 For example, ‘Rate your comfort interpreting the categories and elements of each observation tool’ and ‘Rate the amount of effort (time and mental energy) you expended to use each instrument’. The results suggested that it was likely that each rater has a different mental model of ideal teamwork performance by each professional group. Future qualitative research is needed to further explore these mental models and modify existing tools for teamwork accordingly.40
Objective 3: to determine how the ANTS behavioural marker system has been applied, including the extent of its application
Among the 54 articles that met our inclusion criteria, we assessed the extent of the ANTS behavioural marker system application as follows (online supplemental file 2):
Selection and use of the ANTS behavioural marker system categories and elements
In the included articles, 85% (n=46 of 54) assessed the ANTS at both category and element levels, whereas 57% (n=31 of 54) analysed the ANTS at both total and category levels. Only one study assessed and analysed the ANTS system at only one of the element levels (teamwork) which was related to the research hypothesis of their study.41
Application of the ANTS behavioural marker system in the methods
Table 2 reports the applications of the ANTS behavioural marker system (ie, to guide data collection, analysis or reporting) by study designs (quantitative, qualitative or mixed methods). The majority of included studies (n=47, 87%) used the ANTS system to guide data collection only. Two studies used the ANTS system to guide data analysis only, and two studies used the ANTS system to guide data reporting.
Table 2.
Application of the ANTS system by study design
| Study design | Application of the ANTS system | Comment |
| Quantitative (n=42) |
Data collection: 42 (100%) | The system was used to evaluate the performance. |
| Qualitative (n=4) |
Data analysis: 2 (50%) | The system was used as a base to analyse the NTS of the ICU and medical curricula. |
| Data reporting: 1 (25%) | Observations and in-depth interviews to understand practice of ANTS in Rwanda. | |
| Data collection & data reporting: 1 (25%) | The categories of ANTS as starting points for discussions about each category. | |
| Mixed (n=8) |
Data collection: 5 (62.5%) | The system was used to evaluate the performance. |
| Data reporting: 1 (12.5%) | NTS framework was adapted to the critical care air ambulance setting using the ANTS system as the foundation. | |
| Data collection & data analysis: 2 (25%) | The system was used to evaluate the performance and analyse the results. |
ANTS, Anaesthetists’ Non-Technical Skills; ICU, intensive care unit; NTS, non-technical skills.
The way the ANTS behavioural marker system informed the study methods depended on the study design. For quantitative studies, the ANTS system was used to evaluate participants’ NTS performance. For qualitative studies, the ANTS system was used as a base to guide data analysis and reporting through participant observations, focus group interviews and ethnography. In mixed-methods studies, the ANTS system was used with both qualitative and quantitative methods to examine and analyse NTS.
Training of raters and feedback from participants
The ANTS handbook emphasises that raters should conduct calibration and training in the system before using the ANTS system for teaching and evaluation to ensure that they can provide standardised judgement.38 In addition, formative assessment using the ANTS system needs to provide feedback to trainees, including post-assessment debriefing and feedback sessions. In the included studies, 61% (n=33 of 54) of the studies conducted training of the raters, which mainly included the introduction of the ANTS system and simulation practice. The results of the inter-rater reliability tests were satisfactory. Additionally, 43% (n=23 of 54) of the included studies underwent post-assessment debriefing. Four studies using debriefing as a hypothesis supported its positive effect on NTS ability.42–45 How post-assessment debriefing affects the NTS of trainees in different ways and situations and whether NTS can be retained are aspects that need further exploration and discussion.
Objective 4: to determine the contribution of the ANTS behavioural marker system to improve patient safety
For objective 4, our goal was to determine the contribution of the ANTS system to patient safety, which was assessed based on two aspects: (1) investigation of outcomes and measurement of the association between the ANTS system and outcomes, and (2) whether the ANTS system stimulates new theoretical developments (online supplemental file 2). The frequency of adverse events and errors was not reported in the included studies.
Contribution to the investigation of outcomes and measurement of associations between the ANTS system and outcomes
In total, 18 studies used the ANTS scores as the primary outcome to evaluate the NTS performance of doctors or nurses in the operating room, ICU and delivery room. Almost all of these 18 studies simulated different clinical scenarios, including crisis events or low-cost simulations in low-income countries, and different training methods, such as variable ways of debriefing or multimedia instruction.
Five studies examined the NTS of operating room members during intraoperative incidents in a simulated environment.40 46–49 In these studies, NTS were measured using different behavioural marker systems in different specialties, such as the ANTS system used by anaesthetists, Non-Technical Skills for Surgeons (NOTSS) system, Observational Teamwork Assessment for Surgery tool used by surgeons and Scrub Practitioners’ List of Intraoperative Non-Technical Skills system used by scrub practitioners. Eight studies measured TS and ANTS scores as outcomes. Five of these studies showed a strong association between TS and NTS.
Three qualitative studies and one mixed-methods study based on the ANTS system explored the NTS behavioural markers of an organ procurement team in the ICU,31 nurse anaesthetists in the operating room,32 undergraduate surgical and anaesthesiology curricula,35 and medical air ambulance teams.33
Contribution to new theoretical developments
The qualitative outcomes derived from the application of the ANTS system not only established a robust foundation for analysis but also facilitated the generation of NTS frameworks across various specialties and domains. This foundational work has broadened the scope of NTS application, enabling adaptations and innovations in several fields. Relying on the ANTS system as the starting point, qualitative methods with literature review and focus group interviews were used to identify the NTS frameworks of nurse anaesthetists in the operating room32 and aeromedicine in the critical care environment of air ambulances.33 Behaviour markers of organ procurement in the ICU have been achieved by group discussion and auditing of clinical cases.31 In an ethnographic study, ANTS were measured in an undeveloped, low-income country of Rwanda.34 Another study found a serious lack of formal NTS teaching in undergraduate surgical and anaesthesiology curricula under the NOTSS and ANTS frameworks.35
Discussion
NTS play a vital role in anaesthetic practice. The ANTS behavioural marker system was designed to describe the main NTS, which are important for good anaesthetic practice. In addition to providing a tool for assessing aspects of performance traditionally judged based on gut feelings, the behavioural marker system supplies a common language for discussing NTS and can function as a framework for structuring teaching and debriefing. However, the ANTS behavioural marker system is not widely recognised, especially in low-income and middle-income countries. Reviewing the characteristics of the ANTS behavioural marker system, namely its reliability, validity, application status and contributions to patient safety, may improve the understanding of healthcare personnel regarding NTS and may promote the development of NTS across specialties.
Overall, this review has shown that the ANTS behavioural marker system has been applied across a wide range of studies with diverse objectives, settings, units of analysis, designs and methods. This suggests that the ANTS system is applicable to a wide range of interventions, settings and research designs.
Most studies were based on simulation settings. Simulation training increases the trainees’ self-confidence,50 knowledge51 and TS,52 and improves team behaviour.53 It has many advantages such as the possibility to practise procedures without any risk for patients and to commit errors by trainees and learn from those errors through the repetition of different scenarios.53 One study showed that after just one simulated teaching session, participants showed an ability to translate their newly acquired skills into clinical practice.54 One key point is to explore the link between simulation and real life, especially considering how the ANTS system can be assessed and trained in clinical settings.
Given the importance of NTS in clinical practice, the development of an effective and reliable assessment system is critical. The initial reliability and validity assessments of the ANTS showed that this system was complete, the skills were observable, and the agreement and accuracy of assessments were at acceptable levels.19 Internal consistency of the system appeared sound, and responses to usability were very positive. In articles reporting inter-rater reliability, 69% of the results received substantial agreement. The Standards provided us with a framework to create a hierarchy of studies and thus synthesise a large volume of psychometric data.30 In an attempt to display the overall construct validity of the measures identified, our hierarchy (consistent with the Standards) placed equal weight on all four evidential sources.30 The levels in the hierarchy are based on the number of validity sources reported and not on the actual source or quality of evidence within each source. Overall, the hierarchy is intended to present an overview of the validity testing. It is meant to inform researchers about what testing has been done (online supplemental file 3), such as pilot tests or debriefing of respondents,55 adjustment of Likert scales for rating56 and the development of novel research hypotheses.57 Two additional limitations in the ANTS system were also noted as a result of this review. They include over-reliance on the assessment of a measure in the original study and failure to re-establish validity when modifications are made or the measure is assessed in a new population or scenario.
In recent years, an expert panel on the evaluation of NTS in healthcare, composed of anaesthetists, surgeons, human factor experts and psychologists, met to develop recommendations for a ‘criterion standard’ for the training of NTS assessors in medicine.58 The resulting detailed recommendations describe the length of training, the content and delivery of training, the expertise and experience of assessors, the interval for revalidation or retraining, and the definition of proficiency based on psychometric indicators. Several studies have suggested that knowledge retention after ANTS evaluation needs to be assessed.54 59 60 Moreover, 10 studies measured knowledge retention at different intervals with good results. In the ANTS system handbook, the importance of systematic training before using the system for teaching and valuation was emphasised. The training should consist of the practice of observing and rating, receiving feedback on the scores and discussing the appropriate use of the system.38 While the raters in many of the included studies had similar training and clinical experience, their personal clinical experience may have influenced their assessment of individual performance in each scenario. As reported by Graham et al, an 8-hour programme could not train anaesthetists to reliably use the ANTS as a summative assessment tool.36 Further work may help to determine how much training is needed to be competent in using the ANTS system to assist trainees in assessment and feedback.
In our review, most included studies used the ANTS system to guide data collection only. Almost all included studies assessed ANTS at the category and element levels. Any assessment should be based only on behaviours observed directly, and the rating can then be carried out, first at the element level and then at the more general category level.38 Element-level observations may provide specific feedback on skills, whereas the category level can be used to describe the more general performance. One study that applied the ANTS system to assess the element level, that is, teamwork, showed that some specific teamwork skills proposed by the ANTS model were correlated with trauma team performance.41 Specific teamwork skills and behavioural markers were successfully transferred from one environment to another, indicating the general nature of these behavioural categories.41
With increasing attention being paid to the NTS of medical personnel, behavioural marker frameworks suitable for different specialties have gradually been established. Cultural differences at the organisational, professional or national level should be taken into account when adapting tools for other countries and specialties.61 Thus, NTS behavioural marker systems for an organ procurement team,31 nurse anaesthetists32 and medical air ambulance teams33 were adapted from the ANTS system. Notably, the authors of one study stated that using the ANTS to inform the development of their focus group interview guides may have blinded the research group to other NTS aspects.32 A thorough task analysis may have captured new or other aspects and concepts.32 Revisions of these may become necessary at a later point in time to adjust for changes such as improved knowledge of human factors. Several studies have also mentioned that the ANTS system cannot differentiate between NTS required for different scenarios and cannot capture and evaluate short scenarios,62–64 nor does it distinguish between NTS required in a particular clinical setting and NTS in general. It may be useful to develop other long-term scenarios that specifically use skills related to preoperative evaluation, formulation of anaesthesia plans, consulting other clinicians, and performing invasive procedures, postoperative care, and pain management.63
Our analysis also found that TS and NTS were correlated, not independent, in the process of intraoperative crisis management.64 Yee et al noted that certain parts of the ANTS tool, although designed to assess NTS, were inherently related to technical performance and skills.62 In one study, residents’ theoretical knowledge scores were also moderately correlated with assessment results of all NTS categories, task management, teamwork, situational awareness and decision-making.65 It is also worth considering whether teaching and training NTS to the same extent as TS in educational programmes might improve the performance of anaesthesia residents in the operating room.66 Further studies are necessary to investigate the nature and direction of the relationship between TS and NTS.
Strengths and limitations
To the best of our knowledge, no previous systematic review has illustrated the assessment of validity and reliability of the ANTS behavioural marker system and its application as a tool for the training and assessment of NTS and for improving patient safety.
This study has several limitations. First, non-English-language studies were not searched, which may have led to publication bias. In addition, our study did not include other research outputs such as unpublished papers, grey literature or conference abstracts. However, citation search, as used in this article, is a method of finding relevant research in a field or subject by considering what has been referenced in an article and who has since used the article in question as a reference.67 Moreover, the databases we chose, that is, Web of Science and Scopus, provide the most comprehensive databases of articles that can be tracked using citations, and they allow reference searches.26 67
Implications
This review found that the ANTS behavioural marker system has been widely used as a quantitative evaluation tool and is an effective tool to evaluate the NTS of anaesthetists, which may attract the attention of relevant personnel to use the ANTS system. Based on the results of our systematic review, we recommend that the reliability and validity should be re-established when the ANTS system is used for assessment in a new population or scenario. Although an overlap of NTS may exist among different roles, it remains to be discussed whether the NTS required in distinct scenarios and perioperative stages differ. This has the potential to improve the quality of care and reduce the risk of adverse events. When adapting the ANTS system to various medical disciplines, it is crucial to account for cultural differences. This consideration ensures the system’s effectiveness and relevance across diverse healthcare settings. However, some research has been carried out on whether ANTS can be used as a basis to accurately cover the whole range of NTS; researchers in other disciplines should take this into account when exploring NTS in their fields.32 Although most studies have been conducted in developed countries, studies in African countries have demonstrated the importance of NTS in improving anaesthetists’ performance and patient safety.34 Relevant measures such as simulated training are effective interventions to improve ANTS.54 At present, ANTS research in low-income and middle-income countries is limited, so we suggest extending these studies.
Conclusions
In the 18 years since the ANTS system was first published, 54 published articles were identified using the ANTS behavioural marker system in a meaningful way. These articles indicate that the ANTS system has been used in a wide range of studies with various objectives, units of analysis, designs, methods and innovative research in interdisciplinary fields. Investigating the methods by which the ANTS system can be evaluated and implemented for training within clinical environments is anticipated to significantly enhance ongoing enhancements in staff performance and patient safety.
Supplementary Material
Acknowledgments
We would like to thank Editage (www.editage.com) for English-language editing.
Footnotes
JK and JH contributed equally.
ST and FZ contributed equally.
Contributors: All authors designed the study. JK and JH developed the search strategy. JK and CY screened the articles and assisted in data extraction. JK and JH took the lead in the principal analysis of the data and the writing of the manuscript. FZ, ST, XX, JH and JK contributed to the interpretation and presentation of results. All authors provided feedback on the manuscript and approved its final version. FZ is responsible for the overall content as guarantor.
Funding: This study was supported by Project 333 and the Qinglan Project of the Jiangsu Province.
Competing interests: None declared.
Patient and public involvement: Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review: Not commissioned; externally peer reviewed.
Supplemental material: This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
Data availability statement
All data relevant to the study are included in the article or uploaded as supplemental information.
Ethics statements
Patient consent for publication
Not applicable.
Ethics approval
This study did not involve any human participants or unpublished secondary data. Thus, ethical approval was not required for this study. These findings have been disseminated through peer-reviewed publications and conference presentations.
References
- 1. National Academies of Sciences E, and Medicine; Health and Medicine Division; Board on Health Care Services; Board on Global Health; Committee on Improving the Quality of Health Care Globally . Crossing the global quality chasm: improving health care worldwide. Washington (DC), USA: The National Academies Press, 2018: 23–52. [PubMed] [Google Scholar]
- 2. Panagioti M, Khan K, Keers RN, et al. Prevalence, severity, and nature of preventable patient harm across medical care settings: systematic review and meta-analysis. BMJ 2019;366:l4185. 10.1136/bmj.l4185 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Mazer BL, Nabhan C. Strengthening the medical error "meme pool". J Gen Intern Med 2019;34:2264–7. 10.1007/s11606-019-05156-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Agbabiaka TB, Lietz M, Mira JJ, et al. A literature-based economic evaluation of healthcare preventable adverse events in Europe. Int J Qual Health Care 2017;29:9–18. 10.1093/intqhc/mzw143 [DOI] [PubMed] [Google Scholar]
- 5. Slawomirski L, Auraaen A, Klazinga NS. The economics of patient safety: strengthening a value-based approach to reducing patient harm at national level, No. 96. OECD Health Working Papers, 2017. [Google Scholar]
- 6. Aitken M, Gorokhovich L. Advancing the responsible use of medicines: applying levers for change [abstract]. SSRN Journal 2012. 10.2139/ssrn.2222541 [DOI] [Google Scholar]
- 7. Runciman WB, Webb RK, Lee R, et al. The Australian incident monitoring study. system failure: an analysis of 2000 incident reports. Anaesth Intensive Care 1993;21:684–95. 10.1177/0310057X9302100535 [DOI] [PubMed] [Google Scholar]
- 8. Flin R, Patey R, Glavin R, et al. Anaesthetists' non-technical skills. Br J Anaesth 2010;105:38–44. 10.1093/bja/aeq134 [DOI] [PubMed] [Google Scholar]
- 9. Kirschbaum K, McAuliffe MS, Swanson M. Team communication in the operating room: a measure of latent factors from a national sample of nurse Anesthetists. AANA J 2018;86:11–8. [PubMed] [Google Scholar]
- 10. Engel D, Furrer MA, Wuethrich PY, et al. Surgical safety in radical cystectomy: the anesthetist’s point of view-how to make a safe procedure safer. World J Urol 2020;38:1359–68. 10.1007/s00345-019-02839-y [DOI] [PubMed] [Google Scholar]
- 11. Vaughan KA, Johnson WDJUiA . Progress and challenges in global surgical and anaesthesia care and safety: proceedings of the SAFE-T summit 2018, Vol 33. Update in Anaesthesia, 2019: 13–5. [Google Scholar]
- 12. Assembly UNG . Transforming our world: the 2030 agenda for sustainable development. In: Nations U, ed. New York, 2015. [Google Scholar]
- 13. World Health Organization S-eWHA . Strengthening emergency and essential surgical care and anaesthesia as a component of universal health coverage. In: WHO, ed. Geneva, 2015. [Google Scholar]
- 14. Gaba DM. Human error in anesthetic mishaps. Int Anesthesiol Clin 1989;27:137–47. 10.1097/00004311-198902730-00002 [DOI] [PubMed] [Google Scholar]
- 15. Radhakrishnan B, Katikar MD, Myatra SN, et al. Importance of non-technical skills in anaesthesia education. Indian J Anaesth 2022;66:64–9. 10.4103/ija.ija_1097_21 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Boet S, Larrigan S, Martin L, et al. Measuring non-technical skills of anaesthesiologists in the operating room: a systematic review of assessment tools and their measurement properties. Br J Anaesth 2018;121:1218–26. 10.1016/j.bja.2018.07.028 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Vinagre R, Tanaka P, Tardelli MA. Competency-based anesthesiology teaching: comparison of programs in Brazil, Canada and the United States. Braz J Anesthesiol 2021;71:162–70. 10.1016/j.bjane.2020.12.026 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Fletcher GCL, McGeorge P, Flin RH, et al. The role of non-technical skills in anaesthesia: a review of current literature. Br J Anaesth 2002;88:418–29. 10.1093/bja/88.3.418 [DOI] [PubMed] [Google Scholar]
- 19. Fletcher G, Flin R, McGeorge P, et al. Anaesthetists' non-technical skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth 2003;90:580–8. 10.1093/bja/aeg112 [DOI] [PubMed] [Google Scholar]
- 20. Fletcher G, Flin R, McGeorge P, et al. Rating non-technical skills: developing a behavioural marker system for use in anaesthesia. Cogn Tech Work 2004;6:165–71. 10.1007/s10111-004-0158-y [DOI] [Google Scholar]
- 21. Barré J, Michelet D, Truchot J, et al. Midwifery students' retention of learning after screen-based simulation training on neonatal resuscitation: a pilot study. BMJ Simul Technol Enhanc Learn 2021;7:31–4. 10.1136/bmjstel-2019-000525 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Shear TD, Deshur M, Benson J, et al. The effect of an electronic dynamic cognitive aid versus a static cognitive aid on the management of a simulated crisis: a randomized controlled trial. J Med Syst 2018;43:6. 10.1007/s10916-018-1118-z [DOI] [PubMed] [Google Scholar]
- 23. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. 10.1136/bmj.n71 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Kang J, Hu J, Yan C, et al. Development and applications of the anaesthetists' non-technical skills behavioural marker system: protocol for a systematic review. BMJ Open 2022;12:e065519. 10.1136/bmjopen-2022-065519 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Wright K, Golder S, Rodriguez-Lopez R. Citation searching: a systematic review case study of multiple risk behaviour interventions. BMC Med Res Methodol 2014;14:73. 10.1186/1471-2288-14-73 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Falagas ME, Pitsouni EI, Malietzis GA, et al. Comparison of PubMed, Scopus, web of science, and google scholar: strengths and weaknesses. FASEB J 2008;22:338–42. 10.1096/fj.07-9492LSF [DOI] [PubMed] [Google Scholar]
- 27. Martín-Martín A, Orduna-Malea E, Thelwall M, et al. Google scholar, web of science, and scopus: a systematic comparison of citations in 252 subject categories. J Informetr 2018;12:1160–77. 10.1016/j.joi.2018.09.002 [DOI] [Google Scholar]
- 28. Pranckutė R. Web of science (WoS) and scopus: the titans of bibliographic information in today’s academic world. Publications 2021;9:12. 10.3390/publications9010012 [DOI] [Google Scholar]
- 29. Gale NK, Heath G, Cameron E, et al. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol 2013;13:117. 10.1186/1471-2288-13-117 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. American Educational Research Association APA, National Council on Measurement in Education . Standards for educational and psychological testing, 2014 Edition. 1430 K St., NW, Suite 1200, Washington, DC: American Educational Research Association, 2014: 20005. [Google Scholar]
- 31. Trabucco G, Procaccio F, Marcanti M. Non-technical skills in organ procurement. A preliminary approach to communicative and emotional problems in the intensive care unit. Cells Tissues Organs 2008:197–204. [Google Scholar]
- 32. Lyk-Jensen HT, Jepsen RMHG, Spanager L, et al. Assessing nurse anaesthetists' non-technical skills in the operating room. Acta Anaesthesiol Scand 2014;58:794–801. 10.1111/aas.12315 [DOI] [PubMed] [Google Scholar]
- 33. Myers JA, Powell DMC, Psirides A, et al. Non-technical skills evaluation in the critical care air ambulance environment: introduction of an adapted rating instrument - an observational study. Scand J Trauma Resusc Emerg Med 2016;24:24. 10.1186/s13049-016-0216-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34. Livingston P, Zolpys L, Mukwesi C, et al. Non-technical skills of anaesthesia providers in Rwanda: an ethnography. Pan Afr Med J 2014;19:97. 10.11604/pamj.2014.19.97.5205 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. Lee A, Finstad A, Gawad N, et al. Nontechnical skills (NTS) in the undergraduate surgical and anesthesiology curricula: are we adequately preparing medical students. J Surg Educ 2021;78:502–11. 10.1016/j.jsurg.2020.08.001 [DOI] [PubMed] [Google Scholar]
- 36. Graham J, Hocking G, Giles E. Anaesthesia non-technical skills: can anaesthetists be trained to reliably use this behavioural marker system in 1 day Br J Anaesth 2010;104:440–5. 10.1093/bja/aeq032 [DOI] [PubMed] [Google Scholar]
- 37. Bruppacher HR, Alam SK, LeBlanc VR, et al. Simulation-based training improves physicians' performance in patient care in high-stakes clinical setting of cardiac surgery. Anesthesiology 2010;112:985–92. 10.1097/ALN.0b013e3181d3e31c [DOI] [PubMed] [Google Scholar]
- 38. Flin R, Patey R, Glavin R, et al. Anaesthetists' non-technical skills (ANTS). Br J Anaesth 2010;105:38–44. 10.1093/bja/aeq134 [DOI] [PubMed] [Google Scholar]
- 39. Balki M, Chakravarty S, Salman A, et al. Effectiveness of using high-fidelity simulation to teach the management of general anesthesia for cesarean delivery. Can J Anaesth 2014;61:922–34. 10.1007/s12630-014-0209-7 [DOI] [PubMed] [Google Scholar]
- 40. Phitayakorn R, Minehart R, Pian-Smith MCM, et al. Practicality of intraoperative teamwork assessments. J Surg Res 2014;190:22–8. 10.1016/j.jss.2014.04.024 [DOI] [PubMed] [Google Scholar]
- 41. Westli HK, Johnsen BH, Eid J, et al. Teamwork skills, shared mental models, and performance in simulated trauma teams: an independent group design. Scand J Trauma Resusc Emerg Med 2010;18:47. 10.1186/1757-7241-18-47 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42. Savoldelli GL, Naik VN, Park J, et al. Value of debriefing during simulated crisis management - oral versus video-assisted oral feedback. Anesthesiology 2006;105:279–85. 10.1097/00000542-200608000-00010 [DOI] [PubMed] [Google Scholar]
- 43. Welke TM, LeBlanc VR, Savoldelli GL, et al. Personalized oral debriefing versus standardized multimedia instruction after patient crisis simulation. Anesth Analg 2009;109:183–9. 10.1213/ane.0b013e3181a324ab [DOI] [PubMed] [Google Scholar]
- 44. Boet S, Bould MD, Bruppacher HR, et al. Looking in the mirror: self-debriefing versus instructor debriefing for simulated crises. Crit Care Med 2011;39:1377–81. 10.1097/CCM.0b013e31820eb8be [DOI] [PubMed] [Google Scholar]
- 45. Michelet D, Barre J, Truchot J, et al. Effect of computer debriefing on acquisition and retention of learning after screen-based simulation of neonatal resuscitation. JMIR Serious Games 2020;8:e18633. 10.2196/18633 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46. Phitayakorn R, Minehart RD, Hemingway MW, et al. The relationship between intraoperative teamwork and management skills in patient care. Surgery 2015;158:1434–40. 10.1016/j.surg.2015.03.031 [DOI] [PubMed] [Google Scholar]
- 47. Saleh GM, Wawrzynski JR, Saha K, et al. Feasibility of human factors Immersive simulation training in ophthalmology the London pilot. JAMA Ophthalmol 2016;134:905. 10.1001/jamaophthalmol.2016.1769 [DOI] [PubMed] [Google Scholar]
- 48. Siu J, Maran N, Paterson-Brown S. Observation of behavioural markers of non-technical skills in the operating room and their relationship to intra-operative incidents. Surgeon 2016;14:119–28. 10.1016/j.surge.2014.06.005 [DOI] [PubMed] [Google Scholar]
- 49. Doumouras AG, Hamidi M, Lung K, et al. Non-technical skills of Surgeons and anaesthetists in simulated operating theatre crises. Br J Surg 2017;104:1028–36. 10.1002/bjs.10526 [DOI] [PubMed] [Google Scholar]
- 50. Malmström B, Nohlert E, Ewald U, et al. Simulation-based team training improved the self-assessed ability of physicians, nurses and midwives to perform neonatal resuscitation. Acta Paediatr 2017;106:1273–9. 10.1111/apa.13861 [DOI] [PubMed] [Google Scholar]
- 51. Curran VR, Aziz K, O’Young S, et al. Evaluation of the effect of a computerized training simulator (ANAKIN) on the retention of neonatal resuscitation skills. Teach Learn Med 2004;16:157–64. 10.1207/s15328015tlm1602_7 [DOI] [PubMed] [Google Scholar]
- 52. Lee MO, Brown LL, Bender J, et al. A medical simulation-based educational intervention for emergency medicine residents in neonatal resuscitation. Acad Emerg Med 2012;19:577–85. 10.1111/j.1553-2712.2012.01361.x [DOI] [PubMed] [Google Scholar]
- 53. Mileder LP, Urlesberger B, Szyld EG, et al. Simulation-based neonatal and infant resuscitation teaching: a systematic review of randomized controlled trials. Klin Padiatr 2014;226:259–67. 10.1055/s-0034-1372621 [DOI] [PubMed] [Google Scholar]
- 54. Skelton T, Nshimyumuremyi I, Mukwesi C, et al. Low-cost simulation to teach anesthetists' non-technical skills in Rwanda. Anesth Analg 2016;123:474–80. 10.1213/ANE.0000000000001434 [DOI] [PubMed] [Google Scholar]
- 55. Hards A, Davies S, Salman A, et al. Management of simulated maternal cardiac arrest by residents: didactic teaching versus electronic learning. Can J Anaesth 2012;59:852–60. 10.1007/s12630-012-9752-2 [DOI] [PubMed] [Google Scholar]
- 56. DeMaria S, Samuelson ST, Schwartz AD, et al. Simulation-based assessment and retraining for the anesthesiologist seeking reentry to clinical practice: a case series. Anesthesiology 2013;119:206–17. 10.1097/ALN.0b013e31829761c8 [DOI] [PubMed] [Google Scholar]
- 57. Marshall SD, Mehra R. The effects of a displayed cognitive aid on non-technical skills in a simulated "can't Intubate, can't oxygenate' crisis. Anaesthesia 2014;69:669–77. 10.1111/anae.12601 [DOI] [PubMed] [Google Scholar]
- 58. Hull L, Arora S, Symons NRA, et al. Training faculty in nontechnical skill assessment: national guidelines on program requirements. Ann Surg 2013;258:370–5. 10.1097/SLA.0b013e318279560b [DOI] [PubMed] [Google Scholar]
- 59. Gouin A, Damm C, Wood G, et al. Evolution of stress in anaesthesia registrars with repeated simulated courses: an observational study. Anaesth Crit Care Pain Med 2017;36:21–6. 10.1016/j.accpm.2016.02.008 [DOI] [PubMed] [Google Scholar]
- 60. Michelet D, Barré J, Job A, et al. Benefits of screen-based postpartum hemorrhage simulation on nontechnical skills training a randomized simulation study. Simul Healthc 2019;14:391–7. 10.1097/SIH.0000000000000395 [DOI] [PubMed] [Google Scholar]
- 61. Flin RH, O’Connor P, Crichton M. Safety at the sharp end: a guide to non-technical skills, 1st edn. CRC Press, 2008. [Google Scholar]
- 62. Yee B, Naik VN, Joo HS, et al. Nontechnical skills in anesthesia crisis management with repeated exposure to simulation-based education. Anesthesiology 2005;103:241–8. 10.1097/00000542-200508000-00006 [DOI] [PubMed] [Google Scholar]
- 63. Mudumbai SC, Gaba DM, Boulet JR, et al. External validation of simulation-based assessments with other performance measures of third-year anesthesiology residents. Simul Healthc 2012;7:73–80. 10.1097/SIH.0b013e31823d018a [DOI] [PubMed] [Google Scholar]
- 64. Riem N, Boet S, Bould MD, et al. Do technical skills correlate with non-technical skills in crisis resource management: a simulation study. Br J Anaesth 2012;109:723–8. 10.1093/bja/aes256 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 65. Raksamani K, Jirativanont T, Sareenun P. Correlation of medical knowledge and non-technical skills assessment in anesthesia residents. Siriraj Med J 2020;72:483–7. 10.33192/Smj.2020.65 [DOI] [Google Scholar]
- 66. Neuschwander A, Job A, Younes A, et al. Impact of sleep deprivation on anaesthesia residents' non-technical skills: a pilot simulation-based prospective randomized trial. Br J Anaesth 2017;119:125–31. 10.1093/bja/aex155 [DOI] [PubMed] [Google Scholar]
- 67. Bakkalbasi N, Bauer K, Glover J, et al. Three options for citation tracking: google scholar, Scopus and web of science. Biomed Digit Libr 2006;3:7. 10.1186/1742-5581-3-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
bmjopen-2023-075019supp001.pdf (101.5KB, pdf)
bmjopen-2023-075019supp002.pdf (159KB, pdf)
bmjopen-2023-075019supp003.pdf (113.1KB, pdf)
Data Availability Statement
All data relevant to the study are included in the article or uploaded as supplemental information.