Abstract
Introduction
Medical education often aims to improve either technical skills (TS) or ‘non-technical skills’ (NTS) and how these skills influence adverse events and patient safety. The two skill sets are often investigated independently, and little is known about how TS and NTS influence each other. In this scoping review, we therefore aim to investigate the association between TS and NTS.
Method
We conducted a scoping review of four databases in order to summarize, analyse, and collate findings from the included studies.
Results
In total, 203 of 2676 identified studies were included in the final analysis. The first study was published in 1991, but the majority of studies were published in the last decade. The majority were intervention studies including 41 randomized controlled trials. The the objective structured assessment of technical skills (OSATS) was the most common assessment tool with strong validity evidence within TS, but many variations without validity evidence were used. Conversely, Non-Technical Skills for Surgeons (NOTSS) was the most used tool with strong validity evidence for assessing NTS. However, the majority of studies used non-validated self-assessment tools to investigate NTS. The correlation between TS and NTS was assessed in 46 of 203 studies, whereof 40 found a positive correlation.
Discussion
Our findings echo previous literature suggesting that empirical literature investigating the interaction between TS and NTS lack methodological depth. In this review only a minority of the identified studies (n = 46) investigated this correlation. However, the results strongly indicate a correlation between TS and NTS skills, suggesting that physicians who are proficient in their NTS, also perform well on their TS. Thus, the distinction between them in learning designs may seem arbitrary. While this result is promising, the limited methodological rigour indicates a lack of proper understanding of NTS and how to properly assess them.
Supplementary Information
The online version contains supplementary material available at 10.1186/s12909-024-06419-6.
Introduction
A competent physician needs a broad variety of skills. While the needed competences are multifaceted, skill-based training in medicine often has a narrow focus. Skills are typically divided into technical skills (TS), such as the body-kinetic performance, and non-technical skills (NTS), often referring to human factors such as communication, decision making, leadership, and teamwork [1, 2]. In recent years, human factors gained notoriety for its negative impact on medical performance and threat to patient safety in terms of how these non-technical skills are impacted by environmental factors [3–8]. Thus, research on training initiatives targeting NTS are rapidly emerging in medical education. Here, simulation-based training has been investigated in relation to promoting clinical confidence [9], knowledge [10], collaboration [11], or overall NTS [12]. Similarly, other methods such as team interventions’ effect on team performance [13], and the effect of coaching on well-being and NTS [14] have also been investigated. This rising acknowledgement of NTS has been recognized by stakeholders. In example, employers of veterinarians report that NTS are an equally important feature in their employees, as technical skills [15].
In educational research outside the medical sphere, an interdependence of TS and NTS is well-established [16, 17]. While several authors have pointed towards this interdependence in medical education [18–21], it is less common to investigate the interaction between them. Research often assesses one of the two skills, but studies have found that both skill sets influence each other either directly or indirectly [22, 23].
Understanding the association between TS and NTS may help lay the foundation for more robust education of healthcare professionals, thereby improving overall treatment and enhancing patient safety. To date, only a couple of reviews investigating both TS and NTS have been performed in medical education, but often, they single out a specific phenomenon or medical specialty. Examples of these are reviews of training tools in ophthalmology [24] and urology [25], on the connection between spatial abilities and TS [26], the role of simulation-based training to promote TS and NTS [27, 28], and NTS in surgery and surgical teams [29–32]. Thus, none has, to our knowledge, performed a comprehensive review on the association between NTS and TS in medical education, overall.
The aim of this scoping review was therefore to explore the association between TS and NTS in medical education.
Materials and methods
A scoping review was conducted following the methodological steps outlined by Arksey and O’Malley [33] and PRISMA for Scoping Reviews [34].
Identifying relevant studies
Searches were made on four electronic databases: Embase, PubMed, Web of Science, and Scopus. The search string consisted of a variety of words related to “assessment”, “technical or non-technical skills”, “learning or education or training”, and “medicine or surgery or medical” in abstract or title. This was done to ensure that all studies were related to medical education and had an element of skill assessment. No start date was specified, as the inclusion of studies published over a relevant time span was considered essential for our study. The most recent search was conducted on December 4, 2023, in PubMed.
Study selection
All identified studies were uploaded to EndNote and duplicates were removed. These were then transferred to Rayyan.ai [35] for sorting. Inclusion criteria were:
Peer reviewed studies conducted in the Health Profession Education
Inclusion of both technical and non-technical skills
Assessment of skills
Exclusion criteria were defined during the analysis of the studies based on increased knowledge on the subject.
Overall, exclusion criteria were:
only assessment of either technical or non-technical skills
book chapters or short commentaries
non-English studies
non-health care studies
non-published studies
study protocols
reviews and conference abstracts
One author (LBS) reviewed all studies and removed duplicates. Three blinded reviewers (RDJ, MLG, LBS) independently reviewed the abstracts of a subset of 100 studies. All authors met to discuss inclusion and exclusion of publications, until consensus was reached. After consensus, the remaining studies were full-text screened by one author (LBS) and were either excluded or included in the final analysis.
Charting the data
Key information for each included study was extracted using a charting form. This included: Author(s), year of publication, method(s), setting, participants/population, aim of the study, which NTS and TS was being assessed, what assessment tool was used to assess NTS and TS, main findings, conclusion, correlation between NTS and TS, and which medical specialty was represented in the study. In the present review, self-assessment is used as an umbrella term to cover different self-reported NTS.
Publications were categorized into 3 study types:
intervention studies which describe and assess an intervention,
explorative studies seeking to validate or explore coherence of NTS and TS,
observational studies assessing performance by observing in-clinic behaviour.
Collating, summarizing, and reporting the results
We collated and summarized the data in different illustrations depicting the yearly distribution of published literature, methods and study designs, assessment tools used in the assessment of NTS and TS, and finally quantity of studies finding a correlation between NTS and TS.
Results
We identified 5093 studies matching our criteria. After removal of duplicates in Endnote, 2676 studies were uploaded to Rayyan.ai [35], and another 180 manually identified duplicates were removed. The abstracts of the remaining 2496 studies were screened for eligibility. A total of 2218 studies were excluded as shown in the PRISMA flowchart (Fig. 1). The remaining 278 publications were full text screened, whereof 71 conference abstracts and 11 reviews were detected and excluded due to the lack of original empirical data. Seven additional publications were included after screening reference lists of included studies.
Fig. 1.
Flow chart for studies included and excluded in the study
In total, 203 publications were included in the final analysis. An overview of these publications is presented in Appendix 1.
Year of publication
Figure 2 shows the distribution of published studies from 1991 to 2023; with 2013, being the year with the highest number of publications, and 148 of the 203 included studies being published since this year.
Fig. 2.
Number of included studies per year
Study methods
In total, 87% (n = 176) of the included studies were intervention studies. Of these, 41 studies were randomized controlled trials. The remaining publications were explorative studies (n = 24) and observational studies (n = 3).
Assessment tools
All included studies used assessment tools to evaluate NTS and TS.
As shown in Fig. 3, Objective Structured Assessment of Technical Skills (OSATS) was the most frequently used assessment tool for TS. OSATS measures tissue handling, time and motion, instrument handling, knowledge of instruments, overall flow during the procedure, and knowledge of the procedure [36]. Sixteen studies used variations of OSATS to assess TS [37], but only 9 of these used variations including validity evidence. For the remaining 7 studies, reporting on validity evidence was unspecified or non-validated. Additionally, nine studies used a procedure-specific checklist or assessment tool based on OSATS to assess TS, whereof 5 of 9 included validity evidence. (Table 1).
Fig. 3.
Frequently used assessment tools
Table 1.
OSATS and variations of OSATS
| Validity evidence | No validity evidence | |
|---|---|---|
| OSATS | 52 | - |
| Modified OSATS and variation thereof | 9 | 7 |
| Tool based on OSATS | 5 | 4 |
For assessment of NTS, the Non-technical Skills for Surgeons (NOTSS) tool [38] was the most frequently used assessment tool (n = 18). NOTSS measures situational awareness, decision making, communication & teamwork, and leadership in a surgical setting [38].
A total of 72 studies used tools with strong validity evidence to assess both TS and NTS. However, 62 studies used only one assessment tool with validity evidence in their study to assess either NTS or TS, while the assessment of the other was either non-validated or non-specified. Self-assessment in various forms was the most frequently used form of assessment (n = 78) which was often used to assess the participants’ NTS either in a survey or questionnaire.
Medical specialties represented
Abdominal surgery was the most frequent medical specialty represented in the included publications (n = 47). However, most of the studies did not specify what kind of surgery or medical specialty they assessed. As such, they are represented in “general surgery” or “general medicine”. Resuscitation was categorized independently of medical specialty as it involves different specialties (Fig. 4).
Fig. 4.
Number of studies within each medical specialty
Correlation between NTS and TS
The objective of this review was to map to which degree the current literature investigates technical and non-technical skills. To this end, we found 203 articles that assessed both skills, but only 46 studies tested for a correlation. Here, the majority found a positive correlation, supporting our assumption that technical and non-technical skills correlate.
The 46 studies that tested for correlation between NTS and TS often used Spearman regression or Pearson r correlation. Forty of these (87%) found a positive correlation. The remaining six publications did not find any or found a negative correlation when testing the association between the two skill sets.
Studies that found a positive correlation investigated general surgical skills (n = 15), laparoscopic skills (n = 13), crisis and trauma skills (n = 9), diagnostic skills (n = 2), and robotic surgery (n = 1) within for example paediatrics, urology, OBGYN, and various surgical specialties. The methodology in the 40 studies that found a positive correlation were intervention studies (n = 33), explorative studies (n = 6), and one was an observational study. Within studies on general surgery (n = 9), results showed that NOTSS or NOTECHS had a positive correlation with surgical performance. In the resuscitation-based studies (n = 7), a positive correlation was found between leadership, teamwork, and situational awareness and technical resuscitation skills.
In general, the most common NTS tested for in the studies that found a positive correlation between NTS and TS, were leadership (n = 14) and decision making (n = 14). However, situational awareness (n = 12), teamwork (n = 12), and communication (n = 11) were also commonly tested for in the studies. The most frequently assessed TS with a correlation to NTS was surgical performance (n = 27), either in a simulation-based setting or in the OR.
Discussion
While several studies aimed to assess both TS and NTS, most of the identified studies did not provide an assessment of both constructs, showcasing a lack of methodological rigour [32, 39]. This finding might reflect a broader issue: a limited understanding of how TS and NTS are integrated to influence clinical activities. Our findings support this argument, highlighting not only the absence of a clear, concise definition of NTS but also the insufficient statistical analysis in the current research. Studies by Cleland et al. and Tsuda et al. show how the acquisition of TS happens in relation to other learners and teachers [21, 40]; thereby highlighting that learning of TS is influenced by interpersonal and cognitive skills. In continuation of this, the distinction between the two skill sets might seem arbitrary. This is supported by socio-cultural educational studies emphasizing that the dispositions of the learner and the affordances of the environment form the outcome of learning [23, 41, 42]. These studies were, however, qualitative and cannot establish whether a clear relationship between TS and NTS exists. Within the quantitative studies we saw a lack of theoretical presentation or discussion of how TS and NTS interacts in clinical activities. This gap between the investigation and the theoretical point of departure may impact the development of assessment tools to guide such integration of TS and NTS, and thereby the investigation of the effect of these tools.
While studies conducted outside medical education may point to a correlation and interdependence between NTS and TS [16, 17], the majority of included studies in the present review did not investigate such an interdependence. Of the 192 included manuscripts, we identified 149 studies that did not test for correlation between NTS and TS despite including both skill sets in their analysis. However, many of these speculate if the two skill sets influence each other. The majority of these 149 studies included an intervention based on NTS, which resulted in better technical performance among the participants – but they did not statistically analyse this association and only interpreted its efficiency based on the improvement of TS. To establish how an improvement in TS is influenced by NTS, future studies should simultaneously assess both skill sets and thereby investigate if certain aspects of NTS influence TS and vice versa. This is especially relevant considering the surge of studies on self-regulated learning. In self-regulated learning studies, one of the most frequently used models, the circular model by Zimmerman, sees behaviour as a result of an interplay the individual learner and the environment. Similar is seen in the literature on motor learning arguing that motor control is a result of continues cognitive adaptation to the environment rather than a muscle memory distinct from cognition [43]. Many of these thoughts arise from the insights from Gibson, arguing that different sources of perceptual information present different affordances for learners to execute specific actions [44]. Here, our perception of the environment and task, influence how we execute motor behaviour. This is exemplified in an often-used quote from Gibson (1979, p. 223); “we must perceive in order to move, but we must also move in order to perceive.”. Similarly, Ziegler (2011) argues that learning of new technical skills happens in a way that take the learners intention, attention, and adjustment into account. Here Ziegler highlights self-regulated learning as a useful framework, as the focus on individual and environmental aspects within this framework is complementary and the learning of skills in self-regulated learning can be understood as adaptations to the respective environments [45–47].
Together these theoretical lenses might help future research in medical education, that aim to investigate how different elements of specific skill sets influence or correlate with each other.
Importantly, of the remaining 43 studies that investigated the correlation between TS and NTS, we found that the majority showed a positive correlation, further incentivizing future studies in medical education, and medical educators to account the interdependence of technical and non-technical skills. However, future research should also account for a reconceptualization of non-technical skills, as we also identified common issues with this category during our data retrieval and analysis.
Is knowledge a skill?
During our analysis, we found several publications that defined knowledge as a NTS and mentioned knowledge as a part of their NTS training program. While knowledge is important in medical practice, it seems reasonable to argue that the underlying construct does not theoretically, nor empirically, apply as a skill [48–50]. In this sense, knowledge refers to the process of expanding a frame of mind, rather than an applicable skill [51, 52]. Thus, we argue that knowledge more so is the foundation we base our actions on, rather than being a skill itself. This points to the notion that NTS, in general, is poorly defined and therefore harder, than TS, to assess. We excluded studies that defined knowledge as NTS, without assessing any other aspects of NTS.
Despite a vast amount of research aiming to develop behavioural markers [2, 38, 53], seeking consensus on how to identify and define NTS [18], and systematically reviewing and developing taxonomies or frameworks for NTS [54–56], there is still little consensus on how to assess NTS [4, 57]. This issue affects the ability to compare results across studies, which is also evident in the results of this review as we found that only few publications used tools with validity evidence to assess NTS.
Self-assessment
Our results showed that self-assessment, without any description of validity evidence, was often used to evaluate NTS. Subjective assessment is less reliant than validated psychometric tools or assessment tools used by external assessors [58]. Most common was self-reported confidence on a Likert scale in pre- and post-evaluations. However, psychometric tools assessing mental strain, anxiety, perceived level of competency, comfort, and self-efficacy were also frequently used.
While the accuracy of self-assessment seems to increase with experience [59], there is still a lack of description of validity evidence in tools for self-assessment of NTS amongst novices, where the impact of NTS are arguably most evident [60–63]. Furthermore, Ladonna et al. [64] found that self-doubt affects clinicians at all career stages and emphasize that medical culture must create space for physicians to share their struggles. This highlights the issue of relying on self-assessment as this tool is influenced by the learning culture that the learners are a part of. Hence, it is recommended that research validate the tools they use to assess NTS, using well established frameworks such as Kane’s validity framework [65, 66].
Limitations
This study has several limitations. Firstly, when conducting a review, the identification of all relevant literature on the subject is of essence. To ensure this, advice on constructing a sound literature search was given by a librarian with extensive experience. Additionally, publications not initially identified through the literature search were later found through manual searching, suggesting that our search may not have captured all relevant studies. However, despite our efforts to exhaust all relevant literature, some relevant studies, like a study by Gjeraa et al. who specifically investigated this relationship [67], still evaded our investigation. Unclear definitions of NTS in the literature provided an obstacle to the investigation of the relationship between TS and NTS in the present study. This was evident in that some non-technical assessment tools (i.e., the LBDQ) were not identified in this search. To overcome this obstacle, we applied a broad definition of NTS defined as all skills that are not body-kinetic skills but excluded studies that defined knowledge as NTS without assessing other NTS. Hence, the present study incorporated personal resources skills and cognitive skills in addition to skills such as teamwork and communication which are traditionally defined as NTS.
Conclusions
We aimed to explore the association between NTS and TS. We found that the two constructs showed to influence each other in the majority of the studies identified. This indicates that NTS and TS are interrelated constructs, which is in accordance with much literature on motor learning and cognitive sciences. However, we also found that there is a need for stricter conceptualisation of NTS, and that this construct is often not formally assessed in medical education. As our results suggest, there is a need for better conceptualization of non-technical skills, or rather, a delineation of what these human factors are.
Supplementary Information
Acknowledgements
None.
Funding, conflict of interests, and ethics approval
This study did not receive any external financial support and authors report no conflict of interests. Ethical approval was not required for this study.
Authors’ contributions
All authors contributed to the conception and design of the study. MLG was responsible for writing the first draft of the manuscript and managing the study. LBS conducted the search and was responsible for charting the data and writing the methods section. All authors contributed to reviewing papers, analysing the data, and writing and revising the draft manuscript. Thus, all authors meet conditions 1, 2 and 3.
Funding
This project received no external funding.
Data availability
The author confirms that all data generated or analysed during this study are included in this published review article.
Declarations
Competing interests
The authors declare no competing interests.
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Reader T, Flin R, Lauche K, Cuthbertson BH. Non-technical skills in the intensive care unit. Br J Anaesth. 2006;96(5):551–9. [DOI] [PubMed] [Google Scholar]
- 2.Flin R, Patey R, Glavin R, Maran N. Anaesthetists’ non-technical skills. Br J Anaesth. 2010;104(1):38–44. [DOI] [PubMed] [Google Scholar]
- 3.Kahn K, Pattison T, Sherwood M. Simulation in medical education. Med Teach. 2011;33(1):1–3. [DOI] [PubMed] [Google Scholar]
- 4.Gordon M, Farnan J, Grafton-Clarke C, Ahmed R, Gurbutt D, McLachlan J, et al. Non-technical skills assessments in undergraduate medical education: a focused BEME systematic review: BEME Guide 54. Med Teach. 2019;41(7):732–45. [DOI] [PubMed] [Google Scholar]
- 5.Uramatsu M, Fujisawa Y, Mizuno S, Souma T, Komatsubara A, Miki T. Do failures in non-technical skills contribute to fatal medical accidents in Japan? A review of the 2010–2013 national accident reports. BMJ Open. 2017;7:e013678. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Arora S, Sevdalis N, Nestel D, Woloshynowych M, Darzi A, Kneebone R. The impact of stress on surgical performance: a systematic review of the literature. Surgery. 2010;147(3):319–30. [DOI] [PubMed] [Google Scholar]
- 7.Sinyard RD, Rentas CM, Gunn EGM, Etheridge JC, Robertson JM, Gleason A, et al. Managing a team in the operating room: the science of teamwork and non-technical skills for surgeons. Curr Probl Surg. 2022;59:e101172. [DOI] [PubMed] [Google Scholar]
- 8.Makary MA, Daniel M. Medical error—the third leading cause of death in the US. BMJ. 2016;353:i2139. [DOI] [PubMed] [Google Scholar]
- 9.Coggins A, Desai M, Nguyen K, Moore N. Early acquisition of non-technical skills using a blended approach to simulation- based medical education. Adv Simul. 2017;2:e12. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Iskander M. Systematic review and Meta-Analysis of Computer Based Simulation Training for Non-technical Skills Training and knowledge in Postgraduate Medical Education. J Med Educ. 2019;18(1):38–49. [Google Scholar]
- 11.Jorm C, Roberts C, Lim R, Roper J, Skinner C, Robertson J, et al. A large-scale mass casualty simulation to develop the non-technical skills medical students require for collaborative teamwork. BMC Med Educ. 2016;16:e83. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Hagemann V, Herbstreit F, Kehren C, Chittamadathil J, Wolfertz S, Dirkmann D, et al. Does teaching non-technical skills to medical students improve those skills and simulated patient outcome? Int J Med Educ. 2017;8:101–13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Kilpatrick K, Paquette L, Jabbour M, Tchouaket E, Fernandez N, Al Hakim G, et al. Systematic review of the characteristics of brief team interventions to clarify roles and improve functioning in healthcare teams. PLoS ONE. 2020;15(6):e0234416. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Lovell B. What do we know about coaching in medical education? A literature review. Med Educ. 2018;52:376–90. [DOI] [PubMed] [Google Scholar]
- 15.Danielson JA, Wu T-F, Fales-Williams AJ, Kirk RA, Preast VA. Predictors of employer satisfaction: Technical and non-technical skills. JVME. 2012;39(1):62–70. [DOI] [PubMed] [Google Scholar]
- 16.Kemmis S, Wilkinson J, Edwards-Groves C, Hardy I, Grootenboer P, Bristol L. Changing practices, changing education. Singapore: Springer; 2013. [Google Scholar]
- 17.Schatzki T. Practices and learning. In: Grootenboer P, Edwards-Groves C, Choy S, editors. Practice theory perspectives on pedagogy and education: Praxis, diversity and contestation. Volume 1. Singapore: Springer; 2017. pp. 23–43. [Google Scholar]
- 18.Gordon M, Baker P, Catchpole K, Darbyshire D, Schocken D. Devising a consensus definition and framework for non-technical skills in healthcare to support educational design: a modified Delphi study. Med Teach. 2015;37(6):572–7. [DOI] [PubMed] [Google Scholar]
- 19.Collins JW, Dell’Oglio P, Hung AJ, Brook NR. The importance of Technical and non-technical skills in robotic surgery training. Eur Urol Focus. 2018;4:674–6. [DOI] [PubMed] [Google Scholar]
- 20.Radhakrishnan B, Katikar MD, Myatra SN, Gautam PL, Vinayagam S, Saroa R. Importance of non-technical skills in anaesthesia education. Indian J Anaesth. 2022;66:64–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Cleland J, Walker KG, Gale M, Nicol LG. Simulation-based education: understanding the socio-cultural complexity of a surgical training ‘boot camp’. Med Educ. 2016;50(8):829–41. [DOI] [PubMed] [Google Scholar]
- 22.Cha JS, Anton NE, Mizota T, Hennings JM, Rendina MA, Stanton-Maxey K, et al. Use of non-technical skills can predict medical student performance in acute care simulated scenarios. Am J Surg. 2019;217(2):323–8. [DOI] [PubMed] [Google Scholar]
- 23.Jensen RD, Seyer-Hansen M, Cristancho SM, Christensen MK. Being a surgeon or doing surgery? A qualitative study of learning in the operating room. Med Educ. 2018;52(8):861–76. [DOI] [PubMed] [Google Scholar]
- 24.Lee R, Raison N, Lau WY, Aydin A, Dasgupta P, Ahmed K, et al. A systematic review of simulation-based training tools for technical and non-technical skills in ophthalmology. Eye. 2020;34:1737–59. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Griffin C, Aydın A, Brunckhorst O, Raison N, Khan MS, Dasgupta P, et al. Non-technical skills: a review of training and evaluation in urology. World J Urol. 2020;38:1653–61. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Langlois J, Bellemare C, Toulouse J, Wells GA. Spatial abilities and technical skills performance in health care: a systematic review. Med Educ. 2015;49:1065–85. [DOI] [PubMed] [Google Scholar]
- 27.Rashid P, Gianduzzo TRJ. Urology technical and non-technical skills development: the emerging role of simulation. BJU Int. 2016;117:9–16. [DOI] [PubMed] [Google Scholar]
- 28.Gjeraa K, Møller TP, Østergaard D. Efficacy of simulation-based trauma team training of non-technical skills. A systematic review. Acta Anaesthesiol Scand. 2014;58:775–87. [DOI] [PubMed] [Google Scholar]
- 29.Gabbott B, Tennent D, Snelgrove H. Effect of mental rehearsal on team performance and non-technical skills in surgical teams: systematic review. BJS Open. 2020;4:1062–71. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Ounounou E, Aydin A, Brunckhorst O, Kahn MS, Dasgupta P, Ahmed K. Nontechnical skills in surgery: a systematic review of current training modalities. J Surg Educ. 2018;76(1):14–24. [DOI] [PubMed] [Google Scholar]
- 31.Dedy NJ, Bonrath EM, Zevin B, Grantcharov TP. Teaching nontechnical skills in surgical residency: a systematic review of current approaches and outcomes. Surgery. 2013;154(5):1001–8. [DOI] [PubMed] [Google Scholar]
- 32.Rosendal AA, Sloth SB, Rölfing JD, Bie M, Jensen RD, Technical. Non-technical, or both? A scoping review of skills in Simulation-Based Surgical training. J Surg Educ. 2023;80(5):731–49. [DOI] [PubMed] [Google Scholar]
- 33.Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32. [Google Scholar]
- 34.Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for scoping reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018;169:467–73. [DOI] [PubMed] [Google Scholar]
- 35.Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. Syst Reviews. 2016;5:210. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84(2):273–8. [DOI] [PubMed] [Google Scholar]
- 37.Zevin B, Bonrath EM, Aggarwal R, Dedy NJ, Ahmed N, Grantcharov TP, Development. Feasibility, validity, and reliability of a scale for Objective Assessment of Operative performance in laparoscopic gastric bypass surgery. J Am Coll Surg. 2013;216(5):955–e658. [DOI] [PubMed] [Google Scholar]
- 38.Yule S, Flin R, Maran N, Rowley D, Youngson G, Paterson-Brown S. Surgeons’ non-technical skills in the operating room: reliability testing of the NOTSS Behavior Rating System. World J Surg. 2008;32:548–56. [DOI] [PubMed] [Google Scholar]
- 39.Lee S, Das M, Yeow D. Survey on paediatric intubation experience at a district general hospital. Trends Anaesth Crit Care. 2020;30:e65–6. [Google Scholar]
- 40.Tsuda S, Scott D, Doyle J, Jones DB. Surgical skills training and simulation. Curr Probl Surg. 2009;46(4):271–370. [DOI] [PubMed] [Google Scholar]
- 41.Kemmis S, Wilkinson J, Edwards-Groves C, Hardy I, Grootenboer P, Bristol L. Changing practices, changing education. Volume 1. ed: Springer Science & Business Media; 2013. [Google Scholar]
- 42.Billett S. Learning through health care work: premises, contributions and practices. Med Educ. 2016;50(1):124–31. [DOI] [PubMed] [Google Scholar]
- 43.Jensen RD, Brydges R, Grierson L. Re-examining the integration of routine and adaptive expertise: there is no such thing as routine from a motor control perspective. Adv Health Sci Educ Theory Pract. 2022;27(5):1283–91. [DOI] [PubMed] [Google Scholar]
- 44.Gibson JJ. The ecological approach to visual perception. Hillsdale, NJ: Lawrence Erlbaum Associates; 1986. [Google Scholar]
- 45.Ziegler A, Stoeger H, Grassinger R. Actiotope model and self-regulated learning. Psychol Test Assess Model. 2011;53(1):161–79. [Google Scholar]
- 46.Stoeger H, Sontag C, Ziegler A. Impact of a teacher-led intervention on preference for self-regulated learning, finding main ideas in expository texts, and reading comprehension. J Educ Psychol. 2014;106(3):799–814. [Google Scholar]
- 47.Dresel M, Schmitz B, Schober B, Spiel C, Ziegler A, Engelschalk T, et al. Competencies for successful self-regulated learning in higher education: structural model and indications drawn from expert interviews. Stud High Educ. 2015;40(3):454–70. [Google Scholar]
- 48.Hatano G. Cognitive consequences of practice in Culture specific procedural skills. Q Newsl Lab Comp Hum Cognition. 1982;4(1):14–7. [Google Scholar]
- 49.Mylopoulos M, Regehr G. Cognitive metaphors of expertise and knowledge: prospects and limitations for medical education. Med Educ. 2007;41:1159–65. [DOI] [PubMed] [Google Scholar]
- 50.Kraiger K, Ford JK, Salas E. Application of cognitive, Skill-Based, and affective theories of learning outcomes to New methods of training evaluation. J Appl Psychol. 1993;78(2):311–28. [Google Scholar]
- 51.Furth HG. Piaget and knowledge: theoretical foundations. Chicago: University of Chicago Press; 1981. [Google Scholar]
- 52.Hatano G, Inagaki K. Two courses of expertise. In: Stevenson H, Azuma H, Hakuta K, editors. Child Development and Education in Japan. New York: WH Freeman; 1986. pp. 27–36. [Google Scholar]
- 53.Holland JR, Arnold DH, Hanson HR, Solomon BJ, Jones NE, Anderson TW, et al. Reliability of the behaviorally anchored rating scale (BARS) for assessing non-technical skills of medical students in simulated scenarios. Med Educ Online. 2022;27(1):e2070940. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Dearden E, Mellanby E, Cameron H, Harden J. Which non-technical skills do junior doctors require to prescribe safely? A systematic review. Br J Clin Pharmacol. 2015;80(6):1303–14. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Nicolaides M, Cardillo L, Theodoulou I, Hanrahan J, Tsoulfas G, Athanasiou T, et al. Developing a novel framework for non-technical skills learning strategies for undergraduates: A systematic review. Ann Med Surg. 2012;36:29–40. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 56.Hofmann R, Curran S, Dickens S. Models and measures of learning outcomes for non-technical skills in simulation-based medical education: findings from an integrated scoping review of research and content analysis of curricular learning objectives. Stud Educational Evaluation. 2021;71:e101093. [Google Scholar]
- 57.Johnson AP, Aggarwal R. Assessment of non-technical skills: why aren’t we there yet? BMJ Qual Saf. 2019;28:606–8. [DOI] [PubMed] [Google Scholar]
- 58.Pandey VA, Wolfe JHN, Black SA, Cairols M, Liapis CD, Bergqvist D. Self-assessment of technical skill in surgery: the need for expert feedback. Ann R Coll Surg Engl. 2008;90:286–90. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59.Nayar SK, Musto L, Baruah G, Fernandes R, Bharathan R. Self-Assessment of Surgical skills: a systematic review. J Surg Educ. 2019;77(2):348–61. [DOI] [PubMed] [Google Scholar]
- 60.Eva KW, Hatala RM, LeBlanc VR, Brooks LR. Teaching from the clinical reasoning literature: combined reasoning strategies help novice diagnosticians overcome misleading information. Med Educ. 2007;41:1152–8. [DOI] [PubMed] [Google Scholar]
- 61.Anton NE, Beane J, Yurco AM, Howley LD, Bean E, Myers EM, et al. Mental skills training effectively minimizes operative performance deterioration under stressful conditions: results of a randomized controlled study. Am J Surg. 2018;215(2):214–21. [DOI] [PubMed] [Google Scholar]
- 62.Anton NE, Howley LD, Pimentel M, Davis CK, Brown C, Stefanidis D. Effectiveness of a mental skills curriculum to reduce novices’ stress. J Surg Res. 2016;206(1):199–205. [DOI] [PubMed] [Google Scholar]
- 63.Anton NE, Mulji N, Howley LD, Yurco AM, Tobben D, Bean E, et al. Effects of a novel mental skills curriculum on surgical novices’ attention. J Surg Res. 2017;219:86–91. [DOI] [PubMed] [Google Scholar]
- 64.LaDonna KA, Ginsburg S, Watling C. Rising to the level of your incompetence: what Physicians’ Self-Assessment of their performance reveals about the Imposter Syndrome in Medicine. Acad Med. 2018;93(5):763–8. [DOI] [PubMed] [Google Scholar]
- 65.Kane M, Brennan R. Educational measurement. Validation. 4th ed. Westport: American Council on Education/Praeger; 2006. pp. 17–64. [Google Scholar]
- 66.Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: a practical guide to Kane’s framework. Med Educ. 2015;49:560–75. [DOI] [PubMed] [Google Scholar]
- 67.Gjeraa K, Jepsen R, Rewers M, Østergaard D, Dieckmann P. Exploring the relationship between anaesthesiologists’ non-technical and technical skills. Acta Anaesthesiol Scand. 2016;60(1):36–47. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The author confirms that all data generated or analysed during this study are included in this published review article.




