Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2024 Apr 1.
Published in final edited form as: J Thorac Cardiovasc Surg. 2021 Jul 2;165(4):1462–1469. doi: 10.1016/j.jtcvs.2021.06.052

A Tool to Assess Non-Technical Skills of Perfusionists in the Cardiac Operating Room

Roger D Dias 1,2, William Riley 3, Kenneth Shann 8, Donald S Likosky 4, David Fitzgerald 5, Steven Yule 6,7
PMCID: PMC8720321  NIHMSID: NIHMS1721135  PMID: 34261581

Structured Abstract

Objective(s):

This study aimed to develop the Perfusionists’ Intraoperative Non-Technical Skills (PINTS) tool, specifically to the perfusionists’ context, and test its inter-rater reliability.

Methods:

An expert panel was convened to review existing surgical non-technical skills taxonomies and develop the PINTS tool. During a workshop held at a national meeting, perfusionists completed the PINTS ratings after watching four videos displaying simulated cardiac operations. Two videos showed “good performance”, and two videos showed “poor performance”. Inter-rater reliability (IRR) analysis was performed and intraclass correlation coefficient (ICC) was reported.

Results:

The final version of the PINTS taxonomy contains 4 behavioral categories (decision making, situation awareness, task management and leadership, teamwork and communication) with 4 behavioral elements each. Categories and elements are rated using an 8-point Likert scale ranging from 0.5 to 4.0. A total of 60 perfusionist raters were included and the comparison between rating distribution on “poor performance” and “good performance” videos yielded a statistically significant difference between groups, with a p-value < 0.001. A similar difference was found in all behavioral categories and elements. Reliability analysis showed moderate IRR across overall ratings (ICC coefficient: 0.735; CI 95%: 0.674–0.796; p < 0.001). Similar IRR was found when raters were stratified by experience level.

Conclusions:

The PINTS tool presented moderate inter-rater reliability among perfusionists with varied levels of experience. This tool can be used to train and assess perfusionists in relevant non-technical skills, with the potential to enhance safety and improve surgical outcomes.

Keywords: non-technical skills, cardiac surgery, perfusionist, patient safety, behavioral markers, teamwork, assessment

Graphical Abstract

graphic file with name nihms-1721135-f0001.jpg

INTRODUCTION

In the United States, it is estimated that more than 500,000 cardiac surgery procedures are performed annually, with most of these procedures involving cardiopulmonary bypass (CPB) through an extracorporeal perfusion system operated by a perfusionist.1,2 Over the past five decades, despite considerable improvements in patient safety, the incidence of preventable adverse events continues to be high in cardiac surgery, compared to other surgical specialties.3,4 Among the many factors that impact patient outcomes, intraoperative performance during CPB is strongly associated with both short- and long-term morbidity and mortality among patients undergoing cardiac surgery.5,6 During CPB, the perfusionist plays a critical role in optimizing patient physiologic support while, concurrently, coordinating other complex tasks with the surgical, anesthesia, and nursing subteams.7,8

In addition to having sufficient clinical knowledge and technical skills, perfusionists are required to simultaneously process a vast amount of information received from multiple sources (i.e. auditory, visual, and tactile). They also need to perform tasks under stressful conditions, and efficiently communicate with the cardiac team using closed-loop communication.9,10 While there may be standardization observed in the use of modern CPB circuit safety devices (e.g. line filters, bubble detectors, pressure monitoring, and servo-regulation controls), significant variation still exists in the conduct and clinical management of CPB support.11,12 Such variation may be associated with perfusionist-related errors that impact patients safety and surgical outcomes in the cardiac operating room (OR).13,14 Non-technical skills are cognitive (e.g. decision making, situation awareness) and interpersonal (e.g. teamwork, communication) skills that play a crucial role in clinical performance in several healthcare settings. There is extensive previous literature showing that non-technical skills are as important as technical skills to achieving optimal surgical care.15 Furthermore, large studies have shown that most medical errors and preventable adverse events in the OR occur more as a result of poor non-technical skills rather than deficiencies in medical knowledge or technical skills.16,17 Effective communication, for example, is essential in the cardiac OR and previous studies have shown that surgeon-perfusionist communication, although very frequent, presents lack of systematization and homogeneity which may compromise patient safety.18–20

Previous studies have developed and validated behavioral taxonomies to assess and train non-technical skills of OR providers, such as the Non-technical Skills for Surgeons (NOTSS),21 the Anesthetists’ Non-Technical Skills (ANTS),22 and the Scrub Practitioners’ List of Intraoperative Non-Technical Skills (SPLINTS).23 Nonetheless, no behavioral taxonomy has been developed specifically to assess perfusionists’ intraoperative non-technical skills.24 This study aimed to develop a non-technical skills taxonomy specific for perfusionists, gather validity evidence and assess inter-rater reliability in the context of perfusionists’ intraoperative practice. We aligned our validity and reliability testing with the proposed use of the tool, to provide formative assessment of perfusionists from videos of clinical encounters. Our intended product is an observation-based assessment tool named Perfusionists’ Intraoperative Non-Technical Skills (PINTS).

METHODS

Study Design and Setting

This study involved two sequential phases. In phase 1, an expert panel was assembled to (i) review the current literature on non-technical skills in the OR, (ii) adapt a behavioral taxonomy to the perfusionist’s context, and (iii) generate an initial assessment tool for testing. Subject-matter experts met both in-person and remotely via videoconferencing. Phase 2 involved the observation and rating of simulated cardiac surgery videos by perfusionists during a dedicated 75-minute workshop at the 58th International Conference of the American Society for Extracorporeal Technology (AmSECT) that occurred in St. Louis, MO on March 7th, 2020. This study was approved by the Partners IRB (Protocol #: 2020P000159, Approval Date: 02/06/2020) and a consent procedure was embedded onto an electronic survey.

Population

In phase 1, subject-matter experts included 4 senior perfusionists (WR, DF, KS, DF), 1 organizational psychologist and human factors scientist (SY), 1 physician-scientist and human factors scientist (RDD), and 1 cardiovascular epidemiologist (DL). In phase 2, participants were perfusionists from a wide range of expertise, who were attending the 58th International Conference of the American Society for Extracorporeal Technology (AmSECT).

Procedures

The expert panel reviewed three OR non-technical skills taxonomies: Non-Technical Skills for Surgeons (NOTSS),21 Anesthetists’ Non-Technical Skills (ANTS),22 and Scrub Practitioners’ List of Intraoperative Non-Technical Skills (SPLINTS)23, in addition to existing studies involving human factors and cardiothoracic surgery.25,26 Through synchronous (in-person and videoconferencing) and asynchronous (online collaborative platform) discussions, the panel generated an initial list of perfusion-related behavior categories and corresponding behavior elements. After multiple iterations and by consensus, a final non-technical skills taxonomy was developed (PINTS tool). The initial version of the PINTS tool contained 4 behavioral categories and 27 behavioral elements. After three iterations within the expert panel, a final version was developed, containing 4 categories and 16 elements. Each category and element are rated using an 8-point Likert scale ranging from 0.5 to 4.0 (Figure 1). This scale was selected to allow a wide range of behavior ratings to be provided without the option of a neutral choice.

Figure 1.

Figure 1

Open in a new tab

The Perfusionists’ Intraoperative Non-Technical Skills (PINTS) tool. The left column shows behavioral categories, and the right column shows the behavioral elements within each category. The scale in the bottom shows the 8-point anchoring system used for assessment.

In phase 2, all attendees of the conference workshop were invited to participate in an online survey created using the Research Electronic Data Capture (REDCap) application. The first 15 minutes of the workshop consisted in a lecture addressing non-technical skills in surgery and behavioral marker systems. After agreeing with the consent statement, participants answered two demographic questions and were prompted to complete the PINTS tool immediately after observing each of four simulated cardiac surgery videos. These videos display in-situ high-fidelity cardiac surgery simulations and were produced in 2013 by the American Society of Extracorporeal Technology (AmSECT) and the International Consortium for Evidence--Based Perfusion (ICEBP) as part of the Cardiac Surgical Team Training Video Project. Two videos were designed and scripted to demonstrate “good performance” and two videos to demonstrate “poor performance” of the cardiac surgery team. See Online Supplement containing links to the videos followed by an analysis by a human factors expert (SY) on the specific behaviors (poor and good) displayed in each video.

Data Analysis

All data collected in REDCap was exported in .csv format and analyzed in the statistical software SPSS Statistics 24.0. Data was tested for normality with the Kolmogorov-Smirnov test and since the data distribution was not normal, continuous variables were reported as median and 1st – 3rd interquartile (IQ). Categorical variables were reported as absolute numbers and percentages. Inter-rater reliability analysis was performed using the single measures intraclass correlation estimated by an alpha two-way random model with consistency type. The intraclass correlation coefficient (ICC) was reported with a 95% confidence interval (95% CI). The following references were used to interpret ICC: fair (ICC: 0.50–0.75), good (ICC: 0.75–0.90), and excellent (ICC:>0.90) reliability.27 The Related-Samples Wilcoxon Signed Rank Test was used to compare distribution of behavioral element ratings between poor and good videos. Statistical significance was determined by a p-value of less than 0.05.

RESULTS

A total of 100 participants completed ratings of at least one video. Participants who were not perfusionists (N = 10) and those who did not complete ratings of all four videos (N = 30) were excluded from the analysis. Our final sample included 60 perfusionists, who rated four cardiac surgery simulations using the PINTS tool. The experience in years of these raters is shown in Figure 2.

Figure 2.

Figure 2

Open in a new tab

Distribution of perfusionist raters as absolute number (N) and percentage (%) according to experience in years.

For each video, raters provided scores for 16 behavioral elements and 4 behavioral categories. Twenty ratings per video per rater generated a total of 4,800 ratings. Figure 3 displays the distribution of ratings across the 8-point scale within each behavioral category. Figure 4 shows the variation in rating distribution across different levels of raters’ experience.

Figure 3.

Figure 3

Open in a new tab

Distribution of ratings (percentage) across all possible scores (0.5 to 4.0) for each behavioral category: decision making, situation awareness, task management and leadership, and teamwork and communication.

Figure 4.

Figure 4

Open in a new tab

Heatmap showing the distribution of ratings (percentage) based on raters’ experience in years. The frequency increases in the color spectrum from yellow to red.

In Table 1, the inter-rater reliability measures are displayed for the overall ratings, and stratified by raters’ experience level. Table 2 shows the measures for each behavioral category. The comparison in rating distribution between the “poor performance”, median =1.0 (1st IQ: 0.5 – 3rd IQ: 2.5),and the “good performance”, 3.5 (1st IQ: 3.0 – 3rd IQ: 4.00) videos, for all behavioral elements, across all raters, yielded a statistically significant difference (test statistic = 39.4, p-value < 0.001) between groups. A similar difference was found in all behavioral categories and elements (Figure 5). A graphical abstract summarizing the study is provided in Figure 6.

Table 1.

Inter-rater reliability of the PINTS tool for each experience level

Rater’s Experience ICC 95% CI p-value
Overall 0.735 0.674 – 0.796 < 0.001
< 5 years 0.729 0.663 – 0.794 < 0.001
5–10 years 0.671 0.591 – 0.749 < 0.001
11–15 years 0.802 0.744 – 0.854 < 0.001
> 15 years 0.726 0.663 – 0.789 < 0.001
Open in a new tab

Table 2.

Inter-rater reliability of the PINTS tool for each behavioral category

Rater’s Experience ICC 95% CI p-value
Decision Making 0.757 0.641 – 0.870 < 0.001
Situational Awareness 0.746 0.627 – 0.863 < 0.001
Task Management and Leadership 0.767 0.653 – 0.876 < 0.001
Teamwork and Communication 0.691 0.561 – 0.828 < 0.001
Open in a new tab

Figure 5.

Figure 5

Open in a new tab

Comparison between “poor performance” and “good performance” videos regarding the distribution of ratings. The boxplots display median, 1st and 3rd quartiles, minimum and maximum values.

Figure 6.

Figure 6

Open in a new tab

Graphical Abstract displaying the study methods involving an expert panel to develop the Perfusionists’ Intraoperative Non-Technical Skills (PINTS) tool; the main findings involving 60 perfusionists responding to an online survey after watching 4 simulated cardiac surgery videos; the comparison between videos showing good and poor behavior’s; and the take-home message highlighting the potentials uses of the PINTS tool..

DISCUSSION

In this study, we have developed a non-technical skills taxonomy and assessment tool specific for perfusionists. Our findings demonstrate that the PINTS tool presents a moderate inter-rater reliability, consistent across different levels of clinical experience and different domains of non-technical skills. Additionally, the study results show that the PINTS tool is able to differentiate between poor and good non-technical skills in simulated cardiac surgery videos, corroborating its construct validity.

There is an extensive body of knowledge on the development and testing of non-technical skills assessment tools specifically designed to the OR context.28,29 In a recent systematic review of the literature, Higham et al. reported a total of 25 existing tools that assess non-technical skills of OR providers, either as individuals or teams.24 From these studies, only one instrument, the Pediatric Cardiac Surgery Teamwork classification tool, included perfusionists as observed participants. However, this tool uses the same behavioral taxonomy to assess all four cardiac surgery subteams.30 Findings from previous studies that have created or adapted non-technical skills taxonomies across different clinical settings and roles suggest that profession-specific context needs to be considered.24 Compared with other surgical non-technical skills taxonomies (e.g. NOTSS, ANTS and SPLINTS), the PINTS tool presents five elements that are new: a) discussing patient existing conditions and treatments; b) demonstrating spatial and system awareness; c) participating in team decision making; d) using checklists; and e) transition care (handoffs). The PINTS tool, although grounded by previous surgical behavioral marker systems, intends to capture context-specific behaviors of individual perfusionists, as they play a crucial role as part of the cardiac surgery team.10,26,31 Our results showed that within all four behavioral categories, all scores (from 0.5 to 4.0) were frequently used by raters, although the poorest score (0.5) was used more often (between 18% and 23.2%) than reported by other studies evaluating surgeon’s non-technical skills.32

One of the main limitations of non-technical skills assessment tools in general is the need for standardized training of raters to achieve acceptable inter-rater reliability.33 In fact, previous studies using the NOTSS (for surgeons), ANTS (for anesthesiologists), and SPLINTS (for scrub nurses) tools have found moderate to good inter-rater reliability and requiring extensive rater’s training to achieve excellent levels.21–23 In our study, only a 15-minute lecture was provided to raters and, despite this minimum training, the inter-rater reliability of the PINTS tool was moderate and consistent across four groups of raters stratified based on their years of experience as perfusionists. Interestingly, we found that although junior perfusionist raters (< 5 years of experience) produce similar inter-rater reliability as senior perfusionist raters, they tend to use lower scores to assess the simulated videos for positive and negative behaviors. To our knowledge, no previous study has explored the differences between junior and senior raters in assessing non-technical skills in surgery. Consistent with previous studies investigating the construct validity of observation-based assessment tools in the OR,34,35 our study yielded a significant difference in the PINTS ratings when comparing perfusionist’s non-technical skills in simulated cardiac surgery videos showing different levels of performance.

This study has some limitations that are important to be highlighted. First, concerning potential selection bias and external validity, the raters who used the PINTS tool do not necessarily represent the general population of perfusionists, and the fact that they decided to participate in the workshop may indicate that they are already involved with non-technical skills training or assessments. This may explain why we achieved excellent inter-rater reliability even with a very brief rating training. To minimize this risk, we stratified the analysis by years of experience and showed that even junior raters, which are less likely to have extensive training in non-technical skills, also achieved moderate inter-rater reliability. Second, we used only a limited number of simulated videos of cardiac surgery teams performing a brief segment of the procedure, and additional studies should incorporate more surgical steps for non-technical skills assessment. Additionally, we did not control for the fact that some perfusionists may have already watched the videos in the past; therefore, they may have a different rating compared to first-time viewers. Another limitation is that we did not use an external expert group to further validate the item selection in this initial tool. This limitation may have compromised the item validity of the PINTS tool, and future studies should have an external validation of these items and conduct exploratory factor analysis to determine the better performing items.

Perfusionists are essential members of the cardiac surgery team and their non-technical skills play an imperative role in ensuring patient safety in the OR. The PINTS taxonomy and rating instrument developed and examined in this study can be applied for training and assessment of perfusionists’ non-technical skills for both research and quality improvement initiatives. As an assessment tool, the data gathered from applying this instrument to both novice and senior perfusionists, either in the clinical simulation setting or in the OR, can be used to guide competence-based educational curriculum in surgical non-technical skills, informing knowledge and skills progression, as well as need for remediation. Both ratings and behaviors annotations based on the PINTS taxonomy can be used to facilitate constructive feedback and coach perfusionists towards performance improvement. As per research and quality improvement initiatives, the PINTS tool can be used to systematically assess non-technical behaviors in the OR. The information provided may support initiatives intended to investigate factors that impact quality of care and patient safety in cardiac surgery that extend beyond traditional measures such as pre-operative risk score, institutional case volume and complication rates.

Supplementary Material

Supp.materils

Central Picture Legend

PINTS ratings for “poor performance” and “good performance” simulated cardiac videos.

Central Message

An observation-based behavioral taxonomy, the Perfusionists’ Intraoperative Non-Technical Skills (PINTS), was developed and validated, presenting moderate inter-rater reliability among perfusionists.

Perspective Statement.

Intraoperative non-technical skills, such as decision-making, teamwork, leadership, and communication play a critical role in the cardiac operating room. This study validated the first non-technical skills taxonomy, specifically for the perfusionist context. This tool can be used to train and assess perfusionists in such relevant skills, with the potential to enhance safety and improve surgical outcomes.

Acknowledgments

SOURCE OF FUNDING:

This study was approved by the Partners IRB (Protocol #: 2020P000159, Approval Date: 02/06/2020) and a consent procedure was embedded onto an electronic survey.

Glossary of Abbreviations:

AmSECT

American Society for Extracorporeal Technology

ANTS

Anaesthetists’ non-technical skills

CPB

cardiopulmonary bypass

DM

Decision-making

ICC

Intraclass correlation

ICEBP

International Consortium for Evidence--Based Perfusion

NOTSS

Non-technical skills for surgeons

OR

Operating Room

PINTS

Perfusionists’ Intraoperative Non-Technical Skills

REDCap

Research Electronic Data Capture

SA

Situation awareness

SPLINTS

Scrub practitioners’ list of intraoperative non-technical skills

TML

Task management and leadership

TWC

Teamwork and communication

Footnotes

CONFLICT OF INTEREST:

RD Dias, W Riley, D FitzGerald, D. Fitzgerald, and K. Shann have no conflicts of interest. DS Likosky is a consultant to AmSECT. Dr. Likosky reports funding from the AHRQ and NHLBI

S Yule is a member of the Global Education Council, Johnson and Johnson Institute This project was funded by the National Heart, Lung, and Blood Institute (NHLBI; R01HL146619).

The videos used in this study were produced in 2013 by the American Society of Extracorporeal Technology (AmSECT) and the International Consortium for Evidence--Based Perfusion (ICEBP) as part of the Cardiac Surgical Team Training Video Project. The scenarios scripts were created by a multidisciplinary group composed by cardiac surgery experts (surgeons, anesthesiologists and perfusionists) and human factors/medical simulation experts to demonstrate specific types of behaviors in the operating room. 27

The videos are available on YouTube through the links below:

Videos scripted to display poor behaviors:

Dissection: https://youtu.be/-Lq6nIvviy0

On CPB-Poor: https://youtu.be/KuMHC9KxgTc

Videos scripted to display good behaviors:

On CPB-Improved: https://youtu.be/Fj2nlbmQl1l

Improved Briefing: https://youtu.be/l98D6izEajA

Source: AmSECT : Simulation Videos. Accessed October 14, 2020. https://www.amsect.org/page/simulation-videos-1501

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

REFERENCES

  • 1.Epstein AJ, Polsky D, Yang F, Yang L, Groeneveld PW. Coronary revascularization trends in.the United States, 2001–2008. JAMA. 2011;305: 1769–1776. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Alkhouli M, Alqahtani F, Kalra A, Gafoor S, Alhajji M, Alreshidan M, et al. Trends in Characteristics and Outcomes of Patients Undergoing Coronary Revascularization in the United States, 2003–2016. JAMA Netw Open. 2020;3: e1921326. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Cooley DA, Frazier OH. The past 50 years of cardiovascular surgery. Circulation. 2000;102: IV87–93. [DOI] [PubMed] [Google Scholar]
  • 4.Melly L, Torregrossa G, Lee T, Jansens J-L, Puskas JD. Fifty years of coronary artery bypass grafting. J Thorac Dis. 2018;10: 1960–1967. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Salis S, Mazzanti VV, Merli G, Salvi L, Tedesco CC, Veglia F, et al. Cardiopulmonary bypass duration is an independent predictor of morbidity and mortality after cardiac surgery. J Cardiothorac Vasc Anesth. 2008;22: 814–822. [DOI] [PubMed] [Google Scholar]
  • 6.Axtell AL, Fiedler AG, Melnitchouk S, D’Alessandro DA, Villavicencio MA, Jassar AS, et al. Correlation of cardiopulmonary bypass duration with acute renal failure after cardiac surgery. J Thorac Cardiovasc Surg. 2019. doi: 10.1016/j.jtcvs.2019.01.072 [DOI] [PubMed] [Google Scholar]
  • 7.Baker RA, Bronson SL, Dickinson TA, Fitzgerald DC, Likosky DS, Mellas NB, et al. Report from AmSECT’s International Consortium for Evidence-Based Perfusion: American Society of Extracorporeal Technology Standards and Guidelines for Perfusion Practice: 2013. J Extra Corpor Technol. 2013;45: 156–166. [PMC free article] [PubMed] [Google Scholar]
  • 8.Dias RD, Zenati MA, Conboy HM, Clarke LA, Osterweil LJ, Avrunin GS, et al. Dissecting Cardiac Surgery: A Video-Based Recall Protocol to Elucidate Team Cognitive Processes in the Operating Room. Ann Surg. 2019. doi: 10.1097/SLA.0000000000003489 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Wiegmann D, Suther T, Neal J, Parker SH, Sundt TM . A human factors analysis of cardiopulmonary bypass machines. J Extra Corpor Technol. 2009;41: 57–63. [PMC free article] [PubMed] [Google Scholar]
  • 10.Dias RD, Conboy HM, Gabany JM, Clarke LA, Osterweil LJ, Avrunin GS, et al. Development of an Interactive Dashboard to Analyze Cognitive Workload of Surgical Teams During Complex Procedural Care. IEEE Int Interdiscip Conf Cogn Methods Situat Aware Decis Support. 2018;2018: 77–82. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Ali JM, Miles LF, Abu-Omar Y, Galhardo C, Falter F. Global Cardioplegia Practices: Results from the Global Cardiopulmonary Bypass Survey. J Extra Corpor Technol. 2018;50: 83–93. [PMC free article] [PubMed] [Google Scholar]
  • 12.Likosky DS, Baker RA, Newland RF, Paugh TA, Dickinson TA, Fitzgerald D, et al. Is Conventional Bypass for Coronary Artery Bypass Graft Surgery a Misnomer? J Extra Corpor Technol. 2018;50: 225–230. [PMC free article] [PubMed] [Google Scholar]
  • 13.Svenmarker S, Appelblad M. Reporting of perfusion-related incidents: pitfalls and limitations. Perfusion. 2005;20: 243–248. [DOI] [PubMed] [Google Scholar]
  • 14.Charrière J-M, Pélissié J, Verd C, Léger P, Pouard P, de Riberolles C, et al. Survey: retrospective survey of monitoring/safety devices and incidents of cardiopulmonary bypass for cardiac surgery in France. J Extra Corpor Technol. 2007;39: 142–57; discussion 158–9. [PMC free article] [PubMed] [Google Scholar]
  • 15.Hull L, Arora S, Aggarwal R, Darzi A, Vincent C, Sevdalis N. The impact of nontechnical skills on technical performance in surgery: a systematic review. J Am Coll Surg. 2012;214: 214–230. [DOI] [PubMed] [Google Scholar]
  • 16.Stefl ME. To Err is Human: Building a Safer Health System in 1999. Frontiers of health services management. 2001. pp. 1–2. [PubMed] [Google Scholar]
  • 17.Youngson GG. Intraoperative performance, non-technical skills and surgical safety. Enhancing Surgical Performance. CRC Press; 2015. pp. 3–16. [Google Scholar]
  • 18.Flin R, Youngson GG, Yule S. Enhancing Surgical Performance: A Primer in Non-technical Skills. CRC Press; 2015. [Google Scholar]
  • 19.Santos R, Bakero L, Franco P, Alves C, Fragata I, Fragata J. Characterization of non-technical skills in paediatric cardiac surgery: communication patterns. Eur J Cardiothorac Surg. 2012;41: 1005–12; discussion 1012. [DOI] [PubMed] [Google Scholar]
  • 20.Hazlehurst B, McMullen CK, Gorman PN. Distributed cognition in the heart room: how situation awareness arises from coordinated communications during cardiac surgery. J Biomed Inform. 2007;40: 539–551. [DOI] [PubMed] [Google Scholar]
  • 21.Lawson C, Predella M, Rowden A, Goldstein J, Sistino JJ, Fitzgerald DC. Assessing the culture of safety in cardiovascular perfusion: attitudes and perceptions. Perfusion. 2017;32: 583–590. [DOI] [PubMed] [Google Scholar]
  • 22.Yule S, Flin R, Maran N, Rowley D, Youngson G, Paterson-Brown S. Surgeons’ non-technical skills in the operating room: reliability testing of the NOTSS behavior rating system. World J Surg. 2008;32: 548–556. [DOI] [PubMed] [Google Scholar]
  • 23.Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R. Anaesthetists’ Non-Technical Skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth. 2003;90: 580–588. [DOI] [PubMed] [Google Scholar]
  • 24.Mitchell L, Flin R, Yule S, Mitchell J, Coutts K, Youngson G. Development of a behavioural marker system for scrub practitioners’ non-technical skills (SPLINTS system). J Eval Clin Pract. 2013;19: 317–323. [DOI] [PubMed] [Google Scholar]
  • 25.Higham H, Greig PR, Rutherford J, Vincent L, Young D, Vincent C. Observer-based tools for non-technical skills assessment in simulated and real clinical environments in healthcare: a systematic review. BMJ Qual Saf. 2019;28: 672–686. [DOI] [PubMed] [Google Scholar]
  • 26.Wahr JA, Prager RL, Abernathy JH 3rd, Martinez EA, Salas E, Seifert PC, et al. Patient safety in the cardiac operating room: human factors and teamwork: a scientific statement from the American Heart Association. Circulation. 2013;128: 1139–1169. [DOI] [PubMed] [Google Scholar]
  • 27.Wilson JL, Whyte RI, Gangadharan SP, Kent MS. Teamwork and Communication Skills in Cardiothoracic Surgery. Ann Thorac Surg. 2017;103: 1049–1054. [DOI] [PubMed] [Google Scholar]
  • 28.Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016;15: 155–163. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Agha RA, Fowler AJ, Sevdalis N. The role of non-technical skills in surgery. Ann Med Surg (Lond). 2015;4: 422–427. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Schraagen JM, Schouten T, Smit M, Haas F, van der Beek D, van de Ven J, et al. Assessing and improving teamwork in cardiac surgery. Qual Saf Health Care. 2010;19: e29. [DOI] [PubMed] [Google Scholar]
  • 31.Dias RD, Zenati MA, Stevens R, Gabany JM, Yule SJ. Physiological synchronization and entropy as measures of team cognitive load. J Biomed Inform. 2019;96: 103250. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Crossley J, Marriott J, Purdie H, Beard JD. Prospective observational study to evaluate NOTSS (Non-Technical Skills for Surgeons) for assessing trainees’ non-technical performance in the operating theatre. Br J Surg. 2011;98: 1010–1020. [DOI] [PubMed] [Google Scholar]
  • 33.Johnson AP, Aggarwal R. Assessment of non-technical skills: why aren’t we there yet? BMJ quality & safety. 2019. pp. 606–608. [DOI] [PubMed] [Google Scholar]
  • 34.Yule S, Gupta A, Gazarian D, Geraghty A, Smink DS, Beard J, et al. Construct and criterion validity testing of the Non-Technical Skills for Surgeons (NOTSS) behaviour assessment tool using videos of simulated operations. Br J Surg. 2018;105: 719–727. [DOI] [PubMed] [Google Scholar]
  • 35.Jirativanont T, Raksamani K, Aroonpruksakul N, Apidechakul P, Suraseranivongse S. Validity evidence of non-technical skills assessment instruments in simulated anaesthesia crisis management. Anaesth Intensive Care. 2017;45: 469–475. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supp.materils
NIHMS1721135-supplement-Supp_materils.docx (14.7KB, docx)

RESOURCES