Abstract
BACKGROUND
Methods to assess competency in cardiothoracic training are essential. Here, we report a system that allows us to better assess competency from the perspective of both the trainee and educator. We hypothesized that postprocedural cognitive burden measurement (by the trainee) with immediate feedback (from the educator) could aid in identifying barriers to the acquisition of skills and knowledge so that training curricula can be individualized.
METHODS
The National Aeronautics and Space Administration Task Load Index (NASA-TLX), a validated instrument to measure cognitive load, was administered with an online platform after bronchoscopy, esophagogastroduodenoscopy, and video-assisted thoracoscopic surgery for 11 residents. Immediate postprocedure feedback and standardized debriefing occurred for each procedure.
RESULTS
Mean NASA-TLX scores were highest (indicating greater cognitive load) for esophagogastroduodenoscopy and video-assisted thoracoscopic surgery (P < .001). When comparing subscale measures, mental demand was significantly higher for video-assisted thoracoscopic surgery (P = .026) compared with the other procedures, whereas physical demand was highest for esophagogastroduodenoscopy (P = .018). Self-reported frustration was similar for all case types (P = .247). Cognitive burden decreased with a greater number of procedures for bronchoscopy (P = .027). Significant improvement was noted by the trainee at the end of the rotation in self-assessed procedural competency and preparedness for thoracic board topics (all P < .05). Postprocedure feedback by the attending surgeon correlated with more frequent completion of self-evaluations by the residents.
CONCLUSIONS
Longitudinal assessment of cognitive load in combination with postprocedural feedback identified barriers to skill acquisition for both residents and educators. This information allows for individualized rotation development as a step toward a competency-based curriculum.
The transition to competency-based metrics for surgical advancement has prompted a surge of interest in refining rotation curricula to maximize skill and knowledge acquisition.1,2 That interest is underscored by the knowledge that many residents will not perform the number of procedures required to reach competency during their training, and that each resident may require individualized attention for certain skills.3,4 Thoracic surgery is a subspecialty that is still part of the American Board of Surgery qualifying examination, and therefore is important for both cardiothoracic and general surgery trainees.5 Despite that, general surgery residents are logging fewer thoracic procedures, with senior residents reporting a low comfort level performing thoracotomy and video-assisted thoracic surgery (VATS) procedures.6–8 In addition, with limitations to duty hours and the number of procedures that can be performed within this time, there is a shift to assessing competency rather than assuming competency based on the number of procedures performed. Therefore, there is a need to maximize procedural and knowledge acquisition and tailor it to the needs of a trainee.
Prior studies have examined cognitive load theory as means to enhance graduate medical education and design curricula.9 The concept of cognitive load suggests that the human brain has a limited capacity for working memory and thus additional stressors may impede learning.9 Cognitive load can be divided into intrinsic load, extraneous load, and germane load, and is particularly relevant to the mastery of surgical procedures that require complex higher-level thinking in addition to technical skill performance (Figure 1).9 Cognitive theory in medical education has commonly focused on the volume of information expected to be memorized in medical school and subsequent training.10 However, because procedural training combines physical skills with mental demand, we hypothesized that postprocedural cognitive burden measurement, along with immediate feedback by an attending surgeon, could aid in identifying barriers to the acquisition of skills and knowledge. A novel system such as this would allow educators and trainees to not only recognize which procedures are most challenging but also allow for educators to individualize a training curriculum.
FIGURE 1.
Schematic of the cognitive load theory including examples using thoracic surgery skills and listing of subscales from the National Aeronautics and Space Administration Task Load Index (NASA-TLX) measure of cognitive load.9
The National Aeronautics and Space Administration Task Load Index (NASA-TLX) is a validated instrument developed to assess cognitive load and has been applied extensively to surgical education.11–13 The NASA-TLX is a very commonly utilized instrument in surgical training, although not with immediate two-perspective feedback with the intention to understand individual resident competency.11 Increased cognitive load has been correlated with longer operative time and higher blood loss as well as a greater likelihood of errors.13,14 The objective of this study was to utilize a novel online platform with the NASA-TLX to measure procedure-specific cognitive burden from surgical residents, along with standardized immediate feedback by an attending surgeon, to understand barriers to knowledge acquisition and procedural competency. Here, we present the information that has guided a competency-based curriculum that is individualized for trainees. This is a descriptive account of an educational initiative.
MATERIAL AND METHODS
SETTING AND PARTICIPANTS.
In this pilot program, second year general surgery residents (n = 11) from an academic residency program rotating through a 5-week thoracic surgery rotation at the Veterans Affairs (VA) Pittsburgh Healthcare System between 2018 and 2019 were included. They were proctored by one surgeon (R.D.). This program was supported by a 1-year educational grant. Ethical review and approval were waived for this study by the VA Pittsburgh Healthcare System because it was an education initiative and did not constitute research.
RESIDENT SELF-ASSESSMENT OF COMPETENCY BEFORE AND AFTER ROTATION.
Assessments were administered utilizing the online MedHub platform (MedHub, Minneapolis, MN). Residents completed before and after rotation assessments of self-reported competence in performing each of three key thoracic surgery procedures: bronchoscopy, esophagogastroduodenoscopy (EGD), and video-assisted thoracoscopic surgery (VATS). For VATS, the curriculum focused on patient positioning, identifying anatomic landmarks, and safely entering the thoracic cavity; the subsequent operation (ie, lung resection, esophagectomy, etc.) was not included as a component of the curriculum. The EGD curriculum focused on setting up the equipment, successfully entering the esophagus, navigating and fully inspecting the esophagus, stomach, and duodenum, and retroflexion with examination of the gastroesophageal junction. The bronchoscopy curriculum focused on setting up the equipment, successfully identifying each lobar bronchus, suctioning mucous successfully, and positioning a double-lumen tube. Self-assessment of competence was rated by the trainee on a scale of 1 to 5, with 1 indicating “poor” and 5 indicating “excellent” (Supplemental Table 1). The trainees also rated their perceived competence with thoracic surgery board topics. Self-assessed competency was then compared at the beginning and end of the rotation. Curriculum material was provided to all residents through an online platform (Microsoft OneNote 2016; Microsoft, Redmond, WA); the material included procedural instruction videos, primary literature, and review articles on relevant content (Supplemental Figure 1).
COGNITIVE LOAD ASSESSMENT.
After completion of the procedure (ie, bronchoscopy, EGD, VATS), residents were asked to complete the NASA-TLX instrument available in MedHub. The NASA-TLX instrument is composed of six subscales: mental demand, physical demand, temporal demand, overall performance, effort, and frustration level (Table 1). In addition, the faculty surgeon completed a resident evaluation for each procedure that rated the resident on (1) understanding the indications for the procedure; (2) proper equipment setup; (3) understanding the anatomy; (4) safe conduct of the procedure; and (5) overall ability to perform the procedure independently (Supplemental Table 2). That was available online by mobile device and generally took less than 30 seconds to complete; it was done in the presence of the resident with immediate verbal feedback at the end of the procedure.
TABLE 1.
National Aeronautics and Space Administration Task Load Index Scale
| Subscalea | Description |
|---|---|
| Mental demand | How much mental and perceptual activity was required? Was the task easy or demanding, simple or complex? |
| Physical demand | How much physical activity was required? Was the task easy or demanding, slack or strenuous? |
| Temporal demand | How much time pressure did you feel due to the pace at which the tasks or task elements occurred? Was the pace slow or rapid? |
| Overall performance | How successful were you in performing the task? How satisfied were you with your performance? |
| Effort | How hard did you have to work (mentally and physically) to accomplish your level of performance? |
| Frustration level | How irritated, stressed, and annoyed versus content, relaxed, and complacent did you feel during the task? |
Each subscale is rated from 0 to 10, with 0 being “very low” and 10 being “very high.”
ANALYSIS OF OUTCOMES.
Before and after rotation assessments were compared using the Wilcoxon test for paired, nonparametric data. The Kruskal-Wallis test with Dunn’s multiple comparisons was utilized to compare before and after rotation competency scores and NASA-TLX subscale scores among the three procedures. To compare trends in total NASA-TLX scores in relation to the number of procedures performed, the Friedman repeated measures analysis of variance test was utilized to compare residents completing at least four procedures. A two-tailed P value less than .05 was considered statistically significant. GraphPad Prism 8 (GraphPad Software, La Jolla, CA) was utilized for all statistical analyses.
RESULTS
ROTATION COMPETENCE.
Between 2018 and 2019, a total of 11 residents completed the curriculum. Self-reported competence increased significantly for all procedures as well as for boards preparedness when comparing before and after rotation assessments. Mean boards preparedness score increased from 1.4 before the rotation to 3.1 after the rotation (P = .008; Figure 2). During the rotation, residents completed evaluations for a median of 4 bronchoscopies (interquartile range [IQR], 2 to 5), 8 EGDs (IQR, 4 to 9), and 4 VATS (IQR, 3 to 9). Significant increases in self-reported competency were noted for all procedures: bronchoscopy (1.8 to 3.8, P = .008), EGD (1.3 to 2.8, P = .016), and VATS (1.2 to 2.8, P = .008). Of the three procedures, post-rotation competency to perform bronchoscopy was ranked significantly higher than VATS (P = .027).
FIGURE 2.
Resident self-reported competency before rotation (Pre [black bars]) and after rotation (Post [gray bars]) for bronchoscopy (Bronch), esophagogastroduodenoscopy (EGD), video-assisted thoracoscopic surgery (VATS), and preparation for the board examination (Boards).
OVERALL COGNITIVE LOAD.
Higher scores for the overall performance subscale are favorable; therefore, we compared the sum of the remaining five subscales to determine which components had the highest cognitive load for trainees. Increasing scores indicated greater cognitive load; scores were highest for EGD (27.0 ± 5.7) and VATS (25.9 ± 5.3) when either was compared with bronchoscopy (21.4 ± 7.7, P < .001). For residents who completed at least four procedures, we observed a significant decrease over time in overall cognitive load for bronchoscopy (P = .027; Figure 3). There appeared to be a decrease for EGD, although that was not statistically significant, whereas cognitive load scores for VATS appeared to be relatively constant regardless of the number of procedures.
FIGURE 3.
Trends in National Aeronautics and Space Administration Task Load Index (NASA-TLX) overall scores for (A) bronchoscopy, (B) esophagogastroduodenoscopy (EGD), and (C) video-assisted thoracoscopic surgery (VATS) with number of procedures. Error bars show standard error of the mean for all residents at associated procedure number. P values demonstrate results of Friedman repeated measures test for residents completing at least four procedures.
COGNITIVE LOAD SUBSCALES.
We explored subscale measures to improve our understanding of contributors to increased cognitive burden (Figure 4). We analyzed median scores over all procedures. Compared with bronchoscopy, mental demand was significantly higher for VATS (P = .026) whereas physical demand was significantly elevated for EGD (P = .018). In addition, overall performance, with increasing scores indicating favorable results, was significantly higher for bronchoscopy as compared with EGD (P = .017). There were no significant differences in self-reported frustration (P = .247), effort (P = .337), or temporal demand (P = .451) among the three procedure types.
FIGURE 4.
National Aeronautics and Space Administration Task Load Index (NASA-TLX) subscale scores for bronchoscopy (Bronch), esophagogastroduodenoscopy (EGD), and video-assisted thoracoscopic surgery (VATS). Subscale measures include (A) mental demand, (B) physical demand, (C) temporal demand, (D) overall performance, (E) effort, and (F) frustration. Bars demonstrate median with interquartile range.
COMMENT
The ever-increasing complexity of surgical care and procedures with a push for competency-based curricula necessitate new approaches to surgical education. Two-perspective evaluation (from trainee and faculty) may enhance the ability to individualize surgical education. In particular, subspecialty rotations such as thoracic surgery represent knowledge areas encompassed by the American Board of Surgery examination yet general surgery residents often have limited exposure to these fields.
In this study, we utilized the NASA-TLX scale to measure cognitive burden with three key thoracic surgery procedures—bronchoscopy, EGD, and VATS. We identified relatively persistent cognitive burden for VATS, whereas cognitive burden decreased for bronchoscopy and EGD over the course of a few procedures. Mental demand was significantly higher for VATS when compared with bronchoscopy, and there were no differences among the three procedures with regard to self-reported frustration. Therefore, cognitive load measurement uncovered barriers to achieving procedural competence and was informative to interventions that target educational needs of the residents. Specifically, the rotation has adjusted the focus and instruction toward VATS and EGD rather than bronchoscopy, for which competency was more readily and reliably attained with less cognitive burden. Additional time on simulators was encouraged and greater time was spent on instruction by the educator during these procedures. Because the resident is the sole person for the rotation described in the study, case load was not shifted. Beyond the adjustments made for this specific rotation, the value of this platform is that it can be tailored for any procedure on any rotation to improve the understanding of needs by both the learner and educator.
LONGITUDINAL ASSESSMENT OF RESIDENT LEARNING DURING THORACIC SURGERY ROTATION.
Beyond procedural competency, another benefit of this curriculum is the ability to compare before and after rotation self-assessments by the residents to assess confidence with thoracic surgery knowledge and skills. As expected, we observed a significant increase in resident-reported procedural and content competency by the end of the 5-week rotation. It is expected that residents will report increased confidence after intensive exposure to a subspecialty; however, there are few studies that quantify expected improvement. This observation can have several implications for educators, such as the ability to set concrete goals and provide benchmarks by which the effectiveness of instruction can be compared. In addition, quantification of resident-reported knowledge and skills acquisition specific to a subspecialty rotation might assist in identifying residents who struggle to meet clinical milestones, such as the completion of entrustable professional activities.15 With the move toward competency-based curricula in medical education, tools such as these can be used bidirectionally (ie, from learner and educator) to better understand whether competency is achieved. In addition, the magnitude of resident-reported improvement should be leveraged to improve training strategies and instructional approach.
IMPROVED UNDERSTANDING OF COGNITIVE BURDEN IN SKILL ACQUISITION.
Prior work has examined changes in cognitive burden associated with educational interventions, particularly with regard to surgical skills.16,17 Cognitive load measures have not yet been utilized to quantify progression of skill development during a rotation nor to differentiate among procedures that are unfamiliar to the learner, such as VATS (in this circumstance). We found that of the three procedures analyzed, VATS and EGD had higher cognitive burden when compared with bronchoscopy. Furthermore, the major contributors to excess cognitive burden differed between VATS and EGD.
A recent meta-analysis explored the use of cognitive load measurements in the surgical literature and acknowledged the potential benefits of identifying and addressing excessive cognitive load.11 The researchers point out that although measurement of cognitive load holds value in improving surgeon performance and patient safety, the threshold at which excess cognitive burden impacts performance has not been defined and may differ among surgeons. We specifically focused on surgical residents in an early stage of training as we anticipated that cognitive load would have a greater impact on learning, in contrast with the excess cognitive load experienced by an expert surgeon completing a difficult case, as an example. Such a distinction between novice and expert learners have previously been reported in multiple contexts.18,19 Therefore, we believe that cognitive load measurements uniquely expose intraoperative stressors among general surgery residents performing thoracic procedures.
APPLICATION OF COGNITIVE LOAD TO CURRICULUM DESIGN.
Our primary objective was to establish the feasibility of cognitive load measurement and explore potential differences among procedures. The next steps of this program involve tailored curriculum development based on procedure-specific cognitive burden. For example, VATS was associated with increased intrinsic and germane load that could be targeted by focused teaching efforts during these cases, supplemental review materials, and postprocedure video review.20,21 In addition, a recent study using the NASA-TLX correlated increased difficulty of a case with worse perceived performance and suggested a focus on preoperative planning as a means to reduce the cognitive burden.22 That represents an opportunity to reduce the intrinsic load of a VATS procedure and improve the learning experience. Conversely, the increased physical demand associated with EGD may be more amenable to simulation training.23,24 Indeed, simulation training has been associated with a decrease in cognitive load over time.14 Ultimately, a combination of instructional approaches to accommodate all learner types will be most effective, yet the results of our pilot study underscore that teaching surgical procedures can target procedure-specific obstacles. Focused efforts can refine educational approaches and individualize goals and curricula. For example, the authors found it surprising that endoscopy represented a consistently high cognitive burden, which may be specific to our institution, and will allow us to examine why a traditionally “easy” procedure is challenging to our trainees.
IMPORTANCE OF ATTENDING FEEDBACK.
We initially wanted to review the effects of standard feedback (ie, unstructured postprocedural discussion and completion of a generic end-of-rotation evaluation form) compared with immediate postprocedure feedback (ie, structured procedural debriefing with procedure assessment forms completed by the attending for each case) on cognitive load improvements during the rotation. We found early on, however, that resident self-assessment through completion of the NASA-TLX was minimal in the absence of regular faculty feedback. This finding highlights the importance of faculty engagement in resident self-improvement as well as curriculum development. Although feedback exists in many forms, studies have demonstrated the value of postoperative verbal feedback in improving resident performance.25–28 Further exploration into the role of stakeholder engagement in curriculum design may improve these new teaching and evaluation strategies.
STUDY LIMITATIONS.
This observational study has several limitations. We were unable to account for residents’ prior experience with these procedures, and therefore different levels of exposure to either these same procedures or other similar procedures (eg, colonoscopy) may have influenced initial comfort levels. We also acknowledge that institutional (urban academic center) and faculty (single faculty, apprentice-model rotation) factors may have influenced these results. There was no control group for comparison, as this education initiative was not conceived as a research study. However, we believe that cognitive science measurements are generalizable for curriculum development in surgical and nonsurgical settings. In addition, this is a pilot study limited by time and sample size, and there are no data quantifying the extent of resident participation. Finally, real-time measurements of intraoperative stressors, such as continuous heart rate monitoring, might provide more granular data with regard to specific aspects of the procedures that impact the overall cognitive load score; however, that was not the goal of the study. Variability is a hallmark of surgical education, and therefore this is a reflection of real-life practice.
CONCLUSION.
In conclusion, we demonstrate the effectiveness of administering the NASA-TLX scale to gauge barriers to skills acquisition on a cardiothoracic surgery rotation, allowing for tailoring a rotation to match trainee needs. Future studies will be necessary to target areas of greater cognitive load and reduce the barriers to achieving procedural competence. In addition, as excess cognitive load may not be wholly avoidable, techniques to recognize and effectively navigate excess cognitive burden may improve resident resiliency. Importantly, we found that frequent, structured faculty feedback improved resident participation in self-evaluation. Therefore, the interplay between resident and faculty reflection and documentation is critical for individualized curriculum development. The best results with such a curriculum would come from two way feedback, but that requires a substantial commitment from the faculty. It is important to note that the described curriculum only identifies the barriers to skills acquisition but does not address proficiency. Nonetheless, this approach has applicability to other areas of medical education and will guide competency-based curriculum development at our institution.
Supplementary Material
Acknowledgments
This work was funded by a VA Pittsburgh Health System Medical Education and Patient Safety Grant, VA Career Development Award (CX001771-01A2), and the University of Pittsburgh Dean’s Faculty Advancement Award (R.D.). Dr Huckaby received support from NIH T32HL098036 and the Thoracic Surgery Foundation Nina Starr Braunwald Research Fellowship. Dr Cyr received support from NIH T32GM00851626 and the American College of Surgeons Resident Research Award. Ethical review and approval were waived for this study by the VA Pittsburgh Healthcare System for being an education initiative and not constituting research. The authors wish to acknowledge Maggie Mrozinski for her assistance with data acquisition.
Footnotes
The Supplemental Tables and Supplemental Figure can be viewed in the online version of this article [https://doi.org/10.1016/j.athoracsur.2021.05.086] on http://www.annalsthoracicsurgery.org.
REFERENCES
- 1.Bilimoria KY, Chung JW, Hedges LV, et al. National cluster-randomized trial of duty-hour flexibility in surgical training. N Engl J Med. 2016;374:713–727. [DOI] [PubMed] [Google Scholar]
- 2.Bolster L, Rourke L. The effect of restricting residents’ duty hours on patient safety, resident well-being, and resident education: an updated systematic review. J Grad Med Educ. 2015;7:349–363. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Williams RG, Swanson DB, Fryer JP, et al. How many observations are needed to assess a surgical traineeʼs state of operative competency? Ann Surg. 2019;269:377–382. [DOI] [PubMed] [Google Scholar]
- 4.Stride HP, George BC, Williams RG, et al. Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents. Surgery. 2017;163:488–494. [DOI] [PubMed] [Google Scholar]
- 5.American Board of Surgery. General Surgery: content outline for the qualifying examination. Available at: https://www.absurgery.org/xfer/GS-QE.pdf. Accessed February 9, 2020.
- 6.Ragalie WS, Termuhlen PM, Little AG. Changes in thoracic surgery experience during general surgery residency: a review of the case logs from the Accreditation Council for Graduate Medical Education. Ann Thorac Surg. 2016;102:2095–2098. [DOI] [PubMed] [Google Scholar]
- 7.DeBoard ZM, Paisley M, Thomas DD. Self-appraised readiness of senior and graduating general surgery residents to perform thoracic surgery. J Surg Educ. 2018;75:877–883. [DOI] [PubMed] [Google Scholar]
- 8.Friedell ML, Vandermeer TJ, Cheatham ML, et al. Perceptions of graduating general surgery chief residents: are they confident in their training? J Am Coll Surg. 2014;218:695–703. [DOI] [PubMed] [Google Scholar]
- 9.Van Merriënboer JJG, Sweller J. Cognitive load theory in health professional education: design principles and strategies. Med Educ. 2010;44:85–93. [DOI] [PubMed] [Google Scholar]
- 10.Qiao YQ, Shen J, Liang X, et al. Using cognitive theory to facilitate medical education. BMC Med Educ. 2014;14:79. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Dias RD, Ngo-Howard MC, Boskovski MT, Zenati MA, Yule SJ. Systematic review of measurement tools to assess surgeons’ intraoperative cognitive workload. Br J Surg. 2018;105:491–501. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Hart SG, Staveland LE. Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Adv Psychol. 1988;52:139–183. [Google Scholar]
- 13.Ruiz-Rabelo JF, Navarro-Rodriguez E, Di-Stasi LL, et al. Validation of the NASA-TLX score in ongoing assessment of mental workload during a laparoscopic learning curve in bariatric surgery. Obes Surg. 2015;25:2451–2456. [DOI] [PubMed] [Google Scholar]
- 14.Yurko YY, Scerbo MW, Prabhu AS, Acker CE, Stefanidis D. Higher mental workload is associated with poorer laparoscopic performance as measured by the NASA-TLX tool. Simul Healthc. 2010;5:267–271. [DOI] [PubMed] [Google Scholar]
- 15.Wagner JP, Lewis CE, Tillou A, et al. Use of entrustable professional activities in the assessment of surgical resident competency. JAMA Surg. 2018;153:335–343. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Haji FA, Rojas D, Childs R, de Ribaupierre S, Dubrowski A. Measuring cognitive load: performance, mental effort and simulation task complexity. Med Educ. 2015;49:815–827. [DOI] [PubMed] [Google Scholar]
- 17.Zhou M, Jones DB, Schwaitzberg SD, Cao CGL. Role of haptic feedback and cognitive load in surgical skill acquisition. Proc Hum Factors Ergon Soc. 2007;2:631–635. [Google Scholar]
- 18.Szulewski A, Roth N, Howes D. The use of task-evoked pupillary response as an objective measure of cognitive load in novices and trained physicians: a new tool for the assessment of expertise. Acad Med. 2015;90: 981–987. [DOI] [PubMed] [Google Scholar]
- 19.Schubert CC, Denmark TK, Crandall B, Grome A, Pappas J. Characterizing novice-expert differences in macrocognition: an exploratory study of cognitive work in the emergency department. Ann Emerg Med. 2013;61:96–109. [DOI] [PubMed] [Google Scholar]
- 20.Hayden EL, Seagull FJ, Reddy R. Developing an educational video on lung lobectomy for the general surgery resident. J Surg Res. 2015;196:216–220. [DOI] [PubMed] [Google Scholar]
- 21.Divisi D, Barone M, Zaccagna G, De Palma A, Gabriele F, Crisci R. Video-assisted thoracoscopic surgery lobectomy learning curve: what program should be offered in a residency course? J Vis Surg. 2017;3:143. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Lowndes BR, Forsyth KL, Blocker RC, et al. NASA-TLX assessment of surgeon workload variation across specialties. Ann Surg. 2020;271:686–692. [DOI] [PubMed] [Google Scholar]
- 23.Walsh C, Sherlock M, Ling S, Carnahan H. Virtual reality simulation training for health professions trainees in gastrointestinal endoscopy. Cochrane Database Syst Rev. 2012;6:CD008237. [DOI] [PubMed] [Google Scholar]
- 24.Van Sickle KR, Buck L, Willis R, et al. A multicenter, simulation-based skills training collaborative using shared GI mentor II systems: results from the Texas Association of Surgical Skills Laboratories (TASSL) flexible endoscopy curriculum. Surg Endosc. 2011;25:2980–2986. [DOI] [PubMed] [Google Scholar]
- 25.Liberman AS, Liberman M, Steinert Y, McLeod P, Meterissian S. Surgery residents and attending surgeons have different perceptions of feedback. Med Teach. 2005;27:470–472. [DOI] [PubMed] [Google Scholar]
- 26.Cook MR, Watters JM, Barton JS, et al. A flexible postoperative debriefing process can effectively provide formative resident feedback. J Am Coll Surg. 2015;220:959–967. [DOI] [PubMed] [Google Scholar]
- 27.Porte MC, Xeroulis G, Reznick RK, Dubrowski A. Verbal feedback from an expert is more effective than self-accessed feedback about motion efficiency in learning new surgical skills. Am J Surg. 2007;193:105–110. [DOI] [PubMed] [Google Scholar]
- 28.Vaporciyan AA. How to give effective formative feedback in thoracic surgery education. Thorac Surg Clin. 2019;29:249–257. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.