Abstract
Background:
This study compares surgical residents’ knowledge acquisition of ultrasound-guided Internal Jugular Central Venous Catheterization (US-IJCVC) between in-person and online procedural training cohorts before receiving independent in-person Dynamic Haptic Robotic Simulation training.
Methods:
Three surgical residency procedural training cohorts, two in-person (N=26) and one online (N=14), were compared based on their performance on a 24-item US-IJCVC evaluation checklist completed by an expert physician completed after training. Pre- and post-training US-IJCVC knowledge was also compared for the online cohort.
Results:
No significant change in the pass rates on the US-IJCVC checklist was found between in-person and online cohorts (p = 0.208). There were differences in the Economy of Time and Motion between in-person and online cohorts (p < 0.005). The online cohort had significant increases in US-IJCVC knowledge pre- to post- training (p < 0.008).
Conclusion:
Online training with independent simulation practice was as effective as in-person training for US-IJCVC.
Keywords: central venous catheterization, residency bootcamp, medical simulation, computer-based learning, simulation-based learning
Introduction
The COVID-19 pandemic has required residency programs to re-think the way that skills training is conducted due to both the increased demands placed on medical professionals (1) as well as the impact of social distancing requirements (2). This is particularly important given the fact that the need for highly trained medical professionals has never been so high, due to shortages of health care workers (3). As such, there is a clear need to re-think and re-evaluate how to efficiently and effectively deliver surgical knowledge and skills training in residency programs (4, 5).
One such vital skill taught during residency training is Ultrasound-guided Central Venous Catheterization (US-CVC). While there are multiple insertion sites for US-CVC, the procedure is typically trained in the Internal Jugular (IJ) vein due to its ease of access, anatomy, low complication rate, and use of ultrasound (6, 7). This skill is a crucial element to teach in residency because while more than 5 million central lines are placed in the United States each year (6), CVC has been associated with high complication rates (8), ranging from 2%-26% for thrombotic, 5%-26% for infectious, and 5%-19% for mechanical complications (9). In addition, prior research has linked these complications directly with the skill level of the physician performing the procedure (10) - identifying a clear need to focus physician’s CVC education.
US-IJCVC is typically taught during residency “bootcamp”, including some combination of in-person didactic lectures, ultrasound Rapid Central Venous Assessment (RaCeVA) protocol practice with sterile procedure preparation and draping, in-person central line kit familiarization, US-IJCVC simulation training, and simulation of complication management and alternate vascular access procedures. Procedural skills training is typically taught using high fidelity manikins that require a trained preceptor to provide performance feedback. Thus, these simulators are limited in both the need of a trained professional to provide feedback as well as the fact that they only train residents on one particular patient anatomical variation. To combat these constraints, the Dynamic Haptic Robotic Trainer (DHRT) was developed to simulate multiple patient anatomies during IJ-CVC, provide residents with a more complete and realistic training, and provide automated IJ-CVC performance feedback through a custom Graphical User Interface (GUI) (11) (12). Previous research has shown that the DHRT is as good as manikin-based training (13), and has the added benefit that it can be completed independently of a trained preceptor.
Due to the limitations of the COVID-19 pandemic, both the time required to complete US-IJCVC skills training residency-wide as well as the social distancing and sterilization requirements restricted residency programs ability to provide in-person training – including both in-person manikin-based training and lecture-based instruction. As such, the purpose of this paper was to compare surgical residents’ knowledge acquisition of ultrasound-guided Internal Jugular Central Venous Catheterization (US-IJCVC) between in-person and online procedural training cohorts before receiving independent in-person Dynamic Haptic Robotic Simulation training.
Materials and Methods
The purpose of the current study was to compare surgical residents’ knowledge acquisition of US-IJCVC between an in-person “bootcamp” where the procedural training included didactic lectures, and a demonstration and an online “bootcamp” where the procedural training included online lectures and procedural competency assessments. Both of these groups then attended an in-person independent DHRT training session. Specifically, this study was developed to answer the following research questions (RQ):
RQ1: Are there significant IJCVC knowledge gains between pre- to post-training in the online procedural training cohort?
This research question was developed to understand if surgical residents acquired IJ-CVC knowledge after completing the online procedural training. It was hypothesized that surgical resident’s knowledge acquisition would increase after the online procedural training because prior research has shown that curriculums with online modules and in-person training are associated with higher knowledge and skills (14).
RQ2: Are there differences in US-IJCVC performance on a 24-item checklist between in-person and online procedural training cohorts?
The second research question was developed to understand if there were differences in the passing rates of the 24-item US-IJCVC checklist between the in-person and online procedural training cohorts. It was hypothesized that no difference would exist because prior research has shown that the DHRT system is as effective as manikin training (15).
RQ3: Are there differences in the Economy of Time and Motion in US-IJCVC between in-person and online procedural training cohorts?
The final research question aimed to determine if there was any difference in the Economy of Time and Motion between online and in-person procedural training cohorts gathered from the US-IJCVC check sheet. It was hypothesized that the economy of time and motion would be similar between both cohorts because prior research has shown that surgical skill simulation training is associated with improvements in economy of time and motion (16) and previous research has shown DHRT to be as effective as manikin training (15).
Participants and Procedures
Data for this study was collected over three time periods from the surgical residency program at Hershey Medical Center (HMC). The first two datasets were collected between 2016 and 2017, and each cohort comprised 13 surgical interns from an in-person “boot camp”. The third set of data was collected in 2020 and comprised of 14 surgical interns participating in an online “boot camp”. Thus, the delivery method for didactic instruction, either in-person or online, was the primary difference between cohorts. Table 1 demonstrates the demographics (residency program, previous experience) of the participants. The remainder of this section outlines the procedures for the data collection by cohort type as shown in Figure 1.
Table 1:
Demographics of participants across cohorts
| 2016 | 2017 | 2020 | |||
|---|---|---|---|---|---|
| Gender | Male | 8 | 9 | 7 | |
| Female | 3 | 3 | 7 | ||
| Unknown | 2 | 1 | |||
| Residency Program | General Surgery | 8 | 6 | 14 | |
| Urology | 1 | 1 | |||
| Plastic Surgery | 1 | 1 | |||
| Ortho | 2 | 2 | |||
| Otolaryngocolary | 1 | ||||
| ENT | 1 | ||||
| Unknown | 2 | ||||
| Previous Line Experience | # Previous IJ's placed | 0 | 10 | 12 | 10 |
| 1 | 2 | 1 | 1 | ||
| 2 | - | 2 | |||
| >2 | 1 | 1 | |||
| Previous Training | none | 8 | 3 | 10 | |
| subclavian and/ or femoral | 5 | 6 | 2 | ||
| other | 4 | 2 | |||
Figure 1:
In-person (top) and Online (bottom) Procedural Training Cohort Methodological Process
In-person Cohort (2016 and 2017)
At the start of the study purposes and procedures were explained. Informed consent was obtained according to an Institutional Review Board (IRB) approved protocol. First-year surgical residents (novices) in their first week of their residency program (“residency boot camp”) at HMC participated in the study. Both in-person cohorts were a subset of a larger study that compared needle insertion and procedural skills efficiency between manikin vs DHRT training (15). Participants from both in-person cohorts, 2016 (N=13) and 2017 (N=13) who completed in-person independent DHRT training were evaluated in this study. The remainder of this section highlights the methodology that pertains to the current investigation, see Figure 1.
Residents completed a demographics and prior CVC experience survey. Next, they received an in-person didactic lecture about CVC knowledge and observed a 15-minute US-IJCVC video developed by the New England Journal of Medicine (NEJM) (17). There was no knowledge assessment completed during this part of the course instruction. Following this, the residents received a live in-person training demonstration of central-line placement by a senior surgical resident using a Blue-Phantom Gen II Ultrasound Central Line Training Model (Model #BPH660) manikin. Participants then each had one practice attempt on the manikin. Next, residents observed a 7-minute introductory video on the DHRT system and then began in-person independent DHRT training which included 22 central line needle insertions simulating different patient scenarios over 3 training days. Feedback was provided by the DHRT Graphical User Interface (GUI) which provides performance feedback on the angle of insertion, how close the final needle tip was to the center of the vessel, the number of insertion attempts, and the percent of time the resident aspirated the needle. Residents then completed a full central line procedure on the same Blue-Phantom manikin and were evaluated on the 24-item US-IJCVC evaluation checklist by an expert observer. The checklist that experts used to evaluate residents can be found in https://bit.ly/3xyKpfC. This checklist shows resident’s proficiency on the US-IJCVC procedure and failure requires resident’s additional practice and re-evaluation. The raters on this 24-item checklist were senior residents in the surgical residency program who had received prior training on the rating process and who were trained in Central Line education.
Online Procedural Training Cohort (2020)
Data from an online procedural training cohort was collected in 2020 with first-year surgical residents (N=14) recruited from HMC as part of their “residency boot camp”. Residents were trained in an online environment due to the COVID-19 pandemic. The remainder of this section highlights the methodology that pertain to the current investigation.
At the start of the study, first-year surgical residents were enrolled in the online curriculum designed in Canvas Learning Management System (18). The content, learning objectives, main goals, and assessments were designed by agreement between five medical experts from two hospital institutions, HMC and Cedars Sinai Medical Center (CSMC).
Residents read the purposes and procedures and consent was obtained according to an IRB approved protocol. Residents completed a demographics and prior CVC experience survey. Next, they took a 14-questions pre-test to assess their CVC knowledge before the online training. The pre-test had multiple choice questions about CVC advantages, benefits, location sites, confirmation steps, risks, complications, and removal. There was also an ordering question on the 17 US-IJCVC steps and a picture identification question on a component of the tray. After completion of the pre-test, residents did not receive the correct answers to their questions, they were only notified which were the incorrect or correct questions they had.
Residents then accessed the content which had eight modules as agreed by experts. Experts recorded themselves teaching the content to residents. Final recordings ranged from 2.30 minutes to 15.28 minutes and included 1) CVC information and complication rates, 2) CVC overview based on the NEJM video (17), 3) access sites with the benefits and risks of each insertion site, 4) Central Line Bundle with the six best practices to improve care, 5) RaCeVA and Ultrasound for a quick assessment of the vessels, 6) mechanical procedures for wire and needle troubleshooting, 7) complications types, ultrasound and X-Rays identification, and 8) catheter removal of the line. Each module had 2 in-video multiple-choice questions to measure short-term knowledge gained developed using Kaltura Media Space (19). Settings in Kaltura and Canvas were modified to avoid residents skipping through the videos. Residents then completed a post-test with an unlimited number of attempts and a required passing grade of 80%. The content in the post-test was the same as the pre-test and each post-test attempt. It is important to note that only the score on the first post-test attempt was considered for the purposes of the current study. After successfully passing the post test, residents watch a 7-minute introductory video on the DHRT system which concluded the online training.
After completion of the online training, residents received a live demonstration of a central-line placement using a Blue-Phantom Gen II Ultrasound Central Line Training Model (Model #BPH660). Next, residents began in-person independent DHRT training and manikin training. The in-person independent DHRT training included 10 central line needle insertions and after completion of each needle insertion performance feedback was provided. The number of insertions was reduced from 22 down to 10 for this cohort because the previous data collected demonstrated that the residents had sufficient learning gains and mimicked expert performance around 10 trials {Pepley, 2017 #66}The manikin training included one needle insertion in the same Blue-Phantom manikin. At the end of the procedural skills training, all residents completed a full central line procedure using the same Blue-Phantom Gen II Ultrasound Central Line Training Model (Model #BPH660) manikin and were evaluated on the 24-item US-IJCVC evaluation checklist by an expert observer and is shown in https://bit.ly/3xyKpfC. This 24-item checklist has been previously used in HMC to assess residents’ performance on the manikin (15, 20). The raters on this 24-item checklist were senior residents in the surgical residency program who had received prior training on the rating process and who were trained in Central Line education.
Procedural Skills Performance Metrics
The 24-items US-IJCVC evaluation checklist was used to assess the residents’ performance after in-person independent DHRT training. The metrics from this checklist used in this study have been previously used to assess the residents’ procedural skills in previous studies using the US-IJCVC evaluation checklist (15). These metrics are the following:
Pass Rate:
The individual passing rate was calculated by the averaging the binary scores for the 24 items. For example, if a resident had 21 out of 24 items on the checklist, this translates to an 88% (21/24) individual passing rate. Thus, the pass rate is the total percentage of items each individual completed on the checklist.
Number of insertion attempts:
The number of insertion attempts was noted on the US-IJCVC check sheet by the expert observer. The number of insertion attempts was collected because it relates to the number of times that the resident had an unsuccessful needle insertion which can predict a higher CVC complication rate (21).
Economy of Time and Motion:
The economy of time and motion was scored by the expert observer between 1 (unnecessary/disorganized movements) to 5 (maximum economy of movement and efficiency) based on the residents’ movements to perform the procedure. This metric has also been used in simulation laparoscopic surgery training (22).
Results
This section highlights our results in terms of the research questions. All statistics were analyzed with SPSS (v. 26.0) with a significance level of 0.05.
RQ1: Are there significant IJCVC knowledge gains between pre- to post-training in the online procedural training cohort?
The first research question was developed to evaluate the knowledge acquisition of the online curriculum by comparing if residents acquire knowledge after undergoing the training. To accomplish this, a paired t-test was conducted to determine if there were any differences in the pre- and post- training scores of the online procedural training cohort. Post-test scores used were the scores that surgical residents received on their first attempt in the test. The data had no significant outliers as shown in boxplots, and the assumption of normality was not violated, as assessed by boxplot and the Shapiro-Wilk’s test (p= 0.397). The results of the paired-sample t-test showed that the online procedural training cohort had significantly higher scores in their post-test (M=70.84, SD=10.71) compared to their pre-test (M=58.49, SD=11.71), a statistically significant mean increase of 11.98, 95% CI [3.69, 20.30], t(13)=3.11, p < .008, d=3.74. These results indicate that the residents in the online procedural training cohort improved in their US-IJCVC knowledge from pre- to post-test.
RQ2: Are there differences in US-IJCVC performance on a 24-item checklist between in-person and online procedural training cohorts?
The second research question aimed to understand if passing rates on a 24-item US-IJCVC evaluation checklist differed between the online and in-person procedural training cohorts. All of the residents (100%) in the online procedural training cohort passed the Verification of Proficiency (VOP) on their first attempt, while all but 1 resident (96%) passed the VOP in the 2016-2017 cohort. As such no statistical analyses were completed comparing VOP pass rates. However, we did compare the number of insertion attempts to perform US-IJCVC on a post-test manikin to determine if there were differences between the online and in-person procedural training cohorts. Prior to running any analyses assumptions were checked. Visual inspection of boxplots revealed that there were no significant outliers. However, the assumption of normality was violated, as assessed by the Shapiro Wilk’s test (p < 0.05). As such, the non-parametric Mann-Whitney U Test was computed. Results from the Mann-Whitney U Test showed that there was no statistically significant difference in the number of insertion attempts needed for successful venipuncture on the post-test manikin between the in-person (Mdn = 1.00, M = 1.36, SD = 0.91) and online procedural training cohorts (Mdn = 1.00, M = 1.16, SD = 0.39), (U = 143, z= −0.33, p < 0.74).
RQ3: Are there differences in the Economy of Time and Motion in US-IJCVC between in-person and online procedural training cohorts?
The final research question was developed to evaluate differences between online and in-person procedural training cohorts for the Economy of Time and Motion from the 24-item US-IJCVC evaluation checklist. To answer this research question, a Mann-Whitney U Test was conducted. The results showed that there was a statistically significant difference in the Economy of time and motion between procedural training groups. Specifically, the results showed that the online procedural training cohort (Mdn = 5.00) had a significantly better economy of time and motion compared to the in-person procedural training cohorts (Mdn = 3.00), (U = 22, z= −4.55, p < 0.001). Although results showed that online training might improve economy of time and motion, we still need to further explore this to determine if there is any type of relationship between them.
Discussion
The results for the first research question were expected as our hypothesis was that trainees in online training will have knowledge acquisition pre- to post-module learning. This shows that the online training was effective at teaching CVC knowledge to surgical residents at HMC. These results provide evidence that the online curriculum can replace in-person didactic lectures because it provides a knowledge evaluation method and no in-person requirements, something that in-person lectures do not provide. Results also showcase that surgical residents can receive CVC training even with social distancing limitations.
For the second research question, we expected that passing rates between in-person and online procedural training cohorts will be the same because prior research showed that DHRT is as effective as manikin training. Results were as predicted and there were no statistically significant differences in passing rates between cohorts meaning that online trained residents performed similarly to in-person trained residents in the US-IJCVC evaluation checklist. These results provide evidence that despite the type of CVC training, surgical residents will perform similarly in their procedural skills training. Thus, when having surgical residents go through online procedural training, performance of procedural skills for CVC will not be affected.
The results for the last research question showed that there was a statistically significant difference in the Economy of Time and Motion of in-person and online procedural training cohorts different than what we predicted. Online procedural training cohorts were evaluated to have a better Economy of Time and Motion in their procedural skills training compared to the in-person group providing empirical evidence that online training may improve resident’s economy of motion when related to time and hand motion. Residents in laparoscopic training showed a higher economy of time and motion when trained by simulation (23). Thus, our results provide preliminary evidence that the integration of computer-based learning to simulation training may improve the economy of time and motion of surgical residents. Moreover, research has shown how economy of hand motion is a reliable measure of competency (24). Since residents in the online cohort had higher Economy of Time and Motion, our study provide preliminary evidence that the online curriculum integrated with the DHRT system may provide a reliable measure of resident’s competency. However, further studies are needed to validate these findings and compare economy of time and motion before and after simulation training.
Limitations
While this study shows promising results, there are several limitations that were encountered. The first limitation is that this study was performed at only one large academic institution causing limitations with the surgical resident’s sample size. Another limitation is that experts only provided feedback and agreement of the online curriculum but did not perform the online curriculum. Surgical residents attended the surgical bootcamp, which had time allocated for them to participate in the study. Future work should apply lessons learned from the feedback and satisfaction survey to improve the online curriculum and implement this online training in other large academic institutions. Additionally, further studies should evaluate the relationship between economy of time and motion with online training since our results could have been caused due to a cohort effect. Finally, the curriculum and DHRT trainer studied here is currently only available at two hospital institutions. While the results presented here are only generalizable to the DHRT system, future work should seek to study the broader application of these findings to other training systems.
Conclusion
The goal of this research was to compare surgical residents’ knowledge acquisition of ultrasound-guided Internal Jugular Central Venous Catheterization (US-IJCVC) between in-person and online procedural training cohorts. The first main findings of this study was that surgical residents acquire US-IJCVC knowledge after undergoing an online curriculum as assessed by higher scores in their post-assessments. Second, we found that passing rates for procedural performance for the online procedural training cohort was just as effective as the in-person procedural training cohort. We also found that the economy of time and motion significantly differ from the online to the in-person procedural training cohort showing that the online cohort had a better performance. Lastly, we found that residents can have knowledge gain for CVC even with in-person limitations. Therefore, we can preliminarily conclude that US-IJCVC online training was just as effective as in-person training for surgical resident’s acquisition of knowledge and development of the technical skillset involved in the performance of safe central line placement. Further studies should be conducted to evaluate these conclusions.
Acknowledgements
Research reported in this publication was supported by the National Heart, Lung, and Blood Institute of the National Institutes of Health (NIH) under Award Number RO1HL127316. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. This material is based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. (DGE1255832). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. Professors Miller and Moore are equity owners of Medulate, which has a commercial interest in this project.
References
- 1.Chatterjee SS, Bhattacharyya R, Bhattacharyya S, et al. Attitude, practice, behavior, and mental health impact of COVID-19 on doctors. Indian Journal of Psychiatry. 2020;62(3):257. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Courtemanche C, Garuccio J, Le A, et al. Strong Social Distancing Measures In The United States Reduced The COVID-19 Growth Rate: Study evaluates the impact of social distancing measures on the growth rate of confirmed COVID-19 cases across the United States. Health Affairs. 2020;39(7):1237–46. [DOI] [PubMed] [Google Scholar]
- 3.An assessment of the impact of COVID-19 on job and skills demand using online job vacancy data. 2021; Available from: https://www.oecd.org/coronavirus/policy-responses/an-assessment-of-the-impact-of-covid-19-on-job-and-skills-demand-using-online-job-vacancy-data-20fff09e/.
- 4.Chick RC, Clifton GT, Peace KM, et al. Using technology to maintain the education of residents during the COVID-19 pandemic. Journal of surgical education. 2020;77(4):729–32. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Okland TS, Pepper J-P, Valdez TA. How do we teach surgical residents in the COVID-19 era? Journal of Surgical Education. 2020;77(5):1005–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.McGee DC, Gould MK. Preventing complications of central venous catheterization. New England journal of medicine. 2003;348(12):1123–33. [DOI] [PubMed] [Google Scholar]
- 7.Saugel B, Scheeren TW, Teboul J-L. Ultrasound-guided central venous catheter placement: a structured review and recommendations for clinical practice. Critical Care. 2017;21(1):1–11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Taylor RW, Palagiri AV. Central venous catheterization. Critical care medicine. 2007;35(5):1390–6. [DOI] [PubMed] [Google Scholar]
- 9.Graham AS, Ozment C, Tegtmeyer K, et al. Central venous catheterization. N Engl J Med. 2007;356(21):e21. [DOI] [PubMed] [Google Scholar]
- 10.Kotsis SV, Chung KC. Application of see one, do one, teach one concept in surgical training. Plastic and reconstructive surgery. 2013;131(5):1194. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Milner TD, Okhovat S, McGuigan M, et al. Feasibility of ovine and porcine models for simulation training in parotid surgery and facial nerve dissection. European Archives of Oto-Rhino-Laryngology. 2020:1–9. [DOI] [PubMed] [Google Scholar]
- 12.Pepley DF, Gordon AB, Yovanoff MA, et al. Training surgical residents with a haptic robotic central venous catheterization simulator. Journal of surgical education. 2017;74(6):1066–73. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Chen H-E, Yovanoff MA, Pepley DF, et al. Evaluating surgical resident needle insertion skill gains in central venous catheterization training. Journal of Surgical Research. 2019;233:351–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Berger ER, Kreutzer L, Halverson A, et al. Evaluation of changes in quality improvement knowledge following a formal educational curriculum within a statewide learning collaborative. Journal of surgical education. 2020;77(6):1534–41. [DOI] [PubMed] [Google Scholar]
- 15.Chen H-E, Sonntag CC, Mirkin KA, et al. From the simulation center to the bedside: Validating the efficacy of a dynamic haptic robotic trainer in internal jugular central venous catheter placement. The American Journal of Surgery. 2020;219(2):379–84. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Limberg J, Karnick A, Bagautdinov I, et al. Does variability among surgical skills diminish throughout surgical internship? Analysis of a 5-task surgical simulation assessment program starting Day 1. Surgery. 2020;167(4):704–11. [DOI] [PubMed] [Google Scholar]
- 17.NEJM Central Venous Catheterization Video. Available from: https://www.nejm.org/doi/full/101056/NEJMvcm055053
- 18.Canvas Learning Management System. Available from: https://www.instructure.com/canvas.
- 19.Kaltura MediaSpace. Available from: https://corp.kaltura.com/video-collaboration-communication/enterprise-video-portal/.
- 20.Yovanoff MA, Chen H-E, Pepley DF, et al. Investigating the Effect of Simulator Functional Fidelity and Personalized Feedback on Central Venous Catheterization Training. Journal of surgical education. 2018;75(5):1410–21. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Mansfield PF, Hohn DC, Fornage BD, et al. Complications and failures of subclavian-vein catheterization. New England journal of medicine. 1994;331(26):1735–8. [DOI] [PubMed] [Google Scholar]
- 22.Cruff J. Robotic Surgical Training at Home: A Low-Fidelity Simulation Method. Journal of Surgical Education. 2021;78(2):379–81. [DOI] [PubMed] [Google Scholar]
- 23.Parent RJ, Plerhoples TA, Long EE, et al. Early, intermediate, and late effects of a surgical skills “boot camp” on an objective structured assessment of technical skills: a randomized controlled study. Journal of the American College of Surgeons. 2010;210(6):984–9. [DOI] [PubMed] [Google Scholar]
- 24.Grober ED, Roberts M, Shin E-J, et al. Intraoperative assessment of technical skills on live patients using economy of hand motion: establishing learning curves of surgical competence. The American journal of surgery. 2010;199(1):81–5. [DOI] [PubMed] [Google Scholar]