Performance of Vascular Exposure and Fasciotomy Among Surgical Residents Before and After Training Compared With Experts

This study measures resident surgeon technical and nontechnical skills for trauma core competencies before and after training and up to 18 months later and compares resident performance with the performance of expert traumatologists.

Key Points

Question

Does Advanced Surgical Skills Exposure for Trauma training improve resident performance compared with expert traumatologists?

Findings

In a longitudinal cohort study that used the Trauma Readiness Index, most residents had significantly improved correct procedural steps and anatomy skills and decreased errors after a 1-day Advanced Surgical Skills Exposure for Trauma course, and core competency skills were retained for up to 18 months. Experts performed significantly better than residents.

Meaning

Advanced Surgical Skills Exposure for Trauma provides training in trauma core competencies, and the Trauma Readiness Index can identify those residents who require remedial interventions in trauma technical skills.

Abstract

Importance

Surgical patient outcomes are related to surgeon skills.

Objective

To measure resident surgeon technical and nontechnical skills for trauma core competencies before and after training and up to 18 months later and to compare resident performance with the performance of expert traumatologists.

Design, Setting, and Participants

This longitudinal study performed from May 1, 2013, through February 29, 2016, at Maryland State Anatomy Board cadaver laboratories included 40 surgical residents and 10 expert traumatologists.

Interventions

Performance was measured during extremity vascular exposures and lower extremity fasciotomy in fresh cadavers before and after taking the Advanced Surgical Skills for Exposure in Trauma (ASSET) course.

Main Outcomes and Measures

The primary outcome variable was individual procedure score (IPS), with secondary outcomes of IPSs on 5 components of technical and nontechnical skills, Global Rating Scale scores, errors, and time to complete the procedure. Two trained evaluators located in the same laboratory evaluated performance with a standardized script and mobile touch-screen data collection.

Results

Thirty-eight (95%) of 40 surgical residents (mean [SD] age, 31 [2.9] years) who were evaluated before and within 4 weeks of ASSET training completed follow-up evaluations 12 to 18 months later (mean [SD], 14 [2.7] months). The experts (mean [SD] age, 52 [10.0] years) were significantly older and had a longer (mean [SD], 46 [16.3] months) interval since taking the ASSET course (both P < .001). Overall resident cohort performance improved with increased anatomy knowledge, correct procedural steps, and decreased errors from 60% to 19% after the ASSET course regardless of clinical year of training (P < .001). For 21 of 40 residents (52%), correct vascular procedural steps plotted against anatomy knowledge (the 2 IPS components most improved with training) indicates the resident’s performance was within 1 nearest-neighbor classifier of experts after ASSET training. Five residents had no improvement with training. The Trauma Readiness Index for experts (mean [SD], 74 [4]) was significantly different compared with the trained residents (mean [SD], 48 [7] before training vs 63 [7] after training [P = .004] and vs 64 [6] 14 months later [P = .002]). Critical errors that might lead to patient death were identified by pretraining IPS decile of less than 0.5. At follow-up, frequency of resident critical errors was no different from experts. The IPSs ranged from 31.6% to 76.9% among residents for core trauma competency procedures. Modeling revealed that interval experience, rather than time since training, affected skill retention up to 18 months later. Only 4 experts and 16 residents (40%) adequately decompressed and confirmed entry into all 4 lower extremity compartments,

Conclusions and Relevance

This study found that ASSET training improved resident procedural skills for up to 18 months. Performance was highly variable. Interval experience after training affected performance. Pretraining skill identified competency of residents vs experts. Extremity vascular and fasciotomy performance evaluations suggest the need for specific anatomical training interventions in residents with IPS deciles less than 0.5.

Introduction

Surgical patient outcomes are related to the technical skills of the surgeon. Trauma operative management experience among general surgical trainees has decreased since the Accreditation Council for Graduate Medical Education (ACGME) introduced resident duty hour restrictions in 2003. The increasing use of nonoperative management of many traumatic injuries, reduction in motor vehicle occupant injuries, and decreased incidence of penetrating injuries nationwide has further limited trauma operative experience during training. A recent 20-year review of ACGME case logs demonstrated that graduating general surgery chief residents performed approximately half the number of designated trauma operations compared with 2 decades ago. Peripheral vascular repair and fasciotomy for injury are among American Board of Surgery trauma procedural core competencies. The mean number of trauma and vascular cases submitted to the American Board of Surgery for 2014 graduates of US surgical residencies was 2.1, including 1.0 exposure or repair of peripheral arteries and 0.8 fasciotomy for trauma. Trauma procedure–specific human cadaveric courses, such as the Advanced Surgical Skills for Exposure in Trauma (ASSET) course developed by the American College of Surgeons Committee on Trauma, have been used to fill this training gap. Recently, individual procedure-specific metrics were validated for technical and nontechnical skills for extremity vascular exposure and control and lower extremity fasciotomy. The objective of this study was to evaluate the performance of surgical residents before and immediately after ASSET training, with follow-up evaluation 12 to 18 months later to determine skill retention and competence when compared with expert traumatologists.

Methods

This longitudinal study was conducted at the Maryland State Anatomy Board cadaver laboratories situated at the University of Maryland School of Medicine, Baltimore, from May 1, 2013, through February 29, 2016. Forty surgical residents in postgraduate years 3 to 6 from 13 different training programs in the Mid-Atlantic region volunteered to participate. Free attendance for the ASSET course and travel and accommodation expense reimbursement were offered. Ten expert practicing traumatologists (mean of 14 years in practice) from 6 different US level I trauma centers were offered a study participation payment and travel and accommodation reimbursement. The University of Maryland School of Medicine Institutional Review Board and US Army Office of Research Protection approved the recruitment and consent process. All participants provided written informed consent. Cadaver use was approved by the Maryland State Anatomy Board and the US Army. Individual surgeon data were deidentified for the purposes of analyses.

Metrics

Performance of surgical technical and nontechnical skills were measured during extremity vascular exposures and lower extremity fasciotomy. The primary outcome variable was a previously validated individual procedure score (IPS), with secondary outcomes of IPSs on 5 components of technical and nontechnical skills, Global Rating Scale (GRS) scores, frequency of errors, and time to complete procedures. The validated IPS checklist defined 5 components of technical and nontechnical skills: knowledge, anatomy, management, procedural steps, and technical points. The IPS was the sum of the correct component scores divided by the total possible score. Each procedure had differing numbers of evaluation points, so the IPS was normalized (percentage or fraction correct). A total of 357 items and 23 errors were evaluated for these 4 procedures for each surgeon. For vascular procedures, the Trauma Readiness Index (TRI) was calculated as the sum of 3 IPSs. The 5 GRSs were overall indications and management (GRS1), overall understanding of anatomy (GRS2), technical skills for exposure or decompression (GRS3), readiness to perform the procedure solo (GRS4), and overall global rating of performance (GRS5). Scoring for GRS1 through GRS4 was based on a 1- to 5-point Likert scale, with 1 indicating poor and 5 indicating excellent. Scoring for GRS5 was based on a scale of 1% to 100%. Errors (defined in Table 1) and time to complete each procedure were noted. Performance data were entered into a touch-screen mobile Android tablet app in real time, as described elsewhere. Resident participants were evaluated before and within 4 weeks after ASSET training with follow-up reevaluation up to 18 months later. Experts were evaluated once in an identical manner up to 5 years after ASSET training.

Table 1. Definitions of Critical Technical and Critical Management Errors for Vascular Procedures and Lower Extremity Fasciotomy.

Error	Definition
Vascular Procedures
Critical technical error	Fails to loop vessel proximal to injury or duration is ≥20 min
Critical management error	Delay going to the operating room or inappropriate use of computed tomography or angiography
Noncritical, morbidity, or other management error	Inadequate skin preparation or inadequate investigation (eg, failure to order chest radiography with a gunshot in shoulder and no exit wound)
Fasciotomy
Critical technical error	Incorrectly identified the intramuscular septum; failed to decompress at least 1 of the compartments (anterior, lateral, or superficial) by opening fascia to within 3 cm of the malleoli and 3 cm of the tibial plateau; or in the deep compartment fails to take down soleus fibers from underside of tibia and identify the neurovascular bundle

Performance Evaluation and Data Collection

Two trained evaluators located in the same laboratory (1 anatomist and 1 physician) evaluated performance with a standardized script. Twenty evaluators were trained (12 surgeons, 6 anatomists, and 2 nonsurgeon physicians) in a standardized manner with video illustrations of features of performance and an evaluator handbook in one-on-one sessions. Data collection to assess procedure-specific technical and nontechnical skills used a custom-programmed app. No prompting or comments were made by the evaluators other than repetition of a question or instruction when requested.

Video Recording of Procedures

Surgeons were audio and video recorded from ceiling-mounted pan-tilt-zoom cameras and directional microphones and head-mounted miniature cameras. They performed American Board of Surgery core trauma competencies, including 3 extremity vascular procedures (axillary artery, brachial artery, and femoral artery exposure) and control (a double-vessel loop placed around the correct artery) and a 4-compartment, 2-incision lower extremity fasciotomy on unpreserved cadavers. None of the surgeons were aware of the procedures they would be asked to perform.

Debriefing

Debriefing occurred with discussion of correct incision landmarks, demonstration of procedural steps, and description of key technical points on the cadaver, where needed. For residents, learning bias attributable to repeated interval testing and debriefing on the same 4 procedures was addressed by adding a fifth surprise procedure, carotid artery exposure, when they returned for follow-up reevaluation. The carotid artery exposure is part of the ASSET course curriculum but had not been previously evaluated in these residents as part of this study.

Case Logs and Experience

Case log summaries were requested from each surgeon, and a survey was completed by all study participants identifying numbers of upper and lower extremity vascular procedures and lower extremity fasciotomy performed, years in training, and months since taking the ASSET course.

Statistical Analysis

A linear mixed model with repeated measurements was used that included individual surgeons as a random variable, type of surgeon (resident or expert), evaluator type (anatomist or physician), cadaver body habitus (average, obese, or thin), finding of cadaveric anatomical anomalies, months since taking the ASSET course, and number of trauma patients evaluated_. Multiple pairwise comparisons were adjusted by the Tukey-Kramer method to identify features that affect training for procedure performance and skill retention on follow-up. Additional methods included cluster analysis. Two experts were never ASSET trained but had more than 40 years of clinical practice as attending surgeons at level I trauma centers. For calculation of the interval since training, both were grandfathered to the first American College of Surgeons–sponsored ASSET course in 2010. Scatterplots were prepared using SigmaPlot (Systat Software Inc). SAS statistical software, version 9.3 (SAS Institute Inc) was used for analyses, with P < .05 considered to be statistically significant. A priori sample size calculation required 36 (90%) of 40 residents to be followed up for reevaluation to detect skill degradation of 0.70 and 0.82 SDs with 80% and 90% power, respectively, using a 2-tailed t test with 5% type I error.

Results

Thirty-eight of 40 surgical residents (95%; mean [SD] age, 31 [2.9] years) who were evaluated before and within 4 weeks of ASSET training completed follow-up evaluations 12 to 18 months later (mean [SD], 14 [2.7] months). The experts (mean [SD] age, 52 [10.0] years) were significantly older and had a longer (mean [SD], 46 [16.3] months) interval since taking the ASSET course (both P < .001).

Performance Improved With Training

The mean (SD) TRI for vascular procedures performed by residents improved from evaluation before taking the ASSET course by 31.2% after training (48 [7] before vs 63 [7] after training, P = .004), and this improvement was sustained at follow-up after training (48 [7] before vs 64 [6] 14 months later, P = .002) (Table 2). Comparing pretraining with posttraining and follow-up performance, the residents demonstrated doubling of anatomy skills, a 41.2% increase in correct procedural steps (Table 2), and decreased time to complete the procedures by 2.5 minutes, similar to results previously reported. Modeling revealed that the benefits of ASSET training occurred regardless of clinical year of residency. A cluster analysis for vascular procedures (Figure 1) that assessed anatomy knowledge by correct procedural steps (the 2 IPS components most improved with training) among residents before and after training and experts indicates that 21 of 40 residents (52%) after training were within 1 nearest-neighbor classifier of the experts after taking the ASSET course. The performance ranges overlap considerably when looking at only procedure or only anatomy IPS components; however, combining the 2 factors permits performance comparisons relative to an expert standard. However, the TRIs and IPSs for axillary artery, brachial artery, femoral artery, and fasciotomy were all higher in experts than residents at both time points after training (Table 2).

Table 2. Pretraining and Posttraining Comparison of IPSs and TRIs.

Test	Training			Follow-up 14 mo Later		Experts, Mean (SD)	P Values
							Before vs After, Follow-up, and Experts	After vs 14 mo		After vs Experts		14 mo vs Experts
	Before, Mean (SD)	After, Mean (SD)	Change From Pretraining Values, %	Mean (SD)	Change From Pretraining Values, %
IPS component
Knowledge	48 (7)	58 (7)	20.8	57 (6)	18.7	65 (5)	<.001		.98		.005		.002
Anatomy	47 (11)	71 (9)	51.1	69 (9)	46.8	78 (7)	<.001		.09		.10		.06
Management	44 (7)	46 (9)	4.5	46 (8)	4.5	53 (9)	<.001		.99		.007		.005
Procedural steps	51 (14)	72 (12)	41.2	75 (11)	47.0	85 (7)	<.001		.98		.73		.57
Technical points	57 (12)	71 (13)	24.6	75 (12)	31.6	87 (8)	<.001		.74		.004		.03
TRI	48 (7)	63 (7)	31.2	64 (6)	33.3	74 (4)	<.001		.89		.004		.002
GRS
GRS1 (indications and management)	2.7 (0.64)	3.4 (0.74)	25.9	3.5 (0.76)	22.8	4.5 (0.46)	<.001		.76		<.001		<.001
GRS2 (anatomy)	2.3 (0.75)	3.6 (0.88)	56.5	3.6 (0.89)	56.5	4.3 (0.67)	<.001		.99		.30		.05
GRS3 (technical skills)	2.7 (0.87)	3.6 (0.92)	33.3	3.6 (0.86)	33.3	4.4 (0.55)	<.001		.99		.03		.05
GRS4 (readiness to perform)	2.2 (0.72)	3.6 (0.84)	63.6	3.5 (0.91)	59.1	4.3 (0.62)	<.001		.99		.20		.01
GRS5 (overall), %	63 (11)	81 (11)	18.0	80 (11)	17.0	91 (14)	<.001		.85		.20		.06

Abbreviations: GRS, Global Rating Scale; IPS, individual procedure score; TRI, Trauma Readiness Index.

Figure 1. Cluster Analysis of Correct Procedural Anatomy Skills and Technical Steps by Resident and Expert Surgeons.

Cluster analysis of technical procedural steps score plotted against anatomy skills score on a scale of 0 to 1.0 for axillary, brachial, and femoral artery procedures for 40 residents before and after training compared with experts. Circles indicate 1 nearest-neighbor classifier. After training, 7 residents did not leave the pretraining nearest-neighbor classifier of performance. Twenty-one residents were within 1 nearest-neighbor classifier of expert performance after training.

Errors and Error Recovery

At least 1 critical technical or management error was made in the 4 procedures by 24 residents (60%) before training, which decreased to 8 (19%) after training and was 9 (22%) at follow-up (P < .001). Critical error frequency in the carotid artery procedure in 38 residents was no different from the other 4 procedures performed after training. Pretraining deciles of IPS below a threshold of 0.5 were associated with at least 1 critical error. Five residents with repeated critical technical errors in vascular procedures were identifiable by pretraining performance IPS in the lowest tertile for the axillary artery procedure. Nine residents made at least 1 critical error after training. The 2 residents lost for follow-up reevaluation had critical errors before and immediately after training. With training, residents had increased ability to recognize and correct critical errors. The error recovery rate was 9% before training; the proportion of errors that were recognized and corrected was increased to 28% after training and 29% at follow-up. The number of errors and error recovery rate were not different among the expert cohort, the residents after training, and at follow-up reevaluation. Only 2 of 40 members (5%) of the resident cohort adequately decompressed and confirmed entry (eg, by identifying neurovascular bundle in deep posterior compartment) into all 4 fasciotomy compartments before training, increasing to 13 after training and 10 of 38 (26%) at follow-up evaluation. Only 4 of 10 experts (40%) adequately decompressed and confirmed entry into all fasciotomy compartments. Of expert and resident cohorts, 5 surgeons failed to adequately decompress the anterior fasciotomy compartment, and 12 surgeons failed to adequately decompress the deep posterior fasciotomy compartment (Figure 2).

Figure 2. Number of Lower Extremity Fasciotomy Compartments Decompressed by Residents Before and After Training and Follow-up vs Experts.

Percentage of participants decompressing and confirming entry into lower extremity fascial compartments is plotted for each study arm by the number of compartments adequately decompressed (0, 1, 2, 3, or 4 compartments using the definitions in Table 1).

Variability of Resident Cohort Performance

The individual resident surgeon IPSs for each procedure are shown in Figure 3. Resident performance varied from 31.6% to 76.9% in overall evaluations judged by IPSs. Similar variability in IPSs was also found in the carotid artery procedure evaluated for the first time at a mean of 14 months after ASSET training (Figure 3). Five residents did not improve the skills that were typically most improved by taking the ASSET course, namely, increase in anatomy knowledge, increase in correct procedural steps, decreased errors, and shortened time to complete the procedures. Plots of the pretraining IPSs by tertiles before, immediately after, and at follow-up reevaluation demonstrate that residents in the upper-performing tertile before taking the ASSET course remain in the upper tertile immediately after training and at follow-up (Figure 3) and the residents below the 0.5 decile of IPS do not improve their performance and continue to make critical technical errors despite training.

Figure 3. Tertiles of Resident Pretraining Individual Procedure Scores (IPSs) for 5 Trauma Core Competency Procedures With Follow-up.

The IPSs of 40 residents for each axillary artery, brachial artery, carotid artery (n = 38), and femoral artery exposure and control procedures and lower extremity fasciotomy plotted against the training interval. Color bars indicate change in performance with training of the median score for each IPS tertile at the pretraining visit. ASSET indicates Advanced Surgical Skills for Exposure in Trauma.

Results of Modeling

Study arm (before vs after training) performance improvements in IPSs and GRSs were significant for all procedures. Results of IPS comparisons are given in Table 2. Significant differences were found for axillary artery, brachial artery, and femoral artery procedures between the single evaluation of expert performance and trained residents for knowledge, procedural steps, and technical points at the posttraining and follow-up evaluations. However, experts were no different than trained residents in anatomy scores, procedural steps, or overall subjective GRS5 scores. Modeling revealed that experts and trained residents assessed by TRI, IPSs, and GRS1 through GRS4 performed better than residents evaluated with the same metrics before training. There were no IPS differences after training and at the follow-up evaluation in resident performance on axillary artery, brachial artery, and femoral artery vascular procedures. No differences were seen in IPSs between expert and resident posttraining evaluations for fasciotomy. Linear mixed modeling indicates there were no effects on IPS of year of residency, evaluator type, or time since training. However, resident interval experience with larger numbers of trauma patients evaluated and extremity procedures performed improved IPSs for procedures performed at follow-up 14 to 18 months after training. Cadaver anatomy anomalies affected overall understanding of axillary artery anatomy (GRS2), number of upper extremity procedures performed affected IPSs for brachial artery, and cadaver body habitus affected brachial artery technical skills (GRS3). For femoral artery cadaver, anatomical anomalies affected IPSs, and number of upper extremity procedures performed affected overall technical skills (GRS3). Surprisingly, for fasciotomy, upper extremity and lower extremity interval experience affected GRS1 (knowledge of indications and management) and GRS2 (overall understanding of anatomy in the leg) (GRS2).

Discussion

Residents had surgical procedural performance improvement after attending the ASSET course. Overall, this benefit was unchanged when follow-up reevaluation occurred. This study therefore demonstrated that most residents can be trained in a 1-day ASSET course and these skills can be retained for up to 18 months. Improved knowledge of procedural anatomy, including surface landmarks, procedural steps, and anatomical structural recognition, were identified by IPS components as key lessons learned. Anatomy skills and procedural steps were the only IPS component for which residents were no different from experts after training. It is well recognized that some trainees are unable to achieve basic competencies despite training. Within the resident cohort, there was nearly 100% variability for all evaluations of 5 procedures judged by the IPS (suggesting no learning bias from repeated evaluation because the carotid artery procedure had same variability). Such large variability is not seen among experts, although even experts have variability in performance and outcomes. Learning theories describe 3 phases in acquisition of motor skills (cognitive, integrative, and autonomous). For the 5 residents with no overall improvement in procedural performance, there are many ways that surgeons who have limited technical abilities can acquire and execute psychomotor skills, including deliberate practice and mental rehearsal to improve technical performance. Others have noted 3 tiers of surgical skill. Figure 3 indicates that the residents in the lowest tertile do not change their performance despite training. Our data suggest that the IPS can identify these individuals and the occurrence of critical technical errors despite training and repeated debriefing (including demonstration of how to perform the procedures correctly).

Errors

All residents with critical technical errors were recognizable by pretraining IPS decile less than 0.5. This threshold might be useful to identify the need for additional remedial training interventions. Nonexperts depend on feedback to correct errors, whereas self-correction of surgical errors is correlated with increasing expertise, as confirmed by the increase in error recovery after ASSET training for residents. Surgeons in this study performed solo without any input from evaluators observing the procedures. This circumstance occurs during emergencies, deployment, and mass casualty events; thus, this design to assess individual, unmentored surgeon performance is clinically relevant.

Lower Extremity Fasciotomy

Fasciotomy was a sentinel procedure for residents and experts. Only 4 experts and 16 residents (40%) adequately decompressed and confirmed entry into all 4 lower extremity compartments, even though residents had performed fasciotomy 3 times for the study and received 2 debriefings. However, ASSET training of the entire resident cohort increased successful 4-compartment decompression by 5-fold. Failure of adequate lower extremity fasciotomy is a well-recognized clinical problem with life-threatening consequences in the civilian and military surgical domains.

GRSs vs IPSs

All GRS1 scores (evaluation and management) were higher for experts than residents after training, and the management component score of the IPS showed no change with training. However, management is not part of the ASSET course curriculum. For the IPS, differences occurred between procedures; in general, IPS and GRS had convergent validity. Unlike IPSs, evaluator differences were seen in GRS scores between physicians and anatomists. Procedure-specific checklist scores, as noted by others, have better ability to identify specific interventions amenable to training or refreshing of skills than do GRS scores. Alternatively, GRS scores may capture facets of surgeon performance missed by IPSs that are particularly relevant to higher levels of expertise. The GRS scores, expert performance scores, and increase in scores with training provide convergent validity for the IPS performance metric.

Limitations

Biases occurred because evaluators could not be prevented from knowing when the surgeons had taken the ASSET course or who the experts were. To minimize these biases, residents were paired with different evaluators for each interval assessment so that the same raters did not rate the same resident at each evaluation. This approach does not enable us to determine whether an individual rater showed bias. Analysis of video by evaluators masked to before- and after-training status will in the future address this potential bias. Surgeons were not as stressed as they would be in a busy trauma center (ie, the cadavers had no bleeding, hematoma, or tissue damage as there would be in live trauma patient situations), and there were no assistants. Surgeon performance could have been affected by not having support personnel, who provide not only additional hands for exposure but also often feedback to the surgeon.

Conclusions

ASSET training improved procedural and anatomy skills in most residents, and this skill was retained for up to 18 months. Interval experience after training affected performance. The IPSs ranged from 31.6% to 76.9% among the residents for core trauma competency procedures, suggesting that some residents need remedial training in trauma technical skills, including anatomy and correct procedural steps. Resident skill before training identified competency and errors in vascular procedures of residents after training compared with experts.