Anesthesia Resident Training Experience Minimally Impacts Emergence Time, Making Correlation of Resident Competency With This Operational Metric Difficult

Abstract

Introduction:

Anesthesia residents are deemed competent based on subjective and objective metrics. Knowledge acquisition and procedural skill is often difficult to accurately measure. Inspecting tangible metrics of perioperative efficiency may provide a source for reliable evaluation.

Methods:

Retrospective case-log database review yielded 3072 surgical cases involving residents over 5 years. Primary variable investigated was the time from surgery completion to exit from operating room. Other variables recorded included day of week, attending anesthesiologist name, attending surgeon name, patient age, sex, American Society of Anesthesiologists physical status (ASA PS) classification, and inpatient versus day surgery status.

Results:

After controlling for procedure duration time, inpatient status, ASA PS, surgeon, and attending anesthesiologist, resident training time had high statistical significance. In the fully adjusted model, 1 year of resident training was associated with a reduction in emergence time by 28 seconds. A 1-hour increase in procedure time was associated with an increase in emergence time of 34 seconds.

Conclusions:

Although a statistically significant correlation between anesthesia resident training time and emergence time was demonstrated, the clinical significance is likely low given the relatively small amount of actual time saved. We caution the value of using perioperative metrics (e.g., emergence time) for evaluating anesthesia resident competency, until such metrics have undergone significant validation.

Introduction

The Accreditation Council for Graduate Medical Education (ACGME) has established 6 core competencies for residency training programs.¹ Anesthesia residency engages a multitude of educational pedagogies and training environments. This method of education is frequently difficult to accurately assess, measure, and correlate to end points and outcome improvement.^2–4 In a recent review, Weller et al⁵ evaluated the current state of competency-based medical education in anesthesia programs. They concluded that there is a dearth of studies across the core components of competency-based medical education and a “large number of unanswered questions.” In the current competency-based model, both subjective and objective metrics are used to assess resident competency. Subjective metrics include individual faculty evaluations and group assessments by clinical competency committees (CCC).⁶ Some commonly used objective metrics include in-training examinations (ITE) scores, other written examinations, and case minimums.⁷ These metrics may demonstrate acquisition of medical knowledge and clinical experiences, but may not definitively reflect an individual’s clinical competency. As an example, Sessler et al⁸ investigated the potential correlation between duration of intraoperative hypotension and resident CCC evaluations and ITE scores. Their study concluded that there was no association between 5 hypotension management metrics and CCC evaluations or ITE scores. Anesthesia-controlled time (ACT) metrics are used to assess anesthesia operating room (OR) efficiency.^9,10 The specific impact anesthesia residents have on ACT is mixed. Urman et al¹¹ demonstrated that anesthesiologist-resident care teams have worse induction and emergence times, but improved turnover times, as compared with solo anesthesiologists. Eappen et al¹² investigated the effect of new anesthesia trainees on OR efficiency and concluded that new trainees are unlikely to have any “clinically or economically meaningful adverse effect.”^(p1210) In an earlier work,¹³ we used ACT to assess the correlation between OR turnover time and resident training time (ie, years of residency). Our data demonstrated that mean turnover time was inversely related to resident training time. Based on these findings, we further investigated emergence time as a function of anesthesia resident training level. In a prior study of prolonged emergence, House et al¹⁴ demonstrated an inverse relationship between anesthesia resident training level and prolonged emergence (emergence time more than 15 minutes). Expanding on this prior work, we aim to assess the relationship between level of anesthesia resident training and anesthesia emergence time, and the potential use of these data to assess resident efficiency and possibly competency.

Materials and Methods

Data Collection

This retrospective, observational study polled the OR case-log database maintained by a tertiary care facility. The anesthesiology department consisted of 27 residents (9 residents per year) and 23 faculty members at the main facility investigated. The Drexel University Institutional Review Board ruled this study exempt from review. The nonemergent scheduled adult cases of every nonholiday weekday at every anesthetizing location (ie, no “off-site” locations) from July 1, 2013, through February 28, 2018, constituted the initial observations. Information extracted from the database consisted of the date, day of week, anesthesiology resident name, attending anesthesiologist name, attending surgeon name, patient age, patient sex, American Society of Anesthesiologists physical status (ASA PS) classification, inpatient versus day surgery status, surgery initiation and completion time, and time the patient entered and exited the OR. Resident progression was calculated as “days of training,” determined by each resident’s start date and the date of surgery. Only cases of ASA PS 1, 2, or 3 patients involving an anesthesia resident and attending anesthesiologist were retained. ASA PS 4 and 5 were excluded due to anticipated increase in error due to the complexity of these cases and the difficulty to accurately assign emergence times. Transplant, intrathoracic, obstetric, neurosurgical, and otolaryngological procedures were also excluded based on the same rationale.

During data cleaning, errors were discovered in procedure duration times, emergence times, and resident, surgeon, and anesthesiologist names. Duration and emergence times less than 10 and 3 minutes, respectively, were considered suspicious and were verified or corrected from original records, as were errors in names that appeared to be alternate spellings of previously included individuals. After these corrections were made, an audit of 30 previously unexamined cases was performed. The audit uncovered errors in 1 resident name and 2 ASA PS entries, but no additional errors in duration or emergence times.

Analysis

The distributions of continuous variables were examined with descriptive statistics and histograms. Because emergence time had outliers, values greater than 30 minutes were converted to 30 for analysis. The continuous predictors were sufficiently normal to be used in linear regression, but Spearman correlations were used in bivariate analysis to compensate for remaining deviations from normality. T-tests and analyses of variance were performed to measure levels of association between categorical predictors and the outcome variable. These analyses gave a preliminary sense of the data, but do not account for clustering of cases within residents, surgeons, and anesthesiologists, which was addressed in the multivariate models.

A multivariate mixed model was built using a forward selection process in which covariates were added to the model in order of bivariate significance. Variables with P > .20 were removed from the model. After all variables were added, variables with P > .05 were removed from the model. Correlation due to repeated measures on residents was controlled for by modeling resident as a random effect. The effects of surgeons and attending anesthesiologists were entered as fixed factors. The large number of possible combinations rendered an analysis in which all 3 provider groups were considered as random factors statistically challenging, so only residents (the most critical of the 3) were analyzed in that way. Surgeons and attending anesthesiologists with fewer than 5 cases each were grouped. Due to concerns with data reliability, a sensitivity analysis was run removing ASA PS from the final model. All analysis was performed using Statistical Analysis System version 9.4 (Cary, NC).

Results

A total of 3072 cases were used in the final analysis. The 63 residents involved with the study participated in 1 to 111 cases each, with training times ranging from 0 to 2.99 years at the time of the case. Procedure duration ranged from 0.02 to 4.23 hours. Emergence time before being capped at 30 ranged from 2 to 193 minutes with a median (interquartile range) of 12 (8–16); 79 of the 3072 procedures (2.3%) had times over 30 minutes. Study cases involved 115 unique surgeons with 1 to 202 cases each, and 42 attending anesthesiologists with 1 to 250 cases each. After those with fewer than 5 cases each were grouped, there were 73 unique surgeon IDs and 33 anesthesiologist IDs in the final model. See Table 1 for additional case characteristics.

Table 1.

Characteristics of Cases Used in Analysis

Characteristic	Value, N = 3072
Patient gender, female, n (%)	1619 (52.7)
Patient age, mean (SD)	49.0 (13.86)
Inpatient procedure, n (%)	947 (30.8)
ASA physical status classification, n (%)
I	262 (8.5)
II	1603 (52.2)
III	1207 (39.3)
Resident training, median (IQR), ya	0.69 (0.30–1.71)
Procedure duration, median (IQR), hr	1.03 (0.52–1.70)
Emergence time, median (IQR), min	12 (8–16)

Abbreviations: ASA, American Society of Anesthesiologists; IQR, interquartile range.

^a At the time of the procedure.

Of the continuous covariates, only procedure duration time was significantly correlated with emergence time (r = 0.145, P < .001). Independent-sample t-tests indicated that inpatient emergence times were 97 seconds (95% confidence limit [CL] = 69, 126) longer than outpatient emergence times, and emergence times for male patients were 40 seconds (95% CL = 14, 65) longer than those for female patients. One-way analyses of variance indicated significant differences in emergence times by day of the week (P < .001) and ASA PS (P = .012). Correlation matrices ran on resident year, procedure duration, ASA PS (ordinal), and inpatient status (binary) all yielded correlations below 0.4.

Resident training time was not significantly predictive of emergence time in the unadjusted model. However, after controlling for procedure duration time, inpatient status, ASA PS, surgeon, and attending anesthesiologist, resident training time became highly significant (P < .001). Day of the week was no longer predictive of emergence times after controlling for surgeon and attending anesthesiologist. Resident training trends are reported by time reduction per year, although the model is stratified by days of training. In the fully adjusted model, 1 year of resident training was associated with a reduction in emergence time by 28 seconds (95% CL = 12, 44). A 1-hour increase in procedure time was associated with an increase in emergence time of 34 seconds (95% CL = 17, 50). Inpatient emergence times were an average of 69 seconds (95% CL = 36, 102) longer than outpatient times. Emergence times for male patients were 37 seconds (95% CL = 10, 63) longer on average than those for female patients. Patients in ASA PS 2 had average emergence times 41 seconds (95% CL = 13, 69) shorter than those in ASA PS 3, but there was no significant difference between emergence times between ASA PS 1 and 3. Resident covariance was significant as a random intercept (σ2 = 0.748, P = .001), but not as a random slope. This suggests that although residents differed in mean emergence times (as reflected in their individual intercepts), the slope of the line relating experience with emergence time did not differ significantly between residents. Removing ASA PS from the sensitivity model resulted in a less than 1-second change to the estimated effect of resident training time, indicating that the effect persists regardless of errors in ASA PS.

Discussion

After statistical modeling was performed, resident level of training was statistically inversely correlated to emergence time. As an example: for an ASA PS 3 outpatient female the average emergence time was 11 minutes 54 seconds for a starting first-year clinical anesthesia resident (CA-1). Analyzing data by “days of training” instead of “resident year” attempts to differentiate smaller changes in resident progression than years (eg, emergence time by a CA-1 in the 2nd month versus 12th month, a CA-1 in their last week versus a CA-2 in their first week). In this model, each year of resident training was associated with a reduction in emergence time by 28 seconds. Although a statistically significant correlation was demonstrated, the clinical and logistical significance is likely low given the relatively small amount of actual time saved. The absolute value of this amount of time is unlikely to have impacts on OR efficiency as it pertains to cost saving, as demonstrated in prior studies.^12,15 The presence or absence of this negative correlation may still offer a metric to help assess resident progression toward clinical competency. Timely emergence from anesthesia is an important part of efficient anesthesia delivery, and therefore resident competency. Even though our results do not demonstrate a reduction in emergence time that will have an impact on OR scheduling or overall efficiency and costs, this metric could be useful in assessing residents and developing individualized learning paths. Weller et al¹⁶ evaluated the reliability and validity of using supervisor scores of “amount of resident supervision required” as a tool to make decisions regarding progression to competency. Studies aiming to correlate perioperative workflow data and resident competency seek to identify less subjective means for assessment, with varying results. Emergence time may identify residents who may not be progressing appropriately and target them for additional evaluation and curriculum modification. However, results suggest emergence time has not yet been validated as a competency metric, and further investigation is required for appropriate validation.

The greater conversation to be had is whether reporting metrics by resident is an actual or assumed representation of competency. Given the minimal changes seen in this particular observation, OR metrics may be only appropriate when significant outliers to site-specific averages are observed. Aforementioned studies have investigated many facets of the same strategy. However, we can also posit that most metrics typically measured are the result of many perioperative factors (eg, patient, surgeon, anesthesiologist, nursing, and hospital-related performances and obstacles) that may not be resident related. Yet the desire to connect these to resident performance persists both in daily practice and in academic study. If the academic community collects these investigations and determines that they have substance, then resident assessments need to account for and eliminate sources of bias. Either that, or implications assigned to these trends need to be tempered appropriately or restricted to extreme outliers. If educators decide these investigations show trends too rife with bias, then work must be done within academic centers to distance the ever-increasing scrutiny over OR management and workflow from perceived links to education, as many assume it to be.

Limitations

Data acquisition occurred at a single site. Multicenter review of more anesthesia providers caring for patients in higher volume and variety may provide different findings. Reviewing data over 5 years may not account for broad changes at a hospital or departmental level over seasons and years (eg, policy, perioperative staffing, and faculty changes). The ASA PS 4 and 5 patients were excluded to prevent bias, creating a more homogeneous group of patients and procedures. This likely prevented observing advanced residents in higher acuity situations. The same applies to excluding transplant, cardiac, intrathoracic, obstetric, neurosurgical, and otolaryngological procedures. Special circumstances could introduce delays beyond control of anesthesiology residents, but this exclusion may mask improved resident efficiency more training time. The inclusion criteria were designed to exclude cases that may have inaccurate or misrepresentative emergence times (ie, complex cases, emergent cases). Our data also demonstrated a central tendency toward residents earlier in their training median (interquartile range) for resident training year 0.69 (0.30–1.71). This could be due to more senior residents being at off-site rotations or doing more complex cases that were excluded from the sample. Future studies may benefit from targeting specific ranges for resident training year and capturing a given number of cases to better stratify the data by training level. It should be noted that other outstanding confounders were likely not fully corrected for, such as surgeon preference for emergence or anesthesia attending supervision at emergence. The extent of potential confounders may render perioperative workflow data irrelevant for academic evaluation. Retrospective analysis of times entered into electronic records leaves room for erroneous or inaccurate data entry and subsequently misled interpretation.

Conclusions

Competency-based medical education for anesthesia residency continues to be a developing area of study. Perioperative data trends collected for individual residents may provide a new body of objective data for assessment of clinical efficiency and growth. Trends would need to be assessed in combination with other metrics to truly establish if a resident is in fact appropriately progressing toward competency. Foreseeably, these data could be made available to program directors and integrated into the processes already in place to assess resident milestone progression and competency. The data could be correlated with other metrics such as board pass rate and ITE scores in an attempt to validate emergence time as a valid clinical competency correlate. Specifically, emergence times may not display significant enough information to warrant credible resident evaluation. Further investigation is warranted, given statistically significant findings and identifiable limitations that could be optimized. These findings and others like it should foster discussion among educators regarding whether perioperative workflow (perceived, observed, or calculated) should play a role in resident competency assessment.