Enhancement of anesthesiology in-training exam performance with institution of an academic improvement policy

Abstract

Background

Anesthesiology resident physicians across the United States complete an annual in-training examination (ITE). The ITE evaluates resident knowledge and provides personalized feedback to guide future study in low scoring sections¹. Performance on the ITE correlates with outcomes on the American Board of Anesthesiology (ABA) written board examination².

Over the last several years, declining ITE scores were observed at the University of North Carolina (UNC). In response to this decline, our department reprioritized the ITE by instituting an academic improvement policy (AIP). The AIP employed both reward for satisfactory achievement and consequence for under-performance to elevate the ITE as a “high stakes” examination. Our hypothesis was that implementation of this AIP would improve ITE scores.

Methods

ITE scores were compiled from 150 residents in the Department of Anesthesiology at UNC for graduating classes from 2004–2015. Data is presented as the number of residents scoring below the 20th percentile when compared to the national distribution before and after the AIP. In addition, average USMLE Step 1 three-digit scores for each graduating class were compared to average ITE percentile scores of the corresponding graduating class (USMLE does not provide percentile scores).

Results

Between 2009 and 2013, the number of residents who scored below the 20th percentile on the ITE increased steadily to a peak of 10 in 2011. After implementation of the AIP in July 2011, there was an 80% decrease in those scoring below the 20th percentile, from 10 to 2 residents (p<0.05).

Conclusions

Anesthesiology resident ITE scores improved after implementation of an academic improvement policy.

Introduction

Anesthesiology residents must acquire vast knowledge and skill during their four-year residency training. This knowledge and skill set is tested by the American Board of Anesthesiology (ABA) using a sequenced examination process consisting of a written multiple choice examination (ABA Part 1 examination) and a structured oral examination (ABA Part 2 examination). A resident must pass both to achieve ABA certification. Currently, the ABA Part 1 examination has a pass rate of 85% for first time test takers and is offered annually in July¹.

The ABA In-Training Examination (ITE), taken by anesthesiology residents in the U.S. and Canada every March, is designed to provide an accurate and unbiased evaluation of the resident’s anesthesia knowledge¹. The ITE also provides information comparing each resident to all other examinees at his or her training level, represented as a national percentile score¹. This ranking allows a resident the opportunity to self-assess his or her relative standing compared to their peers. Residents consistently scoring in the lower percentiles of performance on the ITE can be identified early in training and supplemental instruction can be tailored to address needs.

Performance on the ITE is an independent predictor of performance on the ABA certification examinations. Graylee et al. (2010) sought to identify objective predictors for passing the ABA certification examinations in the shortest possible time after graduation (i.e., passing both Part 1 and Part 2 on first attempts)². These investigators studied 2,400 U.S. anesthesiology residents to examine the association of several independent variables, including ITE score at the end of the first clinical anesthesia year (CA-1), with combined success on the ABA Part 1 and Part 2 examinations. Using regression analyses, they found the ITE score after the CA-1 year was a predictor of success in completing ABA certification within one year of graduation (R² = 0.15, P < 0.001) and a moderately strong predictor of high performance on the ABA Part 1 examination (R² = 0.45, P < 0.001)².

A similar correlation was seen in a study by Kearny et al. (2000), which compared resident ABA ITE scores with performance on the Royal College of Physicians and Surgeons of Canada (RCPSC) certification examinations, an exam that also consists of a written examination and an oral examination³. They analyzed ITE percentile scores of 165 former residents who had completed RCPSC certification. For both the written and the oral examinations they found that scores higher than the 60th percentile for any training year were associated with an increased likelihood of success; conversely scores lower than the 20th percentile were associated with an increased likelihood of failure. Post-test probabilities demonstrated a 90% likelihood of success on the RCPSC examinations for residents consistently scoring above the 60th percentile³.

The correlation of ITE percentile scores with board examination success or failure has also been seen in other specialties including surgery, emergency medicine, and pediatrics^{5, 9}. A study conducted by Althouse et al. (2008) found that the predictive power of the Pediatric ITE percentile score increased with each year of training, providing the greatest power in the third year of residency⁴.

Given the predictive value of ITE performance on subsequent certification examinations and a recent decline in the ITE scores at the University of North Carolina (UNC) (see Figure 1), an Academic Improvement Policy (AIP) was designed and implemented July first of 2011 (see Appendix A). Key interventions for low scoring residents included: closer academic guidance and supervised meetings with a faculty advisor, development of a personalized study plan with an emphasis on self-study and reading, meeting with a test taking expert, mandatory departmental didactic conference attendance, and withdrawal of moonlighting privileges. Each intervention is then tailored to the specific needs of the individual resident. Low scoring CA-2 residents do not receive prepayment of the ABA written board examination fee. Low scoring CA-3 residents are at risk for an unsatisfactory report to the ABA for failure in the domain of medical knowledge. Given the timing of the ITE examination, CA-3 residents assigned an unsatisfactory report have a six month extension of their residency.

Figure 1. Class Cumulative ITE scores versus Step 1 Scores.

Graduating class USMLE Step 1 scores are plotted along with average ITE scores. The Step 1 scores do not show a decreasing trend over the same time period that the ITE scores decreased.

Materials and Methods

A total of 150 residents were included in the data set for the study. Eight residents were removed from the study population due to incomplete data. Nineteen residents did not complete their clinical base year at our program; however, they were included if they had ITE scores for all clinical anesthesia years at our institution. Only scores from the clinical anesthesia years were evaluated, as this is the training time that correlates with ABA pass rate and this cohort is affected by the implemented AIP². Residents with scores below the 20th percentile were identified.

ITE scores were obtained from archived reports previously provided by the testing board, which had been stored in paper format in the UNC Anesthesiology Department’s administrative office dating for each graduating class from 2004 – 2015. Each resident’s report contained an individual scaled score and a national percentile for the corresponding year. The reports were reviewed by the administrative staff, and the data were blinded to the authors of this paper. This study was granted exempt status by the UNC IRB.

The average national percentile ITE score was calculated, for each resident from the graduating classes of 2004 – 2015, by averaging the resident’s three clinical anesthesia ITE national percentile scores. These averages were then used to calculate the average national percentile ITE score for each graduating class from 2004 to 2013 (Figure 1). In addition to comparing ITE scores, we compared average USMLE Step 1 scores for each graduating class during the same time period (Figure 1).

In Figure 2, the number of residents scoring below the 20th percentile (stratified by clinical training year) is shown for the testing years of 2009 – 2013. In order to have consistency in our analysis of ITE scores, we focused on ITE scores from 2009 onwards because the test has been given in March since that year. Prior to 2009, the test was administered in July.

Figure 2. Number of Residents scoring below the 20^th percentile on the ITE.

A substantial decrease in the number of residents with low ITE scores is noted after the implementation of a comprehensive AIP for residents in the clinical anesthesia years.

Statistical Analysis

Since our study sample of residents contains scores from an individual resident multiple times as they move through their clinical anesthesia years, the compared groups of examinees partially overlap. Because the optimal approach to analyze the partially correlated data is still to be developed, we chose an unpaired t-test for proportion differences to evaluate the difference in proportions of the examinees with scores below 20^th percentile^{10, 11}. If k₁ and k₂ are the numbers of tests taken in year 1 and year 2, and n₁ and n₂ are the numbers of scores below 20th percentile in year 1 and year 2, then the proportions are p₁ = n₁/k₁, p₂ = n₂/k₂, and the variance of (p₁ − p₂) = p₁(1 − p₁)/k₁ + p₂(1 − p₂)/k₂. The 95% confidence intervals for proportion differences were calculated using the formula: (p₁ − p₂) ± 1.96×sqrt [var (p₁ − p₂)]. Statistical significance of the difference was evaluated via a z-score: z = (p1 − p2)/sqrt [var (p1 − p2)]. P-values < 0.05 were considered statistically significant.

Results

Average ITE percentile scores for each graduating class from 2004-2013 are plotted in Figure 1. A significant decline in the average score is noted in 2006. This decline continues until the implementation of the AIP in 2011. Figure 1 shows average USMLE Step 1 scores, which were maintained at a consistent high level, for each graduating class from 2004 – 2013. These results suggest that the decline in ITE scores was not secondary to a downward shift in the resident cohort’s ability to perform well on standardized tests.

Between 2009 and 2013, the number of residents who scored below the 20th percentile on the ITE increased steadily to a peak of 10 in 2011 (Figure 2). After implementation of the AIP in 2011, there was an 80% relative reduction in those scoring below the 20th percentile from 10 (24%) to 2 residents (5%). This is a 19% absolute reduction (95% confidence interval: 5% to 34%), calculated using the formula provided in the statistical analysis section. The reduction was statistically significant (z-score 2.50, p = 0.01). This improvement was maintained in 2013 as only 4 (10%) residents scored below the 20th percentile. Comparing scores from 2013 to 2011, there is a 14% decrease (95% confidence interval: 0% to 30%; z-score=1.71, p=0.09) in the number of residents scoring below the 20th percentile

Discussion

Given the evidence for correlation between ITE scores and success on the ABA Board Examination, it is important to identify effective interventions to improve the performance of residents who score poorly on the ITE^{2, 3}. The results of our study indicate that the AIP implemented at the UNC Department of Anesthesiology in 2011 was associated with an improvement in the ITE scores of the anesthesiology residents. Notably, an 80% reduction in the number of anesthesiology residents performing below the 20th percentile was seen in the first year after AIP implementation. This improvement was maintained during the second year after the AIP implementation.

One possible contribution to the success of this AIP is that it provides both reward and punishment. Residents who do well on the examination are rewarded with moonlighting privileges and prepayment for their ABA written board examination. Residents who perform below par receive a variety of negative consequences, the most severe being a potential six-month extension of residency.

There has been a major change in the ITE during the time frame used in this study. In 2009, the test administration changed from July to March. This gave third year clinical anesthesia residents the opportunity to take the ITE a few months prior to their actual board certification exam. This change in the timing of the test administration may have theoretically improved ITE raw scores as each resident had an additional 8 months of clinical experience and self-study time prior to taking the exam. However, since all residents have this additional experience, the national percentile ITE scores should not be affected.

There are limitations with the study. As is common in medical education research, the sample size is small. Also, this is an observational study from a single institution. We argue that our findings should be applicable to other centers. The AIP helped create an increase in resident and faculty awareness of declining ITE scores; additionally, the AIP highlights the importance of scoring well on the exam. Our department’s increased emphasis on the importance of resident ITE scores may have contributed to the observed improvement in performance. We also looked at resident cohorts, stratified by test year and year of graduation. We did not trend individual resident ITE performance due to potential of multiple confounding factors that may affects one’s score (such as a personal tragedy interrupting training).

Due to the recent adaptation of the AIP at UNC, we only have two years of data to study its effect on ITE scores. Further study can provide vital information regarding whether or not there is a sustained improvement in scores. De Virgilio et al. (2002) demonstrated improvement in ITE scores after the implementation of an AIP at the University of Virginia but also found that this change was not maintained in some of the study participants for the following year^{7, 8}. Additionally, as our residents progress through training and graduation their performance on the ABA Part 1 examination and Part 2 examination could potentially be used to develop a post-test probability of success in certification based on ITE scores.

Although a significant improvement in ITE scores was seen after implementation of the AIP, there is still important work to be done. The goal of all anesthesiology residencies is to produce competent, well-educated anesthesiologists capable of providing the best care for their patients. The predictive value of ITE scores has been demonstrated before, and here we illustrate a successful method to identify and aid at-risk residents early in their training. While negative consequences are a feature of our AIP, our main focus is to provide residents with the individual support needed to help them become the best anesthesiologists possible.

Conclusion

Prior to the implementation of the AIP in 2011, ITE scores for anesthesiology residents at a major academic center, the University of North Carolina, were trending in a negative direction despite no apparent change in the academic resources provided to residents. Following implementation of an AIP, which rewards residents who perform well on the ITE and provides extra assistance to academically at-risk residents, there was significant improvement in ITE scores. Further study is needed to determine if this improvement is sustained and to refine the AIP as needed to best aid residents in their training.

Appendix A

Academic Improvement Policy

1. All residents, including those in the clinical base year, must take the ITE.

2. Resident scores from the clinical base year are not subject to the AIP.

3. CA-1 and CA-2 residents: Must score at or above the 20th percentile. An ITE score below the 20^th percentile will result in the following:

   1. An unsatisfactory (U) grade reported to the ABA for that 6-month academic period.
   2. Required remediation program consisting of meeting with an academic advisor to develop a personalized study plan, meeting with a test-taking strategist, and mandatory attendance to all departmental didactic conferences.
   3. Withdrawal of moonlighting privileges.
   4. CA-2 residents only: Scores at or below 25th percentile results in no prepayment for the ABA written boards. (If resident scores >25th percentile as CA-3, this fee will be reimbursed.)

4. CA-3 residents: Must score above the 15th percentile. An ITE score at or below the 15th percentile will result in a U grade reported to the ABA for that 6-month academic period.^*