Use of an Integrated, Anatomic-Based, Orthopaedic Resident Education Curriculum: A 5-Year Retrospective Review of Its Impact on Orthopaedic In-Training Examination Scores

Abstract

Introduction

Experts have called for a comprehensive didactic curriculum in orthopaedic residency training. This study examined the effects of an anatomic-based, integrated conference program on annual Orthopaedic In-Training Examination (OITE) scores at a single orthopaedic residency program.

Methods

We implemented a new, integrated, anatomic-based curriculum in January 2005. Differences between scores were analyzed by postgraduate year (PGY) of training. OITE scores (percentile ranking and raw scores) of year 1 (PGY-2) through year 4 (PGY-5) residents exposed to the curriculum (2005–2009) were compared to prior PGY-2 through PGY-5 residents (2000–2004) who had experienced the previous unstructured curriculum. To evaluate for cohort effects, United States Medical Licensing Examination (USMLE) Step I scores for these 2 groups were also compared.

Results

Eight residents were exposed to the new conference program and 8 to the prior conference program. All residents' percentile rankings improved after exposure to the curriculum, although improvement was not statistically significant for all participants. The most dramatic improvements in OITE scores were seen for PGY-4 and PGY-5 residents, which improved from 65th to 91st percentile (P  =  .03) and from 66th to 91st percentile (P  =  .06), respectively. There were no differences between the cohorts in USMLE Step I scores.

Discussion

Initiation of an integrated, anatomic-based, resident conference program had a positive impact on resident performance on the OITE.

Editor's Note: Further information or complete curriculum ^{(26KB, doc)} details can be made available upon request to the authors.

Introduction

The Orthopaedic In-Training Examination (OITE) measures resident knowledge and allows for comparisons of individuals within a program and also among programs nationally.^1,2 Performance on the OITE has been demonstrated to correlate with resident performance on Part I of the American Board of Orthopaedic Surgery Certifying Examination.^3,4 Recent studies about the OITE have focused on its content and the impact of resident study habits and study time on OITE scores.^5,6

Our program has a daily (Monday–Friday) hour long conference, with protected time, and attendance is mandatory for all residents. Active participation is encouraged. Topics were selected at the discretion of lecturing staff, residents, and invited speakers. Before 2005, the conference schedule was largely unstructured. In addition, a resident directed study session, 3 mornings a week, included a sequential review of the current Orthopaedic Knowledge Update (using an American Academy of Orthopaedic Surgeons [AAOS] publication), with discussion led by the chief residents. We hypothesized that the new structured conference format would improve residents' performance on the OITE.

Methods

Setting and Participants

The Geisinger Department of Orthopaedic Surgery has a 4-year residency program located at Geisinger Medical Center, a level 1 trauma center. The program has a census of 2 residents per year and all residents had completed a preliminary year of training at Geisinger Medical Center during the intervention.

Intervention

In 2005, an anatomic-based, integrated, comprehensive conference program replaced our previous conference program. The new conference schedule is based on a 2 year revolving curriculum. Each 6 month conference period is composed of individual 3 to 4 week blocks devoted to 1 anatomic subject (table 1). The anatomic-based approach allows residents to analyze the interaction of parts of a greater whole. With a systems approach, a perspective change occurs in which individual educational topics are centrally focused.⁷ In effect, this is like integrating the smaller pieces of a puzzle together to reveal the clarity of the complete puzzle picture. As applied to our curriculum, this change involved analysis based on anatomic locations: shoulder and elbow, forearm and wrist, hand, hip and pelvis, knee and leg, foot and ankle, and spine. The individual topics within each anatomic focus are integrated. For example, during the anatomic focus on knee and leg, topics in areas such as physical examination, imaging, trauma, pediatrics, joint replacement, sports medicine, and anatomy would be included (appendix provided as online supplemental material). Other subject areas such as basic science, physiology, rehabilitative medicine, and orthopaedic oncology are integrated into each anatomic focus where practicable. Additional educational blocks are added for these and other subjects such as ethics, epidemiology, statistics, applied pharmacology, and practice management. The morning resident-directed Orthopaedic Knowledge Update review was also integrated into the curriculum, with the chapters reviewed corresponding to the current anatomic focus.

TABLE 1.

Sample 6-Months Curriculum

Analysis

The unpaired 2-sample t test (version 9.2; SAS Institute Inc, Cary, NC) was used to determine significant differences in OITE scores. We compared preintervention versus postintervention mean scores and percentile rankings within each resident training year. In addition, a 2-sample t test was used to determine if a significant difference was present between the 2 resident cohorts' United States Medical Licensing Examination (USMLE) Step I scores. P values < .05 were considered statistically significant.

The OITE was administered in accordance with the guidelines recommended by the AAOS Evaluation Committee, which included the use of proctors and a strict time limit of 7 hours. Resident OITE scores were examined for the years 2000–2004 (controls) and 2005–2009 (intervention). Individual scores were deidentified and designated by postgraduate year (PGY) for each specific year of OITE administration. To control for changes over time in the difficulty of the examination, both individual raw test scores as well as nationwide program ranking were examined for each year from 2000 to 2009. We used the yearly percentile rankings for our program, as compared to national data and standard deviations, to calculate percentile rankings for each year in training from 2000–2004 and 2005–2009.

To ensure that a change in OITE scores was not attributed to a change in the quality of residents accepted into the program, the USMLE Step I examination scores of the same resident cohorts were collected. The study was conducted under Institutional Review Board approval.

Results

The means and standard deviations of examination scores and nationwide percentile rankings, according to year in training, for pre-intervention versus postintervention, are shown in table 2. Twenty-six residents were included in the study. During the pre-intervention period, junior residents tended to have lower raw scores but higher national rankings on the OITE than their senior resident counterparts. PGY-2 and PGY-3 mean scores were in the 86th and 82nd percentiles, respectively, compared to mean scores in the 65th and 66th percentiles for the PGY-4 and PGY-5 years, respectively. In the postintervention period, junior residents continued to have lower mean raw scores, but the national rankings of both junior and senior residents were similarly improved (ie, all were 90th percentile or higher).

TABLE 2.

2000–2009 Orthopaedic In-Training Examination (OITE) Test Scores and Program Nationwide Rank Pre and Post Curriculum Implementation^a

Our PGY-4 residents noted a statistically significant improvement in OITE scores postintervention (P  =  .03) compared to national results. PGY-5 residents had a marked improvement in their postintervention scores, although not statistically significantly (P  =  .06). All residents' percentile rankings improved postintervention, although not statistically significantly. We found that mean scores improved from the pre-intervention to postintervention periods for all resident years except PGY-3, but these improvements (+3.3 to +10.6 points) did not reach statistical significance.

Residents in our analysis were reviewed for their USMLE Step I scores and no significant difference was found between the residents in the pre-intervention and postintervention groups (P  =  .85). The average score for the pre-intervention group was 224.2 (SD  =  15.2), and 226.0 (SD  =  16.9) for the postintervention group.

Discussion

Our intervention was associated with an improvement in OITE scores across all years postintervention. The most significant improvement was seen in the PGY-4 and PGY-5 residents with score improvements in the 65th to the 91st percentile and in the 66th to 91st percentile, respectively.

At our institution, resident participation in Orthopaedic Conference begins at the start of the PGY-1 year. This early exposure to a formal didactic program has traditionally benefited the knowledge base of junior residents (PGY-2 and PGY-3). During the preimplementation period, junior residents tended to perform better on the OITE than their senior counterparts in comparison to national averages for year in training (table 2). The effects of the curriculum change were most notable for the senior residents, PGY-4 and PGY-5.

Other studies have shown that a formal curriculum or protected educational block improves in-training examination scores in several specialties.^8–10 In addition, restructuring of surgical rotations, ensuring conference attendance, and implementation of a formal basic science curriculum have also been shown to positively influence examination results.^11–13 The ability of resident staff to successfully oversee a surgical curriculum has also been reported.¹⁴

During the 10 year time period studied, the curriculum change was the only substantive change to the educational program. Our program did experience a leadership change (the residency director assumed the position of department chairman, the assistant residency director became residency director), but because there was no fundamental change in resident education philosophy, curriculum, or conference schedule, this change had little or no impact on the results of this study.

Our study has several limitations, including its single-site administration and small sample size. The latter limits the ability to detect a statistically significant difference even when a large absolute difference was present (beta II error).

Once established, maintenance of the current curriculum did not require any additional time investment for coordination. Informal feedback from the resident complement has been overwhelmingly positive. Further information or complete curriculum details can be made available upon request.

Conclusions

Initiation of an integrated, anatomic-based, resident education curriculum had a positive impact on resident performance on the OITE. We believe that this change reflects an improvement in the medical knowledge imparted at our institution. The structured conference curriculum is practicable, well-accepted by the resident complement, and easily reproducible.