Structured Textbook Review and Individualized Learning Plans Successfully Remediate Underperforming Residents and Improve General Surgery Program Performance on the ABSITE

Abstract

Background

We endeavored to create an evidence-based curriculum to improve general surgery resident fund of knowledge. Global and resident-specific interventions were employed to this end. These interventions were monitored via multiple choice question results on a weekly basis and American Board of Surgery In-Training Examination (ABSITE) performance.

Methods

This study was performed in a prospective manner over a 2-year period. A structured textbook review with testing was implemented for all residents. A focused textbook question-writing assignment and a Surgical Council on Resident Education (SCORE)-based individualized learning plan (ILP) were implemented for residents scoring below the 35th percentile on the ABSITE.

Results

Curriculum implementation resulted in a statistically significant reduction in the number of residents scoring below the 35th percentile, from 50% to 30.8% (P = .023). One hundred percent of residents initially scoring below the 35th percentile were successfully remediated over the study period. Average overall program ABSITE percentile scores increased from 38.5% to 51.4% over a 2-year period.

Conclusion

Structured textbook review and testing combined with a question-writing assignment and a SCORE-focused ILP successfully remediated residents scoring below the 35th percentile and improved general surgery residency ABSITE performance.

Introduction

The American Board of Surgery In-Training Examination (ABSITE) is an instrument used to evaluate general surgery residents’ general fund of knowledge and progression over the course of training. An ABSITE score below the 35th percentile, particularly in the later postgraduate years, places residents at an increased risk of failing the General Surgery Qualifying Examination, as described in the literature and by the American Board of Surgery.^1,2 One systematic review of interventions to improve performance on the ABSITE determined that structured reading programs with multiple-choice questions along with remediation programs are most effective.³ Furthermore, direct involvement of program leadership in the administration of remediation plans positively influenced subsequent exam scores.⁴

Methods

All residents participated in structured textbook reading reviews from March to December for consecutive years. The review consisted of reading assignments of approximately 50 pages per week and covered each textbook in its entirety. During the first year, Schwartz’s Principles of Surgery, 11th edition, was reviewed.⁵ The second-year residents reviewed the Sabiston Textbook of Surgery, 20th edition.⁶ Surgical topics covered in the reading were reviewed on a weekly basis during protected educational time via 12 multiple-choice questions in ABSITE format. Additionally, 3 multiple-choice questions from previous weeks were reintroduced to reinforce retention, resulting in a total of 15 questions per week. Residents were required to individually answer slide-deck projected questions orally in the presence of all residents to facilitate group discussion of topics and address test-taking skills. Residents were divided into teams, and the number of questions answered by each resident was recorded to ensure equal distribution over the course of the year. The number of correct and incorrect responses by the residents was also recorded. Residents who scored below the 35th percentile on the ABSITE were required to write 1 multiple-choice question based on the weekly reading assignment, due 3 days prior to the weekly quiz (Figure 1).

Figure 1.

Above is a textbook review of curriculum items, including (A) an example of weekly textbook reading assignments; (B) an example of a resident-produced, multiple-choice question for a weekly reading assignment quiz; and (C) a spreadsheet detailing the number of multiple-choice questions answered correctly and incorrectly by each resident and team.

Residents performing below the 35th percentile on the ABSITE more than once were entered into an individualized learning plan (ILP). The planned curriculum included all content areas answered incorrectly on the previous year’s ABSITE by that specific resident as provided by the American Board of Surgery ABSITE report. Weekly one-on-one meetings were held between the resident and program leadership to review the Surgical Council on Resident Education (SCORE) modules that corresponded with incorrectly answered topic areas. Residents underwent formative assessment during these sessions, clarifying areas of confusion within the modules and answering corresponding SCORE multiple-choice exam questions with active surveillance and discussion thereof. These sessions typically covered 2–3 SCORE modules per week over the course of approximately 10 months (Figure 2).

Figure 2.

An individualized learning plan (ILP) of curriculum items shows (A) an ABSITE score report detailing incorrect topic answers; (B) an ILP calendar; (C) a corresponding SCORE topic module; and (D) a corresponding SCORE topic multiple-choice question from a question bank.

In all cases, successful remediation was defined as scoring above the 35th percentile during the subsequent year. The percentage data were analyzed using a chi-square test, and the program percentile mean data were analyzed using a t test.

Results

Prior to initiating this curriculum, the 2020 ABSITE was taken by 13 residents: 3 PGY-5, 1 PGY-4, 3 PGY-3, 3 PGY-2, and 3 categorical PGY-1. The overall mean score for the program was 38.5th percentile. Of the non-PGY-5 residents, 5 of 10 scored below the 35th percentile (50%). These 5 residents were tasked with providing 1 multiple-choice question for the weekly review as outlined above. Additionally, 1 of these 5 residents entered an ILP as it was his/her second year of performance below the 35th percentile.

Four out of 5 residents successfully scored above the 35th percentile on the subsequent year’s ABSITE (2021), including the resident enrolled in the ILP. The average increase in percentile score among all remediated residents between 2020 and 2021 was 25.4 percentile points, with the resident who completed the ILP increasing 44 percentile points.

Overall, the 2021 ABSITE exam was taken by 13 residents. The average mean program percentile increased to 42.8. Of the non-PGY-5 residents, 5 of 10 scored below the 35th percentile (42%). This represented 4 new residents scoring below the 35th percentile and 1 who did not successfully remediate (from the prior year) via the question-writing assignment, and he/she was thus placed on an ILP.

The 2022 ABSITE was taken by 16 residents (including 1 preliminary surgery resident), and again we saw 4 out of 5 residents successfully remediated, including the resident enrolled in the ILP. The average overall increase in percentile score among all remediated residents was 25 percentile points, with the 1 resident completing the ILP increasing by 64 percentile points. The mean program percentile increased to 51.4. Of the 13 non-PGY-5 residents, 4 scored below the 35th percentile (31%).

Over a 2-year interval, the percentage of non-graduating residents scoring below the 35th percentile decreased in a statistically significant fashion from 50% to 30.8% (P = .023). Eighty percent of residents were successfully remediated from the question-writing assignment alone (n = 10) with the average percentile score increasing by 25.2 percentile points using this method. Both residents who completed question writing plus an ILP were successfully remediated with average percentile scores increasing 55 percentile points (Table 1). During this time period, the program saw an overall average increase of percentile scores from 38.5 (year 0) to 42.8 (year 1) to 51.4 (year 2) with the implementation of this curriculum (P = .12).

Table 1.

Cumulative Program Performance on ABSITE Over a 3-Year Period

	Residents below 35th percentile (n, % total)	Program performance (average percentile, standard deviation)	Remediation success from previous year (question writing, ILP)
Curriculum initiation	5, 50%	38.5, 22.6	N/A
Year 1	5, 42%	42.8, 21.4	75%, 100%
Year 2	4, 31%	51.4, 20.7	75%, 100%
P value	.023	.12	N/A

Abbreviation: ILP = individualized learning plan

Discussion

This study details a curriculum that successfully remediated underperforming residents and improved overall program performance on the ABSITE over 2 consecutive years. Drawing on the experience of past published reports, we sought to implement clear expectations for remediation by using a 35th percentile cutoff as a normative assessment and to institute study plans that allowed formative assessment by program leadership.^1–4

As a structured reading program has been shown to be most effective, we chose to use textbooks as our primary source for educational review as these references are comprehensive and appropriate for all PGY levels.^5,6 These detailed sources also easily lent themselves to evidenced-based learning strategies of spaced practice and test-taking.⁷ While the ABSITE reports 3 forms of performance (standard score, percentile score, and percent correct), we chose the percentile score as it most closely correlated to the American Board of Surgery Qualifying Examination pass and failure rates. Furthermore, recent publications state that the percentile score is the unanimous choice among program directors for screening fellowship applications.^4,8

The weekly multiple-choice questions were tested in a group setting during protected educational time. We believe this offered multiple advantages over individual test completion at resident discretion, including the following: (1) clarity on question topics could be achieved through discussion, often leading to additional learning points and engagement of teaching by fellow residents; (2) exam-taking strategies and coaching could be provided as the residents verbalized their thought processes for answering each question; and (3) the format ensured that correct answers were obtained from conceptual understanding and not from open-book references or guessing.

We found the assignment of weekly question writing to be effective in remediating residents on the first instance of scoring below the 35th percentile, with a success rate of 80%. This assignment guaranteed that the resident reviewed the weekly reading assignment to some degree. We also received feedback from the residents themselves that the exercise of question writing improved their test-taking ability as they gained insight from the perspective of the question writer. An additional advantage to this weekly test is the low utilization of program leadership time necessary to administer the intervention. This mostly self-directed assignment allowed program leadership to concentrate on ILPs for residents in greater need of intensive remediation secondary to a second instance of scoring below the 35th percentile. Questions written by the residents were reviewed and edited. The remainder of the weekly 15 questions were written by program leadership, then organized and presented using a slide deck.

The SCORE-based ILP proved to be effective in each of the 2 years with an average increase of 55 percentile points on subsequent year ABSITE exams. This tailored plan allowed for residents to focus specifically on modules/subjects for which they performed poorly on the ABSITE and thus redirected residents from an ineffective strategy of restudying material with which they were already familiar. These data reinforce previously published reports detailing successful remediation with active involvement of program leadership. Furthermore, we believe this curriculum has a distinct advantage in preparing residents for board examinations as the American Board of Surgery is actively involved in selecting SCORE content.⁹

While the overarching concepts implemented in this study have been described, we believe that several details of our curriculum and execution are novel and warrant distribution. First, weekly textbook reading with public and collaborative review of multiple-choice questions allows for topic review and test-taking strategy instruction to teach residents en masse. Second, question-writing assignments as remediation provide an opportunity to improve the fund of knowledge with minimal utilization of program resources. Third, SCORE-based ILPs provide an easily accessible and highly board-exam-relevant curriculum, given its construction in conjunction with the American Board of Surgery.

This study was limited by its single-institution nature and short observation period of 2 years. Additionally, data points were limited to percentile scores and did not account for standard scores or percent correct. During the first year of implementation, the prior year PGY-5 residents did not have the opportunity to participate in this curriculum, so for the sake of consistency, the remediation success rate calculation does not include PGY-5 residents. However, all PGY-5 residents during the subsequent 2 years of analysis scored above the 35th percentile in their final year of taking the ABSITE.

Future improvements in our process would include the incorporation of ILPs for all residents, and we would advocate for this system to be dispersed in the widest possible manner as programmatic resources can accommodate. Additionally, we aim to institute a program-focused, “high-yield review,” which would entail reviewing SCORE modules corresponding to topics on the ABSITE that were incorrectly answered by more than 50% of residents in a given year.

Conclusion

Structured, year-long study plans that utilized multiple-choice questions based on a comprehensive information source effectively increased the general surgery resident fund of knowledge and corresponding ABSITE performance. Question-writing assignments and ILPs are simple and effective remediation strategies for residents underperforming on the ABSITE.

Funding Statement

This research was supported (in whole or in part) by HCA Healthcare and/or an HCA Healthcare-affiliated entity.