General Surgery Resident Use of Electronic Resources: 15 Minutes a Day

Abstract

Background:

General surgery resident performance on the American Board of Surgery In-Service Training Exam (ABSITE) has been used to predict American Board of Surgery (ABS) passage rates, selection for remediation programs, and ranking of fellowship applicants. We sought to identify electronic resource study habits of general surgery residents associated with successful test scores.

Study Design:

A single institution, retrospective review of general surgery resident use of two electronic study resources, Surgical Council on Resident Education (SCORE) and TrueLearn (TL), were evaluated for the 7 months prior to the 2019 ABSITE. Metrics included TL question performance, SCORE utilization, and a survey about other reading sources. These metrics were evaluated in three ABSITE percentile groupings: ≥80th, 31^st – 79th, and ≤30th.

Results:

The ≥80th and 31^st– 79th percentile groups scored higher on TL questions at 69% and 67.7% compared to 61.4% for the ≤30th percentile group (p<0.03). The ≥80th percentile group spent on average 14.6 min/day on SCORE compared to 5.0 min/day and 4.7 min/day for the 31^st – 79th and ≤30th percentile group respectively (p<0.04). The ≥80th percentile group spent 34.8 min/session (77 sessions) compared to 19.2 min/session (49 sessions) and 20.7 min/session (43 sessions) in the 31^st −79th and ≤30th percentile group respectively (p=0.009).

Conclusions:

Our nomogram incorporates time spent accessing an electronic content-based resources, SCORE, and performance on an electronic question-based resource as a novel method to provide individualized feedback and predict future ABSITE performance.

Graphical abstract

Introduction

The American Board of Surgery (ABS) first administered the American Board of Surgery In-Training Examination (ABSITE) to general surgery residents in 1975 to evaluate the critical cognitive knowledge required of general surgeons and to allow individuals and programs to assess their relative strengths and weaknesses.^1,2

A standardized curriculum was offered in 2008 to general surgery residency programs via the Surgical Council on Resident Education (SCORE); a consortium of seven U.S.-based surgical organizations: ABS, American College of Surgeons (ACS), American Surgical Association, Association of Program Directors in Surgery, Association for Surgical Education, Residency Review Committee for Surgery of the Accreditation Council for Graduate Medical Education (ACGME), and the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES).³

SCORE’s web portal provides a curriculum with links to content via updated textbook chapters, videos, and a multiple choice question bank that was released nationally in 2009 to enhance access. This curriculum is widely used by general surgery residencies via a subscription service.⁴

As of 2014, the ABSITE has been aligned with the SCORE curriculum and currently is a 250 question multiple-choice examination comprised of 72% patient care topics, 24% medical knowledge topics, and 4% other topics administered annually to general surgery residents.⁵

ABSITE performance is important for individual general surgery residents and general surgery programs because of its correlation with passage of the ABS Qualifying Exam (QE). An ABSITE score below the 35^th percentile anytime during residency is a risk factor for not passing the ABS QE. Until recently, the first time pass rates of both the ABS QE and ABS certifying examinations of at least 65% are used by the ACGME as one of the measures of the educational success of a surgery residency program (6). Additionally, although not its original intent, ABSITE performance is used to rank fellowship applicants and a score below a 30^th percentile score has been used to dictate resident participation in remediation programs.^7–9

Several survey-based studies have demonstrated the variable impact of study frequency, study duration, residency level of training, clinical experience, structured reading programs, didactic conferences, practice tests, and problem based learning on ABSITE performance.^10–17

National resident utilization characteristics of SCORE has been described and its’ usage is correlated with improved ABS QE scores.^18–21 Additionally, utilization of a question-based electronic resource TrueLearn (TL) has been correlated with ABSITE performance.²² However, SCORE usage characteristics and the correlation with ABSITE performance have not been described.

Surgery residents have access to a multitude of resources to prepare for the ASBITE. Our aim is to analyze the surgery residents’ self-directed utilization of electronic resources (SCORE and TL) and the pattern of utilization associated with higher ABSITE performance. In addition a survey of other study resources and habits was performed. These electronic resource metrics were then used to make a nomogram as an attempt to predict resident success on the ABSITE.

Methods

Participants

At a single institution, an IRB-approved retrospective review of all general surgery resident utilization of the electronic resources (SCORE and TrueLearn) was conducted for the seven months prior to the 2019 ABSITE. Utilization characteristics of residents were stratified into three groups based on their 2019 ABSITE percentile; 80^th and above, 31^st to 79^th, and equal to or less than 30^th percentile. Group stratification into ≤30^th and ≥80^th was based on work previously described by deVirgilio et al.¹⁰

SCORE

SCORE metrics analyzed were utilization in total minutes, total number of sessions, minutes averaged per session and per day. Further descriptive SCORE data included day of week access and time of day access. Residents were provided with open access to SCORE. With regard to utilization of this electronic resource, there were no expectations or consequences set by the program for the surgical residents. None of the authors have any affiliation with SCORE nor have they received any financial support from SCORE. SCORE was not involved in the design of this study or in any analysis.

TrueLearn

TrueLearn metrics analyzed were number of total questions answered, percent questions correct, number of unique questions, questions/day, and TL percentile. Residents were provided with open access to TL. Residents were assigned weekly TL quizzes and had open access to create their own quizzes as well. There were no consequences set by the program for the surgical residents with regard to utilization of this electronic resource. None of the authors have any affiliation with TL nor have they received any financial support from TL. TrueLearn was not involved in the design of this study or in any analysis.

Survey

A survey was conducted within five months after the 2019 ABSITE where residents estimated the amount of time per week they used an ABSITE study guide, Surgical Education and Self-Assessment Program (SESAP), read a textbook, or read extant literature. This is a non-validated survey meant to further delineate what residents use to prepare for the ABSITE as not all residents accessed SCORE.

Statistics

The relations between the ABSITE percentile and the SCORE, TL, and survey study variables were separately modeled by linear regression. Number of SCORE visits was log (base 2) transformed prior to analysis to reduce skewness. A 95% level of confidence was assumed (alpha=.05). An optimal predictive model was selected as the model relating ABSITE percentile to TL percent correct and each of the remaining study variables which showed evidence of significant association in the above models, with and without interactions, selecting as optimal the model with the lowest Akaike information criterion in which a likelihood ratio test indicated significant improvement over the single-predictor models. Three-predictor models were also explored, but none improved upon the best two-predictor model. Statistical analyses were performed using R statistical software.²³ The nomogram was produced using the “Regression Modeling Strategies” package. ²⁴

Results

Participants

All 32 general surgery residents (34% female) in our five categorical residents per year program were included in the analysis for this study. This includes two “preliminary” residents and four residents that were in designated research years. This study was conducted over a period of 215 days starting at the beginning of the academic year and was used to calculate “per day” metrics.

SCORE

The >80^th percentile group spent an average 15 minutes/day on SCORE where the 31^st –79^th percentile and ≤30^th percentile group both spent an average 5 minutes/day. SCORE total minutes showed significant evidence of association with ABSITE percentile, where each additional minute was associated with a .006 ABSITE percentile increase, p=.033. This is equivalent to saying that each additional hour was associated with a 0.36 percentile increase, or that each additional 10 hours was associated with a 3.6 percentile increase. SCORE minutes/day showed significant evidence of association with ABSITE percentile, where each additional minute/day was associated with a 1.3 ABSITE percentile increase, p=.033. SCORE minutes/session showed significant evidence of association with ABSITE percentile, where each additional minute/visit was associated with a 0.2 ABSITE percentile increase, p=.020. Number of SCORE visits showed significant evidence of association with ABSITE percentile, where each doubling of the number of visits was associated with a 4.7 ABSITE percentile increase, p=.043 (Table 1).

Table 1.

Association of Study Metric and American Board of Surgery In-Training Exam Percentile.

Metric	p Value
SCORE
Total mins	0.033
Mins/d	0.033
Mins/visit	0.020
No. of sessions	0.043
TrueLearn
% correct	0.026
TL-predicted percentile	0.249
No. of unique questions	0.106
Total questions	0.129
Questions/d	0.131
Survey (min/wk)
Literature	0.048
SESAP	0.418
Textbook	0.243
ABSITE study guide	0.362

ABSITE, American Board of Surgery In-Training Exam; SESAP, Surgical Education and Self-Assessment Program; SCORE, Surgical Council on Resident Education; TL, TrueLearn

Time of day and day of week access associated with ABSITE percentile group are depicted in Figures 1 and 2. High ABSITE performers access SCORE an average 39 times during both 6am to 6pm and 6pm to 6am. Low ABSITE performers access SCORE an average 29 times between 6am and 6pm and 15 times between 6am and 6pm. Additionally, high ABSITE performers also access SCORE more often on weekends. The ≥80^th percentile group accessed SCORE an average of 9 times on Saturday and 12 times on Sunday while the ≤30^th percentile group accessed SCORE an average 3 times and 5 times respectively.

Figure 1.

Surgical Council on Resident Education (SCORE) time of day access. The ≥80^th percentile group accessed SCORE more consistently overall throughout the day and the ≤30^th percentile group was less likely to access SCORE at night. There was no significant difference in average SCORE accesses between 6:00 AM and 6:00 PM but there was significant difference between 6:00 PM and 6:00 AM (p = 0.3, 0.02).

Figure 2.

Surgical Council on Resident Education (SCORE) day-of-week access. The ≥80^th percentile group accessed SCORE more consistently throughout the week and the ≤30^th percentile group accessed SCORE less often during the weekend (p = 0.005). At this institution, Wednesday contains a significant amount of protected education time, which may account for its greatest amount of SCORE access for both groups.

TrueLearn

TL % correct showed significant evidence of association with ABSITE percentile, where each additional % correct was associated with a 1.7 ABSITE percentile increase, p=.026. The average number of TL questions answered by residents was 944. TL predicted percentile, TL number of unique questions, TL total questions, and TL questions/day each showed no significant evidence of association with ABSITE percentile.

Survey

Review of the literature in minutes/week showed significant evidence of association with ABSITE percentile, where each additional minute/week was associated with a 0.6 ABSITE percentile increase, p=.0475. SESAP minutes/week, Textbook minutes/week, and ABSITE study guide minutes/week each showed no significant evidence of association with the ABSITE percentile (Table 2).

Table 2.

American Board of Surgery In-Training Exam percentile group and associated Surgical Council on Resident Education usage characteristics, performance on multiple choice questions, and type of study resource used.

ABSITE percentile	Low ≤30th (n = 12)	Middle 31st – 79 th (n = 11)	High* ≥80th (n = 9)
SCORE
Total time, min, mean± SD	1017 ± 1501	1076 ± 2523	3141 ± 2436
Total visits, n, mean ±SD	43±56	49±12	78±40
Min/d, mean ± SD	5 ± 7	5±12	15±11
Min/session, mean ±SD	21±9	19±12	35±18
Time frame	Daytime/weekday	Daytime/weekday	Night/weekend
Pattern	Inconsistent	Inconsistent	Consistent
TrueLearn
% correct, mean ± SD	61±8	67±7	69±6
Survey
Adjunct resource	Textbook	Study guide	Literature

^*All variables in this chart are statistically different (p < 0.05) in the ≥80^th percentile group as compared with the ≤30^th percentile group.

ABSITE, American Board of Surgery In-Training Exam; SCORE, Surgical Council on Resident Education.

Nomogram

Combining TL % Correct and SCORE Minutes/Visit created an optimal predictive model as predictors of ABSITE percentile as depicted in Figure 3. The likelihood-ratio test showed this model was significantly better than the best single-predictor model, p=.043 and a 27% variance.

Figure 3.

Nomogram for predicting American Board of Surgery In-Training Exam (ABSITE) percentile using Surgical Council on Resident Education (SCORE) minutes/visit and TrueLearn (TL) percent correct. To estimate score, calculate points for each variable by drawing a straight line from the individual’s variable value to the axis labeled “Points.” Sum all points and draw a straight line from the total points axis to the ABSITE Percentile axis.

Discussion

The pattern of SCORE utilization at our institution correlates with published data regarding access by the time of day, frequency of study sessions per month, and duration of study sessions.^18,20 We distinguished SCORE utilization characteristics of low, middle, and high ABSITE percentile groups. This is the first study to the authors’ knowledge to analyze SCORE utilization combined with question bank performance to objectively identify characteristics of high ABSITE performers.

There are distinct differences in how high ABSITE performers access SCORE compared to low performers. Higher ABSITE performers tend to access SCORE more often and in longer sessions averaging 15 minutes per day or 35 minutes per session. Lower performers on the ABSITE average 5 minutes per day or 20 minutes per session. In addition, higher ABSITE performers tend to access SCORE in a steadier pattern throughout the year and peak during the month of ABSITE as compared to a less consistent pattern and decline in use during the month of ABSITE by lower performers.

Even across different medical specialties, there appears to be a minimum average time spent per day associated with demonstrating knowledge on standardized examinations of residents. Knowledge on the study habits of internal medicine residents identified 20 minutes a day of electronic resource utilization was significantly associated with higher in-training exam scores.²⁵

Previous studies have shown that completing an increasing number of questions significantly increases one’s ABSITE percentile.²² While that was not reproducible with our resident cohort, there was a significant association in the percentage of questions answered correctly and ABSITE percentile. TrueLearn was an available electronic question-based resource to obtain objective data for this study but it could be argued that any question-based resource could be used to test knowledge learned. One survey-based study found that 76% of their residents stated they used TL for >50% of ABSITE studying and that 61% used TL as their only study tool with only 15% using practice questions from other sources such as SCORE. They suggested that “Therefore an education resource such as TL may provide an avenue to address gaps in knowledge and to potentially increase the overall number of test questions answered correctly.” ²² The statement “Too often, trainees tend to focus on the self-assessment without having invested time and effort in the actual curricular content.” from Onufer et al. ²⁰ summarizes our concern with a disproportionate amount of time dedicated to question-based resources instead of content-based resources.

Previous studies have also utilized survey data in attempts to answer how general surgery residents’ study with one reporting that up to 15% study 11–15 hours per week. ^15,26 While surveys can be beneficial to ascertain types of resources utilized, they are limited with regard to their accuracy to define the amount of time residents spend actually studying. We performed a survey to attempt to identify additional resources resident use besides the electronic resources described. The only resource that appeared to significantly improve an ABSITE score was reading surgical literature. The culmination of what the high ABSITE performers use to study at our institution speaks to the importance of content resources over only performing an abundance of questions. There is undoubtedly value in using questions to assess knowledge but we postulate they are more beneficial as a building block on top of a foundation of knowledge cultivated from a content-based resource.

There are noticeable differences when high ABSITE performers access SCORE compared to low performers in both the day of week and time of day. Higher ABSITE performers were more likely to access SCORE later at night and on the weekends; generally after regular hour work duties have ended. This data alludes to the habits of the higher ABSITE performers and their drive to acquire the vast general surgery knowledge base outside of the restricted 80 duty hours a week. This pattern may also potentially reflect the impact of focused, non-distracted studying that may be difficult to find during work hours. It has been demonstrated that multi-tasking increases the amount of time needed to learn material and decreases the quality of the learning.²⁷ The highest performers in our study had a steady and consistent versus inconsistent pattern of access throughout the study period. It has been demonstrated that this steady pattern of study or “distributed practice” is more beneficial with regard to its impact on learning compared to “cramming”. ²⁸ Alternatively, this pattern may reflect the value of clinic and OR-based learning. That is, taking time out of the 80 hour work week to read shortens the hands-on learning experience.

There are not many differences between the low and middle performers in many respects including the average time spent accessing SCORE per day or per session. The middle performers scored closer to the high performers on TrueLearn multiple-choice questions. This may further the hypothesis that to perform well on the ABSITE, residents must have a foundation of knowledge on a topic to answer an abstract question instead of simply becoming good at answering questions.

Because our institution SCORE utilization data are similar to national data and it has been reported that residents are not able to predict their own ABSITE scores, we aimed to see if any predictions would be possible by analyzing SCORE access and multiple choice question performance.²⁹ The nomogram in Figure 3 is the first attempt at a novel guide to general surgery residents for the dose of content and performance on questions to predict performance on the ABSITE. This tool could be beneficial for a resident to use as a gauge for their personal studies or for use in remediation programs where residents could aim to reach metrics similar to those that perform well on the ABSITE. The ABSITE’s second goal in the mission statement is to provide assessment of individual’s strengths and weaknesses, however this is only done once per year. Using a study feedback guide throughout the year when following a curriculum could improve one’s ability to score higher on the ABSITE.

There are several limitations of this study that can be improved upon with future research. This is a single institution study with a small sample size. National SCORE usage with ABSITE correlation would strengthen attempts to predict resident performance based on their SCORE usage patterns. We do not delineate time spent on specific content and or questions residents are utilizing during their access of SCORE. Giving the resident the ability to discern SCORE content usage from multiple choice question usage would provide further descriptive data on how SCORE is used and could help in predicting correlation to ABSITE performance and providing individualized feedback to improve performance. We used the arbitrary SCORE session cut off of 8 hours as done by previous studies but shorter sessions could just as easily be a failure to not log off and not represent time actually spent by a user on the web portal. In addition, our survey responses are subject to recall error as they were collected in a range on months after taking the 2019 ABSITE. Also, resident’s likely use resources, such as podcasts, that cannot be objectively measured and contribute in some fashion to the knowledge acquisition for general surgery.

Conclusion

Self-directed utilization of SCORE 15 minutes per day on average or at least 35 minutes per session is associated with an improved ABSITE performance for general surgery residents. While the role of reviewing literature appears important in high ABSITE performers, the role is currently not clearly defined. Our nomogram incorporates time spent accessing an electronic content-based resource (SCORE) per visit and performance on an electronic question-based resource (TrueLearn) is a novel method to predict an individual’s future ABSITE performance on an as-needed basis.