The impact of fellowship‐trained medical toxicology faculty on emergency medicine resident in‐training examination scores

Abstract

Background

The American Board of Emergency Medicine (ABEM) In Training Exam (ITE) gauges residents' medical knowledge and has been shown to correlate with subsequent performance on the ABEM board qualifying examination. It is common for emergency medicine (EM) residencies to employ subspecialty‐trained faculty members with the expectation of improved resident education and subspecialty knowledge. We hypothesized that the presence of subspecialty faculty in toxicology would increase residents' scores on the toxicology portion of the ITE.

Methods

We assessed ABEM ITE scores at our institution from 2013–2022 and compared these to national data. The exposure of interest was the absence or presence of fellowship‐trained toxicology faculty. The primary outcome was performance on the toxicology portion of the ITE, and secondary outcome was overall performance on the exam.

Results

Residents who had ≥1 toxicology faculty were 37% (95% CI: 1.01–1.87) more likely to surpass the national average for toxicology scores, and those who had ≥2 toxicology faculty were 77% (95% CI: 1.28–2.44) more likely to surpass the national average for toxicology scores on the ABEM ITE. With the presence of ≥2 toxicology faculty, there was also an increase in toxicology score by years in training, with residents being 63% (95% CI: 1.01–2.64), 68% (95% CI: 1.08–2.61), and 92% (95% CI: 1.01–3.63) more likely to surpass the national average for toxicology score in first, second, and third years of residency, respectively. There was no significant relationship between the presence of toxicology faculty and the overall ABEM ITE scores.

Conclusions

The presence of fellowship‐trained toxicology faculty positively impacted residents' performance on the toxicology portion of the ABEM ITE but did not significantly impact the overall score. With the presence of ≥2 toxicology faculty we noted an improvement in toxicology scores throughout the 3 years of training, indicating that an individual rotation or educational block is probably less important than spaced repetition through a longitudinal curriculum.

INTRODUCTION

The emergency medicine (EM) physician is expected to be an expert in the management of all acute and life‐threatening medical conditions. In order to be competent in this field, a broad knowledge base is required. EM residencies are designed to train a clinically competent physician, as well as prepare the trainee to demonstrate objective mastery of their field by becoming specialty board certified in EM. As part of this preparation, emergency medicine residents complete an annual In‐Training Examination (ITE), which is developed and administered by the American Board of Emergency Medicine (ABEM) and covers the breadth of emergency medicine. ¹ After successful completion of EM residency, graduates are eligible to pursue EM specialty board certification, which is designed to “objectively and independently confirm that physicians who complete an Emergency Medicine residency demonstrate core knowledge, skills, and abilities needed to practice Emergency Medicine at the highest standards.” ² Both the ITE and Board Qualifying Examination are constructed from the comprehensive list of EM core content contained in the Model of the Clinical Practice of Emergency Medicine (EM Model), which defines the scope of medical knowledge required for emergency physicians. ³ Within the emergency medicine specialty, there are also multiple subspecialties that require additional post‐residency, directed training (or fellowships), as well as subspecialty board certification to provide a higher level of patient care. With the increasing numbers of subspecialists within emergency medicine, it is common for residencies to include subspecialty‐trained faculty members with the expectation of improved patient care and resident education. There is an assumption that trainees' subspecialty knowledge would improve with the addition of subspecialists to train them; however, there are minimal objective data to support this belief.

The ITE gauges residents' medical knowledge and has been shown to correlate with subsequent performance on the ABEM qualifying examination, which is required for emergency medicine specialty board certification. ⁴ , ⁵ Given this association, a considerable amount of emphasis is placed on interventions or factors that can positively influence an individual resident's ITE score or benefit a residency program as a whole. Walter et al. found that participation and engagement in a question bank is associated with improved performance on the ABEM ITE. ⁶ Other factors have also been found to be associated with higher ABEM ITE scores including increased hours of sleep and participation in simulation focusing on certain subcategories. ⁷ , ⁸ In non‐EM programs, case‐based, non‐lecture activities have been shown to have a positive impact on internal medicine (IM) ITE scores. ⁹ Conversely, resident conference attendance and board‐review lectures have not been found to have any significant positive effect on ITE scores. ¹⁰ , ¹¹ In addition to nonclinical education, clinical experiences are a significant part of residency training. However, Frederick et al. found that clinical productivity and patient encounter hours correlate poorly with performance on ABEM exams. ⁴ Additionally, one study found that the actual clinical experience of EM residents differs significantly from the ITE Content Blueprint (which reflects the ABEM EM Model); specifically, toxicologic and environmental disorders are two of the most underrepresented categories. ¹² This finding is not entirely unexpected given the rarity of some disease processes in clinical practice but does emphasize the importance of program adjuncts such as subspecialty faculty and/or nonclinical education methods in these underrepresented areas.

To date, no study has looked specifically at the effect the presence of subspecialty faculty would have on residents' knowledge. One study focused on medical students found an improvement in core content knowledge with completion of a toxicology elective or interest in a pharmacology intensive specialty. ¹³ Studies looking specifically at internal medicine residents did find that completion of a subspecialty elective is associated with an increase in specialty‐specific knowledge, and subspecialty interest is positively associated with higher overall scores on the IM ITE. ¹⁴ , ¹⁵ Another study found that an increase in subspecialty patient encounters correlates with improved IM ITE scores. ¹⁶ However, no similar studies have been performed looking at EM residents, and notably no study to date has specifically looked at the effect of subspecialty faculty exposure on residents' knowledge or ITE scores.

This study aimed to fill this knowledge gap and answer the question: Does the addition of subspecialty fellowship‐trained faculty members improve resident ITE scores overall and/or in the associated subspecialty devoted sections? Specifically, this study aimed to explore the association between the addition of fellowship‐trained medical toxicologists and performance on the toxicology portion and overall scores on the ABEM ITE. We hypothesized that the presence of subspecialty faculty in toxicology would increase residents' scores on the toxicology portion of the ITE. Medical toxicology was chosen for this study as it is the only subspecialty that has a distinct scored content area on the ABEM ITE examination.

METHODS

Study design, setting, and population

This was a quasi‐experimental design that took place at one tertiary care, midwestern, academic EM department. This study was approved and given exempt status by the Institutional Review Board. We assessed ABEM ITE exam scores from calendar years 2013–2022, which encompassed the EM resident graduating cohorts from 2015–2023. The sample of interest was our residents in comparison to residents at the national level. Our institution has hosted a residency in EM since 2007 but has only had fellowship‐trained subspecialists in medical toxicology since the fall of 2016. All ITE scores since 2013 have been collected and stored by the residency program (including itemization of individual topics) to improve board examination preparation, thus providing an ideal site to assess for any association of ITE scores to fellowship‐trained subspecialists in medical toxicology.

Exposure assessment

The exposure of interest was the absence or presence of a fellowship‐trained subspecialist in medical toxicology (defined as toxicology faculty from this point forward). We defined this based on the year of onboarding of each new toxicology faculty member who joined our department. If the faculty member onboarded before the midpoint of the year, residents in that year were considered toxicology exposed. If a faculty member joined the team after the midpoint of the year (e.g. July), then they were considered toxicology unexposed for the rest of that year but toxicology exposed the next calendar year. The primary exposure was defined as any toxicology faculty member present. A secondary exposure was the number of toxicology faculty members present in 1 year, characterized as 0, 1, or ≥2.

With the presence of toxicology faculty, residents have frequent contact throughout their 3 years of training, and contribution of toxicology faculty to resident education comes in multiple forms and provides a consistent exposure over the course of residency. During their first year of residency, interns participate in a required 3‐week toxicology rotation that includes 12 days of small group didactics and case‐based discussion, bedside assessment and consults, call shifts with a toxicology faculty member, and at least 25 h of dedicated toxicology didactic time. In addition to the dedicated toxicology rotation, toxicology faculty consistently provide dedicated lectures at residency‐wide conference, accounting for 4% of didactic education on an 18‐month education cycle, which is consistent with the toxicology content weight on the ABEM ITE blueprint. All toxicology faculty at our institution are also board certified in EM and work clinically and provide direct supervision to residents in the emergency department. Lastly, residents have round‐the‐clock access to toxicology faculty during all of their clinical rotations whenever they have a toxicology‐related case and interact with the toxicology faculty frequently in this capacity throughout their years of training.

Outcome assessment

The primary outcome was test performance on the toxicology portion of the ABEM ITE of our residents in relation to the national average. For the secondary outcome, we assessed the test performance of our residents on the overall ABEM ITE (i.e. all sections of the exam) compared to the national average for each year. Both outcomes were operationalized in three ways as a dichotomized measure: whether the resident achieved (1) a score within 20% points or better of the national score, (2) a score within 10% points or better of the national score, and (3) a score that surpassed the national average. We opted for this assessment as in‐service exams are not evaluated with an overall pass rate, but rather in relation to other scores. The secondary exposure, the comparison with the national average of the overall test score, also served as a robustness check. This allowed us to assess whether any associations observed were primarily impacted by toxicology faculty rather than differences in each resident cohort in our program each year.

Statistical analysis

We assessed the proportion of residents who achieved the outcome each year. We used generalized estimating equations, clustered on a deidentified resident number, to assess the relative risk and 95% confidence intervals (CIs) of having toxicology faculty on resident scores. As a sub‐group analysis, we further assessed this relationship within each year of residency training to evaluate whether there was a potential dose–response relationship in test scores by the duration of toxicology training.

RESULTS

There were 80 residents included in the time period with 227 cumulative test scores. Toxicology questions made up 3.6%–4.9% of overall test questions. The proportion of test scores within 20% points of the national average of the toxicology portion ranged between 72%–100% and between 60%–79% using the 10% point cut‐off. The proportion that surpassed the national average for toxicology scores ranged from 37% for the graduating class of 2015 to 78% for the 2023 graduating class. For the overall score, almost all years had 100% of the residents passing within 20 percentage points of the national score and between 83%–100% using the 10% point cut‐off. The range for the proportion of residents surpassing the national average was 16%–81%. The proportion of residents passing the toxicology portion of the exam and the overall scores by each outcome cut‐off are presented in Figure 1, with residents grouped by graduating year, showing an increase in proportion of residents scoring above the national average over time.

FIGURE 1.

Distribution of test score performance by varying outcome cut‐offs

Using either the 20% or 10% point cut‐offs, there were no statistical differences in test scores in either toxicology‐specific portions or the overall test scores by the presence of toxicology faculty (Table 1). However, those who had at least one toxicology faculty were 37% (95% CI: 1.01–1.87) more likely to surpass the national average for toxicology scores. When further stratifying this, those who had two or more toxicology faculty members at the time of training were 77% (95% CI: 1.28–2.44) more likely to surpass the national average for toxicology scores. There was no significant relationship between the presence of toxicology faculty on any cut‐off points assessing the overall exam score.

TABLE 1.

Distribution of toxicology and total scores by toxicology faculty presence

Toxicology attendings	Total residents	Within 20 points or better of national score		Within 10 points or better of national score		Better than national score
		n (%)	RR (95% CI)	n (%)	RR (95% CI)	n (%)	RR (95% CI)
Toxicology scores
Binary
Any	125	109 (87.2)	1.01 (0.91–1.12)	92 (73.6)	1.09 (0.91–1.31)	69 (55.2)	1.37 (1.01–1.87)
None	102	88 (86.3)	Ref	70 (68.6)	Ref	41 (40.2)	Ref
Categorical
0	102	88 (86.3)	Ref	70 (68.6)	Ref	41 (40.2)	Ref
1	55	47 (85.5)	0.99 (0.87–1.13)	37 (67.3)	0.97 (0.77–1.23)	21 (38.2)	0.94 (0.61–1.47)
≥2	70	62 (88.6)	1.03 (0.92–1.16)	55 (78.6)	1.21 (0.98–1.49)	48 (68.6)	1.77 (1.28–2.44)
Overall scores
Binary
Any	125	124 (99.2)	0.99 (0.98–1.01)	111 (88.8)	0.98 (0.89–1.08)	39 (31.2)	0.69 (0.47–1.02)
None	102	102 (100.0)	Ref	93 (91.2)	Ref	48 (47.1)	Ref
Categorical
0	102	102 (100.0)	Ref	93 (91.2)	Ref	48 (47.1)	Ref
1	55	55 (100.0)	1.00 (0.99–1.00)	51 (92.7)	1.01 (0.91–1.11)	15 (27.3)	0.61 (0.37–1.02)
≥2	70	69 (98.6)	0.99 (0.96–1.01)	60 (85.7)	0.95 (0.83–1.10)	24 (34.3)	0.80 (0.50–1.29)

Abbreviations: RR, relative risk; CI, confidence interval.

In Table 2, we present our findings stratified by year of resident training. Using a dichotomized measure of toxicology faculty, there was no significant relationship between presence of toxicology faculty and toxicology exam scores across any years. However, when we assess the presence of two or more faculty members on toxicology‐specific exam scores, there was an increase in scores by year in training. For example, the likelihood of surpassing the national toxicology score average was 63% (95% CI: 1.01–2.64) greater among first‐year residents, 68% (95% CI: 1.08–2.61) greater in second‐year residents, and 92% (95% CI: 1.01–3.63) greater among third‐year residents with two or more toxicology faculty when compared to having no toxicology faculty.

TABLE 2.

Frequency of residents who surpassed national toxicology and total exam scores, by resident year in training

Toxicology attendings	1st year residents		2nd year residents		3rd year residents
	n (%)	RR (95% CI)	n (%)	RR (95% CI)	n (%)	RR (95% CI)
Toxicology scores
Binary
Any	17 (47.2)	1.15 (0.70–1.89)	27 (58.7)	1.33 (0.85–2.09)	25 (47.2)	1.74 (0.94–3.24)
None	18 (40.9)	Ref	15 (44.1)	Ref	8 (33.3)	Ref
Categorical
0	18 (40.9)	Ref	15 (44.1)	Ref	8 (33.3)	Ref
1	5 (27.8)	0.68 (0.30–1.55)	7 (36.8)	0.84 (0.41–1.68)	9 (50.0)	1.50 (0.72–3.11)
≥2	12 (66.7)	1.63 (1.01–2.64)	20 (74.1)	1.68 (1.08–2.61)	16 (64.0)	1.92 (1.01–3.63)
Overall scores
Binary
Any	12 (33.3)	0.81 (0.46–1.46)	14 (30.4)	0.61 (0.35–1.06)	13 (30.2)	0.56 (0.31–1.00)
None	18 (40.9)	Ref	17 (50.0)	Ref	13 (54.2)	Ref
Categorical
0	18 (40.9)	Ref	17 (50.0)	Ref	13 (54.2)	Ref
1	6 (33.3)	0.81 (0.39–1.71)	5 (26.3)	0.53 (0.23–1.20)	4 (22.2)	0.41 (0.16–1.05)
≥2	6 (33.3)	0.81 (0.39–1.71)	9 (33.3)	0.67 (0.35–1.25)	9 (36.0)	0.66 (0.35–1.26)

Abbreviations: RR, relative risk; CI, confidence interval.

There was no significant relationship between the presence of toxicology faculty on any cut‐off points assessing the total exam score within any resident cohort year.

DISCUSSION

Our findings supported our hypothesis that fellowship‐trained subspecialty faculty members provide educational benefit to trainees, as measured by residents' performance on the toxicology portion of the ABEM ITE. We found that the presence of toxicology faculty positively impacted residents' performance on the toxicology portion of the ABEM ITE but did not significantly impact the overall ABEM ITE score. Importantly, with the presence of two or more toxicology faculty we noted an improvement in toxicology scores throughout the 3 years of training, with increase in scores by years of training and higher likelihood of surpassing the national average as residents progressed, indicating that an individual rotation or educational block is probably less important than spaced repetition through a longitudinal curriculum.

Our findings are consistent with previous studies in non‐EM programs, which have found that an increase in subspecialty exposure in the form of subspecialty electives, subspecialty interest, and increased subspecialty patient encounters are correlated with better ITE scores. ¹⁴ , ¹⁵ , ¹⁶ Other non‐EM studies have also found that case‐based, non‐lecture activities have a positive impact on ITE scores. ⁹ Additionally, it has been found that individualized, structured self‐study as part of a comprehensive academic enrichment program may improve ITE scores. ¹⁷ Altogether, we believe that spaced repetition through formal didactics, discussion of individual cases in real time, and the availability for continued dialogue between the EM resident and subspecialty‐trained faculty members may solidify important concepts for subsequent standardized exams.

LIMITATIONS

This was a study involving a single institution, which may limit the generalizability of our findings. Opportunity for future studies includes similar analyses at other institutions to see if these results would be replicated. Additionally, the improvement in scores may have been due to variability in the cohort enrolled over time; however, we found that our program's overall scores compared to the national average of total scores. For example, while toxicology scores increased, overall scores did not, suggesting that our findings may not have been a result of cohort changes over time. We also considered whether other educational interventions or confounders could have affected residents' scores and did note that Rosh Review ¹⁸ and Foundations curricula were implemented near the beginning of our study period (2014 and 2015, respectively); however, given that these were in place prior to the addition of any toxicology faculty and subsequently remained constant, we feel it is less likely these had a major confounding effect.

Another limitation of this study is the inability to analyze the effect of other subspecialty faculty on test scores due to lack of other subspecialty scored content on the ABEM ITE. Despite having fellowship‐trained faculty with extensive expertise in multiple subspecialties, including medical toxicology, pediatric emergency medicine, sports medicine, and palliative care, we were only able to study the effect of medical toxicology faculty as the ABEM ITE does not index scores for other subspecialty categories. If the ABEM ITE had distinct scored content for other subspecialties, we would have been able to run a similar analysis for other subspecialties, resulting in a more robust study. In the future, it is our hope that the ABEM will collect and release data on scored content for other subspecialties so that valuable information could be added to the literature regarding the effect of different subspecialty faculty on EM residents' exam scores and knowledge.

CONCLUSIONS

We found that the presence of toxicology faculty positively impacted residents' performance on the toxicology portion of the ABEM ITE but did not significantly impact the overall ABEM ITE score. Importantly, with the presence of two or more toxicology faculty we noted an improvement in toxicology scores throughout the 3 years of training, with increase in scores by years of training and higher likelihood of surpassing the national average as residents progressed, indicating that an individual rotation or educational block is probably less important than spaced repetition through a longitudinal curriculum. Opportunities for future studies include expansion to other subspecialties or institutions to assess if these results are replicated.

AUTHOR CONTRIBUTIONS

Study concept and design (CAT, JPV, BJO, NH, DJM), acquisition of data (NH, DJM), analysis and interpretation (CAT, JPV, DJM), drafting of manuscript (CAT, JPV), critical revision of manuscript (CAT, JPV, BJO, NH, DJM), statistical expertise (JPV), study supervision (CAT, DJM).

CONFLICT OF INTEREST

All authors (CAT, JPV, BJO, NH, DJM) report no conflict of interest.

Theiler CA, Vakkalanka JP, Obr BJ, Hansen N, McCabe DJ. The impact of fellowship‐trained medical toxicology faculty on emergency medicine resident in‐training examination scores. AEM Educ Train. 2023;7:e10840. doi: 10.1002/aet2.10840