Association between In-Training Exam Scores and Clinical Exposure during the Covid-19 Pandemic

Abstract

Background

We sought to evaluate changes in ITE scores and associations with clinical work during the COVID-19 pandemic. We hypothesized that residents saw a decrease in clinical encounters during the pandemic and that this would be associated with smaller gains in ITE score.

Methods

We compared ITE score changes with data on patient notes for three classes of pediatric residents at four residency programs: one not exposed to the pandemic during their intern year who entered residency in 2018, one partially exposed to Covid-19 in March of their intern year (2019-2020), and one that was fully exposed to the pandemic, starting residency in June of 2020.

Results

ITE scores on average improved from the PGY1 to PGY2 year in the “no covid” and “partial covid” cohorts. The “full covid” cohort had little to no improvement, on average. The total number of patient encounters was not associated with a change in ITE scores from PGY1 to PGY2. There was a small but statistically significant association between change in ITE score and number of inpatient H+P notes.

Discussion

A drop in ITE scores occurred in pediatric residents who entered residency during the COVID-19 pandemic. This change was largely unrelated to clinical encounter number changes.

Introduction

To achieve certification in General Pediatrics, graduates of pediatric residency programs must pass the American Board of Pediatrics certifying examination.¹ The In-Training Examination (ITE) is a standardized test utilized to assess resident knowledge and monitor individualized progress in knowledge acquisition during training, and ITE scores have been shown to be predictive of board passage.² Changes in the educational environment of residency programs during the COVID-19 pandemic may have had effects on numerous aspects of training, including an associated decrease in first time test takers pediatric boards pass rate (91% in 2018 to 81% in 2021).³ At our own institutions, we anecdotally noted a decrease in pediatric ITE scores. This decrease was observed across content subsections and residency classes but was most pronounced for the post-graduate year-2 (PGY) class who began residency in July of 2020, shortly after the start of the pandemic. Identifying changes in residency training that led to decreased ITE scores may help residency programs address learning environment gaps and better position residents to pass their ABP certification exam.

Historically, factors associated with higher scores on the residency ITE have included performance on prior knowledge-based examinations,⁴ independent reading, and conference attendance.5, 6 Prior studies in surgical and anesthesiology programs have not found correlations between clinical case volume and examination scores.7, 8 One recent study of internal medicine and pediatric residents at a large academic hospital found a positive association between patient encounters in the second year of training and an increase in ITE score; however, this association was not seen for first year residents or for any of the pediatric-specific cohort.⁹ A study in pediatric residents found that the number of inpatient admissions, but not conference attendance or outpatient encounters, was correlated with an increase in ITE scores.¹⁰ Recent literature has also begun to explore resident physician burnout in multiple types of programs (orthopedic surgery, emergency medicine) as a factor in ITE performance, with mixed results.11, 12

We sought to better characterize changes in ITE scores and possible associations with clinical encounter numbers during the COVID-19 pandemic across four geographically diverse academic pediatric residency programs in North Carolina, Michigan, Arkansas, and California. All programs are medium sized with between 14-25 residents per year (Supplementary Table 1). We hypothesized that residents experienced a decrease in clinical encounters during the pandemic and that the number of clinical encounters would be associated with the change in ITE score between a resident’s first and second years.

Methods

We utilized the EPIC^TM electronic medical record (EMR) at each of the four pediatric residency programs to obtain data on patient notes as a proxy for patient encounters for three classes of general pediatric residents. We focused on three class years - one that was not affected by the pandemic during their intern year (“No COVID” or “NC” – entered residency in 2018), one that was partially affected by COVID-19 in March of their intern year (“Partial COVID” or “PC” – 2019) and one that was fully affected by the pandemic, starting in June of their intern year (“Full COVID” or “FC” – 2020). To standardize across programs, we included only residents in the general pediatric residencies at each site and excluded those in combined medicine/pediatrics or combined pediatric/subspecialty residency programs.

Reports were run through EPIC^TM to capture proxies for clinical encounters. Inpatient history and physical (H+P) notes and progress notes were used as a proxy for inpatient encounters. General pediatric outpatient clinic notes were utilized as a proxy for outpatient encounters and included all encounter types (well child checks, acute care visits, and follow-up visits for both new and established patients). Due to separate EMRs, any resident encounters that occurred at community hospitals were not included in the analysis. This accounted for 25% or less of clinical experience depending on the program. Individual ITE scores from July in the PGY-1 and PGY-2 years and pre-residency United States Medical Licensing Examination (USMLE) Step 2 CK scores, or Comprehensive Osteopathic Medical Licensing Examination (COMLEX) scores, were obtained from each program. COMLEX scores were converted to Step 2 CK scores using percentile matching.¹³ Because of a known strong correlation between prior test scores and future test performance, we chose to focus on individual changes in ITE score between the first and second year of residency as opposed to absolute ITE scores3, 12, 14 (Supplementary Figure 1 and Figure 2 and Supplementary Table 3). Overall, we aimed to explore the association of ITE score change with number of patient encounters, represented by number of total inpatient, total outpatient, H+P, and progress notes.

Fig. 2.

Number of patient encounters per resident by encounter type, program and cohort. Note log scale on y-axis.

This study was deemed exempt by each programs’ Institutional Review Board. (UNC -IRB 22-0362, University of Michigan -IRB HUM00216927, UAMS/ACH - IRB 274331, UCI/CHOC-IRB 220334).

Statistical analyses were performed using R (v4.1.1, Vienna, Austria). Association between ITE score and Step 2 CK scores was analyzed with a linear mixed model, treating cohort, program and individual resident as random effects and Step 2 CK score as a fixed effect. Association between ITE score change from first to second year and cohorts was analyzed with a linear mixed model, treating program as a random effect and Step 2 CK score and cohort as fixed effects. Differences between cohorts were estimated as linear contrasts in the resulting model using the emmeans package (v1.7.2), without adjustment for multiple comparisons. Confidence intervals for pairwise comparisons were constructed using the method of Kenward and Roger, as implemented in the pbkrtest package (v0.5.1). Association between ITE score change and number of patient encounters was analyzed with a linear mixed model, treating Step 2 CK score and program as random effects and number of patient encounters as a fixed effect (Supplementary Table 1 and Table 2). The latter quantity was scaled to have mean zero and unit variance prior to model fitting, so that effect sizes are interpreted as (point difference in ITE score) per (1 standard deviation difference in number of patient encounters). All mixed models were fit using the lme4 package (v1.1-27.1), and confidence intervals for regression coefficients computed via the likelihood profile method.

Results

Our total cohort included 231 residents across the four programs, with class size ranging from 14 to 25 (Supplementary Table 1). Across programs and cohorts, Step 2 CK scores were positively correlated with ITE scores in both the PGY1 and PGY2 year (β = 0.73 ITE score points per Step 2 CK score point; 95% confidence interval [CI] 0.63 - 0.83). However, Step 2 CK scores were not correlated with change in individual ITE score from the PGY1 to PGY2 year (β = 0.08 point increment in ITE score per Step 2 CK score point; 95% CI -0.06 - 0.23) (Supplementary Figure 1 and Figure 2)

The differences in ITE score between first and second year varied across cohorts ( Figure 1). ITE scores on average improved from the PGY1 to PGY2 year in the NC (+12.2 points; 95% CI 6.1 - 18.3 points) and PC (+18.2 points; 95% CI 12.1 - 24.3 points) cohorts. The FC cohort had little to no improvement in ITE scores from PGY1 to PGY2 (-0.5 points; 95% CI -6.6 - 5.6 points). The estimated mean difference between the FC cohort and the NC and PC cohort ITE score changes was 12.7 points (95% CI 7.0 - 18.5 points) and 18.7 points (95% CI 13.0 - 24.4 points), respectively.

Fig. 1.

ITE score change between PGY1 and PGY2 year by program and cohort.

The number of patient encounters seen at the programs also differed by cohort, and the pattern was not uniform across programs, with some programs seeing a decline in inpatient encounters and others in outpatient encounters during the pandemic ( Figure 2 and Supplementary Table 4).

After controlling for Step 2 CK score, the total number of patient encounters was not associated with a change in ITE scores from PGY1 to PGY2 (β = 1.8 point increment in ITE score per 1 SD change in number of patient encounters; 95%CI -0.8 - 4.3). There was a statistically significant association between change in ITE score and number of inpatient H+Ps (β = 2.3 point increment in ITE score per 1 SD change in number of inpatient H+P notes; 95% CI 0.1 - 4.8). This association was not observed with any of the other note types ( Table 1).

Table 1.

Effect of number of patient encounters on ITE score change.

covariate	estimate	SE	95% CI
all encounters	1.84	1.33	(-0.8, 4.35)
inpatient encounters	0.67	1.36	(-1.88, 3.26)
inpatient H&Ps	2.35	1.20	(0.07, 4.82)
outpatient encounters	2.56	1.31	(-0.32, 5.06)

Discussion

Our study sought to examine the relationship between ITE scores and clinical encounters in pediatric residents during the COVID-19 pandemic. Residents who matriculated during the COVID-19 pandemic did statistically worse than their colleagues from the prior two classes when comparing the change in their ITE scores even when controlling for prior test taking ability with Step 2 CK scores. This relationship was seen across the programs studied. There was variability in clinical encounter numbers seen across programs, with some programs seeing declines in numbers in certain clinical settings. H+P notes, our proxy for inpatient admission numbers, showed a small statistical correlation with change in ITE score. Overall, there are likely numerous other factors that contributed to ITE score change that were not examined in our study.

Clinical exposure, which in our study was measured by number of patient encounters, was not significantly associated with ITE score improvement. Prior studies have found mixed results when looking at the relationship of clinical experience and ITE scores, with some finding a positive association¹⁰ and others finding a lack of association,7, 8, 14 or an association only for certain class years.⁹ We suspect this mixed relationship is due to the complexity and variability of patient encounters, with some providing more learning than others. This is also likely why inpatient admissions, perhaps the most representative of novel patient encounters, was marginally associated with change in ITE scores in our resident group, mirroring a prior study that found that inpatient admissions was associated with change in ITE scores.¹⁰ The programs in the study reported a change in pattern of admissions during the pandemic, with decreased respiratory viruses and increased mental health admissions.

Although our study focused on patient encounters, many factors likely affected the learning environment of pediatric residencies during the COVID-19 pandemic. Gruppen et al (2019) proposed a learning environment conceptual frame work with the following elements: material, personal, social, and organizational.¹⁵ Numerous elements of the conceptual framework of the learning environment¹⁵ were affected during the COVID-19 pandemic. Within the material dimension, didactic educational sessions changed across the nation as restrictions to in-person gatherings were mandated in many states. The programs in the study all shifted lectures to a virtual format initially. A study of the efficacy of resident virtual didactics has not been done, although a study of medical students suggested a virtual format did not change exam scores.¹⁶ The psychosocial dimension of the learning environment was also altered during the pandemic. In the personal realm, resident study habits may have shifted during the pandemic. Studying outside of work has had mixed effect on ITE scores in prior studies5, 17, 18 ^{, 19} If residents studied less during the pandemic, this may have contributed to their decreased scores.

From a social perspective of the learning environment, relationships with others and sense of community were significantly altered. Globally, healthcare professional burnout increased during the pandemic.²⁰ Burnout has been previously associated with worse performance on the ITE, although the causal direction in this relationship is unknown.11, 21 Pervasive burnout during the pandemic may negatively contribute to the work environment in numerous, immeasurable ways.

One of the limitations of our study is that we were unable to measure conference attendance or engagement, which likely changed during the pandemic and is a factor that has been previously associated with ITE scores.5, 6 The programs in the study had various challenges with tracking conference attendance during the pandemic, and we were unable to consistently compare conference attendance across programs or time. Although we looked at total clinical numbers, we did not have information surrounding complexity of patients or diagnoses a resident encountered, and how this varied across the pandemic years. The programs anecdotally reported a change in type of admissions with decreased respiratory and increased mental health admissions. They also reported a decline in ambulatory experiences during March through June of 2020. We were not able to quantify this pattern. Additionally, by using note reports from the EMR, we did not capture all of the clinical encounters for residents, including encounters without documentation, encounters where another provider documented, or encounters occurring in a different EMR. We also were unable to evaluate hospital census which may also have contributed to change in volume of patient encounters and overall clinical learning environment. In addition, the ITE was created to allow for individualized self-assessment information for a resident and their program, not for summed or aggregated analysis of a total cohort. Where we could, our analyses examined differences and associations within individuals rather than across aggregated cohorts. Finally, while our programs represented geographic diversity, all were medium-sized residency programs. Very small or very large programs may have had differing experiences.

In conclusion, our study demonstrated decreased clinical encounter numbers as well as decreased change in ITE score from the PGY1 to PGY2 year, most notable in the intern class fully affected by the COVID-19 pandemic. There was a small but statistically significant association between number of inpatient H&Ps and change in ITE score, but total clinical encounter numbers were not associated with change in ITE score. Other factors that may have contributed to knowledge gaps not evaluated by our data include changes in conference format, changes in studying habits, and pervasive burnout. As residency programs work to recover from the pandemic, it will be crucial to continue to study the effects of this time on trainees and their ability to rebuild and recover from the pandemic. Program leadership should continue active work on mitigating deleterious effects of the pandemic, as our study suggests clinical encounter numbers are not fully responsible for the notable changes in test scores.

Conflicts of Interest

The authors have no conflicts of interest, competing interests, or corporate sponsors.

Funding

The authors received no funding or grant support for this study.

Appendix A. Supplementary material

Supplementary material