Quantifying Impact of Disruption to Radiology Education During the COVID-19 Pandemic and Implications for Future Training

Abstract

Purpose

The aim of this study was to assess the impact on radiology resident education due to the COVID-19 pandemic in order to inform future educational planning.

Methods

During a 10-week study period from March 16 to May 22, 2020, changes to educational block-weeks (BW) of first through fourth year residents (R1-4) were documented as disrupted in the setting of the COVID-19 pandemic. The first 5 weeks and the second 5 weeks were evaluated separately for temporal differences. Overall and mean disrupted BW per resident were documented. Wilcoxon rank-sum tests were used to assess pairwise differences between classes with Bonferroni-adjusted P-values, as well as differences in the early versus later phase of the pandemic.

Results

Of 373 BW, 56.6% were assigned to virtual curriculum, 39.4% radiology clinical duties, 2.9% illness, and 1.1% reassignment. Scheduling intervention affected 6.2 ± 2.3 (range 1-10) mean BW per resident over the 10-week study period. The R3 class experienced the largest disruption, greater than the R2 classes, and statistically significantly more than the R1 and R4 classes (both P < 0.05). The second half of the pandemic caused statistically significantly more schedule disruptions than the first half (P = 0.009).

Discussion

The impact of COVID-19 pandemic varied by residency class year, with the largest disruption of the R3 class and the least disruption of the R4 class. To optimize future educational opportunities, shifting to a competency-based education paradigm may help to achieve proficiency without extending the length of the training program.

Introduction

The SARS-CoV-2 (COVID-19) coronavirus pandemic substantially impacted many academic training programs in the United States.^{1 , 2} Shortly after the initial COVID-19 outbreak in our state, the Governor of Massachusetts declared a state of emergency on March 10, 2020. Our institution officially moved to defer outpatient elective procedures on March 13, 2020 [reference redacted for blinded review]. In response to the decrease in number of imaging exams and healthcare system-wide risk mitigation polices regarding social distancing and work-from-home guidelines, resident schedules in our radiology training program were adjusted to match clinical demands and to meet institutional safety guidelines.

Residency training in diagnostic radiology is a four-year program with residency years 1 through 4 (R1-4), preceded by an intern year. In 2019, there were 4,587 diagnostic radiology (DR) and interventional and diagnostic radiology (IR/DR) residents.³ Our program consists of [blinded] DR and [blinded] IR/DR at a tertiary care academic medical center affiliated with a medical school. The clinical schedule for each academic year includes rotations on 18 subspecialty services that take place within our institution and affiliated institutions in the city.

During the COVID-19 pandemic, remote learning and social distancing were established. The disruption in radiology residency education has been managed similarly at many institutions: the number of on-site clinical rotations have decreased while distanced learning has increased⁴ However, these gaps in traditional graduate medical education may potentially have lasting effects on residents, and it is not yet known how disruption related to the COVID-19 pandemic will affect clinical competency. In order to predict the impact on residents, first it is necessary to measure the magnitude of educational disruption. In our paper, we aim to describe a standardized method to gauge disruptions to residency education of residents across all years and use these findings to optimize clinical competency and graduation goals moving forward.

Methods

Study Setting and Design

This HIPAA-compliant, Institutional Review Board–exempt, retrospective study was performed at an urban tertiary care academic medical center with [blinded] DR and [blinded] IR/DR residents. The residency program academic year schedule consisted of the 52-week academic year divided into rotations on 18 subspecialty services within our institution and associated institutions in the city. One week of each block was identified as a “block-week.” Typically, residents were assigned to 2 or 4-week blocks on each service. Exceptions included the pediatrics rotation that was 8 weeks in length during R2. R3 resident schedules included the 4-week American Institute for Radiology Pathology (AIRP) Radiologic Pathologic Correlation Course and the American Board of Radiology (ABR) Core Exam in the spring. R4 resident schedules were based on elective interests, mini-fellowships, and completion of remaining graduation requirements in nuclear medicine and breast imaging.⁵ R4 mini-fellowships ranged in length from 3 months up to the entire year minus any remaining requirements.

Intervention and Study Time Periods

The intervention was the modification of the original planned 2019-2020 academic year residency schedule to account for decreased clinical demands and to meet departmental policies and procedures set forth during the COVID-19 pandemic for remote work and social distancing. Schedule changes occurred over a 10-week period between March 16, 2020 and May 22, 2020; the intervention period started the Monday following a state of emergency declaration by our Governor and announcement by the hospital to defer outpatient elective procedures. The intervention period spanned the peak of the COVID-19 pandemic as well as transition to recovery planning at the institutional level. In order to assess for temporal differences, the intervention period was retrospectively divided into the first 5 weeks and the second 5 weeks. The methodology of schedule changes throughout the intervention period was based on real-time shifting needs during the crisis situation: reduced resident staffing models were adopted as part of the departmental response to clinical volumes, aligned with emerging COVID-19 institutional work guidelines. Scheduling changes were made during the intervention with unique considerations for each resident class, including maintaining the rotation experience in emergency radiology rotation for R1s in preparation for after-hours call shifts; scheduling on-site weeks of rotations not yet completed in the academic year, and prioritizing graduation requirements for R4s. Some schedule changes were based on cancelation of off-site rotations, which included the 8-week pediatric rotation, a 2-week subspecialized head and neck imaging rotation, and the 4-week AIRP Radiologic Pathologic Correlation Course.

Data Collection

In early March during the COVID-19 preparation, the chief residents created a new residency schedule for the remainder of the academic year using Excel (Microsoft, Redmond, WA) while working in conjunction with core faculty to determine if and how many trainees were needed on-site based on retrospective clinical imaging volumes of the previous week. If residents were not assigned to an on-site rotation, they were assigned to a “virtual curriculum,” where they participated in assigned online lectures and study material and kept daily educational logs to document their activities. After-hours call shifts were continued despite lower volumes. As part of departmental policy, designated vacation weeks were frozen for all staff; they were excluded from the study. Similarly, weeks designated as leave of absence were excluded.

Primary and Secondary Outcome Measures

The primary outcome measures were total disrupted block-weeks, mean disrupted block-weeks per resident, and block-weeks assigned to radiology clinical duties, virtual curriculum, direct patient care, and illness. Comparison of the original schedule to the modified schedule during the COVID-19 pandemic was made. Block-weeks were marked as “disrupted” if residents did not obtain the originally intended clinical experience. If an on-site rotation was shifted to a different week within the study period, it was not counted as disrupted since the resident still gained the clinical experience. Block-weeks assigned to virtual curriculum, direct patient care, or clinical duties not in the original schedule were considered disrupted. Block-weeks related to illness or investigation of COVID-19 infection were also marked as disrupted. Virtual curriculum block-weeks attributed to internal medicine reassignment preparatory curriculum, a period of time during which volunteer residents prepared for possible internal medicine patient care duties, were noted.

The secondary outcome measures were the number of block-weeks disrupted stratified by residency class and comparison of the first 5 weeks versus the second 5 weeks of the study period.

Statistical Analysis

The number of block-week schedule changes were calculated overall and separately for each residency year using descriptive analysis. Wilcoxon rank-sum tests were used in a pairwise fashion to determine if schedule changes differed among classes with Bonferroni-corrected P-values to account for multiple comparisons. Differences between the first 5 weeks and the second 5 weeks of the pandemic were compared with a Wilcoxon rank sum test. All tests were 2-tailed and P-values < 0.05 were considered statistically significant. Statistical analysis was performed using R statistical programming language (version 4.0.0; The R Foundation for Statistical Computing, Vienna, Austria).

Results

Overall Radiology Scheduling Changes

After excluding 17 weeks associated with vacation and 20 weeks associated with leave of absence, 373 block-weeks in total were included in the study period (Fig 1 ). Of these, 147 block-weeks (39.4%) were assigned to radiology clinical duties (95.9% [141/147] on-site in the hospital and 4.1% [6/147] remote work at home). Four block-weeks (1.1%) were attributed to reassigned residents working on internal medicine inpatient teams for direct patient care. All block-weeks due to illness or investigation for Covid-19 infection took place within the first 5 weeks, disrupting 11 block-weeks (2.9%) with 2 weeks in the R1 class, 8 weeks in the R2 class, and 1 week in the R3 class. The remaining 211 block-weeks (56.6%) were assigned to residents as a virtual curriculum (71.6% [151/211] radiology curriculum and 28.4% [60/211] internal medicine curriculum). Virtual curriculum was assigned to block-weeks impacted by cancellation of off-site programs including 16 BW of pediatric radiology rotation, 2 BW of subspecialized head and neck imaging rotation, and 16 BW of AIRP Radiologic Pathologic Correlation Course.

FIG 1.

Flow diagram showing study selection and results.The flow diagram for the retrospective review details assignment of block weeks (n). All direct patient care and reassign ment duties were in internal medicine (IM).

In total, 244 block-weeks (65.4%) were disrupted, included those assigned to the virtual curriculum, illness, reassignment to direct patient care, and 18 weeks of clinical duties that were not planned in the original schedule. Each resident experienced 6.2 ± 2.3 (range 1-10) disrupted block-weeks on average over the 10-week study period.

Secondary Outcomes

The R3 class had the largest number of block-week changes per resident during the intervention period while the R1, R2, and R4 classes had fewer block-week changes (Fig 2 ). Pairwise comparison of residency years showed a statistically significant difference in disruption to the R3 class compared to the R4 class (P = 0.002) and R1 class (P = 0.011) (Table ). There was no statistically significant difference among the other classes. Comparison of the weekly mean block-week changes showed a statistically significant temporal change in that there was more disruption (P = 0.009) in the latter half of the intervention period (29.4 ± 1.5, range 28-31) than the initial 5 weeks (19.2 ± 5.4, range 10-24).

FIG 2.

Boxplot showing number of block-week changes for each year of residency. The R3 class had the largest number of disrupted block-weeks and was statistically significantly higher than the R4 class (p = 0.002) and R1 class (P = 0.011).

TABLE.

Block-week (BW) schedule changes per class

Class	Median BW changes/resident	Interquartile range (IQR)	Total BW changes/class	Bonferroni comparison*
R1, PGY-2	6.0	5.0-6.0	51 (59.3%)	R3
R2, PGY-3	5.5	5.0-8.75	64 (68.1%)	–
R3, PGY-4	9.0	7.0-9.0	91 (86.7%)	R1, R4
R4, PGY-5	5.0	4.0-5.0	37 (43%)	R3

^⁎Residency classes with statistically significant difference using Bonferroni-corrected p-value significance cutoff, p < 0.05

Discussion

In our analysis of the residency clinical schedule during a 10-week time period corresponding to the COVID-19 pandemic surge at our institution, 65.4% of scheduled block-weeks were disrupted. The interventions associated with disruptive schedule changes included using a reduced staffing model in which approximately one on-site resident was assigned per subspecialty. Subspecialties with very low volume, such as musculoskeletal radiology, only retained staff radiologists and fellows on-site. Low clinical volume was the predominant driver of schedule disruption, resulting in 56.6% of block-weeks assigned to virtual curriculum. Of these, a substantial number of block-weeks (16.1% [60/373]) were assigned to internal medicine virtual curriculum due to nearly half of our residents volunteering for reassignment, whereas a small number (1.1% [4/373]) of block weeks were attributed to performing direct patient care. Illness and investigation for COVID-19 infection accounted for 2.9% of disrupted block-weeks. All instances of illness occurred in the initial phase of the pandemic, when the COVID-19 occupational health protocol took approximately 1 week per occurrence due to waiting on test results.

By using the metric of block-weeks rather than number of individual residents affected, we standardized the amount of time that was attributed to various activities and created a uniform metric to gauge disruption across the residency. Subsequently, we will use disrupted block-weeks time to drive future resource allocation, aka scheduling. Accurately determining the costs, or disruptions to education, is the first step to identifying the opportunities.

Comparison of residency years revealed that the R3 class experienced the largest number of disruptions to their schedule and the R4 class experienced the smallest number of schedule changes. Clinical rotations were often not feasible due to low clinical volume. Furthermore, shared workspaces limited the inability to maintain adequate social distancing. If resident presence was possible, the R4 class was prioritized so that they could meet graduation requirements and complete mini-fellowship responsibilities. Thus, the R4 class experienced the least disruption during the pandemic. In contrast, the R3 class experienced the largest disruption. This is likely due to the deferment of the ABR Core Exam, cancelation of the 4-week AIRP course, and deferment of board review courses.

The R1 and R2 classes did not experience the same amount of disruption as the R3 class. We surmise that this is likely attributed to being in the early stages of training where the rotations were less dependent on preparation for the ABR Core Exam. However, the cancelation of clinical rotations impacted R1 and R2 residents greatly because it was often their initial exposure to some subspecialties, including procedural rotations. Moreover, junior residents were developing independence for call shifts. For this reason, assignments to the emergency radiology rotation were preserved for R1 residents since clinical exposure was considered essential preparation for taking after-hours independent call. Adequate exposure to subspecialty rotations and procedural experience will need to be accounted for in the subsequent academic years of the R1 and R2 residents.

These residency scheduling changes made due to the COVID-19 pandemic will have downstream effects for future scheduling, predominantly to address deficiencies and reconcile deferred clinical and procedural experiences. Individualized scheduling considerations will be important for optimizing the educational experience for each resident, including how each resident will make up a portion of missed rotations during the next academic year. Select future rotations may need to be shortened in duration to accommodate educational needs, as the extension of the overall training period may not be feasible. Performance evaluation of prior on-site clinical experiences and review of each resident's virtual curriculum educational logs during the pandemic will aid in determination of outstanding gaps in knowledge. Competency of trainees for readiness to graduate will require regular knowledge assessments and a heavier reliance on individual evaluation by the Clinical Competency Committee (CCC), a body consisting of the Program Director, Associate Program Directors, and core faculty representatives. This greater emphasis incumbent upon the CCC puts into sharp focus the need for competency-based education.⁶ Quantitative assessments of competency may be implemented as standardized tests at regular intervals; several tools have been developed for this purpose, such as the ACR RadExam⁷ and the ACGME Milestones. Self-directed learning and quality improvement projects may be complementary supplementation to this model.⁸ A competency-based curriculum may be the key to optimizing resident education after the substantial disruption of the pandemic given the subsequent limitations in time imposed by the COVID-19 pandemic.

Competency-based education has existed in the United States since 196⁹ and was implemented in some residency programs in the 1990s.¹⁰ The Royal College of Physicians and Surgeons of Canada implemented a competency-based curriculum called Competence by Design (CBD) based on CanMEDS guidelines, a project initially started in 1993¹¹ to align medical education with population outcomes. CBD is task-driven and depends on completion of Entrustable Professional Activities (EPA) that become progressively more complex as residents progress through the stages of competency. Well-defined milestones consist of observable markers of a resident's ability along a developmental continuum that can be displayed in dashboard format for feedback.¹² The ACGME Milestones project with its planned updates for July 2020 was developed to similarly align the core clinical competencies in radiology education to this end.¹³

In the subsequent stages of the COVID-19 pandemic, a new graduate medical education paradigm will be needed to address the concerns of our current cohort of residents, as well as the incoming first-year radiology residents when the new academic year begins in July 2020. The COVID-19 pandemic is not the only event to have caused disruption of graduate medical education. For example, the devastating aftermath of Hurricane Katrina in 2005 greatly disrupted graduate medical education at Louisiana State University (LSU) and Tulane University,^{14 , 15} requiring widespread accommodations. Given the toll of structural damage, nontraditional sites of training were utilized, including community clinics and smaller community hospitals, which had previously been without trainees. This paradigm can be adapted to the current COVID-19 pandemic, where space constraints due to social distancing guidelines will likely limit on-site clinical opportunities for residents. Emphasis on competency-based residency training after Hurricane Katrina included assigned monthly guidelines, regular assessment of knowledge, and use of simulated patient experiences.¹⁶ Ultimately, the goal of medical training is not only competency, but clinical proficiency and excellence to enable independent practice.

Limitations of this study include the small dataset of residents from a single academic institution that may not be generalizable to other populations of residents, given that the COVID-19 pandemic has affected institutions in different parts of the country differently.⁶ Furthermore, the limited study period of 10-weeks does not capture all changes to residency education that will be warranted. Importantly, schedule disruptions also depended on different characteristics of each class. For example, a greater number of R2s and R3s volunteered for COVID-19 reassignment than R1s and R4s, and all volunteers were placed on internal medicine virtual curriculum in order to prepare for and be available for possible reassignment; these block-weeks spanned the first and second 5 weeks of the study period. While only 4 block-weeks were attributed to direct patient care, the study period did not capture all reassignment weeks.

In addition, lost clinical time may not be the most accurate measure of a resident's experience. Other methods of learning including back-reading previously finalized attending reports or studying material in a virtual curriculum may substantially improve a resident's knowledge base and positively impact resident competencies; these and other methods require further review. The impact of these self-directed learning options and a virtual curriculum are opportunities for future investigations. The landscape of education has changed. Both the current cohort and future residents may experience further disruption due to COVID-19 surges, and they will require the support of an educational framework that can adapt to these challenges.

Conclusion

The standardized metric of block-weeks is generalizable to all years of residency and acts as a framework for approximating disruption in medical education. The COVID-19 pandemic resulted in educational disruption that most impacted the R3 class and least impacted the graduating R4 class. Furthermore, there were higher number of schedule disruptions during the later phase of the pandemic compared to the initial phase during our study. Longitudinal educational planning will be needed in order to support our trainees through the current crisis and through their remaining tenure as residents, likely requiring customized approaches for each residency class and a greater reliance of competency-based education in the subsequent stages of the COVID-19 pandemic so that training need not to be extended.

Author Contributions

JS: conceptualization, methodology, statistical analysis, writing, revision and approval of final draft; CB (biostatistician): statistical analysis, methodology design, editing and approval of final draft; NM/BD/AK/PS (Chief Residents): design, data validation, writing and approval of the final draft, JG/GG (Assistant Program Director and Program Director): conceptualization, design, editing and approval of final draft.

Conflict of Interest

The authors declare no conflict of interest.