COVID-19 Pandemic-Associated Changes in the Acuity of Brain MRI Findings: A Secondary Analysis of Reports Using Natural Language Processing

Abstract

Rationale and Objectives

We aimed to assess early COVID-19 pandemic-associated changes in brain MRI examination frequency and acuity of imaging findings acuity.

Methods

Using a natural language processing model, we retrospectively categorized reported findings of 12,346 brain MRI examinations performed during 6-month pre-pandemic and early pandemic time periods across a large metropolitan health system into 3 acuity levels: (1) normal or near normal; (2) incidental or chronic findings not requiring a management change; and (3) new or progressive findings requiring a management change. Brain MRI frequency and imaging finding acuity level were compared over time.

Results

Between March and August of 2019 (pre-pandemic) and 2020 (early pandemic), our health system brain MRI examination volumes decreased 17.0% (6745 vs 5601). Comparing calendar-matched 6-month periods, the proportion of higher acuity findings increased significantly (P< 0.001) from pre-pandemic (22.5%, 43.6% and 34.0% in acuity level 1, 2, and 3, respectively) to early pandemic periods (19.1%, 40.9%, and 40.1%). During the second 3 months of the early pandemic period, as MRI volumes recovered to near baseline, the proportion of higher acuity findings remained high (42.6% vs 34.1%) compared with a similar pre-pandemic period. In a multivariable analysis, Black (B coefficient, 0.16) and underinsured population (B coefficient, 0.33) presented with higher acuity findings (P< 0.05).

Conclusions

As the volume of brain MRI examinations decreased during the early COVID-19 pandemic, the relative proportion of examinations with higher acuity findings increased significantly. Pandemic-related changes in patient outcomes related to reduced imaging access merits further attention.

Introduction

During the initial phase of the COVID-19 pandemic, stay-at-home orders and social distancing guidelines forced United States health systems to postpone elective tests and procedures. Additionally, many patients voluntarily deferred care due to the fear of contracting COVID-19. The combination of hospital-initiated closure of services and patient-initiated deferral of care led to large overall declines in healthcare encounters.^{1 , 2} As a result, both low-acuity and high-acuity patient visits decreased as patients overall presented to hospitals in more advanced stages of disease.3, 4, 5, 6, 7

With respect to imaging services, multiple recent studies have examined the early effects of the COVID-19 pandemic on radiology practices. Across geographically diverse academic and community health systems of various sizes, imaging volumes decreased 40-60% during the first few months of the pandemic.8, 9, 10 Outpatient imaging volumes were most severely impacted and inpatient examinations shifted somewhat from cross-sectional to radiographic imaging.¹¹ Examination declines varied considerably across body parts and modalities (eg, -80 to -92% for mammography, -44% for abdominal CT, -52% for brain MR).^{12 , 13} Similarly, decreases have been reported in cancer screening efforts.14, 15, 16

Manually reviewing 196 radiology reports, Romero et al recently reported a similar decline in the frequency of abdominal computed tomography examinations performed for suspected appendicitis during the early pandemic; importantly, they also described a shift in the incidence of positive examinations as well as an increase in the relative acuity of imaging findings.¹⁷ Changing patterns of imaging findings in association with pandemic-related examination declines, however, have received little attention in other body regions.

Using natural language processing (NLP), we aimed to categorize changes in the overall acuity of brain MRI examination findings during the early phase of the COVID-19 pandemic (March 1, 2020 to August 30, 2020) across a large metropolitan health system compared with a similar time period 1 year earlier.

Methods

This retrospective study was formally approved by our Institutional Review Board. Procedures for data collection. Our analysis complied with the Health Insurance Portability and Accountability Act (HIPAA) and was performed under a formal exemption for informed consent.

Study Population

Our institutional clinical data warehouse, a patient data repository that annually integrates data for approximately 600,000 patient encounters across our large metropolitan health system (with 5 hospital sites and multiple outpatient imaging centers) was used to search for study-relevant information. We first identified all brain MRI examinations (without and/or with contrast) performed on patients 18 years of age or older between March 1, 2020 and August 31, 2020 (during the early COVID-19 pandemic). A comparison group of brain MRI examinations was identified for similarly identified patients between March 1, 2019, and August 31, 2019. In both pre-pandemic and early pandemic groups, brain MRI examination reports with combined reports of other examinations (eg, MRA of the head) were excluded to facilitate accurate training of the subsequently detailed NLP model used in this study.

Acuity Levels of Imaging Finding

The acuity of imaging findings in each brain MRI examination report was categorized as (1) normal or near-normal, (2) chronic or incidental findings without an anticipated change in management, and (3) new or progressive findings requiring a change in management. Examples of findings associated with this categorization scheme are outlined in Supplementary Table 1.

Natural Language Processing Model

Of all brain MRI reports meeting inclusion criteria, the first 1128 consecutive reports of year 2019 and the first 848 consecutive reports from year 2020 (total of 1976 reports), were set aside to train our task-specific decoder to classify each report into 1 of the 3 labels: “normal or near normal,” “chronic or incidental,” and “new or processive finding.” Each report was independently examined by 2 reviewers (selected from a group of 3 fellowship-trained board-certified neuroradiologists with 3-14 years of post-training experience, 1 neuroradiology fellow, and 2 radiology residents with interest in neuroradiology) and acuity level of imaging findings were independently assigned. All 6 reviewers held meetings after every 500 report reviews to discuss and resolve discrepancies by group consensus.

Additional brain MRI reports were then obtained from our institutional clinical data warehouse for patients meeting similar inclusion and exclusion criteria, but now for exams predating both time periods of interest (ie, between 2012 and 2019). These unannotated reports were only used for training the language model of the transformer encoder in our NLP model. This pretrained language model was then finetuned by the task-specific decoder using the 1976 reports for disease acuity classification using the set of manually classified MRI brain reports. The final resulting NLP model was then used to classify the MRI reports from March through August of each 2019 (pre-pandemic) and 2020 (early pandemic) period examined in this study. Details regarding the NLP model training and the 4 NLP models used to classify the acuity of imaging findings are shown in Supplementary Material 1.

Independent Variables

Patient demographics including age, gender, race, marital status, type of insurance and encounter type (inpatient, outpatient, emergency department) as reported in our institutional electronic medical record were extracted from the database.

Statistical Analysis

Categorical variables were reported as frequencies and percentages, and continuous variables were reported as means and standard deviations. Patient characteristics for included brain MRI examinations and acuity levels for imaging findings in 2019 and 2020 were compared using chi-square tests for categorical variables and 2 sample t-test for continuous variable. Multivariable ordinal logistic regression analyses were performed to assess factors associated with acuity levels of imaging findings. Listed independent variable patient demographics, encounter type and year were included in the model. Interaction between race and year, as well as insurance and year were further assessed in multivariable ordinal logistic regressions. Stata Corp software (Stata/SE 14.2 for Mac (64-bit Intel)) was used for statistical analysis. The baseline established for statistical significance was P < 0.05.

Results

Study Population

A total of 6745 and 5601 brain MRI examinations meeting inclusion criteria were performed between March and August of 2019 and March and August of 2020, respectively (total of 12,346 examinations). Patient characteristics were similar across both years with respect to age at the time of the examination, gender, race, and marital status (Table 1 ). Compared to pre-pandemic 2019, a larger percentage of examinations during early pandemic 2020 were performed during emergency department and inpatient encounters (41.1% vs 36.8%; P < 0.001) and on commercially insured patients (70.3% vs 67.5%; P = 0.006). Of note, fewer examinations were performed on Medicaid patients during early pandemic 2020 (3.7% vs 4.5%; P = 0.006).

TABLE 1.

Patient characteristics for included Brain MRI examinations

	2019(n = 6745)	2020(n = 5601)	P value
Age (mean ± std)	58.1 ± 17.0	58.7 ± 16.5	0.07
Gender, n (%)			0.97
Female	3770 (55.9)	3133 (55.9)
Male	2975 (44.1)	2468 (44.1)
Race, n (%)	Missing = 356	Missing = 369	0.16
Caucasian or White	4000 (62.6)	3215 (61.5)
African American or Black	2062 (32.3)	1709 (32.7)
Other	327 (5.1)	306 (5.8)
Marital status, n (%)	Missing = 729	Missing = 570	0.47
Married or life partner	3428 (57.0)	2902 (57.7)
Single, divorced, separated, or widowed	2588 (43.0)	2129 (42.3)
Encounter type, n (%)	Missing = 27	Missing = 38	<0.001
Outpatient	4245 (63.2)	3278 (58.9)
Inpatient or Emergency Department	2473 (36.8)	2285 (41.1)
Insurance type, n (%)	Missing = 836	Missing = 790	0.006
Commercial	3989 (67.5)	3383 (70.3)
Medicare	1647 (27.9)	1249 (26.0)
Medicaid	265 (4.5)	176 (3.7)
Uninsured	8 (0.1)	3 (0.1)

Bolded P values represent statistical significance.

Inter-annotator Agreement and NLP Model Performance

The overall inter-annotator agreement rate for the 2 initial independent reviews of 1976 brain MRI reports for imaging finding acuity categorization was 85.11%, with a kappa value of 0.7737, confirming that our scheme (Supplementary Table 1) was robust enough to produce statistically significant agreement rates between reviewers across varying levels of experience.

The NLP model developed for this study was used to categorize the same set of 1976 reports and demonstrated an accuracy of 86.19 % accuracy (Supplementary Material 2). This NLP model was used to categorize the remaining 10,370 reports.

Brain MRI Volumes in Pre-pandemic 2019 vs Early Pandemic 2020

Although the total number of brain MRI examination performed across our health system during early pandemic 2020 was less than during the same calendar period in 2019, this decrease occurred predominantly during the very early pandemic months of March through May 2020 with 2347 examinations (representing 41.9% of all March through August 2020 examinations) vs 3464 during the pre-pandemic 2019 period (51.3% of all March through August 2019 examinations), resulting in a relative decrease of 32.2% examinations in 2020. During the months of June through August, a total of 3254 and 3281 examinations were performed during 2020 and 2019, reflecting a return to 99.2% of pre-pandemic baseline volumes. Figure 1 illustrates daily mean examination volumes in 2019 and 2020.

FIG 1.

Daily brain MRI examination volumes during (A) Pre-pandemic March-August 2019 and (B) Early Pandemic March-August 2020 (color version of figure is available online).

Each blue circle denotes the number of brain MRIs performed on a given day. The red line shows a moving 7-day average of the daily number to help visualize trends. The black line denotes the overall mean of number of brain MRI exams.

Brain MRI Imaging Findings During Pre-pandemic 2019 vs Early Pandemic 2020

The volume of brain MRIs performed at our health system across the 3 predetermined acuity levels of imaging findings during both 6-month pre-pandemic and early pandemic periods (as well as in the first and second half of each period) are outlined in Table 2 . In 2019, 22.5%, 43.6% and 34.0% of all included brain MRI examinations were assigned to findings acuity category 1, 2, and 3 respectively. In 2020, the fraction of category 1 and 2 examinations decreased to 19.1% and 40.9%, respectively, and the fraction of category 3 examinations increased to 40.1% (P < 0.001). When the early pandemic 2020 period was divided into first and second 3-month windows (ie, March through May 2020 vs June through August 2020), shifts to the higher acuity category 3 since 2019 persisted, but were more pronounced during very early phase of pandemic with 42.6% of category 3 examinations between March through May 2020 vs 34.1% between March through May 2019 (Table 2). Figure 2 illustrates frequency of acuity levels of imaging findings across the pre-pandemic 2019 and early pandemic 2020 periods.

TABLE 2.

Brain MRI volume and acuity of imaging findings during pre-pandemic and early pandemic March through August 2019 and 2020

	Volume	Acuity levels for Imaging findings*			P value
		1	2	3
Year					<0.001
2020	5601	1067 (19.1)	2289 (40.9)	2245 (40.1)
2019	6745	1515 (22.5)	2938 (43.6)	2290 (34.0)
Earlier (March-May)					<0.001
2020	2347	434 (18.5)	914 (38.9)	999 (42.6)
2019	3464	787 (22.7)	1495 (43.2)	1182 (34.1)
Later (June-August)					<0.001
2020	3254	633 (19.4)	1375 (42.3)	1246 (38.3)
2019	3281	728 (22.2)	1445 (44.0)	1108 (33.8)

Bolded P values represent statistical significance.

^⁎Acuity levels of Imaging findings are as follows: (1) Normal or near normal; (2) chronic or incidental that does not change management; (3) acute or any finding that change management.

FIG 2.

Frequency of levels of imaging finding acuity during pre-pandemic March-August 2019 and early pandemic March-August 2020.

Predictors of Higher Acuity of Brain MRIs Findings

In our multivariable analysis of acuity of imaging findings controlling for other variables, examinations on patients who were older (B coefficient, 0.004; 95% CI: 0.001, 0.007), Black (B coefficient, 0.16; 95% CI: 0.07, 0.25), married (B coefficient, 0.10; 95% CI: 0.02, 0.18) and underinsured (Medicaid and uninsured; B coefficient, 0.33; 95% CI: 0.12, 0.51) were associated with a higher acuity of imaging findings on their brain MRI examination (Table 3 ). Outpatient encounters were associated with a lower acuity of imaging findings (B coefficient, -0.61; 95% CI: -0.70, -0.53).

TABLE 3.

Predictors of higher acuity category of imaging findings

Patient characteristics		B (95% confidence interval)	P value
Age		0.004 (0.001, 0.007)	<0.001
Gender	Female	Reference
	Male	0.01 (-0.70, 0.09)	0.77
Race	Caucasian or White	Reference
	African American or Black	0.16 (0.07, 0.25)	<0.001
	Other	0.09 (-0.09, 0.26)	0.33
Marital status	Not married	Reference
	Married	0.10 (0.02, 0.18)	0.01
Encounter type	Inpatient + emergency department	Reference
	Outpatient	-0.61 (-0.70, -0.53)	<0.001
Insurance	Commercial	Reference
	Medicare	0.09 (-0.004, 0.18)	0.05
	Medicaid or Uninsured	0.33 (0.12, 0.51)	0.001
Year	2019	Reference
	2020	0.22 (0.15, 0.30)	0.001

Bolded P values represent statistical significance.

When evaluating the observed racial and insurance disparity gaps in acuity levels of imaging findings between 2019 and 2020, adjusting for other variables, no significant differences were noted. As such, while the odds of higher levels of acuity findings in 2020 compared to 2019 were increased for certain race and insurance groups, we did not find statistically significant racial and insurance disparity gaps (P > 0.05, Table 4 ).

TABLE 4.

Comparison of racial and insurance disparity gap in acuity levels of imaging findings between 2019 and 2020

	OR for 2020 vs 2019 (95% CI)‡	P value
Comparison of racial disparity gap*		0.16
Caucasian or White	1.19 (1.09, 1.31)
African American or Black	1.33 (1.16, 1.53)
Other	1.57 (1.13, 2.19)
Comparison of insurance disparity gap†		0.63
Commercial	1.22 (1.11, 1.33)
Medicare	1.32 (1.14, 1.53)
Medicaid or Uninsured	1.30 (0.89, 1.91)

^⁎Model evaluates the interaction between race and year controlling for gender, marital status, encounter type, and insurance.

^†Model evaluates the interaction between insurance and year controlling for gender, marital status, encounter type and race.

^‡Odds of increasing 1 unit in acuity level of imaging findings for 2020 compared to 2019 is reported for each race or insurance category.

Discussion

In this retrospective review of 12,346 brain MRI reports categorized into 3 levels of imaging finding acuity, we found that although the volume of our health system brain MRI examinations decreased during the early COVID-19 pandemic, patients during the pandemic presented with higher levels of acuity of imaging findings. Overall, our observed volume reduction but increased acuity of findings in brain MRI examinations comports with findings recently described in abdominal and pelvic CT examinations in patients with suspected appendicitis¹⁷.

The overall decrease in brain MRI volume during the early pandemic is consistent with prior studies,^{18 , 19} and likely attributable to a combination of factors including health system and departmental policies developed in response to social distancing protocols to reduce nonurgent radiology services availability, as well as many patients curtailing outside activities and choosing not to seek medical care due to fear of getting infected with COVID-19 or lost insurance or income as a result of economic crisis associated with COVID-19.²⁰ Higher levels of acuity for imaging findings during the early months of the pandemic are likely a result of similar factors with patients needing the most medical attention presenting more urgently.

Although brain MRI volume recovered to 2019 levels in the later months of 2020, the average disease acuity remained higher than in 2019. If this persists, this could have implications for radiologist staffing, with more complicated examinations requiring more physician time and effort. Several factors may have driven this shift to higher acuity examination findings. First, the care delays many patients experienced during the early months of the pandemic may have resulted in clinical deteriorations. Second, patients requiring screening tests may have continued to delay receipt of preventive imaging services due to fear of getting infected within a healthcare setting. Third, it is also possible that patients with less severe conditions may have chosen other imaging facilities with lower levels of provider-initiated delays (eg, private centers that did not postpone services to the degree of our health system). Fourth, local COVID-19 infection rates in Georgia reached a peak during months of July and August.²¹ As COVID-19 infection is associated with increased risk of acute cerebrovascular accidents and thrombotic events,^{22 , 23} some of the high acuity findings seen in the second half of study may have been direct results of increasing COVID-19 infections in the community. Finally, patients with less severe conditions (eg, mild back pain, minor headache) may have been managed more conservatively to avoid exposure to hospitals and imaging facilities. The latter is likely the result of a change in referring provider ordering behavior and could have serendipitously been a step towards mitigating the ongoing increasing utilization of imaging services, which has been shown to be a trend over the last 2 decades regardless of clinical setting or health conditions.24, 25, 26, 27, 28, 29

The degree that each of these factors—as well as potentially others—contributed to an increase in imaging finding acuity is unknown. As such, further studies are needed to elucidate how patients and clinicians weighed skipping or delaying a nonacute imaging exam due to fear of COVID vs obtaining a negative exam to improve patient anxiety³⁰ or increase clinical confidence in patient management.³¹

Our findings suggest that regardless of the pandemic, patients who are older, Black (vs White), uninsured or covered by Medicaid (vs commercial), married (vs not married), and evaluated in a hospital setting (emergency room and inpatient vs outpatient) were found to have higher imaging finding acuity. Our findings thus comport with prior studies suggesting that racial minorities and underinsured populations often present in the later stages of disease, receive disparate treatment (eg, lower rates of mechanical thrombectomy for acute ischemic stroke), and have worse outcomes.32, 33, 34, 35 Of note, however, such racial and insurance disparities in the acuity levels of imaging findings did not increase in 2020 compared with pre-pandemic levels. In contrast other studies have shown an exacerbation in health disparities and their underlying social determinants of heath during the pandemic.^{36 , 37} Our analysis only included examinations during the first 6 months of the COVID-19 pandemic and only focused on acuity of brain MRI examinations. It is possible that our study did not capture the consequences of an evolving disparity gap in healthcare access and utilization.

One strength of our study includes the use of a NLP model to categorize the acuity levels of imaging findings (as opposed to manual review) allowing the evaluation of far more reports than in a similar investigation¹⁷ focusing on abdominal CT exams. However, our study has several limitations. First, since the study was conducted in a retrospective manner, we were unable to assess of the outcome of the patients who did not seek healthcare during the pandemic. Second, we were unable to evaluate specific provider-level and patient-level reasons for observed changes in the acuity levels of imaging findings. Third, while NLP allowed us to evaluate many radiology reports, the NLP algorithm was not perfect in acuity characterization (86.2% accuracy rate). Nonetheless, that accuracy was concordant with the inter-rater agreement rate of our physician reviewers (85.1%). Finally, in order to optimize NLP performance, we excluded 5762 brain MRI examinations that had additional linked examinations reported together. The most common such studies were MRA of the head and neck performed to evaluate transient ischemic attack or stroke. However, given that a large proportion of patients with transient ischemic attack or stroke also underwent a single brain MRI following a CT angiogram, we expect a similar distribution of imaging findings acuity in those patients.

Conclusion

In summary, while our health system experienced a 17.0% decrease in the volume of brain MRI examinations during the early COVID-19 pandemic, patients undergoing brain MRI presented with findings of higher acuity, which persisted even after the return of examination volumes to pre-pandemic levels. Although we confirmed existing disparities with Black and underinsured patients presenting with higher levels of acuity of imaging findings, those disparities were not exaggerated during the pandemic. Future multi-institutional studies evaluating imaging acuity over longer period (as opposed to 6 months) are needed to evaluate provider- and patient-level factors that contributed to increase in imaging findings acuity during pandemic and further assess the changing health disparity gaps in imaging findings.

Authors’ Contributions

Conceptualization and Design: GS, RD, JAW, RH, JDC; Data curation: TLM, LX, JDC, RH, JWA, CR, DH, GS; Data analysis: GS, JS; Interpretation of results: GS, RD, JAW, RH, JDC; Drafting the manuscript: TM; Revision and final approval of the manuscript: TLM, LX, JDC, RH, JWA, CR, DH, RD, JS, GS