Abstract
Objective
To examine the impact of commonly used case definitions for COVID-19 hospitalizations on case counts and outcomes.
Design, Patients, Setting
Retrospective analysis of all adults hospitalized March 1, 2020-March 1, 2022 at five Massachusetts acute care hospitals.
Interventions
Six commonly used definitions of COVID-19 hospitalization were applied: positive PCR within 14 days of admission, PCR plus dexamethasone administration, PCR plus remdesivir, PCR plus hypoxemia, institutional COVID-19 flag, or COVID-19 ICD-10 codes. Outcomes included case counts and in-hospital mortality. 100 PCR-positive cases were reviewed to determine each definition’s accuracy for distinguishing primary/contributing vs incidental COVID-19 hospitalizations.
Results
Of 306,387 hospital encounters, 15,436 (5.0%) met the PCR-based definition. COVID-hospitalization counts varied substantially between definitions: 4,628 (1.5% of all encounters) for PCR plus dexamethasone, 5,757 (1.9%) for PCR plus remdesivir, 11,801 (3.9%) for PCR plus hypoxemia, 15,673 (5.1%) for institutional flags, and 15,868 (5.2%) for ICD-10 codes. Definitions requiring dexamethasone, hypoxemia, or remdesivir selected sicker patients compared to PCR alone (mortality rates 12.2%, 10.7%, and 8.8% versus 8.3%, respectively). Definitions requiring PCR plus remdesivir or dexamethasone did not detect a reduction in in-hospital mortality associated with Omicron. ICD-10 codes had the highest sensitivity (98.4%) but low specificity (39.5%) for distinguishing primary/contributing vs incidental COVID-19 hospitalizations; PCR plus dexamethasone had the highest specificity (92.1%) but low sensitivity (35.5%).
Conclusions
Commonly used definitions for COVID hospitalizations generate variable case counts and outcomes and differentiate poorly between primary/contributing vs incidental COVID-19 hospitalizations. Surveillance definitions that better capture and delineate COVID-associated hospitalizations are needed.
Keywords: COVID-19, hospitalizations, epidemiology, definitions, public health
INTRODUCTION
The COVID-19 hospitalization rate is a key surveillance metric used by public health officials to estimate the population burden of severe SARS-CoV-2 infections. COVID-19 hospitalization rates are also used by the U.S. Centers for Disease Control and Prevention (CDC), in conjunction with case counts and percentage of inpatient beds occupied by COVID-19 patients, to estimate community COVID-19 risk levels that in turn inform recommendations such as indoor masking.1
CDC initially defined “COVID hospitalizations” as any person hospitalized within 14 days of a positive PCR result for SARS-CoV-2, regardless of the patient’s presenting syndrome or reason for admission.2 This definition initially served well for estimating the burden of severe illness, but widespread vaccination, universal testing, prolonged PCR positivity after infection, increasing rates of prior infection, and new and potentially milder SARS-CoV-2 variants such as Omicron have challenged the validity of this measure as a severity indicator. High community infection rates will lead to some patients hospitalized for reasons other than COVID-19 testing positive for SARS-CoV-2, including patients with mild, asymptomatic, or resolving infections. These so-called “incidental” SARS-CoV-2-positive patients are still counted by the traditional CDC definition as COVID-19 hospitalizations without differentiating them from patients hospitalized specifically for COVID-19.
Public health agencies and hospital officials have therefore proposed, and in many cases implemented, alternative definitions to identify hospitalizations specifically due to COVID-19 illness. These typically require receipt of SARS-CoV-2 therapeutics (e.g. dexamethasone or remdesivir) or need for supplemental oxygen in addition to a positive PCR.3-5 Large cohort studies have also used different approaches for defining COVID-19 hospitalizations, including a positive PCR alone,6-9 International Classification of Disease, Tenth Revision, Clinical Modification (ICD-10-CM) codes for COVID-19,10-18 institutional definitions, or combinations of these.19-25 Notwithstanding the panoply of definitions being used, there are little data comparing estimates of COVID-19 hospitalizations, severity of illness, mortality, and trends between definitions, nor their accuracy in identifying primary/contributing vs incidental infections.
In this study, we assessed the impact of different commonly used definitions for COVID hospitalization on case counts, disease severity, and in-hospital mortality in the Omicron vs pre-Omicron periods in a regional health system. In addition, we evaluated how each definition performed at identifying hospitalizations due to COVID-19 versus hospitalizations with incidental COVID-19 using detailed medical record reviews.
STUDY DESIGN AND METHODS
We performed a retrospective cohort study using electronic health record (EHR) and administrative data from 2 large academic hospitals and 3 community hospitals within the Mass General Brigham healthcare system: Massachusetts General Hospital, Brigham and Women’s Hospital, Salem Hospital/Northshore Medical Center, Newton Wellesley Hospital, and Brigham and Women’s Faulkner Hospital. The study included all adult (≥18 years) hospitalizations, including inpatient admissions and observation stays, as well as Emergency Department (ED) visits ending in death, with admission dates between March 1, 2020 and March 1, 2022. ED visits ending in death were included to ensure capture of all potential cases of severe disease. Transfers between study hospitals and readmissions on the same date as discharge were treated as continuous encounters. The study was approved with a waiver of informed consent by the institutional review board at Mass General Brigham (protocol 2020P001631).
COVID Hospitalization Definitions
We assessed six definitions for COVID hospitalizations modeled after existing strategies currently being used for public health surveillance, clinical monitoring, and/or research:1,3,4,26,27.
“PCR only”: positive PCR for SARS-CoV-2 between 14 days prior to admission and discharge.
“PCR + hypoxemia”: positive PCR (using the same timeframe of 14 days pre-admission through discharge) AND the patient either required supplemental oxygen for any amount of time or had at least one oxygen saturation <94% recorded in vital signs during hospitalization.
“PCR + dexamethasone”: positive PCR AND received at least one dose of dexamethasone during hospitalization.
“PCR + remdesivir”: positive PCR AND received at least one dose of remdesivir during hospitalization.
“COVID-19 flag”: presence of an institutional EHR-based COVID-19 flag maintained for ≥5 days with start and end dates overlapping with the hospitalization. Institutional COVID-19 flags were triggered by a positive PCR result but could be removed by local infection control officials if review of patients’ clinical syndrome, initial and repeat PCR tests, cycle thresholds, prior history of known infections, and SARS-CoV-2 anti-nucleocapsid antibody status suggested the positive PCR was more indicative of remote infection or a false positive result.28
“ICD-10”: an ICD-10 discharge diagnosis code for COVID-19 (U07.1 or J12.82).
Outcomes
Each definition was applied across all encounters to calculate crude hospitalization counts, percentage of hospitalizations related to COVID-19, and in-hospital mortality rates by month. We also assessed how outcomes estimates varied by definition in the pre-Omicron (March 1, 2020-December 16, 2021) vs Omicron (December 17, 2021-March 1, 2022) periods in accordance with when the Omicron variant began to predominate in Massachusetts.29
Assessing Accuracy of Definitions for Primary, Contributing, or Incidental COVID-19
From 15,436 PCR-positive encounters, 100 cases were randomly selected for structured medical record reviews using a standardized data abstraction tool in REDCap (version 12.0.19, Vanderbilt University, 2022) (Supplement). 50 cases were selected at random from the pre-Omicron and Omicron eras. All available notes, medication records, laboratory and microbiology results, radiology reports and images, and pathology reports were reviewed.
Each case was adjudicated into one of three categories: 1) primary COVID-19 admission, 2) contributing COVID-19 admission, and 3) incidental COVID-19 admission. A primary COVID-19 admission was defined as an encounter where the patient presented with a syndrome definitely or probably due to SARS-CoV-2 infection, e.g. COVID pneumonia or COVID-related myocarditis. A contributing COVID-19 admission was defined as any encounter not meeting primary COVID-19 criteria but likely triggered by or related to SARS-CoV-2 infection (e.g. exacerbation of underlying disease such as congestive heart failure, chronic lung disease, or arrhythmia) or an encounter during which the patient presented for non-COVID reasons, developed COVID-19 after admission, and the infection led to complications such as medically prolonged stay, ICU transfer, or death. Of note, receipt of a COVID therapeutic in of itself was not considered evidence of primary or contributing COVID hospitalization. Incidental COVID hospitalizations were those where SARS-CoV-2 was not relevant to the syndrome causing admission and did not cause complications. Positive tests deemed to be false positives or residual RNA from previous infection were categorized as “incidental”.28 Please see supplement table 1 for complete descriptions of these categories and representative examples.
The first 15 cases were reviewed independently by two physician reviewers (C.S. and C.R.); interrater reliability for classifying COVID-relevance categories was moderate to strong (agreement on 13/15 cases, Krippendorff’s alpha 0.77). All 15 cases were discussed between the two reviewers to make final adjudications for the two discrepant classifications and ensure a standardized process moving forward. The remaining 85 cases were reviewed by one physician (C.S); any cases where classifications were unclear (n=12) were subsequently discussed with two additional reviewers (C.R. and M.K.) to achieve consensus.
Statistical Analysis
Patient characteristics and outcomes were compared across groups using chi square tests for categorical variables and ANOVA tests for continuous variables. Comparisons between the pre-Omicron and Omicron eras were performed for each definition by calculating incidence of COVID cases per 100 admissions and incidence of ICU admissions, need for mechanical ventilation, and in-hospital deaths per 100 COVID cases, and then calculating an incidence rate ratio (IRR) for Omicron vs pre-Omicron periods.
The proportions of COVID hospitalizations due to primary/contributing COVID vs incidental COVID per medical record review were compared across definitions and area under the receiver operating curves (AUROCs), sensitivity, specificity, and positive and negative predictive values were calculated.
Analyses were conducted using Stata version 17 (StataCorp, 2021, College Station, TX: StataCorp LLC). For all analyses a p value of < 0.05 was considered statistically significant.
RESULTS
Study Cohort
The study cohort included 306,387 hospital encounters associated with 197,434 unique individuals. 15,436/306,387 (5.0%) encounters met the primary PCR-based definition (positive PCR between 14 days prior to admission and discharge). Compared to hospital encounters without a positive PCR test, those meeting the PCR definition were slightly older (median age 62 vs 60 years), more likely to be male (51.8% vs 43.9%), and less likely to be of white race (59.8% vs 75.2%) (Table 1).
Table 1.
Demographics and Baseline Characteristics for COVID- and Non-COVID-related Hospital Encounters
| Category | COVID Encounters | Non-COVID Encounters |
All Encounters | |
|---|---|---|---|---|
| Description | Positive SARS-CoV-2 PCR −14d to DC |
No Positive SARS- CoV-2 PCR −14d to DC |
||
| Overall, n (%) | 15,436 (5) | 290,951 (95) | 306,387 | |
| Age, median (IQR), y | 62 (46-75) | 60 (40-73) | 60 (40-73) | |
| Sex, n (%) | ||||
| Women | 7,437 (48.2) | 163,236 (56.1) | 170,673 (55.7) | |
| Race, n (%) | ||||
| White | 9,226 (59.8) | 218,923 (75.2) | 228,149 (74.5) | |
| Black | 2,256 (14.6) | 26,602 (9.1) | 28,858 (9.4) | |
| Other | 3,096 (20.1) | 34,052 (11.7) | 37,148 (12.1) | |
| Missing | 858 (5.6) | 11,374 (3.9) | 12,232 (4) | |
| BMI, median (IQR) | 27.9 (24-32.8) | 27.3 (23.6-31.9) | 27.3 (23.6-32) | |
| Comorbidities±, n (%) | ||||
| Cancer | 1,555 (10.1) | 46,645 (16) | 48,200 (15.7) | |
| Congestive heart failure | 2,615 (16.9) | 41,770 (14.4) | 44,385 (14.5) | |
| Chronic Lung Disease | 3,483 (22.6) | 55,302 (19) | 58,785 (19.2) | |
| Diabetes | 4,967 (32.2) | 62,534 (21.5) | 67,501 (22) | |
| Neurologic Disease | 2,187 (14.2) | 33,060 (11.4) | 35,247 (11.5) | |
| Kidney Disease | 3,200 (20.7) | 43,008 (14.8) | 46,208 (15.1) | |
| Elixhauser Mortality Score, median (IQR) | 0 (−3 to 15) | 0 (−2 to 13) | 0 (−2 to 13) | |
| ICD-10 Code for COVID | 13,451 (87.1) | 2,417 (0.8) | 15,868 (5.2) | |
| Time Period | ||||
| Pre-Omicron | 12,012 (77.8) | 264,102 (90.8) | 276,114 (90.1) | |
| Omicron | 3,424 (22.2) | 26,849 (9.2) | 30,273 (9.9) | |
Comorbidities were derived using the Elixhauser index.38,39 “Cancer” includes solid tumor with and without metastases and lymphoma. “Diabetes” includes diabetes with and without complications. “Neurologic disease” includes movement disorders, seizures, and other neurologic conditions. “Kidney disease” includes moderate and severe renal failure.
COVID Hospitalizations, Clinical Characteristics, and Outcomes Across Definitions
Clinical characteristics and outcomes for all six definitions are shown in Table 2. The proportions of encounters meeting criteria for COVID hospitalization were 1.5% for PCR plus dexamethasone, 1.9% PCR plus remdesivir, 3.9% for PCR plus hypoxemia, 5.0% for PCR only, 5.1% for institutional COVID-19 flag, and 5.2% for COVID-19 ICD-10 codes; these proportions varied substantially over time (Figure 1). In-hospital mortality rates ranged from 8.3% for PCR only to 12.2% for PCR plus dexamethasone versus 2.2% for all non-COVID encounters.
Table 2.
Clinical Characteristics and Outcomes of Encounters Meeting Each Definition for COVID Hospitalization
| Category | PCR Only | PCR + Hypox | PCR + dex | PCR + RDV | EHR Flag | ICD-10 | Non-COVID (no +PCR) |
All Encounters |
|---|---|---|---|---|---|---|---|---|
| Total Encounters, n (%) | 15,436 (5) | 11,801 (3.9) | 4,628 (1.5) | 5,757 (1.9) | 15,673 (5.1) | 15,868 (5.2) | 290,951 | 306,387 |
| Omicron Era, n (%) | 3,424 (22.2) | 2,323 (19.7) | 1,066 (23) | 1,496 (26) | 3,585 (22.9) | 3,310 (20.9) | 26,849 (9.2) | 30,273 (9.9) |
| Hospital LOS, median (IQR), d | 5 (3-10) | 6 (4-12) | 7 (4-15) | 7 (4-13) | 5 (2-10) | 5 (3-10) | 3 (2-6) | 3 (2-6) |
| Required ICU Admission, n (%) | 3,088 (20) | 3,020 (25.6) | 1,462 (31.6) | 1,474 (25.6) | 3,003 (19.2) | 3,168 (20) | 28,616 (9.8) | 31,704 (10.4) |
| Any Supplemental Oxygen, n (%) | 9,379 (60.8) | 9,379 (79.5) | 4,251 (91.9) | 4,686 (81.4) | 9,496 (60.6) | 9,997 (63) | 127,896 (44) | 137,275 (44.8) |
| Any Mechanical Ventilation (MV), n (%) | 2,140 (13.9) | 2,140 (18.1) | 1,079 (23.3) | 1,029 (17.9) | 2,057 (13.1) | 2,186 (13.8) | 17,294 (5.9) | 19,434 (6.3) |
| Duration of MV, median (IQR), d | 10 (3-20) | 10 (3-20) | 11 (4-21) | 11 (4-21) | 10 (3-20) | 10 (3-20) | 2 (1-5) | 2 (1-6) |
| Discharge disposition, n (%) | ||||||||
| Home | 10,771 (69.8) | 7,506 (63.6) | 3,067 (66.3) | 4,071 (70.7) | 11,208 (71.5) | 11,142 (70.2) | 244,140 (83.9) | 254,911 (83.2) |
| Facility or Transfer | 3,151 (20.4) | 2,810 (23.8) | 920 (19.9) | 1,100 (19.1) | 2,838 (18.1) | 3,092 (19.5) | 36,236 (12.5) | 39,387 (12.9) |
| Hospice | 236 (1.5) | 218 (1.9) | 75 (1.6) | 78 (1.4) | 221 (1.4) | 225 (1.4) | 4,212 (1.5) | 4,448 (1.5) |
| Died | 1,278 (8.3) | 1,267 (10.7) | 566 (12.2) | 508 (8.8) | 1,406 (9.0) | 1,409 (8.9) | 6,362 (2.2) | 7,640 (2.5) |
Figure 1.
Crude number of hospitalizations (A), proportion of COVID hospitalizations/PCR positive hospitalizations (B) and in-hospital mortality for COVID hospitalization (C) by month for six candidate definitions of COVID-19 hospitalization
Pre-Omicron vs Omicron Time Periods
Overall, 30,273/306,387 (9.9%) encounters occurred during the Omicron era versus 276,114 (90.1%) during the pre-Omicron era. However, amongst PCR-positive encounters, 3,424/15,436 (22.2%) occurred during the Omicron period. Median duration of mechanical ventilation was substantially shorter for Omicron-era COVID encounters across all six definitions (4 vs 11 days for PCR only, 6 vs 13 days for PCR plus dexamethasone). Incidence rate ratios and their respective confidence intervals for ICU admission and need for mechanical ventilation were less than 1 across all definitions in the Omicron period, indicating lower risk of these outcomes during the Omicron vs pre-Omicron periods. In-hospital mortality was lower during Omicron for PCR only, PCR plus hypoxemia, institutional flag, and ICD-10 definitions; however, mortality was similar during pre-Omicron and Omicron periods for PCR + dexamethasone and PCR + remdesivir definitions (Table 3). A sensitivity analysis limiting the pre-Omicron period to November 1, 2020 to December 16, 2021 (when use of dexamethasone and remdesivir to treat SARS-CoV-2 were well-established) yielded similar results (Supplemental Table 4).
Table 3.
Incidence of COVID Hospitalization, ICU admission, Mechanical Ventilation, and In-Hospital Mortality in Omicron vs Pre-Omicron Periods for Six Definitions of COVID Hospitalization
| Outcome | Pre-Omicron | Omicron | Comparison | ||
|---|---|---|---|---|---|
| n | Incidence* | n | Incidence* | IRR (95% CI) | |
| COVID Hospitalization | |||||
| PCR only | 12,012 | 4.35 | 3,424 | 11.31 | 2.60 (2.50-2.7) |
| PCR + hypoxemia | 9,478 | 3.43 | 2,323 | 7.67 | 2.24 (2.14-2.34) |
| PCR + dexamethasone | 3,562 | 1.29 | 1,066 | 3.52 | 2.73 (2.54-2.92) |
| PCR + remdesivir | 4,261 | 1.54 | 1,496 | 4.94 | 3.20 (3.02-3.4) |
| COVID flag | 12,088 | 4.38 | 3,585 | 11.84 | 2.71 (2.61-2.81) |
| ICD-10 code for COVID | 12,558 | 4.55 | 3,310 | 10.93 | 2.40 (2.31-2.5) |
| ICU Admission | |||||
| PCR only | 2,621 | 21.82 | 467 | 13.64 | 0.63 (0.57-0.69) |
| PCR + hypoxemia | 2,572 | 27.14 | 448 | 19.29 | 0.71 (0.64-0.79) |
| PCR + dexamethasone | 1,194 | 33.52 | 268 | 25.14 | 0.75 (0.66-0.86) |
| PCR + remdesivir | 1,180 | 27.69 | 294 | 19.65 | 0.71 (0.62-0.81) |
| COVID flag | 2,569 | 21.25 | 434 | 12.11 | 0.57 (0.51-0.63) |
| ICD-10 code for COVID | 2,718 | 21.64 | 450 | 13.60 | 0.63 (0.57-0.69) |
| Mechanical Ventilation | |||||
| PCR only | 1,820 | 15.15 | 320 | 9.35 | 0.62 (0.55-0.69) |
| PCR + hypoxemia | 1,820 | 19.20 | 320 | 13.78 | 0.72 (0.63-0.81) |
| PCR + dexamethasone | 885 | 24.85 | 194 | 18.20 | 0.73 (0.63-0.86) |
| PCR + remdesivir | 824 | 19.34 | 205 | 13.70 | 0.71 (0.61-0.83) |
| COVID flag | 1,754 | 14.51 | 303 | 8.45 | 0.58 (0.52-0.66) |
| ICD-10 code for COVID | 1,876 | 14.94 | 210 | 6.34 | 0.63 (0.56-0.71) |
| In-Hospital Mortality | |||||
| PCR only | 1,085 | 9.03 | 193 | 5.64 | 0.62 (0.54-0.73) |
| PCR + hypoxemia | 1076 | 11.35 | 191 | 8.22 | 0.72 (0.62-0.84) |
| PCR + dexamethasone | 439 | 12.32 | 127 | 11.91 | 0.97 (0.79-1.18) |
| PCR + remdesivir | 383 | 8.99 | 125 | 8.36 | 0.93 (0.76-1.14) |
| COVID flag | 1204 | 9.96 | 202 | 5.63 | 0.57 (0.49-0.66) |
| ICD-10 code for COVID | 1199 | 9.55 | 210 | 6.34 | 0.66 (0.57-0.77) |
For COVID Hospitalization, incidence is the number of COVID cases per 100 encounters; for ICU admission, Mechanical Ventilation, and In-Hospital Mortality, incidence is the number of outcome cases per 100 COVID encounters based on given definition.
Distinguishing Primary/Contributing vs Incidental Infections
Of 100 cases reviewed, 45 met criteria for primary COVID hospitalizations, 17 were COVID contributing, and 38 were COVID incidental (including 19 with PCR results deemed to be a false positive or residual RNA from a previous recovered infection based upon patients’ clinical syndrome, PCR cycle threshold values, repeat test results, and timing of recent infections. Proportions of primary, contributing, and incidental cases differed between pre-Omicron and Omicron subgroups (p<0.001), with 30/50 (60%) vs 15/50 (33.3%) primary, 8/50 (16%) vs 9/50 (18%) contributing, and 12/50 (24%) vs 26/50 (52%) incidental in the pre-Omicron vs Omicron subgroups, respectively.
The performance characteristics for each definition are summarized in Figure 2 and Table 4. PCR plus remdesivir had the highest PPV (90.0%, 95% CI 76.3-97.2) and AUROC (0.74, 95% CI 0.66-0.82) for a COVID-primary or -contributing hospitalization, but at best moderate sensitivity (58.1%, 95% CI 44.8-70.5) and negative predictive value (56.7%, 95% CI 43.2-69.4). The ICD-10 based definition had the highest sensitivity (98.4%, 95% CI 91.3-100) and negative predictive value (93.8%, 95% CI 69.8-99.8) but poor specificity (39.5%, 95% CI 24.0-56.6) and fair positive predictive value (72.6%, 95% CI 61.8-81.8). In general, performance of the other definitions was poor to moderate with AUROCs for primary/contributing hospitalizations ranging from 0.57 (95% CI 0.47-0.66) for PCR plus hypoxemia to 0.69 (0.61-0.77) for ICD-10 codes. The performance for each definition stratified by pre-Omicron vs Omicron can be found in Supplement Table 2.
Figure 2.
Performance of five definitions of COVID-19 hospitalization versus medical record review for Primary and Primary/Contributing COVID Hospitalization
Table 4.
Performance of Candidate Definitions for Identifying Primary or Contributing COVID Hospitalization Versus Manual Chart Review
| Definition | Sensitivity (95% CI) | Specificity (95% CI) | AUROC (95% CI) | PPV (95% CI) | NPV (95% CI) |
|---|---|---|---|---|---|
| PCR only | -* | -* | -* | 62.0 (51.8-71.5) | -* |
| PCR + hypoxemia | 79.0 (66.8-88.3) | 34.2 (19.6-51.4) | 0.57 (0.47-0.66) | 66.2 (54.3-76.8) | 50.0 (29.9-70.1) |
| PCR + dexamethasone | 35.5 (23.7-48.7) | 92.1 (78.6-98.3) | 0.64 (0.56-0.71) | 88.0 (68.8-97.5) | 46.7 (35.1-58.6) |
| PCR + remdesivir | 58.1 (44.8-70.5) | 89.5 (75.2-97.1) | 0.74 (0.66-0.82) | 90.0 (76.3-97.2) | 56.7 (43.2-69.4) |
| Institutional COVID-19 flag | 98.4 (91.3-100) | 26.3 (13.4-43.1) | 0.62 (0.55-0.70) | 68.5 (57.8-78.0) | 90.9 (58.7-99.8) |
| ICD-10 | 98.4 (91.3-100) | 39.5 (24.0-56.6) | 0.69 (0.61-0.77) | 72.6 (61.8-81.8) | 93.8 (69.8-99.8) |
Sensitivity, specificity, AUROC, and NPV left blank for definition 1 as all reviewed encounters met the PCR-only criteria
DISCUSSION
We found substantial variation in COVID hospitalization counts and outcomes across six commonly used definitions for COVID hospitalizations. Crude COVID hospitalization counts varied up to 3-fold between the most inclusive definition (ICD10-based) versus the most restrictive definition (PCR plus dexamethasone). Definitions based upon receipt of COVID therapeutics identified encounters with significantly higher rates of ICU admission, mechanical ventilation, and death. None of the definitions examined reliably differentiated between primary/contributing vs incidental COVID hospitalizations when compared to detailed chart review; the most accurate definition (PCR plus remdesivir) had a very high positive predictive value but identified less than two-thirds of primary/contributing cases.
Our findings demonstrate the challenge of conducting surveillance for severe SARS-CoV-2 infections in the contemporary context. Hospitalization with a positive PCR alone is a poor proxy for severe SARS-CoV-2 infections at this stage of the pandemic. None of the alternative definitions we assessed, however, were both sensitive and specific for severe SARS-CoV-2 infection.
Some of the definitions we evaluated may nonetheless still be useful depending on the purpose of the analysis. The original PCR-based CDC definition, ICD-10-based definition, and institutional COVID flag-based definition were sensitive and produced large cohorts, albeit with lower severity of illness overall and low specificity for primary or contributing COVID-19 infections. These metrics mirror trends in the prevalence of COVID-19 in the local community and accurately reflect the absolute count of cases being managed by hospitals. They are imperfect measures, however, of the incidence of severe COVID-19, particularly during the Omicron period.
Conversely, definitions that incorporated hypoxemia or receipt of anti-COVID therapeutics identified smaller cohorts with higher severity of illness, had greater specificity for primary/contributing infections, and yielded more stable mortality estimates in the pre-Omicron vs Omicron eras. These definitions’ low sensitivity renders them poor proxies for estimating the total burden of severe disease, but their high specificity may make them useful candidates for tracking relative changes in the burden of severe disease over time. These characteristics may also make these definitions useful as inclusion criteria for observational studies of inpatient COVID-19 cohorts, since hospitalizations flagged by these definitions are enriched for COVID-primary or -contributing hospitalizations and experience higher incidence rates of many common study outcomes such as ICU admission or death. However, we urge caution for two reasons: risk of selection bias which can distort the magnitude and direction of measured associations if components of the definition used for inclusion qualify as “collider” variables30, and their performance will likely change over time as indications, availability, and alternative therapies evolve. Public health agencies and researchers can also consider using multiple definitions with different sensitivities and specificities to provide both “conservative” and “liberal” estimates of the burden of severe COVID-19.
We found that ICD-10 codes had high sensitivity and good negative predictive value but poor specificity and moderate positive predictive value in our study. Because our medical record reviews were conducted among PCR-positive hospitalizations, the true positive predictive value of ICD-10 codes might be even lower. This stands in contrast to early assessments of COVID-19 ICD-10 codes which reported excellent positive and negative predictive values for ICD-10 codes when compared to PCR data in all-comers and critically ill patients, respectively.8,24,31-34 In retrospect, the excellent performance for ICD-10 codes in these studies was likely due to the use of PCR positivity as the gold standard for COVID hospitalization, as well as the newness of the epidemic, focal use of testing, low healthcare utilization for non-COVID care (hence fewer incidental cases), and fewer false positives due to prior infections. We advise caution when interpreting studies which identify COVID hospitalizations using ICD-10 codes during the current era.
The finding that all definitions had poor to moderate AUROCs for distinguishing incidental vs primary/contributing COVID-19 underscores the complexity and variability of COVID-19 presentations and the challenge of disentangling the attributable morbidity of SARS-CoV-2 in specific patient encounters. EHR-based approaches using the simple definitions assessed in our study, perhaps unsurprisingly, were ill-equipped to identify such nuance. Our study draws attention to the need to develop better surveillance definitions that more accurately capture and characterize the full spectrum of COVID-associated illness in hospitalized patients. Algorithms that incorporate a wider array of EHR data may better distinguish primary vs incidental COVID hospitalizations, but this comes at the cost of generalizability and the broad applicability that is essential for public health surveillance.35
Many frequently reported COVID outcomes such as need for ICU admission, use of mechanical ventilation, and in-hospital death were significantly less common during the Omicron vs pre-Omicron eras despite much higher case incidence rates during the Omicron era. Prior studies have speculated that this is due to higher rates of population immunity from vaccination and prior infections, a broader armamentarium of therapeutics, and/or lower intrinsic severity for Omicron vs prior variants.36,37 Our study also suggests that a fourth contributing factor is the dramatic increase in community incidence during the initial Omicron surge leading to a large increase in the number of hospitalized patients with incidental COVID-19 and a consequent decrease in the percentage of COVID-19 hospitalizations with severe disease.
Our study has several limitations. First, it was conducted using EHR data from a single healthcare system; larger studies with more geographic diversity are needed. Second, it included only adult patients; results cannot be extended to pediatric populations. Third, only a small number of cases were manually reviewed to characterize each definition’s capacity to distinguish primary/contributing vs incidental infections. Determining the role of COVID-19 in hospitalization can be subjective; this was mitigated by use of a standardized data collection tool and discussion of difficult cases with three clinicians to reach consensus. Fourth, the performance of the definitions we evaluated likely fluctuated over the examined period and will continue to change in the future as new variants emerge, therapeutic strategies evolve, and re-infections become more common. Therefore, ongoing periodic reassessment of definitions for COVID-19 hospitalization will be needed to determine their appropriateness to inform public health surveillance, policy recommendations, and research.
CONCLUSIONS
Estimates of COVID admissions, severity of illness, in-hospital mortality, and trends are significantly affected by how COVID hospitalizations are defined. The traditional PCR-based definition identifies many incidental cases and is associated with less severe illness compared to definitions that incorporate hypoxemia or COVID therapeutics. Most definitions demonstrated improvements in in-hospital mortality rates in the Omicron vs pre-Omicron periods, but definitions which required dexamethasone or remdesivir did not . Medical record reviews demonstrated that no definition accurately differentiated between primary/contributing vs incidental hospitalizations, although positive PCR plus remdesivir or dexamethasone had a high positive predictive value for primary/contributing hospitalizations. An ICD-10-based definition had excellent sensitivity but poor positive and negative predictive values. These findings have important implications for public health surveillance and research, including highlighting the need for improved surveillance definitions that better capture and characterize the full spectrum of COVID-associated disease in hospitalized patients.
Supplementary Material
ACKNOWLEDGMENTS
Financial Support
C.N.S. received grant support from the National Institutes of Health (1F32GM143862-01). M.K. and C.R. received grant support from the Centers for Disease Control and Prevention (6U54CK000484-04-02).
Footnotes
Potential Conflicts of Interest
C.R. reports royalties from UpToDate, Inc., and consulting fees from Cytovale and Pfizer on unrelated topics. M.K. reports royalties from UpToDate, Inc. The other authors report no potential conflict of interest.
REFERENCES
- 1.Science Brief: Indicators for Monitoring COVID-19 Community Levels and Making Public Health Recommendations. Centers for Disease Control and Prevention; March 4, 2022. 2022. [PubMed] [Google Scholar]
- 2.COVID-NET: COVID-19 Associated Hospitalization Surveillance Network, Centers for Disease Control and Prevention., https://www.cdc.gov/coronavirus/2019-ncov/covid-data/covid-net/purpose-methods.html. Accessed April 25, 2022. [Google Scholar]
- 3.Fillmore NR, La J, Zheng C, et al. The COVID-19 Hospitalization Metric in the Pre- and Post-vaccination Eras as a Measure of Pandemic Severity: A Retrospective, Nationwide Cohort Study. Infect Control Hosp Epidemiol 2022:1–24. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Murray S, CR, Wachter RM. Is a patient hospitalized “with” covid or “for” covid? It can be hard to tell. The Washington Post. 2022. https://www.washingtonpost.com/outlook/2022/01/07/hospitalization-covid-statistics-incidental/. Accessed 3/22/22. [Google Scholar]
- 5.Fatima S. Here’s why the state is changing the way it reports COVID hospitalization data. The Boston Globe. 2022. https://www.bostonglobe.com/2022/01/07/nation/heres-why-state-is-changing-way-it-reports-covid-hospitalization-data/. Accessed 3/22/22. [Google Scholar]
- 6.Gershengorn HB, Patel S, Shukla B, et al. Predictive Value of Sequential Organ Failure Assessment Score across Patients with and without COVID-19 Infection. Ann Am Thorac Soc 2022;19:790–798. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Garg S, Patel K, Pham H, et al. Clinical Trends Among U.S. Adults Hospitalized With COVID-19, March to December 2020 : A Cross-Sectional Study. Ann Intern Med 2021;174:1409–1419. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Wu G, D'Souza AG, Quan H, et al. Validity of ICD-10 codes for COVID-19 patients with hospital admissions or ED visits in Canada: a retrospective cohort study. BMJ Open 2022;12:e057838. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Myers LC, Mark D, Ley B, et al. Validation of Respiratory Rate-Oxygenation Index in Patients With COVID-19-Related Respiratory Failure. Crit Care Med 2022;50:e638–e642. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Song Z, Zhang X, Patterson LJ, Barnes CL, Haas DA. Racial and Ethnic Disparities in Hospitalization Outcomes Among Medicare Beneficiaries During the COVID-19 Pandemic. JAMA Health Forum 2021;2:e214223–e214223. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Montgomery MP, Hong K, Clarke KEN, et al. Hospitalizations for COVID-19 Among US People Experiencing Incarceration or Homelessness. JAMA Netw Open 2022;5:e2143407. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Brown HK, Saha S, Chan TCY, et al. Outcomes in patients with and without disability admitted to hospital with COVID-19: a retrospective cohort study. CMAJ 2022;194:E112–E121. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Whitfield GP, Harris AM, Kadri SS, et al. Trends in Clinical Severity of Hospitalized Patients With Coronavirus Disease 2019-Premier Hospital Dataset, April 2020-April 2021. Open Forum Infect Dis 2022;9:ofab599. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Bottle A, Faitna P, Aylin PP. Patient-level and hospital-level variation and related time trends in COVID-19 case fatality rates during the first pandemic wave in England: multilevel modelling analysis of routine data. BMJ Qual Saf 2022;31:211–220. [DOI] [PubMed] [Google Scholar]
- 15.Rose AN, Baggs J, Wolford H, et al. Trends in Antibiotic Use in United States Hospitals During the Coronavirus Disease 2019 Pandemic. Open Forum Infect Dis 2021;8:ofab236. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Kadri SS, Sun J, Lawandi A, et al. Association Between Caseload Surge and COVID-19 Survival in 558 U.S. Hospitals, March to August 2020. Ann Intern Med 2021;174:1240–1251. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Verma AA, Hora T, Jung HY, et al. Characteristics and outcomes of hospital admissions for COVID-19 and influenza in the Toronto area. CMAJ 2021;193:E410–E418. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Musshafen LA, El-Sadek L, Lirette ST, Summers RL, Compretta C, Dobbs TE III. In-Hospital Mortality Disparities Among American Indian and Alaska Native, Black, and White Patients With COVID-19. JAMA Network Open 2022;5:e224822–e224822. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Bhavani SV, Verhoef PA, Maier CL, et al. Coronavirus Disease 2019 Temperature Trajectories Correlate With Hyperinflammatory and Hypercoagulable Subphenotypes. Crit Care Med 2022;50:212–223. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Auld SC, Harrington KRV, Adelman MW, et al. Trends in ICU Mortality From Coronavirus Disease 2019: A Tale of Three Surges. Crit Care Med 2022;50:245–255. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Burnim MS, Wang K, Checkley W, Nolley EP, Xu Y, Garibaldi BT. The Effectiveness of High-Flow Nasal Cannula in Coronavirus Disease 2019 Pneumonia: A Retrospective Cohort Study. Crit Care Med 2022;50:e253–e262. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Wongvibulsin S, Garibaldi BT, Antar AAR, et al. Development of Severe COVID-19 Adaptive Risk Predictor (SCARP), a Calculator to Predict Severe Disease or Death in Hospitalized Patients With COVID-19. Ann Intern Med 2021;174:777–785. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Pineles BL, Harris AD. In-Hospital Mortality in a Cohort of Hospitalized Pregnant and Nonpregnant Patients With COVID-19. Ann Intern Med 2021;174:1779–1780. [DOI] [PubMed] [Google Scholar]
- 24.Khera R, Mortazavi BJ, Sangha V, et al. A multicenter evaluation of computable phenotyping approaches for SARS-CoV-2 infection and COVID-19 hospitalizations. NPJ Digit Med 2022;5:27. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Chomistek AK, Liang C, Doherty MC, et al. Predictors of critical care, mechanical ventilation, and mortality among hospitalized patients with COVID-19 in an electronic health record database. BMC Infect Dis 2022;22:413. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Massachusetts Department of Public Health- COVID-19 Interactive Data Dashboard. https://www.mass.gov/info-details/covid-19-response-reporting#covid-19-interactive-data-dashboard-. Accessed Jan 15, 2022.
- 27.Shappell CN, Klompas M, Kanjilal S, Chan C, Rhee C. Prevalence, Clinical Characteristics, and Outcomes of Sepsis Caused by Severe Acute Respiratory Syndrome Coronavirus 2 Versus Other Pathogens in Hospitalized Patients With COVID-19. Critical Care Explorations 2022;4:e0703. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Rhee C, Baker MA, Kanjilal S, et al. Prospective Clinical Assessments of Hospitalized Patients With Positive SARS-CoV-2 PCR Tests for Necessity of Isolation. Open Forum Infect Dis 2021;8:ofab194. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.The Broad Institute. Tracking the Omicron variant in Massachusetts. 2021. https://www.broadinstitute.org/news/tracking-omicron-variant-massachusetts. Accessed April 22, 2022. [Google Scholar]
- 30.Admon AJ, Bohnert ASB, Cooke CR, Taylor SP. Beyond Confounding: Identifying Selection Bias in Observational Pulmonary and Critical Care Research. Ann Am Thorac Soc 2022;19:1084–1089. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Kadri SS, Gundrum J, Warner S, et al. Uptake and Accuracy of the Diagnosis Code for COVID-19 Among US Hospitalizations. JAMA 2020;324:2553–2554. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Lynch KE, Viernes B, Gatsby E, et al. Positive Predictive Value of COVID-19 ICD-10 Diagnosis Codes Across Calendar Time and Clinical Setting. Clin Epidemiol 2021;13:1011–1018. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Bodilsen J, Leth S, Nielsen SL, Holler JG, Benfield T, Omland LH. Positive Predictive Value of ICD-10 Diagnosis Codes for COVID-19. Clin Epidemiol 2021;13:367–372. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Bosch NA, Law AC, Peterson D, Walkey AJ. Validation of the International Classification of Diseases Code for COVID-19 among Critically Ill Patients. Ann Am Thorac Soc 2022;19:861–863. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Klann JG, Strasser ZH, Hutch MR, et al. Distinguishing Admissions Specifically for COVID-19 From Incidental SARS-CoV-2 Admissions: National Retrospective Electronic Health Record Study. J Med Internet Res 2022;24:e37931. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Modes ME, Directo MP, Melgar M, et al. Clinical Characteristics and Outcomes Among Adults Hospitalized with Laboratory-Confirmed SARS-CoV-2 Infection During Periods of B.1.617.2 (Delta) and B.1.1.529 (Omicron) Variant Predominance - One Hospital, California, July 15-September 23, 2021, and December 21, 2021-January 27, 2022. MMWR Morb Mortal Wkly Rep 2022;71:217–223. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Iuliano AD, Brunkard JM, Boehmer TK, et al. Trends in Disease Severity and Health Care Utilization During the Early Omicron Variant Period Compared with Previous SARS-CoV-2 High Transmission Periods - United States, December 2020-January 2022. MMWR Morb Mortal Wkly Rep 2022;71:146–152. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Elixhauser Comorbidity Software Refined for ICD-10-CM Healthcare Cost and Utilization Project (HCUP). 2021. https://www.hcup-us.ahrq.gov/toolssoftware/comorbidityicd10/comorbidity_icd10.jsp. Accessed October 11, 2022.
- 39.Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Med Care 1998;36:8–27. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.