Abstract
OBJECTIVES:
The respiratory rate-oxygenation (ROX) index is a fraction of oxygen saturation, Fio2, and respiratory rate that has been validated to predict receipt of invasive mechanical ventilation in patients receiving high-flow nasal cannula (HFNC). This study aimed to validate ROX in a cohort of inpatients with COVID-19–related respiratory failure.
DESIGN:
Retrospective validation of the ROX index. We calculated sensitivity, specificity, positive predictive value, negative predictive value, and 95% CIs of ROX for invasive mechanical ventilation any time during hospitalization.
SETTING:
Twenty-one hospitals of Kaiser Permanente Northern California, an integrated healthcare delivery system.
PATIENTS:
We identified adults with positive severe acute respiratory syndrome coronavirus 2 polymerase chain reaction test within 3 weeks of, or during, hospitalization between February 1, 2020, and December 31, 2020. We calculated ROX at 12 hours after HFNC initiation. We grouped patients as low (≥ 4.88), intermediate (< 4.88 and ≥ 3.85), or high (< 3.85) risk using previously published thresholds.
INTERVENTIONS:
None.
MEASUREMENTS AND MAIN RESULTS:
We identified 1,847 patients who had no limitation of life support. Of these, 525 (31.7%) received invasive mechanical ventilation any time during hospitalization and 511 died (27.7%). The sensitivity, specificity, positive predictive value, and negative predictive value of 12-hour ROX threshold (< 3.85) predicting invasive mechanical ventilation were 32.3% (95% CI, 28.5–36.3%), 89.8% (95% CI, 88.0–91.4%), 59.4% (95% CI, 53.8–64.9%), and 74.1% (95% CI, 71.8–76.3%), respectively.
CONCLUSIONS:
The 12-hour ROX index has a positive predictive value (59.4%) using threshold of less than 3.85 for COVID-19 patients needing invasive mechanical ventilation. Our health system has embedded ROX into the electronic health record to prioritize rounding during periods of inpatient surge.
Keywords: COVID-19, mechanical ventilation, respiratory rate-oxygenation index
Prior studies show that patients with acute hypoxemic respiratory failure can benefit from oxygen delivered via high-flow nasal cannula (HFNC) (1). Accordingly, HFNC has been commonly used to treat patients with severe COVID-19 pneumonia (2). Tools to predict which patients with COVID-19 requiring HFNC will require invasive mechanical ventilation are needed to prevent delays in care and allocate resources appropriately. The respiratory rate-oxygenation (ROX) (3, 4) index is a fraction of oxygen saturation divided by Fio2 as the numerator and respiratory rate as the denominator. ROX index has been validated to predict receipt of invasive mechanical ventilation in patients receiving HFNC for any cause before COVID-19 existed (3). A previous prospective validation study showed that ROX index had better performance with increasing time from initiation of HFNC (2-, 6-, and 12-hr) and could be used to identify both low- and high-risk patients (3). Data evaluating performance of ROX in COVID-19 is limited. We aimed to validate ROX in a large cohort of COVID-19 patients receiving HFNC.
METHODS
We identified adults (≥ 18 yr) with a positive severe acute respiratory syndrome coronavirus 2 polymerase chain reaction test within 3 weeks prior to, or during, a hospitalization between February 1, 2020, and December 31, 2020, across 21 hospitals of Kaiser Permanente Northern California, an integrated healthcare delivery system. We narrowed to patients receiving HFNC oxygen. If patients required invasive mechanical ventilation, they had to have greater than or equal to 1 HFNC recording before intubation. We excluded patients with limitation of life support based on code status order before HFNC initiation. We decided not to exclude patients who received noninvasive ventilation before or after HFNC initiation because the goal was to determine accuracy of ROX regardless of whether noninvasive ventilation was delivered.
For each patient, we calculated ROX index, choosing the value closest to, but preceding, the 12-hour timestamp after HFNC initiation (3). We chose the 12-hour timepoint because 1) ROX is more accurate over time (3) and 2) death was shown to increase after 12 hours (3). We grouped patients as low (≥ 4.88), intermediate (< 4.88 and ≥ 3.85), or high (< 3.85) risk according to previous studies (3). We calculated sensitivity, specificity, positive predictive value, negative predictive value, and 95% CIs of ROX for invasive mechanical ventilation any time during hospitalization using two risk thresholds (< 4.88 and < 3.85). We evaluated the percent of patients by risk group who had 1) invasive mechanical ventilation within 24 hours of HFNC initiation, 2) invasive mechanical ventilation anytime during hospitalization, or 3) inpatient death.
To confirm the relationship between ROX risk group and risk of subsequent clinical outcome, we developed a Cox model. Exposure was ROX risk group and outcome was invasive mechanical ventilation or death. We censored patients if discharged alive. We report unadjusted/adjusted hazard ratios (95% CI) with low-risk group as reference. Covariates included age, sex, Comorbidity Point Score, version 2 (5), vitals (heart rate, blood pressure), severity of illness score (Epic Deterioration Index), and laboratory values (blood gas, bicarbonate, troponin, d-dimer). Epic Deterioration Index (range, 0–100) is a validated severity of illness score developed by Epic that is based on clinical data including vital signs, laboratory values, and nursing assessments.
The Kaiser Permanente Northern California Institutional Review Board approved the project (1262727-51), waiving the need for informed consent. We used STATA/SE 14.2 (StataCorp LLC, College Station, TX).
RESULTS
We identified 2,039 patients who received HFNC, including 1,847 who had no limitation of life support. Of these, 657 (35.6%) received subsequent noninvasive ventilation, 525 (31.7%) received invasive mechanical ventilation any time during hospitalization, and 511 died (27.7%). The median time from admission to HFNC initiation was 20.8 hours (interquartile range [IQR], 4.1–56.1 hr). Based on the 12-hour ROX value, 1,176 patients (63.7%) were low risk, 353 (19.1%) were intermediate, and 318 (17.2%) were high (Table 1). Among those receiving invasive mechanical ventilation, median time from HFNC initiation to invasive mechanical ventilation was 83.5 hours (IQR, 27.8–178.9 hr).
TABLE 1.
Baseline Characteristics of Patients Hospitalized for COVID-19 Who Required High-Flow Oxygen and Did Not Have Limitation of Code Status
| Variable | Respiratory Rate-Oxygenation Index at 12 hr After Initiation of High-Flow Nasal Cannula | ||
|---|---|---|---|
| Low Risk ≥ 4.88, n = 1,176 | Intermediate Risk ≥ 3.85 to < 4.88, n = 353 | High Risk < 3.85, n = 318 | |
| Age | 61.6 (15.7) | 62.2 (14.2) | 60.0 (14.2) |
| Male | 716 (60.9%) | 212 (60.1%) | 222 (69.8%) |
| Race | |||
| Hispanic | 489 (41.6%) | 150 (42.5%) | 146 (45.9%) |
| White | 338 (26.3%) | 104 (26.4%) | 85 (23.6%) |
| Asian | 239 (20.3%) | 75 (21.3%) | 59 (18.5%) |
| Black | 114 (9.7%) | 28 (7.9%) | 34 (10.7%) |
| Comorbidity Point Score, version 2a | 29.5 (35.7) | 27.8 (34.2) | 23.0 (28.6) |
| Comorbidities | |||
| Diabetes with and without complications | 441 (37.5%) | 131 (37.1%) | 111 (34.9%) |
| Peripheral vascular disease | 396 (33.7%) | 125 (35.4%) | 94 (29.6%) |
| Chronic pulmonary disease | 306 (26.0%) | 80 (22.7%) | 64 (20.1%) |
| Chronic renal insufficiency | 236 (20.1%) | 59 (16.7%) | 43 (13.5%) |
| Congestive heart disease | 106 (9.0%) | 31 (8.8%) | 19 (6.0%) |
| Cerebrovascular disease | 92 (7.8%) | 29 (8.2%) | 20 (6.3%) |
| Malignancy | 58 (4.9%) | 24 (6.8%) | 23 (7.2%) |
| Dementia | 57 (4.9%) | 14 (4.0%) | 13 (4.1%) |
| Rheumatologic disease | 26 (2.2%) | 8 (2.3%) | < 5 |
| Metastatic disease | 13 (1.1%) | 5 (1.4%) | < 5 |
| Mild liver disease | 17 (1.5%) | < 5 | < 5 |
| Severe liver disease | 11 (0.9%) | < 5 | < 5 |
| Peptic ulcer disease | 8 (0.7%) | < 5 | < 5 |
| Hemiplegia | 8 (0.7%) | < 5 | < 5 |
| Acquired immunodeficiency disorder | 6 (0.5%) | < 5 | < 5 |
| Average Epic Deterioration Indexb on day of initiation of high-flow oxygen | 40.6 (9.9) | 42.6 (10.1) | 48.2 (10.2) |
| Average oxygen saturation/Fio2 on day of initiation of high-flow oxygen | 163 (91) | 216 (84) | 119 (78) |
| Average respiratory rate on day of initiation of high-flow oxygen | 23 (4) | 25 (4) | 29 (4) |
| Admit from home | 1,107 (94.1%) | 342 (96.9%) | 304 (95.6%) |
Comorbidity Point Score, version 2 is an externally validated comorbidity index developed by Kaiser Permanente Northern California. It is calculated using all diagnoses incurred by a patient in 12 mo prior to hospitalization (range, 0–1,014). A score of 0–39 correlates to 1-yr mortality of 0.3% and 40–64 correlates to 5.3%.
Epic Deterioration Index is a widely used, validated severity of illness score developed outside of Kaiser Permanente Northern California that is embedded into the electronic health record. It is calculated based on clinical data including vital signs, laboratory values, and nursing assessments (range, 0–100).
Continuous variables are displayed with mean (sd). Categorical variables are displayed with n (%).
The sensitivity, specificity, positive predictive value, and negative predictive value of 12-hour ROX threshold of less than 4.88 predicting invasive mechanical ventilation were 58.1% (95% CI, 54.0–62.2%), 73.8% (95% CI, 71.3–76.2%), 50.7% (95% CI, 46.8–54.5%), and 79.2% (95% CI, 76.7–81.5%), respectively. The corresponding values using 12-hour ROX threshold of less than 3.85 were 32.3% (95% CI, 28.5–36.3%), 89.8% (95% CI, 88.0–91.4%), 59.4% (95% CI, 53.8–64.9%), 74.1% (95% CI, 71.8–76.3%), respectively.
Figure 1 displays percent of patients by risk group who had 1) invasive mechanical ventilation within 24 hours of HFNC initiation, 2) invasive mechanical ventilation anytime during hospitalization, and 3) death. Adverse outcomes increased with ROX risk group. Of the 1,176 low-risk patients, 245 (20.8%) were intubated during the hospitalization; of the 353 intermediate-risk patients, 151 (42.8%) were intubated during the hospitalization; and of the 318 high-risk patients, 189 (59.4%) were intubated during the hospitalization. Similarly, death occurred in 235 (20.0%) of low, 126 (35.7%) of intermediate, and 150 (47.2%) of high risk.
Figure 1.
This figure shows the percent of patients with COVID-19-related respiratory failure by respiratory rate-oxygenation (ROX) risk group at 12 hours who experience various outcomes. The bars indicate the percent of patients who: 1) received mechanical ventilation within 24 hours of high flow cannular initiation, 2) received mechanical ventilation anytime during hospitalization, and 3) died in the hospital.
In survival analysis, the unadjusted hazard ratio was 2.1 (95% CI, 1.7–2.6) for intermediate risk and 2.9 (95% CI, 2.3–3.5) for high risk, both compared with low risk. The adjusted hazard ratio was 2.2 (95% CI, 1.7–2.7) for intermediate risk and 3.0 (95% CI, 2.4–3.7) for high risk, both compared with low risk.
DISCUSSION
We found that 12-hour ROX index has a positive predictive value (59.4% for < 3.85 threshold) for COVID-19 patients needing invasive mechanical ventilation and could risk stratify patients for both early invasive mechanical ventilation and death. This positive predictive value is lower than the original ROX validation cohort (81%) (3), which could be due to viral pneumonia being an infrequent diagnosis in that cohort (~12%). During periods of COVID-19 surge, when hospitals are under tremendous strain, tools like ROX index can guide resource and staff allocation. Our health system has embedded ROX into the electronic health record to identify high-risk patients.
There is no acceptable cutoff to call a positive predictive value adequate. One must interpret the positive predictive value in the context of the clinical scenario and cost/benefit of the intervention. For example, we have shown in previous work in the New England Journal of Medicine that a model implemented with a positive predictive value of only 10% can be associated with a decrease in mortality (6). The positive predictive value of 59.4% in this study correlates to a number needed to evaluate (7) of 1.7 (calculated as 1/positive predictive value), which is excellent to use as an electronic screening tool, especially when it is combined with bedside clinical assessment. Additionally, the intervention of rounding prioritization costs nothing because patients would be rounded on anyway. Although the positive predictive value of ROX index was lower in COVID-19 patients, we felt that it was adequate to embed in the electronic health record because it would be used as a rounding prioritization tool, not a definitive answer about whether to intubate or not. We did not explore changing the model threshold for COVID-19 patients because we preferred embedding one model with one threshold across patients. Otherwise, the model would need to electronically account for diagnoses, which are inaccurate early in hospitalization.
This study fills important gaps in COVID-19 risk stratification. We report on a substantially larger cohort of patients receiving HFNC across multiple centers (8–10), which bolsters confidence in our findings. The range of number of patients and hospitals in previous studies is between 62 and 255 patients and 1–5 hospitals, respectively (8–10). We also examine a highly contemporary cohort, including those treated after the Randomized Evaluation of COVID-19 Therapy (RECOVERY) trial established dexamethasone as standard of care in COVID-19 pneumonia (11). Prior smaller studies included data through June 2020 (8–10). We also performed survival analysis to confirm that intermediate- and high-risk ROX groups have different likelihoods of invasive mechanical ventilation or death compared with low-risk ROX group. Further research is needed to assess ROX at later time points, the predictive capability of each part of the ROX index, and the utility of using noninvasive ventilation in COVID-19.
This study has important limitations. Respiratory rate may not be accurate (12), which can alter ROX values. Factors not accounted for in ROX should guide decisions to intubate, including patients’ work of breathing and secretions. Because this was a validation study, we used the two thresholds that were previously used to develop and validate ROX prior to the COVID-19 pandemic. It may be beneficial in the future to study a stricter threshold. Strengths include its large, population-based sample spanning all waves of the pandemic and evaluating multiple outcomes across ROX risk groups.
CONCLUSIONS
ROX index at 12 hours after initiation of HFNC for COVID-19 has adequate positive predictive value using the stricter threshold of less than 3.85. Our integrated healthcare system has embedded ROX index into the electronic health record to use as a rounding prioritization tool to identify high-risk patients during periods of inpatient surge.
ACKNOWLEDGMENTS
We thank Patricia Kipnis, PhD, for providing statistical consultation.
Footnotes
Supported, in part, by The Permanente Medical Group and Kaiser Foundation Hospitals.
Dr. Liu’s institution received funding from the National Institutes of Health (NIH) (R35GM128672); he received support for article research from the NIH. The remaining authors have disclosed that they do not have any potential conflicts of interest.
REFERENCES
- 1.Frat JP, Thille AW, Mercat A, et al. ; FLORALI Study Group; REVA Network: High-flow oxygen through nasal cannula in acute hypoxemic respiratory failure. N Engl J Med. 2015; 372:2185–2196 [DOI] [PubMed] [Google Scholar]
- 2.Demoule A, Vieillard Baron A, Darmon M, et al. : High-flow nasal cannula in critically ill patients with severe COVID-19. Am J Respir Crit Care Med. 2020; 202:1039–1042 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Roca O, Caralt B, Messika J, et al. : An index combining respiratory rate and oxygenation to predict outcome of nasal high-flow therapy. Am J Respir Crit Care Med. 2019; 199:1368–1376 [DOI] [PubMed] [Google Scholar]
- 4.Roca O, Messika J, Caralt B, et al. : Predicting success of high-flow nasal cannula in pneumonia patients with hypoxemic respiratory failure: The utility of the ROX index. J Crit Care. 2016; 35:200–205 [DOI] [PubMed] [Google Scholar]
- 5.Escobar GJ, Gardner MN, Greene JD, et al. : Risk-adjusting hospital mortality using a comprehensive electronic record in an integrated health care delivery system. Med Care. 2013; 51:446–453 [DOI] [PubMed] [Google Scholar]
- 6.Escobar GJ, Liu VX, Schuler A, et al. : Automated identification of adults at risk for in-hospital clinical deterioration. N Engl J Med. 2020; 383:1951–1960 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Romero-Brufau S, Huddleston JM, Escobar GJ, et al. : Why the C-statistic is not informative to evaluate early warning scores and what metrics to use. Crit Care. 2015; 19:285. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Zucman N, Mullaert J, Roux D, et al. ; Contributors: Prediction of outcome of nasal high flow use during COVID-19-related acute hypoxemic respiratory failure. Intensive Care Med. 2020; 46:1924–1926 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Chandel A, Patolia S, Brown AW, et al. : High-flow nasal cannula therapy in COVID-19: Using the ROX index to predict success. Respir Care. 2021; 66:909–919 [DOI] [PubMed] [Google Scholar]
- 10.Fink DL, Goldman NR, Cai J, et al. : Ratio of oxygen saturation index to guide management of COVID-19 pneumonia. Ann Am Thorac Soc. 2021; 18:1426–1428 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Horby P, Lim WS, Emberson JR, et al. : Dexamethasone in hospitalized patients with COVID-19. N Engl J Med. 2021; 384:693–704 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Badawy J, Nguyen OK, Clark C, et al. : Is everyone really breathing 20 times a minute? Assessing epidemiology and variation in recorded respiratory rate in hospitalised adults. BMJ Qual Saf. 2017; 26:832–836 [DOI] [PMC free article] [PubMed] [Google Scholar]