Predictive Accuracy of the ROX Index for Re-Intubation in Mechanically Ventilated Patients With COVID-19

Abstract

BACKGROUND:

Several studies have suggested that high-flow nasal cannula (HFNC) is useful for respiratory support after extubation in subjects with COVID-19 pneumonia, whereas 18% subsequently needed to undergo re-intubation. This study aimed to evaluate whether the breathing frequency (f)-ratio of oxygen saturation (ROX) index, which has been shown to be useful for predicting future intubation, is also useful for re-intubation in subjects with COVID-19.

METHODS:

We retrospectively analyzed mechanically ventilated subjects with COVID-19 who underwent HFNC therapy after extubation at 4 participating hospitals between January 2020–May 2022. We evaluated the predictive accuracy of ROX at 0, 1, and 2 h for re-intubation until ICU discharge and compared the area under the receiver operating characteristic (ROC) curve of the ROX index with those of f and S_pO2/F_IO2.

RESULTS:

Among the 248 subjects with COVID-19 pneumonia, 44 who underwent HFNC therapy after extubation were included. A total of 32 subjects without re-intubation were classified into the HFNC success group, and 12 with re-intubation were classified into the failure group. The overall trend that the area under the ROC curve of the ROX index was greater than that of the f and S_aO2/F_IO2 was observed, although there was no statistical significance at any time point. The ROX index at 0 h, at the cutoff point of < 7.44, showed a sensitivity and specificity of 0.42 and 0.97, respectively. A trend of positive correlation between the time until re-intubation and ROX index at each time point was observed.

CONCLUSIONS:

The ROX index in the early phase of HFNC therapy after extubation was useful for predicting re-intubation with high accuracy in mechanically ventilated subjects with COVID-19. Close observation for patients with < 7.44 ROX index just after extubation may be warranted because of their high risk for re-intubation.

Introduction

Re-intubation sometimes occurs in patients with COVID-19 who had been liberated from mechanical ventilation. Re-intubation has been associated with higher mortality; therefore, efforts to avoid this adverse event are important for improving outcomes in patients with COVID-19 liberated from mechanical ventilation.¹ Several studies showed that the use of high-flow nasal cannula (HFNC) after extubation substantially decreased the rate of re-intubation for these subjects. However, about one in 5 was re-intubated after the failure of HFNC therapy.^2-4

HFNC therapy is useful for preventing re-intubation.² But if the trial of HFNC therapy fails and the subjects eventually need to be reintubated, then the mortality may be worse than that in subjects who have undergone primary intubation.^5,6 If we can identify patients at high risk for re-intubation in the early period after extubation, we can use more aggressive therapies to prevent re-intubation.⁷ Also, we can more intensively and frequently check their respiratory status than usual, which may enable us to detect clinical deterioration earlier. However, there is no established tool for identifying patients with COVID-19 at high risk for re-intubation.

In this study, we hypothesized that the breathing frequency (f)-ratio of oxygen saturation (ROX) index, which can be simply calculated from f, S_pO2, and F_IO2,⁸ can predict the risk of re-intubation for subjects with COVID-19 with HFNC after extubation in the early phase.⁹ The ROX index is known to be useful for predicting the failure of HFNC in subjects with acute respiratory failure including COVID-19.^10-13 However, it remains unclear whether the ROX index is also useful for predicting re-intubation in subjects who have been extubated. Thus, this study aimed to evaluate the predictive accuracy of the ROX index for re-intubation after HFNC therapy for extubated subjects with COVID-19.

QUICK LOOK.

Current knowledge

Re-intubation has been reported to be associated with increased mortality. Efforts to identify subjects at high risk for re-intubation after high-flow nasal cannula (HFNC) therapy are clinically important. However, there has been no established tool for identifying subjects with COVID-19 at high risk of re-intubation after HFNC therapy.

What this paper contributes to our knowledge

The area under the receiver operating characteristic curve of the breathing frequency-ratio of oxygen saturation (ROX) index for the re-intubation after extubation was > 0.80 at 0, 1, and 2 h after extubation. The subjects with score < 7.44 of the ROX index immediately after extubation had high risk for future re-intubation. The ROX index may be useful as a predictor for re-intubation of extubated subjects with COVID-19, and close observation may be needed for subjects with a score < 7.44 of the ROX index after extubation.

Methods

We conducted a multi-center retrospective observational cohort study for consecutive mechanically ventilated adult subjects with pneumonia due to COVID-19 who were admitted to the ICUs of 4 participating hospitals (Hiroshima University Hospital, Otsu City Hospital, National Hospital Organization Kyoto Medical Center, and Yao Tokushukai General Hospital) from January 2020–May 2022. Subjects who received HFNC therapy after extubation were included in this analysis. Subjects were excluded if they were < 18 y old or had obtained do-not-resuscitate (DNR) or do-not-intubate (DNI) orders before extubation. Clinicians did not trial noninvasive ventilation prior to re-intubation. This study was approved by the ethics review committee of Hiroshima University Hospital (E-2679). The full study protocol can be accessed at the following URL: https://www.hiroshima-u.ac.jp/system/files/181095/E-2679-1.pdf, in Japanese. This study was supported by a grant from the Ryokuhukai and AMED under grant number JP20fk0108544 and JP22fk0108654.

Data were retrospectively collected from the electronic health records of the participants. Data included age, sex, body mass index, comorbidities, Acute Physiology and Chronic Health Evaluation (APACHE) II score within 24 h after ICU admission, Sequential Organ Failure Assessment (SOFA) score within 24 h after ICU admission, SOFA score within 24 h after extubation, and clinical course after extubation. Spontaneous breathing trial (SBT) settings, including F_IO2, PEEP, duration of SBT, and how many times attempted SBT before extubation, were also collected. Physiological parameters, including S_pO2 and f, and mechanical ventilator settings, including F_IO2 and O₂ flow, were also extracted from medical records to calculate the ROX index in the early phase after the start of HFNC therapy followed by extubation (0, 1, and 2 h after HFNC therapy). The ROX index was calculated based on a previous reference.⁸ The early phase was defined as within 2 h after the start of HFNC therapy because the guidelines recommend that the first reevaluation of respiratory status should be done within 2 h after attempting initiating HFNC therapy.¹⁴ The number of f data was retrospectively collected from impedance cardiography monitors on electronic medical records. In this study, we retrospectively collected the values of S_pO2, f, and F_IO2 and calculated the ROX index. The ROX index was newly calculated for this study, but these 3 indices for the ROX index had already been measured by clinicians who were unaware of the outcome of the subjects at that time. Adverse events were not observed in this study.

The primary outcome was re-intubation after HFNC therapy. After subjects were extubated, intensivists examined their respiratory status at least every 30 min by examining vital signs, breathing patterns, results of blood gas examinations (if needed), etc. They judged the need for re-intubation under the following circumstances: signs of respiratory distress (f > 24 breaths/min and use of accessory respiratory muscles), persistent hypoxia (S_pO2 < 90%), hypercapnia with acidosis (pH < 7.35), and circulatory failure or coma despite maximum HFNC support (HFNC: F_IO2 1.0 at 60.0 L/min).¹⁵ We defined subjects who were discharged from hospital without re-intubation as the success group and subjects who were re-intubated prior to hospital discharge as the failure group.

In all participating ICUs, a spontaneous awakening trial and SBT were performed before extubation. We followed the established guidelines for the application of HFNC and the judgment of failure.¹⁶ We identified SBT failure based on both objective and subjective indices according to the guidelines.¹⁷ Objective indices included tachypnea (f > 35 breaths/min), tachycardia (> 140 beats/min), and hypoxemia (S_pO2 < 90%), whereas subjective indices included agitation and increasing respiratory efforts. We used HFNC comprehensively including the subjects’ f, the presence of signs of respiratory distress, and S_pO2 (usually 90–95%). We used the Airvo 2 (Fisher & Paykel Healthcare, Auckland, New Zealand), Precision Flow (Vapotherm, Exeter New Hampshire), or Monnal T60 (Air Liquide Medical Systems, Antony, France) HFNC systems.

We determined the sample size to ensure an acceptable predictive accuracy of the ROX index for the failure of HFNC therapy (areas under the ROC curve ≥ 0.7),^18,19 with 70% statistical power (β power) and 5% of an α error. The ratio was determined from a previous reference.¹ We used pROC package in R software (R Foundation for Statistical Computing, Vienna, Austria) for the calculation of our sample size with the estimation of 10% drop-out rate. In this manner, we determined that at least 47 subjects were needed.

Fisher exact test and Mann-Whitney U test were used to compare categorical and continuous variables, respectively. One subject was excluded from the analysis because the subject was liberated from HFNC within 2 h of extubation and thus had no ROX index data at 2 h. Multivariate logistic regression analysis was performed using the SOFA score within 24 h after extubation and age as adjustment variables because it was substantially associated with the occurrence of re-intubation based on a previous reference.^20,21 The area under the ROC curve of the ROX index was compared with the f and S_pO2/F_IO2. The DeLong test was used for comparison. Pearson correlation coefficients between the ROX index at each time point and the time until re-intubation were calculated. Statistical analyses were performed using JMP Pro 16 (JMP Statistical Discovery, Cary, North Carolina) and R software (The R Foundation for Statistical Computing). The R software was used only for sample size calculation and the analysis regarding linear regression line of the ROX index versus time until re-intubation; all other analyses were performed using JMP Pro 16. Statistical significance was set at P < .05.

Results

Among the 248 subjects who received mechanical ventilation for COVID-19 pneumonia between January 2020–May 2022, 151 were extubated. Among them, 107 were excluded because they did not receive HFNC therapy after extubation (n = 106) or had DNR or DNI orders before extubation (n = 1). Data from the remaining 44 subjects were analyzed (Fig. 1).

Fig. 1.

Subject flow chart. HFNC = high-flow nasal canula; DNR = do not resuscitate; DNI = do not intubate.

The baseline characteristics of the analyzed study participants are summarized in Table 1. Among the 44 subjects, 32 (72%) were categorized into the success group and 12 (28%) into the failure group. Among the 12 subjects in the failure group, 10 had respiratory insufficiency, whereas the remaining 2 subjects had non-respiratory insufficiency. The distribution of the time from extubation until re-intubation for the subjects who failed is shown in Figure 2. Notably, in the failure group, most subjects were re-intubated within 48 h after extubation (92%). The SBT setting is summarized in Supplemental Table 1 (See related supplementary materials at http://www.rcjournal.com). The maximum level of F_IO2 during SBT is also shown in Supplemental Figure 1 (See related supplementary materials at http://www.rcjournal.com).

Table 1.

Baseline Characteristics

Fig. 2.

The distribution of the time from extubation until re-intubation.

The values of ROX index for success and failure group are shown in Figure 3. The median (interquartile range) ROX index at 0, 1, and 2 h after extubation was 7.70 (6.22–11.07), 8.35 (6.47–9.52), and 7.67 (6.53–11.55), respectively, in the failure group and 11.75 (9.78–15.22), 12.95 (10.00–15.66), and 13.06 (9.51–16.67), respectively, in the success group. Lower ROX index values were significantly associated with the occurrence of re-intubation in the univariate and multivariate logistic regression analyses (Table 2). The sensitivity, specificity, positive predictive value, and negative predictive value of the ROX index at each time point are presented in Table 3. The cutoff points at which specificity > 0.95 were 7.44, 6.22, and 7.33 at 0, 1, and 2 h, respectively, whereas the cutoff points at which sensitivity > 0.95 were 12.36, 15.46, and 17.52 at 0, 1, and 2 h, respectively.

Fig. 3.

Summary of ratio of oxygen saturation index values for the success and failure groups. Ratios of oxygen saturation index values (at 0, 1, and 2 h) are shown in A, B, C. One subject was excluded at 2 h because the subject was liberated from high-flow nasal cannula therapy within 2 h after extubation. ROX = ratio of oxygen saturation.

Table 2.

Univariate and Multivariate Logistic Regression Analysis of Ratio of Oxygen Saturation Index for Future Re-intubation at Each Time Point

Table 3.

Sensitivity, Specificity, Positive Predictive Value, and Negative Predictive Value of Ratio of Oxygen Saturation Index at Each Time

We plotted the ROC curves of the ROX index at 0, 1, and 2 h to predict future re-intubation. The areas under the ROC curve of the ROX index were 0.81 (0.62–0.92), 0.81 (0.61–0.92), and 0.80 (0.59–0.92), at 0, 1, and 2h, respectively. For comparison with the ROX index, we also plotted the ROC curves of f and S_aO2/F_IO2. The areas under the ROC curve of f were 0.77 (0.59–0.89), 0.76 (0.56–0.89), and 0.71 (0.50–0.86) at 0, 1, and 2 h, respectively. The areas under the ROC curve of S_aO2/F_IO2 were 0.68 (0.51–0.81), 0.69 (0.52–0.82), and 0.74 (0.56–0.86) at 0, 1, and 2 h, respectively, (Fig. 4 A-C). The overall trend that area under the ROC curve of the ROX index was greater than those of f and S_aO2/F_IO2 was observed, although there was no statistical significance at any time points (vs f: 0 h, P = .45; 1 h, P = .31; and 2 h, P = .053) (vs S_aO2/F_IO2: 0 h, P = .07; 1 h, P = .08; and 2 h, P = .41) (Fig. 4 D).

Fig. 4.

Receiver operating characteristic curves of the ratio of oxygen saturation index at 0, 1, and 2 h after extubation. Receiver operating characteristic (ROC) curves of the ratio of oxygen saturation (ROX) index at (A) 0 h, (B) 1 h, and (C) 2 h after extubation. For comparison, ROC curves of the breathing frequency and S_aO2/F_IO2 were also plotted at each time point. The areas under the ROC curves of the 3 indices at each time point are shown in (D). One subject was excluded from the analysis of the ROX index at 2 h because one subject was liberated from high-flow nasal cannula therapy within 2 h after extubation. ROX = ratio of oxygen saturation; f = breathing frequency; S/F = S_aO2/F_IO2.

We also evaluated the relationship between the ROX index at each time point and time until re-intubation using only the data of the reintubated subjects after extubation (Fig. 5). Overall, a trend of positive correlation between the time until re-intubation and the ROX index at each time point was observed, which means that the subjects with lower ROX index values were more likely to be re-intubated in a shorter time after extubation.

Fig. 5.

Scatter plot and linear regression line of the ratio of oxygen saturation index versus time from extubation until re-intubation at (A) 0 h, (B) 1 h, and (C) 2 h. ROX = ratio of oxygen saturation.

Discussion

In this retrospective study, the ROX index in the early phase of HFNC therapy after extubation was useful for predicting re-intubation with high accuracy in mechanically ventilated subjects with COVID-19. The specificity at the cutoff point of 7.44 of the ROX index immediately after extubation was 0.95, which means that extubated subjects with < 7.44 of the ROX index were likely to be re-intubated in the future. Whereas several studies have shown that the ROX index is useful for predicting the need for future-intubation in subjects with COVID-19,¹⁰ our study is the first to illustrate that the ROX index has potential as a predictor for future re-intubation after HFNC therapy for extubated subjects with COVID-19.

Interestingly, the cutoff point of 7.44 for the ROX index for predicting re-intubation in our study was greater than that of the ROX index for predicting primary intubation in a previous study as well as other studies focusing on subjects with COVID-19.^8,20 This may be explained by the difference in medical conditions between the subjects who underwent primary intubation and re-intubation. For example, subjects who undergo mechanical ventilation often develop neuromuscular disorders such as ICU-acquired weakness after extubation. Such subjects may have a high risk for re-intubation compared with those who have not undergone mechanical ventilation and have just received primary intubation, even if the value of the ROX index is relatively high. However, further research is needed to investigate why the cutoff point of the ROX index is different between primary intubation and re-intubation.

The areas under the ROC curve of the f and S_aO2/F_IO2 were not stable among the 3 time points after extubation. The areas under the ROC curve of f at 0 and 1 h were relatively greater at 2 h, whereas the area under the ROC curve of the S_aO2/F_IO2 at 2 h was greater at 0 and 1 h. This may be explained by a simple physiological response to acute respiratory failure. Physiologically, an increase in f is more sensitive than a decrease in S_pO2 for hypoxemia. An increase in f is the primary compensation as a response to hypoxia for increasing oxygen uptake and maintaining blood oxygen levels,^22,23 which can be expressed as a better predictive accuracy of f than S_aO2/F_IO2 in the earlier phase. Moreover, in the later phase, as the stage of acute respiratory failure progresses, the S_aO2/F_IO2 may become more useful than f for predicting further deterioration of subjects with acute respiratory failure. In this context, the ROX index can reflect both changes in the values of f in the early phase and S_aO2/F_IO2 in the later phase. Particularly, the predictive accuracy of the ROX index, which can be calculated by combining f and S_aO2/F_IO2, can be greater than that of either f or S_aO2/F_IO2. The areas under the ROC curve of the ROX index showed a trend of consistently higher predictive accuracy than that of f and S_aO2/F_IO2 after extubation, although there was no significance. The predictive accuracy for re-intubation might be improved by combining the ROX index with other clinical information, such as the respiratory status during SBT, although we did not analyze it in this study.

In this study, some differences in baseline characteristics between the 2 groups were observed. First, the length of ICU stay was longer in the failure group than in the success group. We assumed a possibility that the event of re-intubation extended the ICU stay of subjects in the failure group. Still, our retrospective study cannot clarify the relationship between cause and result, and another prospective study is needed to answer this clinical question. Second, unexpectedly, the prevalence rate of diabetes mellitus was higher in the success group than in the failure group. We do not know the mechanism for the difference, even though there is a possibility that an unknown confounding factor exists between the 2 groups. As our study was based on retrospective data with a small sample size, further prospective studies with larger sample sizes are needed.

During the pandemic, ICU medical staff had to manage and transfer a considerable number of critically ill patients due to limited ICU capacity; sometimes, when the ICU had reached capacity, they had to make decisions to transfer patients who were extubated just a few hours from the ICU to the general wards. The risk assessment for future re-intubation is an essential factor in decision making. However, the assessment should be done quickly because the delayed transfer of patients from the ICU can lead to delayed ICU admission of other patients. In this context, taken together with the simplicity of the calculation for ROX index, our results, which show the acceptable predictive accuracy of the ROX index in the early phase after extubation, suggest a possibility that the ROX index may be useful to identify patients at low risk who can be transferred to the general ward in the early phase after extubation under circumstances of crowded ICUs. However, further studies are required to confirm this.

Our study has several limitations. First, our study results were based on a small sample size, and the number of adjustment factors was limited to 2 (SOFA score and age). Thus, the influence of other confounding factors could not be excluded completely. In this study, we calculated the required sample size with 70% statistical power (β power), although the value of 80% is usually used for sample size calculation. Moreover, the final number of participants included in our study was 44, and not 47 as planned; thus, our sample size might be insufficient, although the validity of our study was generalized with data enrollment from multiple medical sites. Further prospective studies with larger sample sizes are required. Second, the time from extubation until re-intubation in most subjects in our study was within 48 h (Fig. 2), and whether the ROX index is useful for predicting re-intubation in the late phase (after 48 h) is unknown. Third, in this study, we did not analyze the change in ROX index between each time point, although the decision of intubation is often based on the course of hypoxia as well as the present respiratory status in actual clinical practice. Fourth, we could not exclude the selection bias because the final decision on the use of HFNC was made according to the clinical preference of each participating hospital. Finally, we did not analyze the relationship between the causes for re-intubation and the predictive accuracy of the ROX index because of our small sample size. Investigating the association between the causes of re-intubation and the ROX index values in a future study would be of great interest.

Conclusions

The ROX index in the early phase of HFNC therapy after extubation was useful for predicting re-intubation with high accuracy in mechanically ventilated subjects with COVID-19. We may need close observation for subjects with < 7.44 points of an ROX index just after extubation because of their high risk for re-intubation.