A simple nomogram for predicting failure of non-invasive respiratory strategies in adults with COVID-19: a retrospective multicentre study

Abstract

Background

Non-invasive respiratory strategies (NIRS) including high-flow nasal cannula (HFNC) and non-invasive ventilation (NIV) have become widely used in patients with COVID-19 who develop acute respiratory failure. However, use of these therapies, if ineffective, might delay initiation of invasive mechanical ventilation (IMV) in some patients. We aimed to determine early predictors of NIRS failure and develop a simple nomogram and online calculator that can identify patients at risk of NIRS failure.

Methods

We did a retrospective, multicentre observational study in 23 hospitals designated for patients with COVID-19 in China. Adult patients (≥18 years) with severe acute respiratory syndrome coronavirus 2 infection and acute respiratory failure receiving NIRS were enrolled. A training cohort of 652 patients (21 hospitals) was used to identify early predictors of NIRS failure, defined as subsequent need for IMV or death within 28 days after intensive care unit admission. A nomogram was developed by multivariable logistic regression and concordance statistics (C-statistics) computed. C-statistics were validated internally by cross-validation in the training cohort, and externally in a validation cohort of 107 patients (two hospitals).

Findings

Patients were enrolled between Jan 1 and Feb 29, 2020. NIV failed in 211 (74%) of 286 patients and HFNC in 204 (56%) of 366 patients in the training cohort. NIV failed in 48 (81%) of 59 patients and HFNC in 26 (54%) of 48 patients in the external validation cohort. Age, number of comorbidities, respiratory rate–oxygenation index (ratio of pulse oximetry oxygen saturation/fraction of inspired oxygen to respiratory rate), Glasgow coma scale score, and use of vasopressors on the first day of NIRS in the training cohort were independent risk factors for NIRS failure. Based on the training dataset, the nomogram had a C-statistic of 0·80 (95% CI 0·74–0·85) for predicting NIV failure, and a C-statistic of 0·85 (0·82–0·89) for predicting HFNC failure. C-statistic values were stable in both internal validation (NIV group mean 0·79 [SD 0·10], HFNC group mean 0·85 [0·07]) and external validation (NIV group value 0·88 [95% CI 0·72–0·96], HFNC group value 0·86 [0·72–0·93]).

Interpretation

We have developed a nomogram and online calculator that can be used to identify patients with COVID-19 who are at risk of NIRS failure. These patients might benefit from early triage and more intensive monitoring.

Funding

Ministry of Science and Technology of the People's Republic of China, Key Research and Development Plan of Jiangsu Province, Chinese Academy of Medical Sciences.

Introduction

Acute respiratory failure is an important cause of death in patients with COVID-19.¹ Non-invasive respiratory strategies (NIRS), which include high-flow nasal cannula (HFNC) and non-invasive ventilation (NIV), are now widely used in these patients.2, 3, 4 NIRS can decrease the need for invasive mechanical ventilation (IMV) in patients with acute respiratory failure,5, 6 but patients who do not respond to NIRS have poor outcomes.7, 8, 9 This lack of response might be particularly important in patients with COVID-19, since the availability of health-care services can be strained during pandemics. Therefore, early identification of patients with COVID-19 who are unlikely to respond to treatment with NIRS would be beneficial.

Previous studies in patients with other respiratory conditions have reported that low pH, low Glasgow coma scale score, and low oxygenation, and high heart rate, high respiratory rate, and high tidal volume are associated with NIV failure.10, 11, 12, 13, 14 Similarly, a number of clinical and oxygenation variables, including no clinical improvement in oxygenation or decrease in respiratory rate, have been associated with HFNC failure and subsequent need for IMV;8, 9 however, most of these variables were of limited value in identifying patients who would require subsequent intubation. In patients with acute respiratory failure and pneumonia, the respiratory rate–oxygenation (ROX) index, based on oxygen saturation measured by pulse oximetry (SpO₂), fraction of inspired oxygen (FiO₂), and respiratory rate, can help to identify the risk of NIRS failure and intubation.6, 15 However, which indicators are useful in identifying patients with COVID-19 with a high risk of NIRS failure is unknown. The objective of this study was to develop and validate a simple nomogram and online calculator for predicting the risk of NIRS failure in patients with COVID-19 presenting with acute respiratory failure.

Research in context.

Evidence before this study

We searched MEDLINE and medRxiv for papers published up to Nov 8, 2020, without language restrictions, using the search terms: ((COVID-19 or SARS-CoV-2 or novel coronavirus) and (acute hypoxemic respiratory failure or ARDS or ALI) and (non-invasive ventilation [NIV] or high-flow nasal cannula oxygen [HFNC]) and (nomogram or predictor or prediction)). Many studies have shown that various forms of non-invasive respiratory support (NIRS), including NIV or HFNC, are widely used in patients with COVID-19 with acute respiratory failure. In several studies of other respiratory conditions, NIRS decreased the need for invasive mechanical ventilation (IMV) in some patients with hypoxaemic respiratory failure, but patients who did not respond to these therapies had poor outcomes. As such, an important aim should be early identification of patients at high risk of NIRS failure (and thus in need of intubation or IMV) to prevent death. In hypoxaemic patients without COVID-19, an easily implemented scale that includes heart rate, acidosis, consciousness, oxygenation, and respiratory rate appeared to be effective in predicting NIV failure. The respiratory rate–oxygenation index (defined as the ratio of pulse oximetry oxygen saturation/fraction of inspired oxygen to respiratory rate) can help to identify the risk of HFNC failure and subsequent intubation in patients with acute respiratory failure and pneumonia. However, no indicators have been developed to identify the risk of NIRS failure in patients with COVID-19.

Added value of this study

Our results are consistent with and build on previous prediction models in non-COVID-19 patients. To our knowledge, this study is the first to develop a simple nomogram and online calculator that can identify patients with COVID-19 and acute respiratory failure with a high probability of NIRS failure.

Implications of all the available evidence

By identifying patients at risk of NIRS failure, clinicians could ascertain those who might benefit from early monitoring and early interventions. However, any benefits from such a strategy, which might include early intubation, require confirmation in clinical trials.

Methods

Study design and populations

This retrospective, multicentre observational study was done in 23 hospitals designated for patients with COVID-19 in Wuhan (Hubei province), Huangshi (Hubei province), Shenzhen (Guangdong province), and Jiangsu province (appendix p 2). The study was approved by the ethics committee of Wuhan Jinyintan Hospital (Wuhan, China; approval number KY-2020-10.02). Informed consent was waived due to the retrospective and observational nature of the study.

We enrolled patients with the following inclusion criteria: age 18 years or older; laboratory-confirmed infection with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2);¹⁶ and acute respiratory failure (defined as partial pressure of arterial oxygen (PaO₂)/FiO₂ ≤300 mm Hg) treated with HFNC or NIV and admitted to intensive care units (ICUs) as a consequence of acute respiratory failure. Patients with missing data for at least one NIRS failure predictor in the nomogram after univariable analysis to identify relevant predictors were excluded. Model development and internal validation involved a cohort of patients from 21 hospitals. A second cohort from two different hospitals was used for external validation of the model.

The two hospitals selected for external validation were general medical hospitals accepting patients with COVID-19. One of the hospitals (Wuhan Third Hospital, Wuhan, China) had 650 beds, 35 ICU beds, and 1684 hospitalised patients with COVID-19 during the study period. The second hospital (Wuhan Red Cross Hospital, Wuhan, China) had 400 beds, 15 ICU beds, and 1182 hospitalised patients with COVID-19. The 21 hospitals used for development of the model had a median of 1400 beds (IQR 493–2609) and 23 ICU beds (13–43). Overall, each of the 21 hospitals had a median of 554 patients (IQR 229–1021) admitted with COVID-19. Data were collected throughout the enrolment period for both cohorts.

Data collection and study outcomes

Days from symptom onset to hospital admission, and demographic variables and comorbidities at ICU admission were recorded. Overall severity of illness was assessed by acute physiology and chronic health evaluation (APACHE) II score, based on the worst values for all relevant variables recorded during the first 24 h of ICU admission.¹⁷ Vital signs, Glasgow coma scale score,¹⁸ sequential organ failure assessment (SOFA) score,¹⁹ clinical respiratory variables including respiratory rate, FiO₂, SpO₂, and blood gases were recorded on the first day of HFNC or NIV use (after HFNC or NIV was started). ROX index was defined as the ratio [(SpO₂/FiO₂)/respiratory rate], in units of breaths per min. We recorded the dates of initiation, any switching of ventilation therapy, and the duration of HFNC, NIV, and IMV. Use of vasopressors, steroids, and antivirals, and 28-day mortality after ICU admission were also recorded, as well as negative nucleic acid testing for SARS-CoV-2 within 28 days after ICU admission. NIV was delivered by face mask or nasal mask under bi-level positive pressure ventilation. Patients who received HFNC followed by NIV were included in the HFNC group, and patients who received NIV followed by HFNC were included in the NIV group.

The primary outcome was NIRS failure, defined as the subsequent use of IMV or death within 28 days after ICU admission. Death within 28 days after ICU admission was a secondary outcome. Patients who transitioned from HFNC to NIV or from NIV to HFNC, but did not receive IMV or die, were considered as being successfully treated with NIRS.

Statistical analysis

Descriptive statistics were reported as frequencies and proportions for categorical variables, and medians and IQRs or means and SDs for continuous variables. Differences between medians or means were assessed with the Mann-Whitney U test and between proportions with the χ² test. Absolute differences were calculated with the R software package pairwiseCI.²⁰

Univariable logistic analysis was used to identify clinically relevant variables associated with NIRS failure collected on day 1 of NIRS therapy in the whole training cohort, and separately in the HFNC and NIV groups in the cohort. Odds ratios (ORs) and 95% CIs were calculated per 1-unit increase for all continuous variables, and for presence of a factor for categorical variables. Variables showing a univariable relationship with overall NIRS failure (or HFNC or NIV failure; p<0·10) were entered into multivariable logistic regression models and backwards stepwise selection was done with improvement in goodness of fit assessed by a reduction in the Akaike information criterion. We excluded variables if the number of events was too small to calculate ORs. To prioritise generalisability and simplicity of the risk model, we planned to exclude any identified risk factors with the need for laboratory parameters (eg, SOFA and APACHE II scores). Given that most of the risk factors identified in the logistic regression models for the HFNC and NIV groups overlapped (appendix pp 12–13), a final model was created from the whole training cohort. A nomogram and an online calculator based on the selected final model was constructed from the overall data of the training cohort. The final multivariable model to predict probability of NIRS failure at time t was derived with the formula: t=S0(t)exp(β₁X₁ + β₂X₂…), where β are the regression coefficients and X are the reported values of the covariates showing association in multivariable regression.²¹ S0(t) is the baseline survival function, estimated from the data. Regression coefficients were used to construct the variable axes in the nomogram and S0 was used in the translation from total points to predicted probability.

To assess the ability of the nomogram model to discriminate patients who will respond to NIRS, a concordance statistic (C-statistic; equal to the area under the receiver operating curve) and 95% CIs were calculated and compared with that of each independently associated variable in the training cohort. The roc.test function in R was used to generate p values between C-statistics. C-statistics were also calculated and compared between the model and each independent variable in the HFNC and NIV groups in the training cohort. As a sensitivity analysis, we also evaluated the discriminatory ability of the HFNC and NIV models based on multivariable analysis of each treatment subtype. To analyse the agreement between nomogram predictions and actual observations in the training cohort, bootstraps of 1000 resamples (with replacement) were set and calibration curves were created. To assess the clinical usefulness of the predictive nomogram, decision curve analysis was done by quantifying the net benefits at different threshold probabilities of NIRS failure. Net benefit was defined as the proportion of true-positives minus the proportion of false-positives, standardised by the relative harm of a false-positive and false-negative result.²¹ The Akaike information criterion was calculated to assess the goodness of fit of the model. Cox-Snell R ² and Nagelkerke R ² were calculated to assess the prediction accuracy of the logistic regression model.

Cross-validation was applied to internally validate the stability of the model, by randomly splitting the patients in the training cohort into ten equal samples. Nine of these samples were used to construct logistic regression models and the model coefficients were applied to the remaining sample. This process was repeated 10 times and the mean C-statistic plus SD values corresponding to each iteration were calculated.

To assess external validity, the model was applied to our independent dataset from two hospitals. External validity of the model was assessed with the C-statistic, calibration, and decision curve analysis in patients treated with HFNC and NIV separately.

All statistical analyses were done with RStudio (version 1.2.5019), and a p value of less than 0·05 was considered to indicate statistical significance.

Role of the funding source

The funders of the study had no role in study design, data collection, data analysis, data interpretation, or writing of the report.

Results

From Jan 1 to Feb 29, 2020, 739 patients were eligible for inclusion in the training cohort; 87 of these patients were excluded because of missing data for at least one variable found to be associated with NIRS failure in univariable analysis (appendix p 3), with 652 patients in the final cohort. Baseline characteristics are shown in table 1 and variables with missing data are shown in the appendix (p 4). In the training cohort, 366 (56%) of 652 patients were initially supported by HFNC, and 286 (44%) by NIV (appendix pp 6–9). NIRS failed in 415 (64%) patients; 288 (44%) subsequently received IMV, and 127 (19%) died without intubation. NIV was unsuccessful in 211 (74%) of 286 patients, and HFNC in 204 (56%) of 366 patients. Death within 28 days after ICU admission was reported in 355 (54%) patients (178 [49%] in the HFNC group and 177 [62%] in the NIV group).

Table 1.

Demographic, respiratory, and treatment variables in patients with NIRS failure or success

		Training cohort					External validation cohort
		Overall (n=652)	NIRS success (n=237)	NIRS failure (n=415)	Absolute difference (95% CI)	p value*	Overall (n=107)	NIRS success (n=33)	NIRS failure (n=74)	Absolute difference (95% CI)	p value*	p value†
Admission variables
Age, years		65 (56–72)	58 (50–68)	68 (61–74)	10 (5 to 12)	<0·0001	66 (57–73)	57 (46–67)	66 (61–74)	9 (2 to 17)	<0·0001	0·81
Sex
	Female	229 (35%)	92 (39%)	137 (33%)	..	..	41 (38%)	14 (42%)	27 (36%)	..	..	..
	Male	423 (65%)	145 (61%)	278 (67%)	6 (−2 to 10)	0·14	66 (62%)	19 (58%)	47 (64%)	6 (−15 to 27)	0·62	0·52
Acute physiology and chronic health evaluation II score		11 (7–14)	7 (5–11)	12 (10–16)	5 (4 to 6)	<0·0001	12 (9–15)	9 (7–12)	14 (11–18)	5 (3 to 7)	<0·0001	0·053
Hypertension		268 (41%)	94 (40%)	174 (42%)	2 (−5 to 10)	0·20	49 (46%)	16 (48%)	33 (45%)	−3 (−25 to 17)	0·36	0·36
Diabetes		124 (19%)	40 (17%)	84 (20%)	3 (−3 to 10)	0·29	23 (21%)	7 (21%)	16 (22%)	1 (−17 to 18)	0·40	0·55
Coronary heart disease		84 (13%)	31 (13%)	53 (13%)	0 (−5 to 5)	0·10	16 (15%)	2 (6%)	14 (19%)	12 (2 to 27)	0·12	0·039
Chronic lung disease		31 (5%)	10 (4%)	21 (5%)	1 (−3 to 4)	0·64	6 (6%)	0	6 (8%)	8 (2 to 14)	0·14	0·71
Other comorbidities		54 (8%)	23 (10%)	31 (7%)	−3 (−7 to 2)	0·33	9 (8%)	2 (6%)	5 (7%)	1 (−2 to 4)	0·71	0·96
Number of comorbidities		1 (0–2)	1 (0–1)	1 (0–2)	0 (0 to 1)	<0·0001	1 (0–1)	1 (0–1)	1 (0–2)	0 (0 to 1)	0·12	0·51
Duration of symptom onset to hospital admission, days		9 (6–13)	7 (4–11)	10 (7–14)	3 (1 to 4)	<0·0001	9 (6–11)	9 (5–11)	10 (8–12)	1 (−1 to 4)	0·37	0·86
Data collected on the first day of NIRS
Sequential organ failure assessment score		3 (2–5)	2 (1–3)	4 (3–6)	2 (1 to 3)	<0·0001	5 (4–7)	3 (3–4)	6 (4–6)	3 (1 to 4)	<0·0001	<0·0001
Glasgow coma scale score		15 (15–15)	15 (15–15)	15 (15–15)	−1 (−1 to 1)	<0·0001	15 (15–15)	15 (15–15)	15 (14–15)	−1 (−2 to 0)	<0·0001	0·0020
Heart rate, beats per min		92 (81–104)	90 (81–100)	93 (81–107)	3 (0 to 8)	0·025	95 (84–113)	90 (83–105)	99 (84–115)	9 (0 to 16)	0·037	0·029
Respiratory rate, breaths per min		24 (20–29)	21 (20–25)	25 (21–30)	4 (3 to 5)	<0·0001	30 (25–35)	27 (24–33)	30 (25–35)	3 (−1 to 4)	<0·0001	<0·0001
Mean arterial pressure, mm Hg		95 (87–103)	95 (89–103)	96 (86–103)	1 (−2 to 3)	0·37	93 (85–104)	90 (83–96)	97 (86–109)	6 (0 to 12)	0·42	0·46
pH		7·44 (7·40–7·48)	7·44 (7·42–7·47)	7·44 (7·37–7·48)	−0·00 (−0·02 to 0·01)	0·10	7·47 (7·40–7·50)	7·48 (7·43–7·50)	7·47 (7·38–7·50)	−0·03 (−0·07 to 0·01)	0·68	0·069
PaO2/FiO2 ratio, mm Hg		116 (67–215)	182 (121–322)	90 (53–145)	−92 (−132 to −74)	<0·0001	92 (74–149)	149 (105–193)	79 (65–99)	−70 (−119 to −40)	<0·0001	0·10
PaCO2,		36 (32–42)	36 (32–39)	35 (31–43)	−1 (−3 to 1)	0·98	32 (27–39)	37 (32–41)	30 (26–36)	−6 (−12 to 0)	<0·0001	<0·0001
Respiratory rate–oxygenation index		6·1 (4·0–10·8)	10·4 (6·8–16·8)	4·7 (3·5–7·5)	−5·7 (−6·7 to 4·5)	<0·0001	4·6 (3·4–6·0)	6·3 (5·0–8·7)	3·8 (3·2–5·0)	−2·8 (−4·3 to −1·3)	<0·0001	<0·0001
Treatment and outcome
Only NIV		237 (36%)	46 (19%)	191 (46%)	27 (20 to 33)	<0·0001	59 (55%)	11 (33%)	48 (65%)	32 (2 to 42)	0·0020	<0·0001
	Duration of NIV, days	5 (2–8)	6 (4–10)	4 (2–8)	−3 (−4 to −1)	0·040	4 (3–7)	5 (3–7)	4 (3–7)	−1 (−4 to 1)	0·93	0·83
Only HFNC		246 (38%)	126 (53%)	120 (29%)	−24 (−33 to −20)	<0·0001	29 (27%)	14 (42%)	15 (20%)	−22 (−52 to −11)	0·017	0·034
	Duration of HFNC, days	6 (4–10)	9 (5–11)	4 (2–7)	−3 (−4 to −1)	<0·0001	7 (3–11)	7 (7–11)	4 (2–10)	−2 (−5 to 0)	0·17	0·65
Both HFNC and NIV		169 (26%)	65 (27%)	104 (25%)	−2 (−10 to 3)	0·51	19 (18%)	8 (24%)	11 (15%)	−9 (−27 to 8)	0·24	0·071
	Duration of HFNC and NIV, days	10 (6–16)	11 (8–19)	9 (6–15)	−2 (−2 to −5)	0·031	7 (5–14)	12 (8–14)	6 (4–9)	−5 (−10 to −1)	0·045	0·12
IMV		288 (44%)	0	288 (69%)	69 (65 to 74)	<0·0001	31 (29%)	0	31 (42%)	42 (27 to 53)	<0·0001	0·0030
	Duration of IMV, days	7 (4–13)	..	7 (4–13)	..	..	4 (2–7)	..	4 (2–7)	..	..	0·0040
Use of steroids		305 (47%)	76 (32%)	229 (55%)	23 (15 to 30)	0·58	0	..	..	..	..	..
Use of antivirals		502 (77%)	208 (88%)	294 (71%)	−18 (−23 to −11)	<0·0001	0	..	..	..	..	..
Use of vasopressors		46 (7%)	3 (1%)	43 (10%)	9 (6 to 12)	<0·0001	5 (5%)	0	5 (7%)	7 (1 to 13)	0·0050	0·44
Death before IMV		127 (19%)	0	127 (31%)	31 (26 to 35)	<0·0001	43 (40%)	0	43 (58%)	58 (43 to 69)	<0·0001	0·0010
Negative nucleic acid testing for severe acute respiratory syndrome coronavirus 2 within 28 days after ICU admission		214 (33%)	122 (51%)	92 (22%)	−29 (−37 to −22)	<0·0001	14 (13%)	8 (24%)	6 (8%)	−16 (−33 to −2)	<0·0001	0·0040
Death within 28 days after ICU admission		355 (54%)	0	355 (86%)	86 (82 to 89)	<0·0001	67 (63%)	0	67 (91%)	91 (84 to 97)	<0·0001	0·12
Length of stay in the ICU by 28th day after admission, days		14 (6–28)	27 (14–28)	9 (5–19)	−18 (−20 to −14)	<0·0001	8 (4–18)	19 (9–28)	7 (4–10)	−8 (−12 to −4)	<0·0001	<0·0001

Characteristics are summarised as median (IQR) or frequency (%). NIRS=non-invasive respiratory support. PaO₂=partial pressure of arterial oxygen. FiO₂=fraction of inspired oxygen. PaCO₂=partial pressure of arterial carbon dioxide. NIV=non-invasive ventilation. HFNC=high-flow nasal cannula. IMV=invasive mechanical ventilation. ICU=intensive care unit.

^*p value for difference between patients with NIRS failure versus NIRS success.

^†p value for training cohort versus validation cohort for overall characteristics.

In the training cohort, age, APACHE II score, days from symptom onset to hospital admission, and mortality within 28 days after ICU admission were significantly higher in patients with NIRS failure versus those who showed clinical improvement (table 1). Additionally, patients in whom NIRS failed had significantly higher SOFA score, respiratory rate, heart rate, and likelihood of vasopressor use, and lower Glasgow coma scale score, ROX index, and PaO₂/FiO₂ ratio, on day 1 of NIRS.

During the enrolment period, 123 patients were eligible for inclusion in the external validation cohort; 16 patients were excluded because of missing data (appendix p 3), with 107 patients in the final cohort. 48 (45%) patients were enrolled in the HFNC group and 59 (55%) in the NIV group (appendix pp 6–9). NIRS failed in 74 (69%) patients; 31 (29%) subsequently received IMV, and 43 (40%) died before intubation. HFNC failed in 26 (54%) of 48 patients and NIV in 48 (81%) of 59 patients. Death within 28 days after ICU admission was reported in 67 (63%) patients (20 [42%] in the HFNC group and 47 [80%] in the NIV group). Patients in whom NIRS failed had significantly higher SOFA score, respiratory rate, heart rate, and likelihood of vasopressor use, and lower Glasgow coma scale score, ROX index, and PaO₂/FiO₂ on day 1 of NIRS.

In both the training and validation cohorts, patients who received NIV were more severely ill (higher APACHE II and SOFA scores, lower ROX index, and higher mortality within 28 days after ICU admission) than those who received HFNC (appendix pp 10–11).

Univariable and multivariable logistic regression showed that independent risk factors for NIRS failure were increased age, increased number of comorbidities, low ROX index, low Glasgow coma scale score, and use of vasopressors on the first day of NIRS (table 2 ). The multivariable model had an Akaike information criterion of 618·3, a Cox-Snell R ² of 0·317, and a Nagelkerke R ² of 0·434.

Table 2.

Factors associated with non-invasive respiratory support failure in univariable and multivariable analyses

	Univariable models		Multivariable model
	OR (95% CI)	p value	OR (95% CI)	p value
Respiratory rate–oxygenation index*	0·81 (0·78–0·85)	<0·0001	0·81 (0·78–0·85)	<0·0001
Age, years*	1·05 (1·04–1·07)	<0·0001	1·04 (1·03–1·06)	<0·0001
Glasgow coma scale score*	0·75 (0·64–0·87)	<0·0001	0·76 (0·64–0·89)	<0·0001
Heart rate, beats per min*	1·01 (1·00–1·02)	0·0040	1·01 (1·00–1·02)	0·131
Respiratory rate, breaths per min*	1·10 (1·07–1·13)	<0·0001	0·99 (0·95–1·02)	0·424
Vasopressor use (yes or no)	9·02 (2·77–29·39)	<0·0001	7·84 (2·22–27·65)	<0·0001
Symptom onset to hospital admission, days*	1·07 (1·04–1·10)	<0·0001	1·02 (0·99–1·05)	0·176
Number of comorbidities*	1·48 (1·27–1·73)	<0·0001	1·21 (1·01–1·45)	0·030

OR=odds ratio.

^*Per 1-unit increase.

Based on the final multivariable model, a nomogram and an online calculator were generated by assigning a weighted score to each of the factors associated with NIRS failure (figure 1 ). The total score was calculated as:

Figure 1.

Characteristics in the nomogram to predict probability of NIRS failure in patients with severe acute respiratory syndrome coronavirus 2 pneumonia

Patient prognostic values are located on the axis of each variable; a line is then drawn upwards at a 90° angle to determine the number of points for that particular variable. The sum of these numbers is located on the total score axis, and a line is drawn at a 90° angle downward to the NIRS failure risk axis to determine the likelihood of failure of non-invasive respiratory therapies. Alternatively, failure risk can be ascertained from the online calculator. Vasopressor use was represented on the axis at an arbitrary value of 1 (no use=0). NIRS=non-invasive respiratory support.

$$([\text{age}\times 0.0817]-1.633])+(7.819-[0.521\times \text{Glasgow coma scale}])+(10-[0.385\times \text{ROX}])+3.844(\text{if use of vasopressors})+(0.359\times \text{number of comorbidities})$$

Probability of NIRS failure was calculated as:

$$(0.02354\times {[\text{total score}]}^2)-(0.00079\times {[\text{total score}]}^3)-(0.11954\times [\text{total score}])+0.13527$$

The appendix (p 16) describes use of the nomogram to determine total score and probability of NIRS failure in a patient from the development cohort.

We assessed the ability of our final model to discriminate patients unlikely to respond to NIRS using C-statistics. The nomogram for predicting NIRS failure in the training cohort had a C-statistic of 0·84 (95% CI 0·81–0·87), which was significantly higher than the C-statistic obtained for each variable in the model (table 3 ). In the NIV and HFNC groups of the training cohort, the nomogram had a C-statistic of 0·80 (0·74–0·85) for predicting NIV failure, and 0·85 (95% CI 0·82–0·89) for predicting HFNC failure. The C-statistic remained stable in both internal validation (NIV group mean 0·79 [SD 0·10]; HFNC group mean 0·85 [0·07]) and external validation (NIV group value 0·88 [95% CI 0·72–0·96], HFNC group value 0·86 [0·72–0·93]; table 3, appendix p 18). Furthermore, the discriminatory ability of individual HFNC and NIV models was similar to the final models of whole data in each cohort (appendix pp 15–16).

Table 3.

C-statistics for the nomogram and model variables in the training and external validation cohorts

	Training cohort						External validation cohort
	Overall NIRS (n=652)		NIV (n=286)		HFNC (n=366)		Overall NIRS (n=107)		NIV (n=59)		HFNC (n=48)
	C-statistic (95% CI)	p value	C-statistic (95% CI)	p value	C-statistic (95% CI)	p value	C-statistic (95% CI)	p value	C-statistic (95% CI)	p value	C-statistic (95% CI)	p value
Nomogram	0·84 (0·81–0·87)	..	0·80 (0·74–0·85)	..	0·85 (0·82–0·89)	..	0·88 (0·81–0·95)	..	0·88 (0·72–0·96)	..	0·86 (0·72–0·93)	..
Respiratory rate–oxygenation index	0·80 (0·76–0·82)	0·0037	0·76 (0·70–0·82)	0·12	0·82 (0·77–0·86)	0·024	0·77 (0·67–0·88)	0·012	0·71 (0·52–0·89)	0·0090	0·78 (0·64–0·92)	0·16
Age	0·68 (0·64–0·72)	<0·0001	0·65 (0·57–0·72)	<0·0001	0·70 (0·65–0·76)	<0·0001	0·71 (0·60–0·83)	0·0033	0·78 (0·61–0·95)	0·039	0·66 (0·49–0·82)	0·014
Glasgow coma scale score	0·44 (0·42–0·46)	<0·0001	0·43 (0·39–0·46)	<0·0001	0·45 (0·42–0·48)	<0·0001	0·42 (0·34–0·49)	<0·0001	0·37 (0·27–0·46)	<0·0001	0·49 (0·39–0·59)	<0·0001
Use of vasopressor	0·54 (0·53–0·56)	<0·0001	0·55 (0·52–0·58)	<0·0001	0·54 (0·52–0·56)	<0·0001	0·53 (0·51–0·56)	<0·0001	0·52 (0·49–0·55)	<0·0001	0·56 (0·50–0·62)	<0·0001
Number of comorbidities	0·61 (0·57–0·65)	<0·0001	0·57 (0·50–0·64)	<0·0001	0·62 (0·57–0·68)	<0·0001	0·55 (0·44–0·66)	<0·0001	0·45 (0·28–0·63)	<0·0001	0·63 (0·48–0·78)	0·0030

C-statistic=concordance statistic. NIRS=non-invasive respiratory support. NIV=non-invasive ventilation. HFNC=high-flow nasal cannula.

The calibration plots (apparent and bias-corrected) overlapped with the ideal line in the training and validation cohorts, showing adequate agreement of the predictive nomogram with actual observations (figure 2 ). The benefit derived from applying the nomogram in clinical practice, according to the decision curve method, is depicted in the appendix (p 18). Threshold probabilities for the standardised net benefit associated with application of the nomogram in detecting NIRS failure ranged from 0·00 to 0·94 in the training cohort (NIV group, 0·00 to 0·93; HFNC group, 0·00 to 0·92), and 0·00 to 0·89 in the validation cohort (NIV group, 0·00 to 0·93; HFNC group, 0·00 to 0·99).

Figure 2.

Calibration curves for the nomogram

The calibration method with bootstrapping was used to illustrate the association between actual NIRS failure and predicted NIRS failure. Calibration plots show the apparent (actual), bias-corrected (adjusted), and ideal (100% agreement) curves with bootstrapping samples. Bootstrapping involved 1000 repetitions. Nomogram-predicted probability of NIRS failure is plotted on the x-axis; the observed probability of NIRS failure is plotted on the y-axis. HFNC=high-flow nasal cannula. NIV=non-invasive ventilation. NIRS=non-invasive respiratory support.

Discussion

In this study, we developed and validated a nomogram and online calculator for the early prediction of NIRS failure in patients with COVID-19. The nomogram, based on age, number of comorbidities, ROX index, Glasgow coma scale score, and use of vasopressors on day 1 of NIRS, had a discriminatory ability (C-statistic) of 0·84 (95% CI 0·81–0·87) in predicting NIRS failure. Patients in whom NIRS fails have a high risk of death. Thus, early prediction of NIRS failure could help clinicians to appropriately allocate critical care resources, and identify high-risk patients for entry into clinical trials.

In respiratory conditions other than COVID-19, several studies have shown that intubation after initial use of NIV11, 22 or HFNC²³ (or NIRS generally²⁴) is associated with worse outcomes, such as increased mortality, in patients with acute respiratory failure. Predicting the outcome of NIRS is particularly important in patients with COVID-19, given the limited resources available during the pandemic. Although previous studies have suggested that the criteria for initiating HFNC or NIV and their outcomes might differ,²⁵ we found that most independent risk factors for NIRS failure overlapped in the HFNC and NIV groups in the training cohort (appendix pp 12–13). Indeed, 26% of patients in the training cohort received both HFNC and NIV at various times in the course of their disease. Furthermore, the discriminatory ability of the HFNC and NIV models was similar to the final models in each cohort. Therefore, we analysed HFNC and NIV together as NIRS to make the nomogram easier to use. We verified a previous predictive scale for NIV failure in non-COVID-19 patients (based on heart rate, acidosis, consciousness, oxygenation, and respiratory rate)²⁶ with our data in the training cohort and found a C-statistic of 0·83 (95% CI 0·81–0·87), which was similar to our final model. However, PaO₂/FiO₂ was available only in a portion of our patients. The need for these blood gases and manual score calculation might restrict application of the previously proposed scale during the COVID-19 pandemic.

The present study showed that NIRS failed in 64% of cases in the training cohort and 69% of cases in the external validation cohort; these values are higher than the failure rate previously reported for HFNC (28–38%)5, 6 and NIV (39–50%).5, 14, 26, 27 The high failure rate in the current study could be due in part to the severity of hypoxaemia (median PaO₂/FiO₂ 116 mm Hg [IQR 67–215) on day 1 of NIRS in the training cohort) as compared with that on day 1 of previous studies (with HFNC, 149 mm Hg [SD 72]⁵ to 160 mm Hg [64];²⁷ and with NIV, 157 mm Hg [89]⁵). Blood gases were only available in 64% of patients receiving NIRS in the training cohort and 69% of patients in the validation cohort. Thus, low PaO₂/FiO₂ values in the present study might be partially due to the fact that only patients with severe COVID-19 were likely to have these measurements taken due to limited resources. This could have led to underestimation of average PaO₂/FiO₂. Another explanation could be insufficient supply of invasive ventilators during the pandemic, leading to the use of non-invasive approaches in patients with low PaO₂/FiO₂ who might otherwise have been intubated and ventilated.

28-day mortality in the training and external validation cohorts was 54% and 63%, within the range of 16–78% previously described in patients with COVID-19 admitted to an ICU.2, 4, 28 However, patients in whom NIRS failed had substantially higher mortality than that reported previously in other respiratory conditions after NIV or HFNC failure.6, 15, 26 In both the training and validation cohorts, mortality rate was significantly higher in the NIV group than in the HFNC group. Given the criteria for initiating HFNC and NIV, it is not unexpected that patients in the NIV group had more severe acute respiratory failure and higher mortality than those in the HFNC group.29, 30

To make the prediction model simple and rapid to use in the clinical setting, we only focused on risk factors that did not require laboratory parameters. In patients with pneumonia, a ROX value of 4·88 or higher is a determinant of HFNC success, and a value of less than 3·85 a determinant of HFNC failure, after 12 h of therapy, with uncertainty of success for values between these thresholds.6, 15 In the present study, our nomogram included ROX as a continuous variable to predict the risk of NIRS failure in patients with COVID-19. Age has been shown to be an independent risk factor for death in patients with COVID-19 in a previous ICU cohort (n=344),³ but it has not been commonly reported in association with NIRS failure. In our study, age was an independent risk factor for NIRS failure. The predictive accuracy of age alone in our study was represented by a C-statistic of 0·68 (95% CI 0·64–0·72) in the training cohort and 0·71 (0·60–0·83) in the validation cohort, showing that age itself had moderate discriminative power. In agreement with previous research on predicting NIV failure in hypoxaemic patients,²⁶ we also found that Glasgow coma scale was an independent risk factor for NIRS failure, but with low discriminative power.

Comorbidities have an important effect on outcomes in COVID-19. Consistent with a previous study of 323 hospitalised patients with COVID-19,³¹ we found that the number of comorbidities was independently associated with NIRS failure. Although only 7% of patients in our training cohort and 5% in our validation cohort received vasoactive drugs on the first day of NIRS, our results are consistent with previous findings, that use of vasoactive drugs was associated with NIRS failure.¹⁴ Median duration from symptom onset to hospital admission in our cohort of ICU patients was 9 days, which was higher than previously reported in 1590 hospitalised patients (mean 4·4 to 4·7 days).³² However, time between symptom onset and admission was not independently associated with NIRS failure in multivariable analysis. With respect to clinical utility, decision curve analysis also indicated that the nomogram was feasible in clinical practice, reflected by the positive net benefit associated with application of the nomogram over a broad range of threshold probabilities of NIRS failure (0·00 to 0·94 in the training cohort, and 0·00 to 0·89 in the validation cohort).

Our study has several important limitations. First, this was a retrospective study completed during a pandemic, and the critical nature of the pandemic did not allow us to obtain more detailed clinical information, such as tidal volume during NIV; higher tidal values could be associated with increased risk of self-inflicted lung injury and worsen ventilator-induced lung injury.³³ Second, we only enrolled patients admitted in January and February, 2020, when medical resources were overwhelmed by the surge of COVID-19 cases. The median duration from symptom onset to hospital admission or ICU admission and the practice of respiratory support might be different after a pandemic period, or in other countries. Third, agitation and intolerance to masks might have had prognostic implications against the tested parameters (eg, ROX index) in determining failure of NIRS. However, these data were unavailable in our retrospective study. Finally, although our nomogram and online calculator can identify patients at risk of NIRS failure, our study cannot determine whether an alternative management strategy for these patients would improve outcomes.

In conclusion, our nomogram and online calculator are simple to use and able to predict the risk of failure in patients with COVID-19 treated with HFNC and NIV. The nomogram and online calculator can be used to identify patients with a high probability of NIRS failure. These patients might benefit from early triage and more intensive monitoring. The benefits of such a strategy, which might include early intubation, would require confirmation in randomised control trials.

Data sharing

Individual participant data that underlie the results reported in this Article will be shared after de-identification (text, tables, figures, and appendices) to researchers conditional upon receipt of an approved study proposal along with evidence of approval of the proposal by an accredited ethics committee. Proposals should be directed toliulingdoctor@126.com. To gain access, data requestors will need to sign a data access agreement.