Performance and comparison of assessment models to predict 30-day mortality in patients with hospital-acquired pneumonia

Abstract

Background

Hospital-acquired pneumonia (HAP) is the most common hospital-acquired infection in China with substantial morbidity and mortality. But no specific risk assessment model has been well validated in patients with HAP. The aim of this study was to investigate the published risk assessment models that could potentially be used to predict 30-day mortality in HAP patients in non-surgical departments.

Methods

This study was a single-center, retrospective study. In total, 223 patients diagnosed with HAP from 2012 to 2017 were included in this study. Clinical and laboratory data during the initial 24 hours after HAP diagnosis were collected to calculate the pneumonia severity index (PSI); consciousness, urea nitrogen, respiratory rate, blood pressure, and age ≥65 years (CURB-65); Acute Physiology and Chronic Health Evaluation II (APACHE II); Sequential Organ Failure Assessment (SOFA); and Quick Sequential Organ Failure Assessment (qSOFA) scores. The discriminatory power was tested by constructing receiver operating characteristic (ROC) curves, and the areas under the curve (AUCs) were calculated.

Results

The all-cause 30-day mortality rate was 18.4% (41/223). The PSI, CURB-65, SOFA, APACHE II, and qSOFA scores were significantly higher in non-survivors than in survivors (all P < 0.001). The discriminatory abilities of the APACHE II and SOFA scores were better than those of the CURB-65 and qSOFA scores (ROC AUC: APACHE II vs. CURB-65, 0.863 vs. 0.744, Z = 3.055, P = 0.002; APACHE II vs. qSOFA, 0.863 vs. 0.767, Z = 3.017, P = 0.003; SOFA vs. CURB-65, 0.856 vs. 0.744, Z = 2.589, P = 0.010; SOFA vs. qSOFA, 0.856 vs. 0.767, Z = 2.170, P = 0.030). The cut-off values we defined for the SOFA, APACHE II, and qSOFA scores were 4, 14, and 1.

Conclusions

These results suggest that the APACHE II and SOFA scores determined during the initial 24 h after HAP diagnosis may be useful for the prediction of 30-day mortality in HAP patients in non-surgical departments. The qSOFA score may be a simple tool that can be used to quickly identify severe infections.

Introduction

Hospital-acquired pneumonia (HAP)/ventilator-associated pneumonia (VAP), is the most common hospital-acquired infection in China and is associated with substantial morbidity and mortality.^[1] Despite the remarkable advances in new antibiotic therapies, the mortality of HAP remains high. To select the appropriate treatment, including empiric antibiotic regimens, and monitoring strategies for these patients, the clinical severity and local antibiotic resistance data should be carefully evaluated. The guidelines for HAP/VAP produced by the American Thoracic Society/Infectious Diseases Society of America in 2016 recommended different initial empiric antibiotic therapies for patients with different mortality risks, and the risk factors included the use of ventilation due to HAP and septic shock.^[2] This method of risk classification is imprecise. Patients with atypical disease presentation in the early stage of HAP/VAP may be misdiagnosed. Moreover, there are patients who use mechanical ventilation for long periods or intermittently due to primary diseases in China, and some of them are not critically ill. These conditions may lead to inappropriate clinical decisions. Therefore, the guidelines for HAP/VAP produced by the Chinese Thoracic Society in 2018 specifically mentioned these conditions; it is necessary to choose an appropriate assessment system for these patients.^[1] An assessment model to stratify the risk of mortality could help with the evaluation of clinical severity, the prediction of the clinical outcome, and the reduction of heterogeneity in clinical trials. There are several available assessment models for the assessment of organ dysfunction and the prediction of mortality in intensive care unit (ICU) patients and community acquired pneumonia (CAP) patients,^[3–8] but no specific risk assessment model has been well validated in patients with HAP. The object of this study was to investigate the published risk assessment models that could potentially be used in HAP patients and to provide data to improve their prognosis.

Methods

Ethics

This study was conducted in accordance with the Declaration of Helsinki and was approved by Peking University Third Hospital Medical Science Research Ethics Committee (No. M2019486). The need for informed consent was waived.

Study setting and design

This retrospective study was conducted in Peking University Third Hospital with 1891 beds. Data from 23 patients from non-surgical departments who developed a first HAP episode between March 2012 and March 2017 were collected. Patients who were younger than 18 years of age, who were perinatal, who had coexisting hospital acquired extrapulmonary infection, including urinary infection, bloodstream infection, central nervous system infection, etc, and who lacked critical data were excluded from the study [Supplementary Figure 1].

Definition of HAP

The criterion for the diagnosis of HAP was pneumonia that developed after the patient was hospitalized for more than 48 h. Pneumonia was defined as the presence of a new or progressive infiltrate on radiography plus at least two of three clinical features (fever greater than 38°C, leukocytosis or leukopenia, and purulent secretions).^[1]

Severity assessment models

The pneumonia severity index (PSI) was proposed by Fine et al. The PSI is the first assessment model for CAP. It includes three demographic variables, five comorbidities, five physical examination variables, and seven laboratory tests.^[4]

The consciousness, urea nitrogen, respiratory rate, and blood pressure, age ≥65 years (CURB-65) score was proposed by Lim et al. C, U, R, and B stand for consciousness, urea nitrogen, respiratory rate, and blood pressure, while 65 represents age.^[5]

The Acute Physiology and Chronic Health Evaluation II (APACHE II) score is the most widely used critical illness evaluation system in clinical ICUs. It is comprised of three sections, namely acute physiology, age, and chronic health evaluation. The acute physiology score consists of 12 physiological variables.^[6]

The Sequential Organ Failure Assessment (SOFA) score was proposed by the European Society of Intensive Care Medicine. The SOFA score is mainly used to describe the occurrence and development of multiorgan dysfunction syndrome. Six systems, namely, the respiratory system, nervous system, hepatic system, cardiovascular system, coagulation system, and renal system, are included in the model.^[7]

The Quick Sequential Organ Failure Assessment (qSOFA) score includes altered mentation, systolic blood pressure (≤100 mmHg), and respiratory rate (≥22/min).^[8]

Data collection

All the data needed to calculate the scores during the initial 24 h after HAP diagnosis were collected. Demographic data, admission diagnoses of the patients, comorbidities, and 30-day mortality were also collected. The scores were determined by the worst values.

Statistical analysis

Non-normally distributed continuous data are presented as median (Q₁, Q₃) and categorical data are shown as numbers and percentages, and differences between survivors and non-survivors were tested with Mann-Whitney U test or Chi-square test. The mortality rates among risk strata were compared with Chi-square tests for trend. The discriminatory power of the five scores with regard to classifying survivors and non-survivors was tested by constructing receiver operating characteristic (ROC) curves, and the areas under the curve (AUCs) were calculated. The estimated AUC values were compared by Delong's test. A two-tailed P value < 0.05 was considered statistically significant. The analyses were performed using IBM SPSS Statistics version 26.0 (IBM Corp., Armonk, NY, USA) and MedCalc Version 19.4.1 (MedCalc Software Ltd., Ostend, West Flanders Province, Belgium).

Results

Patient characteristics

During the study period, 254 patients were diagnosed with HAP. Eleven patients were not included because of an incorrect diagnosis was determined upon review of the clinical records. Twenty patients missing critical data were excluded. In total, 223 patients diagnosed with HAP were included. The all-cause 30-day mortality was 18.4% (41/223). The baseline characteristics of the patients grouped by their survival status are provided in Tables 1 and 2. Age (73.5 [62.0, 82.0] years vs. 79.0 [68.0, 88.0] years, Z = −2.097, P = 0.036), neoplastic disease (17.0% vs. 39.0%, χ² = 9.729, P = 0.002), cardiovascular disease (21.4% vs. 56.1%, χ² = 20.037, P < 0.001), and renal disease (20.9% vs. 39.0%, χ² = 6.004, P = 0.014) were statistically different between 30-day survivors and 30-day non-survivors. All of the scores were significantly lower in survivors than in non-survivors (PSI: 99 [82, 118] vs. 141 [121, 165], Z = −6.167, P < 0.001; CURB-65: 2 [1, 2] vs. 3 [2, 3], Z = −5.035, P < 0.001; SOFA: 2 [0, 3] vs. 6 [3, 9], Z = −7.192, P < 0.001; APACHE II: 11 [8, 14] vs. 21 [15, 25], Z = −7.268, P < 0.001; qSOFA: 0 [0, 1] vs. 1 [1, 2], Z = −5.885, P < 0.001).

Table 1.

Characteristics of patients with HAP.

Characteristics	30-day survivors (n = 182)	30-day non-survivors (n = 41)	Statistics	P
Demographics
Male, n (%)	128 (70.3)	27 (65.9)	0.316∗	0.574
Age (years) median, (Q1, Q3)	73.5 (62.0, 82.0)	79.0 (68.0, 88.0)	−2.097†	0.036
Comorbidity, n (%)
Neoplastic disease	31 (17.0)	16 (39.0)	9.729∗	0.002
Cardiovascular disease	39 (21.4)	23 (56.1)	20.037∗	<0.001
Liver disease	24 (13.2)	7 (17.1)	0.422∗	0.516
Renal disease	38 (20.9)	16 (39.0)	6.004∗	0.014
Cerebrovascular disease	64 (35.2)	13 (31.7)	0.177∗	0.674
Immunocompromised status	4 (2.2)	2 (4.9)	0.918∗	0.338
Pulmonary disease	84 (46.2)	25 (61.0)	2.942∗	0.086

^∗χ² value.

^†Z value.

HAP: Hospital-acquired pneumonia.

Table 2.

Scores of HAP survivors and non-survivors with different severity assessment models.

Assessment models	30-day survivors (n = 182)	30-day non-survivors (n = 41)	Z	P
PSI	99 (82, 118)	141 (121, 165)	−6.167	<0.001
CURB-65	2 (1, 2)	3 (2, 3)	−5.035	<0.001
SOFA	2 (0, 3)	6 (3, 9)	−7.192	<0.001
APACHE II	11 (8, 14)	21 (15, 25)	−7.268	<0.001
qSOFA	0 (0, 1)	1 (1, 2)	−5.885	<0.001

Data were presented as median (Q₁, Q₃). HAP: Hospital-acquired pneumonia; PSI: Pneumonia severity index; CURB-65: Consciousness, urea nitrogen, respiratory rate, blood pressure, and age ≥65 years; SOFA: Sequential organ failure assessment; APACHE II: Acute Physiology and Chronic Health Evaluation II; qSOFA: Quick sequential organ failure assessment.

Mortality by severity class

The numbers of patients and the 30-day mortality across the severity classes of each assessment model are shown in Table 3. The mortality of HAP patients increased steadily from low to high risk in each assessment model (PSI: from 5.7% to 50.9%; CURB-65: from 6.6% to 38.5%; SOFA: from 1.3% to 27.6%; APACHE II: from 1.2% to 65.6%; qSOFA: from 13.5% to 48.4%; all P < 0.001).

Table 3.

Numbers and mortality of HAP patients with different severity classes using different severity assessment models.

Assessment models	Patients (n = 223)	30-day survivors (n = 182)	30-day non-survivors (n = 41)	χ2	P
PSI				38.310	<0.001
Low	70	66 (94.3)	4 (5.7)
Intermediate	98	89 (90.8)	9 (9.2)
High	55	27 (49.1)	28 (50.9)
CURB-65				24.526	<0.001
Low	91	85 (93.4)	6 (6.6)
Intermediate	67	57 (85.1)	10 (14.9)
High	65	40 (61.5)	25 (38.5)
APACHE II				56.221	<0.001
Low	85	84 (98.8)	1 (1.2)
Intermediate	106	87 (82.1)	19 (17.9)
High	32	11 (34.4)	21 (65.6)
SOFA				23.281	<0.001
Low	78	77 (98.7)	1 (1.3)
High	145	105 (72.4)	40 (27.6)
qSOFA				21.501	<0.001
Low	192	166 (86.5)	26 (13.5)
High	31	16 (51.6)	15 (48.4)

Data were presented as n (%). PSI score: Grades I–III (0–90) in low risk group, grade IV (91–130) in intermediate risk group, grade V (>130) in high risk group. CURB-65 score: 0–1 in low risk group, 2 in intermediate risk group and ≥3 in high risk group. APACHE II score: 0–10 in low risk group, 11–20 in intermediate risk group, and >20 in high risk group. SOFA score: <2 in low risk group, ≥2 in high risk group. qSOFA score: <2 in low risk group, ≥2 in high risk group. HAP: Hospital-acquired pneumonia; PSI: Pneumonia severity index; CURB-65: Consciousness, urea nitrogen, respiratory rate, blood pressure, and age ≥65 years; APACHE II: Acute Physiology and Chronic Health Evaluation II; SOFA: Sequential Organ Failure Assessment; qSOFA: Quick Sequential Organ Failure Assessment.

Discriminatory ability for 30-day mortality

In patients with HAP, the AUC for the PSI for the prediction of 30-day mortality was 0.808 (95% confidence interval [CI] 0.728–0.889, P < 0.001); the AUC for CURB-65 was 0.744 (95% CI 0.660–0.827, P < 0.001); the AUC for the SOFA score was 0.856 (95% CI 0.796–0.915, P < 0.001); the AUC for the APACHE II score was 0.863 (95% CI 0.806–0.920, P < 0.001); and the AUC for the qSOFA score was 0.767 (95% CI 0.686–0.848, P < 0.001). The ROC curves for the five assessment models for the prediction of 30-day mortality are shown in Figure 1. Table 4 shows the AUC and Youden index values for each assessment model for the prediction of 30-day mortality in HAP patients. Supplementary Table 1 shows the performance of each assessment model when applying the commonly used cut-off values to predict 30-day mortality in HAP patients. The sensitivity of the SOFA score (0.976) and the specificity of the qSOFA score (0.912) were high when applying the commonly used cut-off values of these scores to predict 30-day mortality in HAP patients. Table 5 shows the pairwise comparison of AUCs. The discriminatory abilities of the APACHE II and SOFA scores were better than those of the CURB-65 and qSOFA scores (ROC AUC: APACHE II vs. CURB-65, 0.863 vs. 0.744, Z = 3.055, P = 0.002; APACHE II vs. qSOFA, 0.863 vs. 0.767, Z = 3.017, P = 0.003; SOFA vs. CURB-65, 0.856 vs. 0.744, Z = 2.589, P = 0.010; SOFA vs. qSOFA, 0.856 vs. 0.767, Z = 2.170, P = 0.030).

Figure 1.

ROC curves for the five assessment models for the prediction of 30-day mortality in HAP patients. HAP: Hospital-acquired pneumonia; ROC: Receiver operating characteristic; PSI: Pneumonia Severity Index; CURB-65: Consciousness, urea nitrogen, respiratory rate, blood pressure, and age ≥65 years; SOFA: Sequential Organ Failure Assessment; APACHE II: Acute Physiology and Chronic Health Evaluation II; qSOFA: Quick Sequential Organ Failure Assessment.

Table 4.

AUCs and Youden index values of different severity assessment methods for predicting 30-day mortality in HAP patients.

Assessment models	AUC	95% CI	Youden index	P
PSI	0.808	0.728–0.889	0.557	<0.001
CURB-65	0.744	0.660–0.827	0.390	<0.001
SOFA	0.856	0.796–0.915	0.546	<0.001
APACHE II	0.863	0.806–0.920	0.562	<0.001
qSOFA	0.767	0.686–0.848	0.442	<0.001

AUC: Area under the curve; HAP: Hospital-acquired pneumonia; CI: Confidence interval; PSI: Pneumonia severity index; CURB-65: Consciousness, urea nitrogen, respiratory rate, blood pressure, and age ≥65 years; SOFA: Sequential Organ Failure Assessment; APACHE II: Acute Physiology and Chronic Health Evaluation II; qSOFA: Quick Sequential Organ Failure Assessment.

Table 5.

Pairwise comparison of AUCs of different severity assessment methods for predicting 30-day mortality in HAP patients.

Assessment models	PSI	CURB-65	SOFA	APACHE II
CURB-65	0.115 (1.577)
SOFA	0.312 (1.010)	0.010 (2.589)
APACHE II	0.209 (1.255)	0.002 (3.055)	0.801 (0.252)
qSOFA	0.407 (0.829)	0.590 (0.539)	0.030 (2.170)	0.003 (3.017)

Data were presented as P values (Z values). AUC: Area under the curve; HAP: Hospital-acquired pneumonia; PSI: Pneumonia severity index; CURB-65: Consciousness, urea nitrogen, respiratory rate, blood pressure, and age ≥65 years; SOFA: Sequential Organ Failure Assessment; APACHE II: Acute Physiology and Chronic Health Evaluation II; qSOFA: Quick Sequential Organ Failure Assessment.

Determination of cut-off values

We defined cut-off values for the SOFA, APACHE II, and qSOFA scores as the scores that were associated with the maximum Youden index values. If the score was not an integer, we selected the nearest integer as the cut-off value. The cut-off value for the SOFA score was 4. The sensitivity and specificity were 0.756 and 0.780, the positive predictive value (PPV) was 0.429, and the negative predictive value (NPV) was 0.928 when the new cut-off values were applied to predict 30-day mortality in HAP patients. The new cut-off value for the APACHE II score was 14, the sensitivity and specificity were 0.854 and 0.703, the PPV was 0.393, and the NPV was 0.955. The new cut-off value for the qSOFA score was 1, the sensitivity and specificity were 0.854 and 0.588, the PPV was 0.318, and the NPV was 0.947.

Discussion

Our study included patients diagnosed with HAP in our hospital from 2012 to 2017 and assessed the accuracy of the above five severity assessment methods in patients with HAP. Our results showed that all five scores within 24 h after HAP diagnosis were significantly higher in non-survivors than in survivors. All five scores had discriminatory power for predicting 30-day mortality in HAP patients. Among them, the SOFA and the APACHE II scores performed better than CURB-65 and the qSOFA score with regard to predicting the 30-day mortality of HAP patients. In our study, we defined new cut-off points using the scores that were associated with the maximum Youden index values. The sensitivities of the APACHE II and qSOFA scores were higher, while the specificity of the SOFA score was higher with regard to predicting the 30-day mortality of HAP patients.

Despite the remarkable advances in new antibiotic therapies, the mortality due to HAP remains high. The 30-day mortality of HAP patients reported by a multicenter epidemiological study in China was 22.3% in 2013.^[9] In our study, the 30-day mortality of HAP patients in our hospital in the past 5 years was 18.4% (41/223), which is basically consistent with the reported mortality. Assessment of the severity of illness is important in the management of HAP, and it can be helpful for selecting the appropriate antibiotics and more active treatments with the aim of reducing the 30-day mortality rate. A number of assessment models have already been developed. However, to the best of our knowledge, no specialized assessment models have been developed and validated for patients with HAP, and there are few studies on the use of scoring systems to assess severity and mortality in HAP patients.

Discrimination is the ability of a test tool to correctly separate patients into different groups. The AUC is frequently used to measure the discriminatory power, and higher AUC values indicate better discriminatory abilities of test tools. An AUC between 0.75 and 0.92 is regarded as good, whereas less than 0.75 indicates poor discriminatory accuracy, and less than 0.5 indicates that the test tool has no discriminatory power.^[10] By constructing ROC curves, we found that the APACHE II and the SOFA scores performed better than the other scores with regard to predicting the 30-day mortality of HAP patients. These results are consistent with some reports. Although few studies have been conducted on HAP, some studies have suggested that the SOFA and APACHE II scores can predict mortality in VAP patients.^[11–15] HAP can develop from a local infection into sepsis.^[1] Sepsis is the main cause of death resulting from an infection, and it is usually characterized by multiple organ dysfunction.^[16] In addition, HAP often occurs in the ICU, and patients often suffer from multiple comorbidities. Therefore, multiple organ dysfunction should be considered in the assessment of HAP. Organ dysfunction is associated with death; although the SOFA score was designed to assess organ dysfunction,^[7] it is also able to predict mortality. The variables in the APACHE II score are also comprehensive. The APACHE II score includes physiological variables, laboratory test results, and past medical history of the patients. It is widely used in patients in intensive care units and surgical patients to predict prognosis. Studies have also suggested that an elevated APACHE II score is one of the risk factors for inappropriate initial antibiotic treatment in HAP patients,^[17] which may result in higher mortality. The PSI and the CURB-65 did not show an advantage over other scores with regard to predicting 30-day mortality in HAP patients, which may be related to the fact that they were originally designed for outpatient evaluation.

We found that the sensitivity and specificity were poor when applying the commonly used cut-off values of these scores to predict 30-day mortality in HAP patients, so we defined new cut-off values. The qSOFA score was originally designed as a simple tool to quickly screen for severe infections, and it is especially suitable for rapid assessment outside the ICU and at a patient's bedside. Studies by Seymour et al^[18] suggested that the qSOFA score was a good predictor of in-hospital mortality for patients outside of the ICU who were suspected of having infections. Considering its design intent and simplicity, although the AUC of the qSOFA score was not better than that of the other scores in our study, we also defined its cut-off value.

We defined cut-off values for the SOFA, APACHE II, and qSOFA scores as the scores associated with the maximum Youden index values. If the score was not an integer, we selected the nearest integer as the cut-off value. In our study, the new cut-off values for the SOFA, APACHE II, and qSOFA scores were 4, 14, and 1, respectively. The results are similar to those of previous studies.^[11,19] To the best of our knowledge, there is no report of the cut-off values that should be adopted when these scores are used to predict mortality in HAP patients. In reports of VAP, the cut-off value for the APACHE II score ranged from 16 to 20,^[11,19] and the cut-off value for the SOFA was 4.^[11] When the new cut-off values were applied to predict the 30-day mortality of the HAP patients, the sensitivities and specificities of the SOFA and APACHE II scores were high. The specificity of the SOFA score was higher, which may help avoid overestimation of the severity. The sensitivity of the APACHE II score was higher, which may help alert clinicians to the possibility of critical illness, thereby reducing mortality. In addition, we found that the sensitivity of the qSOFA score was also high when applying the new cut-off value. The qSOFA score is simpler, requiring no laboratory results, and it may be more suitable for the rapid assessment of HAP patients outside the ICU and at their bedsides, especially when laboratory data are unavailable. However, the specificity of the qSOFA score was low, suggesting that the severity of the disease may be overestimated in some patients if the qSOFA score is used alone. Therefore, clinicians should observe the changes in the patient's condition more carefully and be prepared to adjust the clinical treatment to avoid the prolonged administration of combined antibiotics.

There are some limitations of this study. As this study is a retrospective study, some data needed to calculate the scores were unavailable, which may have affected the accuracy of the scoring results. With regard to the outcome measures, the assessment of the prognosis of HAP patients may need to take into account ICU admission, length of stay, and so on. However, due to the limited number of ICU beds and slow turnover in our hospital, some patients who need to be admitted to the ICU may not be admitted in time. Therefore, ICU admission was not considered as an outcome measure in this study. In addition, the included patients were mostly elderly patients with chronic diseases, and some patients had been in the hospital for a long time because of the difficulty of nursing at home due to the various kinds of catheters implanted for their chronic diseases. If we considered the length of hospital stay as one of the outcome measures of the study, the accuracy of the study could be affected. Therefore, the length of hospital stay was not considered as an outcome measure in the study. Moreover, this is a single-center study, and the representativeness of the results still needs to be further verified. Patients with VAP were not included in this study because the number of VAP patients was not sufficient for analysis, but cases are being collected and will be studied in the future.

Conclusions

These results suggest that the APACHE II and SOFA scores determined during the initial 24 h after HAP diagnosis may be useful for the prediction of 30-day mortality in HAP patients in non-surgical departments. The qSOFA score may be a simple tool that can be used to quickly identify severe infections, and it is especially well suited to rapid assessments outside the ICU and at a patient's bedside. With the gradual aging Chinese society, HAP, which is a disease with high morbidity and mortality, deserves more attention. The accurate assessment of the severity may help to select the appropriate treatment. More prospective clinical studies with larger sample sizes are also needed to assess and improve the prognosis of such patients.

Funding

This work was supported by grants from the University of Michigan Health System-Peking University Health Science Center Joint Institute for Translational and Clinical Research (No. BMU2019JI006), and Peking University Third Hospital (No. BYSYDL2019007).

Conflicts of interest

None.