Analysis and Validation of a Diagnostic Nomogram for Predicting the Risk of Acute Respiratory Failure for Non-HIV Related Pneumocystis Jirovecii Pneumonia Patients

Abstract

Objective

Pneumocystis Pneumonia (PCP), primarily affecting individuals with weakened immune systems, is a severe respiratory infection caused by pneumocystis jirovecii and can lead to acute respiratory failure (ARF). In this article, we explore the risk factors of ARF and propose a prognostic model of ARF for PCP patients.

Methods

In this multi-center, retrospective study in 6 secondary or tertiary academic hospitals in China, 120 PCP patients were screened from the Dryad database for the development of a predictive model. A total of 49 patients from Peking University People’s Hospital were collected for external validation. Crucial clinical features of these patients are selected applying univariate and multivariate logistic regression analysis. We established an intuitive nomogram. Receiver operating characteristic (ROC) curve, calibration curve, decision curve analysis (DCA) and clinical impact curve (CIC) were plotted to evaluate the model’s performance.

Results

A cohort of 120 patients formed the training cohort for the development of the model, with 49 patients constituting the test cohort. Univariate and multivariate logistic regression analysis identified five risk factors associated with ARF, which are age, fever, dyspnea, high neutrophil count and use of antibiotics. A nomogram was then proposed based on these factors. The area under the ROC curve (AUROC) in the development group has reached 0.8576, while the validation group has an AUROC of 0.7372, indicating commendable ability for predicting ARF. In addition, results for Hosmer-Lemeshow test indicate the effectiveness of our model. Furthermore, DCA and CIC curves demonstrate excellent clinical benefit.

Conclusion

We present a nomogram for predicting ARF in non-HIV related PCP patients. The prognostic model may provide references in clinical medicine, promote timely treatment and improve therapeutic outcomes of PCP patients.

Introduction

Pneumocystis pneumonia (PCP) arises as a severe respiratory infection instigated by the fungus pneumocystis jirovecii.^1–3 Primarily, it affects patients undergoing immunosuppression, such as individuals with HIV infection, autoimmune diseases, hematological diseases and so on.¹ As antiretroviral and prophylactic therapy come into existence and achieve great success, the occurrence of PCP among HIV patients has been decreasing.⁴ The incidence of non-HIV related PCP is gradually increasing,^2,3,5 and is associated with a higher rate of mortality. It has been reported that the mortality rate in non-HIV PCP patients significantly exceeds that of HIV patients by more than 20%.^6–8

PCP patients often suffer from respiratory symptoms, such as hypoxemia and dyspnea.^8,9 In a considerable proportion of cases, hypoxemia can progress rapidly to acute respiratory failure (ARF) or even acute respiratory distress syndrome (ARDS). Soo Jung Kim et al¹⁰ found in their study that 50%–64% of non-HIV-infected PCP patients required mechanical ventilation during hospitalization. Once progressing to mechanical ventilation, up to 62% of patients died.¹¹ Thus, we believe that early identification of patients who may progress to ARF can alert clinicians to be vigilant with such patients, providing guidance for timely initiation of adequate respiratory support to improve their outcomes. Some studies have already been carried out among PCP patients with ARF. Elie Azoulay et al developed a prediction score of ARF for PCP patients, providing a reference for the progression of PCP in hematologic patients.¹² Ji Soo Choi et al found that PCP PCR negative conversion and oxygenation index are independent risk factors for poor prognosis in PCP patients with ARF.¹³ However, currently, there is limited knowledge about early prediction of ARF in non-HIV-related PCP patients. Prognostic model is a valuable method that can assist clinicians in implementing timely treatment measures and appropriate treatment intensity for PCP patients. Given the challenges associated with collecting PCP cases, statistical analysis using data from public databases may be of help.^14–16

In this article, we present a nomogram for predicting ARF of non-HIV-related PCP patients with machine learning on clinical data. Analyses on training as well as test cohorts prove its clinical effectiveness and robustness.

Materials and Methods

Study Design and Patients

Through the Dryad public database, a retrospective cohort study on a multicenter population of immunocompromised individuals with pneumonia was screened out, with the selection of PCP patients in this study as the training cohort (n = 120). More information about the cohort can be found from https://doi.org/10.5061/dryad.mkkwh70x2. The test cohort included immunosuppressed patients diagnosed with PCP in Peking University People’s Hospital from September 1, 2016, to August 31, 2022. The flow chart was shown in Figure 1. This study was approved by the Ethics Committee of Peking University People’s Hospital (No. 2022PHB457). Informed consent was obtained by telephone communication (due to the impact of the COVID-19 pandemic) and was approved by the Ethics Committee of Peking University People’s Hospital.

Figure 1.

Flow chart of patients enrollment.

The diagnosis of PCP referred to the guidelines of EORTC/MSGERC:¹⁷ A. PCP cases required the presence of at least two out of three clinical criteria: a. Respiratory symptoms, including cough and/or dyspnea, and hypoxemia; b. Typical radiological findings, such as ground-glass opacities on chest CT scans; c. Diffuse interstitial infiltrates on chest X-rays. B. Confirmation of PCP diagnosis was based on positive results from qPCR or metagenomic next-generation sequencing of bronchoalveolar lavage fluid specimens. C. Other infectious etiologies were excluded.

Inclusion criteria were as follows: 1) Immunocompromised status (continuous use of corticosteroids or immunosuppressants for ≥2 weeks); 2) Diagnosed with PCP post-admission; 3) Age ≥18 years.

Exclusion criteria included: 1) Absence of laboratory examinations and chest CT scans; 2) HIV positive; 3) Receiving treatment for PCP for more than 5 days, either trimethoprim/sulfamethoxazole (15/75 mg/kg/day) or clindamycin plus primaquine; 4) No BALF retained.

Data Collection and Processing

We collected variables potentially associated with the occurrence of ARF in patients with PCP. These variables encompassed demographic data, symptoms, comorbidities, laboratory test results (routine blood tests, liver function, renal function, electrolytes, PCT, CRP, coagulation function), characteristics of chest CT scans, and medication history.

Statistical Analysis

Continuous variables in a normal distribution were presented as the mean (± S.D.) while the median (25th–75th percentile) in a non-normal distribution. Continuous data with a non-normal distribution were analyzed by Wilcoxon rank-sum test, while T test with a normal distribution. Categorical variables were analyzed by chi-squared test. A p-value < 0.05 was considered statistically significant. All variables were sequentially incorporated into univariate logistic regression analyses, and those with p<0.05 were included in multivariate analysis to develop a predictive model. A nomogram was plotted. Additionally, the accuracy of the model was evaluated using a bootstrap-based calibration process,^18,19 while Hosmer-Lemeshow test was used to compare actual and predicted probabilities of ARF. A non-significant result in the Hosmer-Lemeshow test indicates good model fit. Receiver operating characteristic (ROC) curve, calibration curve, decision curve analysis (DCA) and clinical impact curve (CIC) were plotted to evaluate the model’s performance. All analyses were conducted in SPSS Statistics 27.0, Prism GraphPad 9, and R 4.2.2.

Results

Characteristics of the Dataset

In this study, 120 PCP patients from the public database were assigned to the training cohort, and 49 PCP patients from Peking University People’s Hospital were assigned to the test cohort. Baseline characteristics of patients with or without ARF in train cohort are shown in Table 1. It showed no significant differences between the two groups in age and gender. In terms of laboratory tests, there were no significant differences between the two groups in lymphocyte count, hemoglobin, lactate dehydrogenase, albumin, and creatinine levels. However, the ARF group had higher levels of neutrophil (NEUT) (6.10 vs 7.19, p=0.016) and blood urea nitrogen (BUN) (5.57 vs 6.80, p=0.028) level. In terms of treatment, the ARF group used more antibiotics (p=0.01) and antiviral drugs (p=0.01).

Table 1.

Baseline Characteristics of Patients with or Without Acute Respiratory Failure

	N-ARF (n=26)	ARF (n=94)	p-value
Age			0.072
Youth	10/26	18/94
Middle-age	6/26	40/94
Old-age	10/26	36/94
Gender(M/F)	16/10	46/48	0.802
Complication
Asthma	1/26	3/94	0.162
ILD	10/26	35/94	0.909
HBP	11/26	30/94	0.602
CHD	3/26	10/94	0.195
DM	4/26	27/94	0.06
Anemia	1/26	8/94	0.424
CRF	4/26	10/94	0.505
Nephrotic syndrome	4/26	26/94	0.201
Cirrhosis	0/26	1/94	0.597
CTD	14/26	42/94	0.407
Tumor	2/26	7/94	0.966
Solid organ transplantation	2/26	7/94	0.966
Radiation Pneumonia	2/26	0/94	0.007**
Cerebrovascular diseases	1/26	1/94	0.327
Laboratory tests
NEUT(×109/L)	6.57(±2.61)	7.56(±3.84)	0.016*
LYM(×109/L)	0.58(0.40–1.01)		0.111
HGB(g/L)	117(±22.03)	0.6(0.40–0.90)	0.343
PLT(×109/L)	182(113–257)	200(126–264)	0.613
ALB(g/L)	32.32(±5.08)	30.53(±5.86)	0.123
ALT(U/L)	25(18–47)	28(18–51)	0.302
AST(U/L)	20(14–30)	30(21–49)	0.127
BUN(mmol/L)	5.97(4.90–12.20)	6.75(5.45–11.50)	0.028*
CRE(μmol/L)	64.75(49.27–138.50)	66.95(50.60–100.27)	0.597
Treatment
Antibiotics	15/26	77/94	0.01*
Antiviral Drugs	1/26	26/94	0.01*
Antiaspergillus	12/26	50/94	0.848
Antipseudomonas	20/26	84/96	0.099

Note: *p<0.05, **p<0.01.

Abbreviations: ARF, Patients with acute respiratory failure; nARF, Patients without acute respiratory failure; NEUT, Neutrophil; LYM, Lymphocyte; HGB, Hemoglobin; PLT, Platelet; ALB, Albumin; ALT, Alanine aminotransferase; AST, Aspartate aminotransferase; BUN, Blood urea nitrogen; CRE, Creatine.

Diagnostic Indicator Selection and Regression Model

Univariate and multivariate logistic regression analyses were performed on 33 baseline characteristics for all patients. The results were depicted in Table 2, revealing that age (40–59; OR, 0.194; 95% CI, 0.048–0.787; p = 0.022), fever (OR, 11.922; 95% CI, 1.898–74.880; p = 0.008), dyspnea (OR, 7.056; 95% CI, 2.429–27.281; p = 0.001), NEUT (OR, 1.202; 95% CI, 1.012–1.427; p = 0.036) and use of antibiotics (OR, 7.056; 95% CI, 2.013–24.725; p = 0.002) were independently associated with developing ARF.

Table 2.

Univariate and Multivariate Logistic Regression Analysis

Univariate logistic regression analysis				Multivariate logistic regression analysis
Variables	OR	95% CI	p-value	OR	95% CI	p-value
Age(years)						0.07
<40	Ref			Ref
40–59	2.639	0.891–7.815	0.08	0.194	0.048–0.787	0.022
≥60	2.639	0.891–7.815	0.08	0.475	0.123–1.39	0.281
Gender	1.118	0.468–2.670	0.802
Fever	7.222	1.599–32.626	0.01	11.922	1.898–74.880	0.008
Cough	0.687	0.183–2.581	0.578
Expectoration	1.248	0.497–3.134	0.637
Chest pain	0.543	0.047–6.240	0.624
Dyspnea	6.468	2.391–7.493	<0.0001	7.056	2.429–27.281	0.001
Antibiotics	3.322	1.299–8.492	0.012	7.056	2.013–24.725	0.002
Antiviral Drugs	9.559	1.231–74.195	0.031
Asthma	0.261	0.035–1.948	0.190
ILD	0.949	0.388–2.320	0.909
HBP	0.787	0.320–1.936	0.602
CHD	3.659	0.453–29.536	0.224
DM	3.253	0.903–11.719	0.071
Anemia	2.326	0.277–19.492	0.437
CRF	0.655	0.187–2.287	0.507
Nephroticsy-ndrome	2.103	0.661–6.69	0.208
CTD	0.692	0.290–1.655	0.408
Cerebrovas-culardiseases	0.269	0.016–4.451	0.359
Heart rate	1.001	0.987–1.104	0.911
Respiratory-rate	1.135	1.014–1.271	0.028
NEUT	1.144	1.001–1.308	0.049	1.202	1.012–1.427	0.036
LYM	0.725	0.351–1.501	0.387
HGB	0.989	0.968–1.011	0.341
PLT	1.000	0.995–1.004	0.599
ALB	0.941	0.871–1.017	0.161
ALT	0.993	0.985–1.000	0.139
AST	1.002	0.993–1.011	0.381
BUN	1.085	0.986–1.195	0.276
CRE	1.002	0.995–1.008	0.937
Aspergillus	1.089	0.457–2.596	0.526
Antipseudo-monas	2.52	0.819–7.790	0.107

Note: *p<0.05, **p<0.01.

Abbreviations: NEUT, Neutrophil; LYM, Lymphocyte; HGB, Hemoglobin; PLT, Platelet; ALB, Albumin; ALT, Alanine aminotransferase; AST, Aspartate aminotransferase; BUN, Blood urea nitrogen; CRE, Creatine.

Diagnostic Nomogram for the Probability of ARF

The study finally incorporated age, fever, dyspnea, NEUT, and antibiotics to make a ARF predictive model. The model is visualized using a nomogram in Figure 2.

Figure 2.

Nomogram of the predictive model for PCP. When using a nomogram, locate the position of each variable on the corresponding horizontal axis and draw a vertical line to the top score axis to determine the score for that variable. Then sum the scores of all variables and draw a line from the total score to the probability axis at the bottom of the nomogram to determine the predicted risk of ARF. Youth: age < 40 years; Middle-age: age 40–59 years; Old-age: age ≥ 60 years.

Additionally, we also utilized a test cohort of PCP patients (n = 49) from our center as an external validation to test the performance of the model. Table 3 illustrates the clinical characteristics of significant factors in train and test cohort. Major difference lies in fever and use of antibiotics.

Table 3.

Clinical Characteristics in the Train and Test Group

	Train cohort (n=120)	Test cohort (n=49)	p-value
Age(years)			0.522
<40	26/120	15/49
40–59	46/120	15/49
≥60	46/120	19/49
Fever	112/120	29/49	<0.0001
Dyspnea	97/120	40/49	0.904
Antibiotics	92/120	46/49	0.009
ARF	94/120	36/49	0.496

Performance of the Nomogram

We then evaluated the performance of the nomogram on train and test cohorts through ROC and calibration curves, which are displayed in Figure 3. In the training cohort, the AUC was 0.8576, with a Hosmer-Lemeshow test K2 of 5.254 and a p value of 0.730, indicating statistical significance of the model. The AUC for the test cohort was 0.7372. Upon performing the Hosmer-Lemeshow test, the K2 and p values were 6.472 and 0.594, respectively, indicating that the model was still meaningful in the test cohort.

Figure 3.

ROC and calibration curves for train and test cohorts. (a) ROC analysis of the training cohort; (b) Calibration curve of the train cohort; (c) ROC analysis of the test cohort; (d) Calibration curve of the test cohort.

Furthermore, DCA and CIC curves are depicted on both the training and test cohorts. We calculated the optimal critical values for the probability thresholds of the train and test cohorts based on the maximum Youden index, which were 75% and 60%, respectively. As was shown in Figure 4a, DCA indicated that the nomogram was clinically useful. With a critical value of the probability threshold at 75%, the corresponding net benefit for the train set was about 0.2. When the critical value of the probability threshold for the test cohort was 60%, the corresponding net benefit for the test cohort was about 0.15. Furthermore, CIC curves based on the DCA were plotted to evaluate the clinical effect of the model more intuitively. Under the high-risk threshold from 0.4 to 1 (Figures 4b and c), the “number high risk” curve was close to the “number high risk with events” curve, indicating that the model has good predictive ability. Therefore, integrating the analysis of ROC, calibration curve, DCA, and CIC, we believe that the model has good discriminative performance and clinical effectiveness.

Figure 4.

DCA and CIC analysis of train and test cohorts. (a) DCA plots of the training cohort (blue) and test cohort (red). With reference to previous studies, our study set the incidence of ARF in PCP at 50%. The black horizontal line represents no intervention for all patients and a net benefit of 0. The gray line represents intervention for all patients. CIC of the training cohort (b) and test cohort (c). The red line represents the number of people at high risk judged by the model at different probability thresholds; Blue represents the number of people that the model judged to be at high risk and had an outcome event.

Discussion

Nomogram is mainly used for diagnosis or poor prognosis of PCP in the research.^20–24 In immunosuppressed hosts, PCP is a life-threatening disease. Therefore, it is very important to investigate to improve its prognosis. Studies have been focused on risk factors for predicting death in PCP patients. Serum albumin, corticosteroid use before diagnosis, PEEP level on day 3 of ARF, pneumothorax, duration of fever after admission, and CD4+ T cells ≤100/μL were found to be risk factors for death in PCP patients.^25,26 Considering that non-HIV related PCP is prone to ARF, early intervention can help improve their prognosis. However, there are few relevant studies at present. Therefore, our study established a predictive model for the occurrence of ARF in PCP patients. Apart from other nomogram studies of PCP, we introduced DCA and CIC analysis to evaluate the clinical performance of our model and found good results. This demonstrates the robust clinical applicability of our model. For physicians, our probability prediction model can be used to quickly and conveniently assess clinical events and help them initiate subsequent interventions.

Previous study found that older age was one of the poor prognostic factors for PCP.²⁷ Additionally, a prospective observational study demonstrated an independent association between age and both the 90-day mortality rate and disease severity in PCP patients.²⁸ Therefore, age was included in the multivariate analysis in the study although it was not significant in the univariate analysis. In the present study, age was found to be a protective factor for patients aged 40–59 during hospitalization. This in turn suggests that younger and older PCP patients are more likely to develop ARF compared to the middle-aged group.

In our model, dyspnea and fever have been incorporated as risk factors of ARF. It is well known that PCP patients often exhibit fever, progressive dyspnea, and dry cough.²⁹ Based on this study, early attention to symptoms of dyspnea and fever is crucial in identifying potential ARF patients.

Use of antibiotics is an independent risk factor of ARF in our study. We speculate the possible reasons are as follows. PCP patients undergoing antibiotic treatment may present with concurrent health issues, such as other lung infections or chronic diseases, which could increase the risk of ARF. Additionally, the treatment with different antibiotics may have pulmonary side effects that exacerbate lung conditions,³⁰ thus increasing the risk of ARF. The immunosuppressed population commonly uses antibiotics prophylactically, so they are also more susceptible to opportunistic infections.³¹

However, there are some defects in the prognostic model. Firstly, this model is based on a multi-center retrospective study of a public database, and the retrospective data of the public database may be processed due to privacy protection and other reasons, which may affect the accuracy of the prediction model. Moreover, the variable settings of the public dataset are fixed and cannot be changed. For example, we were not able to count types of antibiotics/antivirals due to the public dataset. Secondly, PCP patients were often coinfected with other microorganisms, which can confound our results. Thus, the use of antibiotics may also be a confounding factor. The above factors may limit the performance of the model. However, in clinical practice, it is uncommon to encounter patients diagnosed exclusively with PCP, making it challenging to exclude the aforementioned confounding factors. We believe that when applying this model in a clinical setting, it is essential to account for the presence of co-infections. It is anticipated that future prospective studies, utilizing larger sample sizes, will continue to refine and validate the model.

Conclusion

We constructed a nomogram to predict the risk of ARF in non-HIV related PCP patients, which included age, fever, dyspnea, NEUT, and antibiotics.

Funding Statement

The trial was funded by the National and Provincial Key Clinical Specialty Capacity Building Project 2020 (Department of the Respiratory Medicine).

Data Sharing Statement

This study utilized retrospective cohort study data, which was sourced from the Dryad digital repository: https://datadryad.org/. The entry doi: 10.5061/dryad.mkkwh70x2. The dataset was provided by Li, Lijuan, Hsu, Steven H., Gu, Xiaoying et al. The utilization of this dataset was in strict accordance with the terms and conditions as stipulated by Dryad and under the ethical guidelines of our institution.

Ethics Approval and Informed Consent

This study was approved by the Medical Ethics Committee of Peking University People’s Hospital (approval no.: 2022PHB457). Informed consent was obtained from all participating patients. All experiments were performed in accordance with the Declaration of Helsinki.

Consent for Publication

All authors confirm our joint participation and agreement for the publication of this manuscript.

Author Contributions

All authors made a significant contribution to the work reported, whether that is in the conception, study design, execution, acquisition of data, analysis and interpretation, or in all these areas; took part in drafting, revising or critically reviewing the article; gave final approval of the version to be published; have agreed on the journal to which the article has been submitted; and agree to be accountable for all aspects of the work.

Disclosure

The authors declare that they have no competing interests.