Prediction of pulmonary embolism by an explainable machine learning approach in the real world

Abstract

In recent years, large amounts of researches showed that pulmonary embolism (PE) has become a common disease, and PE remains a clinical challenge because of its high mortality, high disability, high missed and high misdiagnosed rates. To address this, we employed an artificial intelligence–based machine learning algorithm (MLA) to construct a robust predictive model for PE. We retrospectively analyzed 1480 suspected PE patients hospitalized in West China Hospital of Sichuan University between May 2015 and April 2020. 126 features were screened and diverse MLAs were utilized to craft predictive models for PE. Area under the receiver operating characteristic curves (AUC) were used to evaluate their performance and SHapley Additive exPlanation (SHAP) values were utilized to elucidate the prediction model. Regarding the efficacy of the single model that most accurately predicted the outcome, RF demonstrated the highest efficacy in predicting outcomes, with an AUC of 0.776 (95% CI 0.774–0.778). The SHAP summary plot delineated the positive and negative effects of features attributed to the RF prediction model, including D-dimer, activated partial thromboplastin time (APTT), fibrin and fibrinogen degradation products (FFDP), platelet count, albumin, cholesterol, and sodium. Furthermore, the SHAP dependence plot illustrated the impact of individual features on the RF prediction model. Finally, the MLA based PE predicting model was designed as a web page that can be applied to the platform of clinical management. In this study, PE prediction model was successfully established and designed as a web page, facilitating the optimization of early diagnosis and timely treatment strategies to enhance PE patient outcomes.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-024-75435-9.

Introduction

Pulmonary embolism (PE) represents a critical pulmonary circulatory disorder resulting from blockages in the pulmonary artery trunk or its branches, posing a significant threat to personal health. Manifesting with either subtle clinical symptoms or atypical features, PE often leads to missed diagnosis and misdiagnosis, causing patients to delay treatment and exacerbate disease progression^1–3. A North American study reported a detection rate for PE of less than 5%⁴. At present, computed tomography pulmonary angiography (CTPA) stands as the primary clinical diagnostic method for PE. However, its application may yield adverse effects, particularly renal function impairment in patients with a history of renal disease or allergies^5,6. In addition, CTPA is time-consuming, expensive, and inaccessible in certain conditions especially grassroots healthcare units. Therefore, establishing an efficient predictive model to identify high-risk PE patients before the onset of clinical event is imperative and impactful.

The concept of “medical engineering integration” advocates for the amalgamation and application of medicine, engineering, and biology, leveraging the complementary strengths of diverse disciplines. Machine learning, a methodology enabling computer systems to execute specific tasks through pattern recognition and reasoning rather than explicit instructions, has significantly contributed to this integration by studying algorithms and statistical models⁷. Machine learning spans diverse algorithms, encompassing supervised techniques like support vector machines (SVM), Bayesian learning, decision trees, regressions, as well as unsupervised methods such as k-means clustering, reinforcement learning (e.g., Q-learning), and specialized models like neural networks^8,9. In the realm of disease risk prediction, the prevalent approach involves framing risk assessment as a statistical classification problem and leveraging pertinent models for resolution. Notably, SVM, neural networks, random forests (RF), and naive Bayes emerge as commonly utilized algorithms in disease risk prediction, recognized for their efficacy based on literature reviews, practical applications, and methodological characteristics¹⁰.

The synergy between machine learning and medicine has revolutionized disease prevention and treatment¹¹. Notably, the China PAR model evaluates both 10-year and lifelong risks of cardiovascular diseases, which provides a practical assessment tool for primary cardiovascular disease prevention in China¹². As for PE Predicting, previous work focused on the capability of MLA in aiding interpret radiology results to increase the accuracy and timeliness of PE diagnosis^13–15. However, these models were limited to patients who had already undergone a formal diagnostic workup for PE. Therefore, in this study we intend to study suspected patients at risk of PE and apply diverse MLAs to analyze PE-associated variables including demographic profiles, clinical status, laboratory findings, and imaging characteristics, aiming to construct an optimal predictive model and verify its predictive capacity through Shapley Additive exPlanation (SHAP) values, which not only helps to improve the clinical management of PE patients, but also decreases the unnecessary CTPA.

Methods

Data collection and processing of data

According to included criteria (1.High risk factors for venous thromboembolism (VTE), with or without the clinical symptoms; 2.The medical records were complete.) and excluded criteria (1.Children under the age of 18 and pregnant women were excluded, N = 28; 2.CTPA reports were not available, N = 934; 3.CTPA did not clearly diagnose, N = 53.), we conducted a retrospective review of medical records pertaining to 1480 hospitalized individuals at West China Hospital of Sichuan University between May 2015 and April 2020. 1480 suspected PE patients progressed to the subsequent phase. Among these, 70% (n = 1036) were allocated to the training set, while the remaining 30% (n = 444) comprised the test set for PE prediction. Comprehensive data encompassing demographic profiles, clinical status, laboratory findings, and imaging characteristics were collected, and 126 features were screened into the next step. To construct the PE prediction model, we employed the Lasso model and multivariate logistic regression techniques to discern the most optimal combination of features in the training set. Finally, we evaluated the performances of MLAs by AUC in the test set and elucidated the optimal prediction model by SHAP values (shown in Fig. 1).

Fig. 1.

Flow diagram of the patient selection and data processing.

Machine learning

The MLAs used in this study specifically included logistic regression (LR), support vector machine (SVM), multilayer perceptron (MLP), random forest (RF), extreme gradient boosting (XGBoost) and light gradient boosting machine (LightGBM). We assessed the predictive performance according to the AUC. To ensure the robustness and reliability of the results, bootstrap methods with 1,000 bootstrap replicates were used to the derive 95% confidence interval (CI). Further, we used SHAP summary plot to delineate the positive or negative effects of the top features in the prediction model, and used SHAP dependence plot to explain how a single feature affects the output of the prediction model¹⁶. Finally, we developed a clinical web interface for pulmonary embolism prediction (https://xingyu87.pythonanywhere.com/). This platform allows for the determination of pulmonary embolism probability by inputting specific variable values. Notably, it visually showcases the SHAP values, providing insight into each variable’s contribution to pulmonary embolism prediction.

Statistics

Continuous variables with normal distribution were presented as mean ± standard deviation (SD), while non-normal variables were reported as median (Q1, Q3). Categorical data were described as numbers (n) and percentages (%). Statistical analyses included Student’s t-test or one-way analysis of variance (ANOVA) for normally distributed continuous variables, and the Kruskal-Wallis test for non-normally distributed continuous variables. Fisher’s exact test or the χ2 test was employed for comparisons involving categorical data. Significance was determined at p-values < 0.05.

Results

We conducted a retrospective analysis involving 1480 hospitalized suspected PE patients at West China Hospital of Sichuan University from May 2015 to April 2020. Demographic and clinical variables were listed in Table S1. Differences between the training and test sets were non-significant for all variables (shown in Table S2). All data during hospitalization were analyzed by Lasso model, and features selected by Lasso were presented as Figure S1. By combining multivariate logistic regression with clinical practice, seven variables emerged as significant predictors of PE : D-dimer, activated partial thromboplastin time (APTT), fibrin and fibrinogen degradation products (FFDP), platelet count, albumin, cholesterol, and sodium (shown in Table S3). Utilizing these selected features as input variables, we constructed diverse machine learning algorithm (MLA) models, including LR, SVM, MLP, RF, XGBoost and LightGBM, and employed GridSearch along with 5-fold cross-validation for model training and evaluation (shown in Table S4). Among these, RF demonstrated the highest efficacy in outcome prediction, exhibiting the largest AUC of 0.776 (95% CI 0.774–0.778) as shown in Fig. 2.

Fig. 2.

Comparison of AUCs among machine learning models.

The SHAP summary plot of the RF model was presented in Fig. 3, which illustrates how higher SHAP values for specific features corresponded to an increased likelihood of PE development. Each patient had a dot assigned on the line for every feature, color-coded according to their respective feature values. These dots stacked vertically to illustrate density, with red denoting higher values and blue indicating lower values. To elucidate how individual feature influences the RF prediction model’s output, the SHAP dependence plot was displayed in Fig. 4. The y-axis represents specific feature SHAP values, with values exceeding zero indicating a higher probability of PE. Finally, we developed an online decision-support tool available on https://xingyu87.pythonanywhere.com/ (shown in Fig. 5 and Supplementary video), and the risk index of PE will be generated automatically after entering specific values of patients.

Fig. 3.

SHAP summary plot of the RF model.

Fig. 4.

SHAP dependence plot of the RF model.

Fig. 5.

Web page of pulmonary embolism prediction.

Discussions

Acute pulmonary embolism (PE) stands as the third leading cause of cardiovascular death globally, following coronary heart disease and stroke. Statistics indicate that the short-term mortality rate for untreated PE reaches as high as 30%^17,18. Given PE potential to swiftly become life-threatening, early detection or advanced prediction plays a pivotal role in optimizing care, particularly for patients with complication such as cancer^19,20. In this study, we demonstrated the efficacy of machine learning models in preemptively identifying patients at high risk of experiencing PE before its occurrence.

Through constructing predictive models using diverse MLAs, we observed that RF exhibited superior performance in PE prediction. RF consists of numerous random decision trees, utilizing random sample from the original data and subsets of features to develop the model. Each tree provides a classification, essentially ‘voting’ for a particular class. The collective outcome of these trees determines the forest’s classification, ensuring RF maintains consistent and stable performance levels^21,22. Compared with previous research associated with machine leaning-based PE prediction^23,24, the scope of research population in our study was much broader, including a variety of factors that contribute to the risk of VTE. Additionally, we utilized more features to build different machine learning models, which makes our prediction results more reliable and accurate.

More importantly, to date, there is no literature evidence of using SHAP values to explain PE prediction models. In this study, we used SHAP values to explain the contribution of machine learning in PE prediction, fully integrating engineering and medicine. First, the SHAP summary plot highlighted several key features critical for predictions, such as D-dimer, APTT, FFDP, and platelet count, aligning with earlier research outcomes^25–27. Additionally, this analysis identified further factors not conventionally recognized as provocative risk elements associated with heightened PE risk, including albumin²⁸, Sodium²⁹, and cholesterol³⁰. This underscores the intricate interplay between the hemostatic and internal environment systems^31–33.

The MLA developed in our study offers distinct advantages over alternative risk stratification methods, such as adjusted D-dimer and scoring systems. Unlike these methods, our algorithm doesn’t require additional clinician inputs or disrupt workflow, which can automatically screen a broad inpatient population solely based on electronic health record (EHR) data. D-dimer, sensitive in detecting fibrinolysis of intravascular thrombus, lacks specificity and can be elevated in inflammatory states and other conditions^34,35. The Wells criteria and revised Geneva score represent the most commonly used risk scores, yet they were designed for inpatients suspected of existing PE and not tailored for predicting future occurrences of PE. Furthermore, these scoring systems have only been validated for assessing PE risk in outpatients, not in hospitalized patients^36,37. Additionally, Stals MAM³⁸ noted remarkably low reproducibility in the scoring method.

In contrast, our MLA model not only enhances future PE prediction but also facilitates early detection preceding the confirmatory diagnostic process, consequently reducing unnecessary CTPA. However, there are still some areas that need to be improved in the future. For example, we need to include more clinical data in multi-centers, and minimize bias that can arise from unbalanced training, suboptimal architecture design or selection, and uneven application of models, which not only increases the reliability and accuracy of the predictive model, but also avoids the influence of the model caused by hospital factors^39,40. In addition, with the popularization of PE prediction models, we try to build the best PE prediction models according to the classification of causes especially for non-thrombotic PE, so as to maximize the clinical outcome of PE patients^41,42. Finally, there are various limitations in integration of the web-based tool into clinical workflows. These algorithms necessitate meticulous data processing, training on potentially extensive datasets, and iterative refinement aligned with real clinical scenarios. In addition, integration of MLA into clinical practice introduces critical ethical considerations, including issues of liability concerning medical errors, comprehension among healthcare professionals regarding how these algorithms generate predictions, as well as concerns regarding privacy, security, and patient data control^43,44.

Conclusions

Our prediction model based on MLA was executed within a premier hospital setting, showcasing enhanced performance in identifying PE risk within suspected patient population. Remarkably, this predictive modeling tool has been designed as a simple web page that can be applied to the platform of clinical management. We only need to input some specified features of the patient into the system, and the risk index for pulmonary embolism is automatically generated within minutes. Our machine learning-based predictive model can not only help doctors to detect suspicious patients in time and decrease unnecessary CTPA, but also improve the clinical outcome of PE patients.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Author contributions

Q.Z. and R.H. performed the experiments, analyzed the data, and contributed equally for the whole study. Y.X. was involved in statistical analysis. Q.Z. and R.H. were involved in drafting of the manuscript. Y.X., Z.L. and W.Z. provided critical comments and edited the manuscript. All authors were involved in critical revision of the manuscript for important intellectual content. All authors provided administrative, technical and material support.

Funding

This study received the following funding: the San Hang Program of the Second Military Medical University; the Changjian Program of First Affiliated Hospital the Second Military Medical University to Wei Zhang. The Project of Disciplines of Excellence, Shanghai Municipal Health Commission (Grant No. 20234Z0017).

Data availability

The authors confirm that the data supporting the findings of this study are available within the article and its supplementary materials.

Declarations

Competing interests

The authors declare no competing interests.

Statement of ethics

This study was approved by the Ethics Committee on Biomedical Research, West China Hospital of Sichuan University, and we promised the whole process was conducted according to the principles expressed in the Declaration of Helsinki. All patients wrote informed consent to participate this study. Because this study involved no more than minimal risk to patients, the need for informed consent was waived by the Ethics Committee on Biomedical Research (No.2021 − 828), West China Hospital of Sichuan University. We promised all efforts were made to protect patient privacy and anonymity.