A multimodal integration pipeline for accurate diagnosis, pathogen identification, and prognosis prediction of pulmonary infections

Abstract

Pulmonary infections pose formidable challenges in clinical settings with high mortality rates across all age groups worldwide. Accurate diagnosis and early intervention are crucial to improve patient outcomes. Artificial intelligence (AI) has the capability to mine imaging features specific to different pathogens and fuse multimodal features to reach a synergistic diagnosis, enabling more precise investigation and individualized clinical management. In this study, we successfully developed a multimodal integration (MMI) pipeline to differentiate among bacterial, fungal, and viral pneumonia and pulmonary tuberculosis based on a real-world dataset of 24,107 patients. The area under the curve (AUC) of the MMI system comprising clinical text and computed tomography (CT) image scans yielded 0.910 (95% confidence interval [CI]: 0.904–0.916) and 0.887 (95% CI: 0.867–0.909) in the internal and external testing datasets respectively, which were comparable to those of experienced physicians. Furthermore, the MMI system was utilized to rapidly differentiate between viral subtypes with a mean AUC of 0.822 (95% CI: 0.805–0.837) and bacterial subtypes with a mean AUC of 0.803 (95% CI: 0.775–0.830). Here, the MMI system harbors the potential to guide tailored medication recommendations, thus mitigating the risk of antibiotic misuse. Additionally, the integration of multimodal factors in the AI-driven system also provided an evident advantage in predicting risks of developing critical illness, contributing to more informed clinical decision-making. To revolutionize medical care, embracing multimodal AI tools in pulmonary infections will pave the way to further facilitate early intervention and precise management in the foreseeable future.

Graphical abstract

Public summary

• •Multimodal integration pipeline could diagnose pulmonary infections based on clinical characteristics, CT images, and laboratory test results.
• •The pipeline achieved diagnostic performance comparable with senior physicians.
• •In addition to assessing pulmonary infections, the automated system could also predict progression to critical illness and facilitate early intervention.

Introduction

Lower respiratory tract infections are one of the leading causes of death worldwide and contribute to a significant disease burden, which impacted 344 million people and resulted in 2.18 million deaths according to the 2021 Global Burden of Diseases study.^1,2 Pneumonia—attributable to bacteria, viruses, and fungi—is recognized as the most pervasive infectious disease globally, further aggravated by the COVID-19 pandemic.³ In addition, Mycobacterium tuberculosis also poses a significant lung infection risk, with 6.2 million people diagnosed with pulmonary tuberculosis (PTB) worldwide.^4,5 There are variations in the treatment strategies for different pathogens, but a staggering 62% of patients lack a definitive pathogen diagnosis.^6,7,8 The traditional diagnostic gold standard requires obtaining respiratory samples for microbial smears or cultures, which is plagued by insensitivity and protracted processing time.^3,9 Once pulmonary infections patients receive delayed treatment and develop severe disease, their mortality rate soars to 50%.¹⁰ Hence, a rapid and accurate method for pulmonary infection diagnosis, pathogen detection, and severe disease prediction is urgently needed in clinical scenarios.

The diagnosis of pulmonary infections calls for a multifaceted approach encompassing clinical assessments, laboratory tests, and other indicators.^3,11 The predominant symptoms, cough and expectoration, are common clinical manifestations. Additionally, peripheral blood leukopenia has significance in bacterial infection, signaling critical illness and an unfavorable prognosis. Chest computed tomography (CT) also acts as a crucial supportive tool for diagnosing pulmonary infections, offering three-dimensional representations of pulmonary structures.¹² Discrepancies exist in the imaging manifestations of different types of pathogenic infections. For example, the imaging of bacterial pneumonia typically reveals alveolar infiltrates or solid changes, whereas the imaging of viral pneumonia is characterized mostly by ground-glass opacities (GGOs) alongside solid shadows.¹¹ In cases of severe pneumonia, the percentage of lesion-involved areas in the chest CT images of patients frequently exceeds 50%.¹³ However, the radiographic manifestations of pneumonia exhibit an overlap across diverse pathogens, and the imaging signs of specific pathogenic infections can vary among individuals. This phenomenon complicates the accurate diagnosis of pulmonary infections for physicians and renders the non-invasive identification of the pathogen even more challenging.

The rapid development of artificial intelligence (AI) technology has catalyzed innovation regarding medical tasks.^{14,15,16,17,18} AI models have garnered remarkable results in skin cancer subtype classification, diabetic retinopathy diagnosis, respiratory disease diagnosis, and prognosis assessment.^{19,20,21,22,23,24} With the evolution of AI technology, a trend of multimodal fusion has emerged, providing a more comprehensive approach that resonates with real-world clinical applications.^25,26,27,28 In terms of pulmonary infections, a surge of intelligent diagnostic models has been noted in recent years.^{29,30,31,32,33,34,35,36,37} For instance, advanced diagnostic architectures employing the DenseNet-121 deep learning network have been established to identify viral, non-viral, and COVID-19 pneumonia based on chest X-ray (CXR) images, demonstrating impressive discrimination capabilities (area under the curve [AUC]: 0.867–0.966).²⁹ Another convolutional neural network model using CXR images was constructed to diagnose PTB with an AUC of 0.992 (95% confidence interval [CI]: 0.961–1.000), and the performance of the pre-trained network was 11% higher than that of the untrained network.³⁰ However, CXR is a two-dimensional imaging technique with inherent limitations regarding diagnostic utility. Furthermore, an AI model based on the quantitative analysis of CT images was constructed to automatically localize typical foci of pneumonia, such as GGOs and solid changes, accurately distinguishing pneumonia with remarkable preformance (AUC: 0.975).³¹ Additionally, investigations have integrated CT images with clinical information from diverse cohorts to rapidly diagnose pulmonary infections patients, outperforming the single-modality model (AUC: 0.92 vs. 0.80).³² However, the majority of studies focus narrowly on individual pathogen but ignore the complexity of multiple infection types. Moreover, merely utilizing unimodal imaging data for detecting pneumonia have curtailed the ability of these AI models to make a more detailed and accurate diagnosis compared to the use of multimodal information, which integrates data from various dimensions.

In this study, we developed a multimodal integration (MMI) pipeline to assist with the precise diagnosis of bacterial, fungal, and viral pneumonia as well as PTB and to facilitate pathogen prediction (Figure 1). In addition, crucial features were filtered to predict the probability of severe pneumonia. The MMI system holds the potential to assist physicians with improving clinical efficiency and personalizing the fine management of patients with pulmonary infections.

Figure 1.

Overview of the multimodal integration (MMI) pipeline for pulmonary infection management

(A) Patients were identified and divided into training (WCH, N = 19,046), validation (WCH, N = 2,432), internal (WCH, N = 2,433), and external testing (CSJH; N = 196) datasets for the MMI system.

(B) Development of the MMI system through the integration of multimodal data. It integrated multimodal information, including unstructured text, images, lab assay results, and structured text to construct the system.

(C) Application of the MMI system in clinical settings for the diagnosis of infectious diseases, identification of pathogens, and risk prediction of critical patients, showcasing the actual utility of the AI model in managing pulmonary infections. CSJH, Chengdu ShangJin Nanfu Hospital; ICU, intensive care unit; WCH, West China Hospital.

Results

Participant characteristics

An extensive large dataset of 161,258 chest CT scans from 54,581 patients with associated clinical information and laboratory examinations was assembled from West China Hospital (WCH) of Sichuan University and Chengdu ShangJin Nanfu Hospital (CSJH). The data from WCH were employed for training, validation, and internal testing, while the data from CSJH served as an external testing set to ensure the robustness of the model. The study population (subset 1, primary prediction) consisted of 24,107 patients to discriminate pulmonary infections (Table S1).

To advance the retrospective diagnosis of infection subtypes and prognosticate severe acute respiratory failure, we developed the multimodal system to predict four infections, specific infectious pathogens, and severe pneumonia utilizing distinct subsets (Figure S1). Subset 2 (infection classification, N = 13,361) was utilized to construct and validate the performance of the MMI system in predicting primary infections. The infections were categorized into four types: bacterial pneumonia (BP), fungal pneumonia (FP), viral pneumonia (VP), and PTB. Additionally, subset 3 (virus subtypes prediction, N = 2,520) and subset 4 (bacteria subtypes prediction, N = 535) were employed to identify the specific infectious pathogens associated with the viral and bacterial pneumonia. Furthermore, subset 5, consisting of 2,672 patients with severe pneumonia, was thoroughly examined. Severe pneumonia was defined based on criteria including clinical need for transfer to the intensive care unit (ICU), the requirement for mechanical ventilation, or death. These patients were identified based on their progression to critical outcomes after admission. This subset was specifically selected to evaluate the performance of our system in predicting critical outcomes that occurred 14 days after admission, specifically targeting patients at a higher risk of developing acute respiratory failure.

MMI system development

The proposed disease recognition system incorporated clinical medical records, laboratory test results, and CT images to expedite the identification of pulmonary infections. To effectively extract information from medical records text, we leveraged bidirectional encoder representations from transformers (BERT), an advanced network-based pre-training technique for natural language processing.^38,39 Additionally, to capture the spatial features of the CT image findings, we utilized image-based backbone networks known as Swin-transformer structures.^40,41 The Swin-transformer is an architecture based on transformer that hierarchically process input tokens. This hierarchical processing allows for the incorporation of larger patch sizes and deeper models, thus enhancing performance in image recognition tasks while maintaining computational efficiency.

To further bolster the performance and exploit the complementary information from textual and image features, this multimodal system integrated basic demographic details, chief complaints, laboratory test results, and CT scans to assess infectious diseases. Employing an attention architecture, we amalgamated the unimodal feature spaces learned from clinical and image data into a unified representation, forming the multimodal features. By extracting discriminative features from the clinical and image data, this system captured the specific information characteristics of each modality. These features were then integrated into a shared feature space, allowing the capture of intricate relationships and leveraging complementary information across different modalities. This integration augmented disease identification accuracy, empowering the MMI system to integrate multiple sources of information and effectively make precise predictions.

Accurate diagnosis of pulmonary infections

In this primary prediction task, our findings demonstrated that both clinical features and image features alone exhibited significant discriminative power in distinguishing infections from cases without infections, thus achieving a satisfactory baseline performance (Figure 2A; Table S2). Specifically, the clinical model achieved an AUC of 0.879 (95% CI: 0.870–0.885), while the image model achieved an AUC of 0.926 (95% CI: 0.922–0.930) on the internal testing cohort. These results underscore the potential of both clinical and image features as valuable discriminators for infections amid lower respiratory tract conditions. Nevertheless, integrating these features resulted in a substantial improvement in the performance of the MMI system. When combined, the accuracy of MMI system increased significantly to 0.849 (95% CI: 0.844–0.855), with a sensitivity of 0.866 (95% CI: 0.857–0.874), specificity of 0.838 (95% CI: 0.829–0.848), and AUC of 0.935 (95% CI: 0.932–0.939) (Table S2). These findings emphasized the critical role of multi-information integration and the effectiveness of fusion strategies in enhancing the accuracy of infection identification. The combination of clinical and image features furnished a more comprehensive and more robust framework to accurately discriminate pulmonary infections from other diseases with heightened accuracy.

Figure 2.

Performance of the MMI system in identifying patients with pulmonary infections

(A) Comparison of the receiver operating characteristic (ROC) curves of the clinical model, image model, and MMI system for detecting pulmonary infections in the validation, internal testing, and external testing datasets.

(B–D), The ROCs for predicting four pulmonary infection types (bacterial pneumonia, fungal pneumonia, viral pneumonia, and PTB) utilizing the clinical model (B), image model (C), and MMI system (D) in the validation, internal testing, and external testing datasets. BP, bacterial pneumonia; FP, fungal pneumonia; PTB, pulmonary tuberculosis; VP, viral pneumonia.

Remarkably, the multimodal system demonstrated exemplary performance in identifying infections within the external testing set. It achieved a mean accuracy of 0.919 (95% CI: 0.898–0.937), signifying a remarkably high level of overall correctness (Table S2). Additionally, the AUC exhibited a significant improvement, increasing from 0.830 (95% CI: 0.792–0.867) of the image model to 0.888 (95% CI: 0.856–0.916) of the MMI system. This notable increase further corroborated superior generalization ability of the MMI system in handling unknown data distributions and highlights its robustness in accurately distinguishing between infected and non-infected cases.

Performance of MMI system in single and mixed infections

It is notable that the current AI models tend to focus on single infection, often neglecting mixed infections. This narrow focus will potentially jeopardize their efficacy and impede their practical applications. To solve this problem, we utilized multilabel system to distinguish between single infection type and more than one infection type. This approach enabled us to assess the performance of the model in a more diverse and clinically relevant setting.

When comparing the diagnosis of single-infection patients to that of mixed infections, the latter posed greater challenges, as indicated in Table S3. Remarkably, the performance of the single infection model yielded an impressive AUC of 0.949 (95% CI: 0.943–0.954), while the mixed infections cases still achieved a respectable AUC of 0.876 (95% CI: 0.861–0.890). These results highlighted the capability of our multilabel multimodal model to deliver superior performance in both the relatively simpler cases of single infection and the more complex cases of mixed infections.

Precise classification of infectious diseases

Pulmonary infections encompass various subtypes, such as bacterial, fungal, and viral pneumonia and PTB. Nevertheless, accurate differentiation of respiratory tract infections presents a formidable challenge, even for experienced physicians in tertiary hospitals. When considering clinical features alone, the model achieved an accuracy of 0.788 (95% CI: 0.777–0.798) and an average mean AUC of 0.850 (95% CI: 0.844–0.856) in the internal testing dataset (Figures 2B; Table 1). In a similar vein, exclusively employing image features, the model achieved an accuracy of 0.821 (95% CI: 0.812–0.830) and an average mean AUC of 0.894 (95% CI: 0.890–0.900) (Figure 2C). Significantly, amplification of the model’s efficacy by supplementing information and purging noisy or incomplete data from these modalities culminated in peak performance. The MMI system realized an accuracy of 0.848 (95% CI: 0.838–0.856), a sensitivity of 0.846 (95% CI: 0.830–0.860), a specificity of 0.847 (95% CI: 0.837–0.858), and an AUC of 0.910 (95% CI: 0.904–0.916) in the internal testing dataset (Figure 2D). In external testing dataset, the AUC of the MMI system yielded 0.887 (95% CI: 0.867–0.909). These results demonstrated the iterative improvements achieved by optimizing the model and incorporating relevant information. By refining the fusion of multiple modalities, we obtained optimal performance in accurately identifying and differentiating the subtypes within the four-class task. A delineation of performance breakdown for the individual four-class task is presented in Table 1 and Figure S2.

Table 1.

Performance of the MMI system in identifying BP, FP, VP, and PTB

	Dataset	Sensitivity (95% CI)	Specificity (95% CI)	Accuracy (95% CI)	AUC (95% CI)
Clinical model	Validation	0.830 (0.816–0.847)	0.760 (0.747–0.772)	0.786 (0.777–0.795)	0.860 (0.855–0.867)
	Internal testing	0.785 (0.764–0.804)	0.779 (0.767–0.792)	0.788 (0.777–0.798)	0.850 (0.844–0.856)
	External testing	0.706 (0.644–0.765)	0.875 (0.846–0.904)	0.788 (0.755–0.821)	0.840 (0.824–0.860)
Image model	Validation	0.838 (0.822–0.854)	0.832 (0.820–0.843)	0.832 (0.824–0.842)	0.896 (0.891–0.901)
	Internal testing	0.847 (0.829–0.861)	0.806 (0.795–0.819)	0.821 (0.812–0.830)	0.894 (0.890–0.900)
	External testing	0.883 (0.853–0.920)	0.711 (0.674–0.749)	0.755 (0.727–0.788)	0.856 (0.840–0.872)
MMI system	Validation	0.836 (0.820–0.853)	0.850 (0.840–0.861)	0.846 (0.838–0.855)	0.905 (0.900–0.910)
	Internal testing	0.846 (0.830–0.860)	0.847 (0.837–0.858)	0.848 (0.838–0.856)	0.910 (0.904–0.916)
	External testing	0.880 (0.819–0.944)	0.800 (0.766–0.839)	0.835 (0.807–0.865)	0.887 (0.867–0.909)

Multimodal data interconnection

The primary objective of the multimodal system was to elucidate associations and shared information across diverse modalities. Then we aimed to gain new insights into the AI-based diagnosis of pulmonary infections and potentially identify new biological signatures conducive to this process. To accomplish this, a spectrum of various fusion approaches was employed, including early fusion, intermediate fusion, and late fusion, and we conducted extensive experiments to evaluate their effectiveness (Figure S3).

In these experiments, cross-attention fusion methods were employed to amalgamate the image and clinical results into a single layer, thereby engendering multi-information, patient-level result features (Figure 3). The classic early fusion model achieved a mean AUC of 0.910 (95% CI: 0.904–0.916) coupled with 0.848 (95% CI: 0.838–0.856) accuracy in the internal testing dataset. It is noteworthy that the intermediate fusion approach yielded a preeminent AUC of 0.923 (95% CI: 0.919–0.927) and an accuracy of 0.870 (95% CI: 0.862–0.879). Similarly, the late fusion approach garnered an AUC of 0.917 (95% CI: 0.913–0.923) and an accuracy of 0.859 (95% CI: 0.852–0.868) (Figure S4; Table S4). Overall, the intermediate approach evinced superior performance compared with the classic early fusion and late fusion models in both validation cohorts for the infections identification tasks. These results highlighted the effectiveness of the intermediate fusion approach in integrating multimodal data and elevating the accuracy for infection identification.

Figure 3.

Performance of the MMI system using different fusion approaches in identifying pulmonary infections types

(A) Structure diagram of different fusion strategies: early fusion, intermediate fusion, and late fusion. Further details were provided in Figure S4.

(B) Evaluation of the three fusion approaches to predict pneumonia types (average performance, bacterial pneumonia, fungal pneumonia, viral pneumonia, and pulmonary tuberculosis) in the internal testing dataset. The performance measures were visualized through boxplots representing the statistical distribution of AUCs, including maximum, minimum, median, and upper and lower quartiles of a set of data (first/third quartile) as well as outliers. A t test was used to calculate the p value between approaches: ∗∗∗∗p < 0.0001.

Comparison of the MMI system against physicians

We conducted a comparative study involving four board-certified physicians divided into two groups based on their experience (the junior group, consisting of physicians with less than 10 years of professional experience, and the senior group, consisting of physicians with over 10 years of experience). In order to ensure a fair comparison, all physicians were provided with a uniform set of information encompassing CT scans, demographic details, chief complaints, and laboratory assay results. The dataset comprised 400 cases including 94 cases of viral pneumonia, 55 cases of fungal pneumonia, 108 cases of bacterial pneumonia, and 143 cases of PTB. Each physician was charged with the independent assessment of whether patients had been diagnosed with one of the four distinct infectious diseases. Subsequently, their performance was juxtaposed with that of the MMI system. The results indicated that the AI system surpassed each junior physician and demonstrated performance comparable to the senior physicians (Figure 4). Furthermore, the superiority of the AI system’s performance was evident numerically as well. When comparing the weighted errors in patient diagnosis between the AI system and the physicians, the AI system yielded 13.52%. Conversely, the range of weighted errors by expert physicians varied from 6.45% to 25.17%, with a mean of 8.98% for the senior group and 24.10% for the junior group (Table S5).

Figure 4.

Benchmarking the MMI system vs. physicians in diagnosing pulmonary infections

(A) The diagnostic performance of the MMI system and physicians for BP, FP, VP, and PTB.

(B) Weighted error results based on penalty scores.

(C–E), The confusion matrix of the MMI system (C), junior physicians (D), and senior physicians (E) to differentiate among various types of pulmonary infections.

Pathogen identification of pulmonary infections

Furthermore, the MMI system also had the capability to play a pivotal role in pinpointing specific pathogens to aid precision treatment decisions. Subset 3 covered a total of 11,375 CT scans from 2,520 patients, all of whom underwent molecular testing for a spectrum of 9 common respiratory viruses: influenza B virus, human metapneumovirus (HMPV), coronavirus, parainfluenza virus, respiratory syncytial virus (RSV), influenza A virus, H1N1pdm09, adenovirus, and rhinovirus. Subset 4 comprised 2,278 CT scans from 535 patients who had been diagnosed with 9 bacterial subtypes: methicillin-resistant Staphylococcus aureus (MRSA), Mycobacterium tuberculosis, Pseudomonas maltophilia, Escherichia coli, Haemophilus influenzae, Klebsiella, Staphylococcus, Pseudomonas aeruginosa, and Acinetobacter baumannii. This comprehensive testing approach ensured the accurate identification of pathogens. For the challenging task of discerning viruses and bacteria, the system utilized transfer learning alongside a pre-trained transformer architecture initially trained on different tasks involving four types of infections and then meticulously fine tuned on specific datasets to recognize the distinctive features and patterns indicative of pathogens from CT images and clinical records.

The performance of the MMI system in accurately identifying viral infections from CT images was exceptional, as evidenced by its mean AUC score of 0.850 (95% CI: 0.839–0.859) on the validation set and 0.822 (95% CI: 0.805–0.837) on the testing set for the overall nine-way classification (Figures 5A and 5B). This remarkable performance underscored the high accuracy and reliability of the system in correctly identifying cases of viral pneumonia with minimal false positives. To gauge the discriminatory power of the model, we calculated the AUC for each viral subtype class, which quantified the ability of the MMI system to differentiate between viral pneumonia cases and other individuals, with AUC values spanning from 0.700 to 0.949 across the nine subgroups, which showcased its excellent capability to differentiate among the various virus subtypes.

Figure 5.

Performance of our AI system in identifying different subtypes of viruses and bacterial subtypes

(A) Mean ROCs demonstrating MMI system accuracy in identifying viral infections within the internal validation and testing cohorts.

(B) Boxplots showing MMI system performance for classifying nine viral subtypes.

(C) Mean ROCs highlighting MMI system accuracy in identifying bacterial infections within the internal validation and testing cohorts.

(D) Boxplots presenting MMI system performance for classifying nine bacterial subtypes. A.baumannii, Acinetobacter baumannii; E.coli, Escherichia coli; HMPV, human metapneumovirus; H.influenzae, Haemophilus influenzae; MRSA, Methicillin-resistant Staphylococcus aureus; MMI, multimodal integration;M.tuberculosis, Mycobacterium tuberculosis; P.maltophilia, Pseudomonas maltophilia;  P.aeruginosa, Pseudomonasaeruginosa; RSV, respiratory syncytial virus.

By leveraging our multimodal AI system, we processed the CT images and generated predictions for the specific pathogen subtypes in each scan (Figures 5C and 5D). To evaluate the performance of our system across different bacterial subtypes, we utilized boxplot visualizations to juxtapose the AUC scores obtained from the system on both the validation cohort and testing cohort. The specific bacteria diagnostic system achieved an AUC of 0.847 (95% CI: 0.834–0.861) and 0.803 (95% CI: 0.775–0.830) on the validation and testing cohorts. The MMI system detected MRSA with an AUC exceeding 0.900. The MMI system could accurately distinguish among different bacteria subtypes based on CT images and clinical records, holding promise for aiding healthcare workers to make prompt and accurate diagnoses.

Prognosis prediction for critical illness

Regarding more actual clinical deployment, the MMI system demonstrated extraordinary performance in identifying patients with severe pneumonia. The multimodal model exhibited a robust ability to differential among varying degrees of disease severity, with an AUC of 0.918 (95% CI: 0.905–0.930) in the internal testing dataset (Figures 6A and 6B), paving the way for a deeper understanding of the critical factors influencing disease prognosis. Then, through comprehensive analysis, we estimated the most influential factors that played a crucial role in the progression toward critical illness, emphasizing the necessity of considering multiple data dimensions when assessing patient outcomes (Figures 6C and 6D). Notably, non-quantized multimodal features score (M-score) played a significant role in predicting the progression to critical illness, aligning with existing research identifying them as determinants of disease outcomes. Liver biochemical markers, including albumin, serum lactate dehydrogenase (LDH), and indirect bilirubin, emerged as important prognostic indicators, elucidating the association between liver dysfunction and adverse clinical outcomes.⁴² Coagulation markers, including thrombin time (TT), activated partial thromboplastin time (APTT), and platelet count, were also prognostic factors, reflecting disturbances in the coagulation system observed in critical illness.⁴³ Furthermore, markers related to electrolyte and acid-base balance, alongside inflammation markers including C-reactive protein, lymphocyte count, and neutrophil count, contributed significantly to predicting clinical prognosis, emphasizing their importance in assessing disease severity and predicting outcomes, and were consistent with previous research.⁴⁴

Figure 6.

Risk factors and clinical prognosis analysis for critical illness

(A) ROC for a binary classification of progression to critical illness in the internal testing dataset.

(B) Diagnostic accuracy for critical illness progression prediction represented by a confusion matrix.

(C and D) Predictive features for critical illness progression through Shapley Additive exPlanation (SHAP) value analysis.

(E) Correlative analysis of predictive features influencing progression to critical illness.

(F) When patients were stratified into high-risk and low-risk groups, the Kaplan-Meier curves demonstrated a significant difference in survival probabilities.

These analyses revealed that clinical parameters and coagulation markers played complementary roles in prognosis, without significant correlations observed among the leading prognostic indicators (Figure 6E). In a specific clinical case, elevated levels of TT and indirect bilirubin levels were found to be associated with more severe outcomes. Notably, LDH and APTT factors were particularly valuable for predicting non-critical disease. The high-risk group, characterized by these prognostic markers, exhibited significantly lower survival probability than the low-risk group, with a statistically significant p value of less than 0.001 based on the log-rank test (Figure 6F). These findings not only reaffirmed the expansive capability of the multimodal model in predicting critical illness but also pinpointed the specific markers pivotal for an accurate prognostic assessment using real-world examples (Figure 7). Although CT images in critically ill patients had shown specific manifestations, AI system recognized unique features that were previously unreadable to the human eye. These results indicated that accurate prognostic prediction required an integration of clinical data with lung lesion characteristics. This synergy enhanced our understanding and aided in the development of more accurate prognostic predictive tools.

Figure 7.

Illustration of the AI system for prognosis estimation of pneumonia patients

(A and B) Real-world cases for prognostic evaluation. One patient with critical illness (patient 1) and the other with non-critical disease (patient 2), to demonstrate how lung lesion characteristics and clinical features serve as input variables to predict prognosis. The influence of these inputs on risk assessment was depicted, with pink indicators increasing the projected risk on the right and blue indicators decreasing it on the left.

(C) Comparative CT scans of non-critical and critical pulmonary infection cases.

Discussion

In this study, the MMI pipeline was constructed to facilitate accurate diagnosis of pulmonary infections based on a large-scale dataset and tested on an external cohort. Integrating radiological, clinical, and laboratory multimodal features, the MMI system enabled discriminative diagnosis of bacterial, fungal, and viral pneumonia and PTB. Meanwhile, a critical illness prediction model incorporating deep learning features and multidimensional parameters was established to provide timely intervention for high-risk patients. This pipeline is poised to significantly expedite the diagnostic process for pulmonary infectious diseases and mitigate the overall disease burden.

Pulmonary infections constitute a substantial global disease burden, epitomized by the high morbidity and mortality rates. Obstacles in ascertaining the pathogens lead to the misuse of antibiotics, thereby exacerbating the proliferation of drug-resistant bacteria.⁴⁵ Existing studies mostly focus on the diagnosis of specific pathogens, which is crucial for decision-making regarding specific types of pneumonia in pandemics.^46,47,48,49 A deep learning-based automated detection algorithm developed using 60,989 CXR scans was used to identify active PTB and performed better than human experts.⁵⁰ Another AI model based on a multicountry chest CT dataset automatically located the parietal pleura and lung parenchyma and classified COVID-19 pneumonia patients with an accuracy of 90.8%.⁵¹ However, there are still frequent types of pneumonia that require accurate and rapid diagnosis. A model based on 432 CT scans was developed to distinguish active PTB from community-acquired pneumonia.⁵² Recently, the Pneumonia-Plus model based on a dataset of 2,763 patients was leveraged to identify bacterial, fungal, and viral pneumonia with a mean AUC of 0.822.⁵³ By contrast, our study has advanced the field by developing an AI model that not only broadens the spectrum of detectable diseases to include bacterial, fungal, and viral pneumonia and PTB but also achieves superior diagnostic precision with an elevated AUC of 0.910. Moreover, the MMI system precisely targeted frequent pneumonia pathogens, thus optimizing therapeutic interventions. Despite the constrained dataset of patients with partially pathogen-positive infections resulting decent preformance for pathogen prediction, these insights can still function as a critical reference for clinicians, guiding the decision-making for pathogen-specific treatments to enhance antibiotic efficacy.

Meanwhile, accurate disease diagnosis necessitates integrated multimodal data analysis, encompassing clinical features, imaging, and laboratory tests.^25,54,55 However, previous studies were often limited by the relatively small number of labeled samples.^30,32 Here, our approach entailed the construction of a multimodal fusion AI model, leveraging a substantial dataset, which yielded performance on par with advanced physicians. The MMI system architecture was refined based on IRENE, a transformer-based representation learning model with unified processing of multimodal input for clinical diagnostics.²⁷ Unprocessed laboratory test indicators, clinical complaints, demographic information (such as age and sex), and chest images were automatically coded and embedded. It demonstrated superior diagnostic accuracy for pulmonary diseases compared with the single-dimensional model (AUC: 0.887 vs. 0.840). In this study, this normalized multimodal fusion architecture was employed to diagnose pulmonary infections, which also surpassed the performance of single-dimension models. The development of AI technologies, including large language models, heralds a future where medical AI models will incorporate a broader array of clinically significant multimodal data, such as breath sounds, thereby facilitating precise medical interventions.^56,57,58

Then we compared three feature fusion strategies: early, late, and intermediate fusion. Different fusion architectures could affect the diagnostic efficiency of the model.^59,60,61 Early fusion utilized global information from different sources, providing a holistic feature representation. However, it may also introduce redundancy or inconsistency, given the crucial disparities among information sources, potentially overlooking unique information specific to each source. Late fusion preserved the independence of each information source, but it lacked the capacity to share global information during the feature extraction phase, potentially hindering the exploitation of local advantages from one source to another. In contrast, intermediate fusion sought to merge features from multiple sources during the feature extraction process, generating a unified feature set for the final classification. This approach enhanced information propagation among sources and interactively optimized the feature extraction process.^60,61 Moreover, a multistage fusion strategy was implemented in the proposed intermediate fusion architecture, allowing multimodal features to interact throughout the entire feature extraction process.

In addition, early warning of severe illness remains another tough challenge in the management of pulmonary infections. Previous studies have confirmed that radiomics could quantify the severity of pneumonia, and multiomics biomarkers, including clinical parameters, microbiome signatures, and radiomics, have been identified as prognostic indicators for patients with pneumonia.^{62,63,64,65,66,67} For example, a regression model constructed with age, BMI, CD4⁺ T lymphocytes, and serum interleukin-6 levels enabled effective prediction of the risk of developing severe disease in patients with viral pneumonia.⁶² The presence of Staphylococcus, Ralstonia, and Enterococcus was strongly associated with acute respiratory distress syndrome by microbiome investigations.⁶³ Additionally, a neural network model based on CT images was utilized to assess disease severity in patients with viral pneumonia (AUC: 0.75).⁶⁷ The present study has pinpointed multiple risk factors from multimodal characteristics as potential predictors of severe pneumonia. The integration of diverse data sources will allow for a comprehensive approach to predicting critical illness, paving the way for effective risk stratification of patients and preemptive medical interventions.

Notably, co-infection with multiple pathogens increases the public health burden.⁶⁸ The present study provides an initial exploration of predicting co-infection, but the intricate mechanism by which mixed infections affect imaging performance remains unclear. The advent of single-cell technology makes it possible to unravel the dynamic interplays in the disease microenvironment, which may be a promising opportunity.^{69,70,71,72,73} Several studies have elucidated that T cells and natural killer cells in patients with bacterial pneumonia complicated by sepsis exhibited activation and depletion characteristics, respectively, and that the proportion of plasma cells is significantly increased.⁷¹ In contrast, cell subsets such as DC_c4-LILRA4, B_c05-MZB1-XBP1, and Neu_c3-CST7 in viral pneumonia are closely associated with disease severity.⁷² Furthermore, gene expression profiles have revealed distinct patterns: CCL7, CCL8, CCL13, and IFIT2 are highly expressed in viral pneumonia, while BAG3, HIF1A and IL1B are upregulated in other types of pneumonia (including bacterial infections, etc.), and CCL2, RNASE1, and MAFB genes were highly expressed in different types of pneumonia.⁷³ These specific molecular events forge new insights for the deep excavation of biological mechanisms underpinning macroscopic imaging observations, providing a pivotal foundation for more nuanced diagnostic and therapeutic decision-making.

This study had several limitations that should be acknowledged. First, mixed infections present as a prevalent yet intricate challenge in clinical scenarios. We constructed a preliminary prediction model, but quantitative evaluation of mixed infections needs to be explored in conjunction with high-throughput molecular detection methods. Second, the practical application of AI models necessitates enhancements of both usability and accuracy. The integration of a predictive AI model with a clinical workflow must be seamless to achieve optimal results.⁷⁴ The existing models still require further optimization to meet the demands of medical applications. Last but not the least, future research should focus on validating these findings using a broader range of central datasets, in this way reinforcing the generalizability of the AI system.

In summary, this study pioneered a multitask model based on multimodal clinical data from over 20,000 patients. The MMI system fused clinical, imaging, and laboratory assay features to diagnose various types of pulmonary infections and pathogens. It also incorporated key indicators to warn critically ill patients. This comprehensive approach could not only enhance the accuracy of diagnostics, but also guide the accurate use of medication in clinical settings. It holds the potential to evolve into a robust tool for precise interventions among patients with pulmonary infections, propelling the advancement into a new epoch of clinical care.

Materials and methods

For materials and methods, see the supplemental information.

Published Online: May 22, 2024

Lead contact website

https://yjs.cd120.com/contents/498/1777.html.