Multimodal text guided network for chest CT pneumonia classification

Abstract

Pneumonia is a prevalent and serious respiratory disease, responsible for a significant number of cases globally. With advancements in deep learning, the automatic diagnosis of pneumonia has attracted significant research attention in medical image classification. However, current methods still face several challenges. First, since lesions are often visible in only a few slices, slice-based classification algorithms may overlook critical spatial contextual information in CT sequences, and slice-level annotations are labor-intensive. Moreover, chest CT sequence-based pneumonia classification algorithms that rely solely on sequence-level coarse-grained labels remain limited, especially in integrating multi-modal information. To address these challenges, we propose a Multi-modal Text-Guided Network (MTGNet) for pneumonia classification using chest CT sequences. In this model, we design a sequential graph pooling network to encode the CT sequences by gradually selecting important slice features to obtain a sequence-level representation. Additionally, a CT description encoder is developed to learn representations from textual reports. To simulate the clinical diagnostic process, we employ multi-modal training and single-modal testing. A modal transfer module is proposed to generate simulated textual features from CT sequences. Cross-modal attention is then employed to fuse the sequence-level and simulated textual representations, thereby enhancing feature learning within the CT sequences by incorporating semantic information from textual descriptions. Furthermore, contrastive learning is applied to learn discriminative features by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. Extensive experiments on a self-constructed pneumonia CT sequences dataset demonstrate that the proposed model significantly improves classification performance.

Introduction

Pneumonia is a serious respiratory disease with symptoms ranging from fever and cough to respiratory failure in severe cases. Timely and accurate Computed Tomography (CT) imaging technology is crucial for early diagnosis and treatment¹. While CT provides detailed lung information, manual interpretation by radiologists is time-consuming and susceptible to subjective biases, resulting in diagnostic inconsistencies^2,3. Moreover, pneumonia caused by different pathogens exhibits varied CT features, making accurate classification and diagnosis highly challenging. Viral pneumonia, including COVID-19, typically presents as ground-glass opacities with multifocal or patchy patterns. Bacterial pneumonia commonly manifests as consolidation, characterized by increased density in the alveoli filled with exudate or inflammatory fluid, resulting in a homogeneous and dense appearance. Additionally, viral pneumonia may exhibit asymmetric lesions on CT images, often affecting one or both sides of the lungs unevenly. In contrast, lesions in bacterial pneumonia may be more symmetric, typically appearing similarly on both sides of the lungs. Figure 1 presents chest CT images of bacterial pneumonia, viral pneumonia, and a healthy individual from top to bottom.

Fig. 1.

The images in chest CT sequence of different pneumonia patients. The three rows show the lung images of bacterial, viral patients and healthy persons from top to bottom, respectively.

With the advancement of artificial intelligence (AI), AI technology can automatically learn and extract features from CT images, enabling faster recognition of different types of pneumonia. Traditional methods rely on manual feature extraction with classifiers like decision trees or SVMs^4–6 , but their dependency on expert knowledge limit performance. Deep learning, particularly convolutional neural networks (CNNs), has significantly advanced automated diagnosis by learning hierarchical features from images^7–9. For instance, ResNet-based models aggregated predictions from selected slices¹⁰, while dual-branch networks integrated segmentation and classification¹¹. Pneumonia lesions in CT sequences dynamically change and may spread to surrounding areas of the lung as the condition progresses. These lesions exhibit direct or indirect interactions. Graph neural networks (GNNs) have shown excellent performance in capturing correlations between nodes¹², making them useful for extracting relevant information in medical image analysis. In¹³, the authors constructed a graph network by calculating the Euclidean distance between feature vectors of local lesion regions to predict pneumonia types from CT slices. Mao et al. proposed a graph convolutional network framework called ImageGCN¹⁴, aiming to better represent X-rays by considering their relationships. ImageGCN utilized relational information, such as gender and age, to enhance image representation. This framework was applied to disease recognition tasks in chest X-ray images, demonstrating superior performance in both disease recognition and localization tasks. The above studies employ CNNs and GNNs to construct automated pneumonia classification methods based on CT slices, which often emphasize global features while neglecting local lesion details¹⁵. In addition, slice-based methods overlook the spatial contextual information in CT sequences¹⁶.

Slice-based classification requires doctors to label each CT slice, which is time-consuming. In contrast, sequence-based classification only needs annotations at the sequence level. As a result, some researchers have focused on using CT sequences for pneumonia classification. Wang et al.¹⁷ proposed DeCovNet, a weakly supervised deep learning framework for COVID-19 classification and lesion localization using 3D CT volumes. They first segmented lung regions with a pre-trained UNet, then fed the 3D lung data into a deep neural network to predict COVID-19. To reduce the cost of manual annotation and enhance model interpretability, Han et al.¹⁸ proposed AD3DMIL, an attention-based 3D multi-instance learning method. It generates 3D instances around potential infection areas and uses attention pooling to assign weights, predicting COVID-19 positivity through a Bernoulli distribution. Wang et al. proposed a joint deep learning model for 3D lesion segmentation and classification, establishing a large CT database to reveal differences in lesions between COVID-19 and community-acquired pneumonia (CAP).

Pneumonia diagnosis based on CT sequence-level data lacks detailed label information, resulting in lower diagnostic accuracy compared to slice-based diagnosis. Recently, using multi-modal data—such as clinical symptoms, diagnostic reports, and other medical records—has become a growing trend^19,20. Wang et al.²¹ combined multi-channel images (lung window images, high absorption images, low absorption images) to provide more visual features. They treated chest CT scans as video sequences and used recurrent neural networks (RNNs) to automatically extract multiple image features from multi-channel image slices, thereby improving image analysis efficiency. While combining diagnostic report semantics with CT sequence visuals can improve pneumonia diagnosis, it initially requires radiologists to generate diagnostic reports based on CT images, which inevitably increases their workload. To accurately simulate the clinical workflow, the model should rely exclusively on CT sequences during the testing phase. However, previous studies that employed multimodal data for training have largely overlooked the importance of performing unimodal testing.

Therefore, this paper introduces a multi-modal text-guided pneumonia classification algorithm that leverages CT sequences and incorporates these descriptive reports. The algorithm utilizes these reports as training data to enhance the learning of CT sequence features, thereby improving the model’s classification performance. In summary, our main contributions are as follows:

• We design the Sequential Graph Pooling Network (SGPNet) for CT sequence encoding, which gradually selects important slice features as nodes to build a new sub-graph and obtain a CT sequence-level representation. This approach fully considers the temporal relationships between slices and emphasizes significant slices.

• Modal transfer aims to generate simulated textual features from CT sequences, designed for multi-modal training and single-modal testing. This module not only simulates the process of actual clinical diagnosis but also fully utilizes the intuitive semantic information in CT descriptions. To effectively utilize multi-modal data, cross-modal attention is proposed to facilitate interaction between CT sequences and text descriptions.

• To maximize the similarity between positive sample pairs across different modalities and minimize the similarity between negative samples, this section has designed a Siamese network for multi-modal contrastive learning. The experimental results show that the MTGNet model significantly improves pneumonia classification accuracy of CT sequences and provides an effective method for pneumonia classification tasks based on coarse-grained CT sequence labels.

Materials and methods

Dataset and preprocessing

The data collected from chest CT scans is saved in DICOM. In the preprocessing process, DICOM data was converted into PNG format through preprocessing. The resulting images have a resolution of 512 × 512 pixels and contain various key parameters, including image resolution, density values, scanning parameters and other patient-related information. We perform center cropping on the image by extracting a square within an inner circle, resulting in a 362*362 pixel image. This step eliminates most irrelevant parameter information while preserving data pertinent to the patient’s lung window. Finally, to accommodate neural network training, the image is resized to 224*224 pixels.

In the experiment, CT sequences often contain lung windowless slices, typically all-black images, at the beginning and end, which provide no meaningful diagnostic information. Therefore, these slices were manually removed during data preprocessing to enhance the quality and relevance of the input data. Then take 32 slices from each patient’s CT sequence at regular intervals.

Table 1 lists the relevant information of the multi-modal dataset. In the experiments, the dataset was split into 70% training, 15% validation, and 15% testing sets using stratified random sampling.

Table 1.

The constructed multi-modal dataset.

Category	CT sequences	CT diagnosis reports
Virus	302	302
Bacterial	280	280
Health	222	222
Total	804	804

Ethical statement

All methods were performed in accordance with the relevant guidelines and regulations. Since we used anonymized data, and did not constitute human subject research, the need for written informed consent was waived by Beijing University of Technology Ethics Committee on Science and Technology. This study was approved by the Beijing University of Technology Ethics Committee.

Framework of MTGNet

As illustrated in Fig. 2, the proposed MTGNet model consists of five components: CT sequence encoder based on sequential graph pooling network (SGPNet), CT description encoder, modal transfer, cross-modal attention and multi-modal contrastive learning. Firstly, we construct feature extraction networks for CT sequences and CT descriptions. Here, we have designed the Sequential Graph Pooling Network (SGPNet) for CT sequence encoding, which gradually selects important slice features as nodes to build a new sub-graph and obtain a CT sequence-level representation. The CT description encoder was designed to learn a representation of the CT description for text-based pneumonia classification. Subsequently, to achieve multi-modal training and single modal testing, we constructed a modal transfer module to generate representations of simulated text features from CT sequence feature representations. To effectively utilize multi-modal data, cross-modal attention is proposed to facilitate interaction between CT sequence and text descriptions. Finally, we applied multi-modal contrastive learning to maximize the similarity between positive sample pairs across different modalities and minimize the similarity to facilitate representation learning of multi-modal data between negative samples.

Fig. 2.

The architecture of the MTGNet model.

CT sequence encoder based on SGPNet

To obtain feature representations at the CT sequence level, this study proposes a Sequential Graph Pooling Network (SGPNet) to encode CT sequences. Figure 3 illustrates the structure of SGPNet. Each CT slice in the entire CT sequence has its slice-level global features extracted by a backbone network. Subsequently, each slice-level feature is treated as a node for constructing an attentional temporal graph. SGPNet can gradually select important slice features as nodes to build a new graph, transforming the attentional temporal graph into a subset of the initial graph. Through several iterations of this selection process, a representation of CT sequence-level features is ultimately obtained.

Fig. 3.

The architecture of SGPNet, including attention sequence graph and the node pooling layer.

Backbone

For each slice in CT sequence, the feature representation at the slice level is obtained from the output of the last fully connected layer before the final layers of ResNet-50. After removing redundancy, this process results in a feature matrix with 32 slices, each having a slice-level feature dimension of 2048.

SGPNet

As illustrated in Fig. 3, SGPNet consists of two main components: constructing attentional sequential graphs and node pooling layers. For constructing attentional sequential graphs, we utilize the -dimensional feature matrix , which represents sequence features obtained from backbone, to build a graph . In this graph denotes the adjacency matrix, indicating whether a slice is connected to the slices before or after it, thus facilitating the handling of dynamic temporal data. Specifically, if node i corresponds to a slice adjacent to the k slices before and after, then are set to 1. Otherwise, they are set to 0. Figure 4 illustrates a schematic diagram of the adjacency relationships for slices with index 124, where . The updated feature embedding , after passing through the graph network, is defined as follows:

1

here represents the set of adjacent nodes to node and denotes the parameter matrix. Additionally, denotes the normalized importance weight coefficient of node relative to node , calculated as follows:

2

where is:

here represents the importance representation of node with respect to node and a is a weight vector. By introducing the attention mechanism, the model can better represent the temporal relationship between adjacent slices.

Fig. 4.

Neighbor relationship of slices for constructing adjacency matrix.

The node pooling layer is designed to select feature nodes corresponding to key slices for constructing a new subgraph. This layer operates by gradually removing non-critical nodes using a trainable projection vector . First, all nodes X in the graph are projected onto the vector p, yielding . The element measures how much information node can retain in the direction of vector p. Next, y is sorted, and the top l values are selected, resulting in the index set , where denotes the position indices of the selected slices in the CT sequence. The adjacency matrix of the new graph is then extracted by indexing: and the feature matrix is updated to . Finally, is computed, and the pooled feature matrix representation is updated to . After applying several sequential graph pooling modules, only one node remains in the final graph, serving as the ultimate feature representation of the CT sequence. This representation aggregates information from all slices in the sequence and is subsequently used for final classification.

The attention sequential graph aggregates feature information between adjacent slices by weighting and propagating the information of each node to its neighboring nodes. This process minimizes the information loss that could occur during node pooling operations. The node pooling layer allows the projection vector p to be trained through backpropagation, which enhances the network’s flexibility and maximizes the retention of information from all slices.

CT description encoder

Figure 5 illustrates the structure of the CT description encoder. The CT reports are segmented using PKUSEG²², and each segment is processed to 100 words as model input. Word representations are initialized using 300-dimensional embeddings, pre-trained on a domain-specific biomedical corpus to capture relevant pharmaceutical semantics. These embeddings form a word embedding matrix for subsequent processing. To encode the textual information, we integrate a Bidirectional Long Short-Term Memory (Bi-LSTM) network with TextRCNN²³. The Bi-LSTM captures contextual semantics by modeling both forward and backward dependencies, which is essential for identifying diagnostic patterns such as lesion progression and spatial relationships (e.g., “asymmetric lower lobe lesions”). Following the Bi-LSTM, we apply TextCNN to extract hierarchical features. TextRCNN combines bidirectional recurrent networks, which learn long-range dependencies (e.g., symptom-lesion correlations), with convolutional layers that identify salient local patterns, effectively balancing global and local semantic features. The final output is an 812-dimensional feature vector, which serves as the compact semantic representation of the CT description.

3

here represents the number of classes: viral pneumonia, bacterial pneumonia, and healthy. The function is an indicator function that takes values of 0 or 1. It equals 1 if the true class of sample i is c, and 0 otherwise. denotes the predicted class probability distribution for the i-th sample and the c-th class.

Fig. 5.

The structure of CT description encoder.

Multi-modal contrastive learning

To maximize the similarity between positive sample pairs in different modalities and minimize the similarity between negative samples, this section has designed a Siamese network for multi-modal contrastive learning. The main principle of Siamese networks is to extract features from the paired input data— the CT sequence feature representation and the CT description representation —through a shared subnetwork. These paired representations are then input into a metric learning layer, where their similarity is measured using Euclidean distance. Subsequently, the two feature representations are further dimensionally reduced through the shared fully connected layer, using multiple linear layers and ReLU activation functions to gradually reduce the dimension to 32. The results obtained after unifying and into the same feature embedding space are represented as and , where is 32. The multi-modal contrastive loss can be calculated as follows:

4

here, the label indicates whether a sample pair is positive or negative: represents a positive sample pair, while represents a negative sample pair. The Euclidean distance of the sample pair is denoted by . The margin is a hyperparameter and set to 2 in this paper.

Modal transfer

We propose a modality transfer module to enable multi-modal training and single-modal testing. Specifically, during training, both visual and textual modalities are used as inputs, while during testing, only the visual modality (i.e., the CT sequence) is required for prediction. This not only simulates the process of actual clinical diagnosis but also fully utilizes the more intuitive semantic information features in CT description. To achieve this, we define a modality transfer function to explicitly learn the transition from the visual modality to the textual modality. Here, represents the visual feature of the CT sequence, while is the CT description representation and is a simulated textual representation approximately generated from the visual features via modality transfer to implicitly describe the content of the CT sequence. The parameter denotes the parameters of the modality transfer function. The encoders of CT sequence and CT description respectively yield a 2048-dimensional feature representation and an 812-dimensional CT description feature representation, thus we design the modality transfer function as a two-layer fully connected network. The function is trained using mean square loss:

5

Cross-modal attention

In the above section, we generated a simulated textual representation from the visual modality representation. However, the ultimate goal of this study is to classify pneumonia using a single visual modality. Therefore, it is necessary to integrate the information from the simulated textual representation into the feature representation of the CT sequence to achieve text-guided effects. In this section, we utilize a cross-modal attention mechanism to facilitate interaction between the CT sequence and the CT description.

The calculation process of cross-attention is depicted in Fig. 6. Firstly, the dimensionality of the CT sequence feature and the simulated textual representation are adjusted to and , respectively. According to the attention mechanism, it is necessary to determine the query () vector, key () vector, and value () vector:

where , , are weight matrices, , , are bias terms.

Fig. 6.

Cross-modal attention mechanism.

After multi-head attention operation, each attention head generates a result enhanced by textual information:

6

where represents the output of the i-th attention head. Subsequently, the outputs of all attention heads are concatenated,

7

here is the number of attention heads. Eventually, to obtain a fused representation of the simulated textual representation and the CT sequence feature representation, the result of the fusion from multiple heads is projected back to the original representation space,

8

To obtain the enhanced feature representation of the CT sequence, the final output is then adjusted back to .

The final pneumonia class prediction is performed at the sequence level, and the training loss for this part is ultimately represented as:

9

Throughout this process, the visual feature representation of the chest CT sequence is enhanced by integrating semantic information from the CT description, thus improving the model’s ability to express features from the CT sequence.

Loss optimization

With the introduction of the above modules, the model is trained end-to-end. The objective function consists of minimizing the following components: the text classification loss of the CT description encoder, the modality transfer loss, the CT sequence classification loss after multi-modal information fusion, and the multi-modal contrasting loss. Therefore, the loss function for the model is defined as follows:

10

here , , and are hyperparameters representing the weights assigned to each loss term, set to 0.4, 0.3, and 0.3, respectively, in the experiments.

Results and discussion

Implementation details

In the experiments, we need to set some hyperparameters, including the number of adjacent slices when constructing adjacency matrix A, the number of nodes eliminated in each pooling step, batch size, learning rate, epoch number, and dropout rate. The values are listed in Table 2. Additionally, we initialized the network weights with pre-trained parameters from the ImageNet dataset and employed an early stopping strategy. Training was halted when the validation set’s loss did not decrease for 20 consecutive epochs to prevent overfitting. The evaluation metrics include accuracy, precision, recall, and F1 score. Accuracy serves as the most direct metric to gauge model performance. F1-Score is a harmonic metric of precision and recall.

Table 2.

Hyperparameter settings.

Hyperparameter	Parameter settings
	6
Batchsize	4
Learning rate	1e−4
Epoch	100
Dropout	0.5

Pneumonia classification based on single CT sequences

In our MTGNet, the SGPNet can serve as a CT sequence encoder to learn a representation of CT sequences for pneumonia classification. To assess the effectiveness of SGPNet, we compare it with five traditional CT sequence-based pneumonia classification methods, including ResNet50+LSTM²⁴, ResNet+GRU²⁵, DeCoviNet¹⁷ DrCT²⁶, and AD3D-MIL¹⁸. As demonstrated in Table 3, the SGPNet outperforms the baselines, achieving the highest values in accuracy(0.919), precision(0.918), recall(0.916), and F1 Score(0.917). The experimental results suggest that the SGPNet is superior to traditional CT sequence-based classification methods, effectively capturing the spatial and temporal relationships between adjacent slices in the CT sequence.

Table 3.

Comparative results of SGPNet with other CT sequence-based pneumonia classification methods.

Method	Accuracy	Precision	Recall	F1 score
ResNet50+LSTM	0.869	0.866	0.864	0.865
ResNet+GRU	0.875	0.872	0.870	0.871
DeCovnet	0.888	0.883	0.883	0.883
DrCT	0.894	0.891	0.890	0.890
AD3D-MIL	0.900	0.898	0.898	0.900
SGPNet	0.919	0.918	0.916	0.917

Additionally, Fig. 7 presents the confusion matrices of different CT sequence-based pneumonia classification methods. Compared to traditional CT sequence-based pneumonia classification methods, the SGPNet correctly predicted 147 samples, achieving the highest accuracy. Evidence suggests that SGPNet effectively distinguishes between different types of pneumonia and healthy samples, providing more accurate predictions when only CT sequences are used.

Fig. 7.

Confusion matrices of different CT sequence-based pneumonia classification methods.

Pneumonia classification based on multi-modal data

To verify the effectiveness of the proposed multi-modal model MTGNet, we conducted a comparison experiment with classic multi-modal classification models, including MUTU²⁷, DeCovNet¹⁷, AD3D-MIL¹⁸ and TandemNet2²⁸. The MUTU model was proposed by Mahdi et al.²⁷ for dynamic gesture recognition using knowledge transfer-based 3D CNNs. TandemNet2 is a text-guided image recognition model based on visual information distillation. The results are shown in Table 4, where MTGNet outperforms all other methods, achieving the highest accuracy, precision, recall, and F1-Score, with values of 0.944, 0.950, 0.941, and 0.944, respectively. This demonstrates the superiority of the proposed method, primarily due to its ability to accurately learn from CT sequences guided by detailed lesion information in CT descriptions. Figure 8 shows the confusion matrix of different classification methods. The results indicate that MTGNet misclassified only 9 samples in the test data. Comparing the confusion matrices of other CT-based pneumonia classification methods, the results reveal that MTGNet effectively distinguishes CT sequences of different pneumonia types and healthy individuals, demonstrating significant performance improvement. Moreover, compared to the SPGNet using only single-modality CT sequence data, the MTGNet has improved accuracy by 2.5%, precision by 3.2%, recall by 2.5%, and F1 score by 2.7%. This suggests that the semantic information from the textual descriptions of CT images enriches the visual information, thereby improving the overall pneumonia classification. To identify the regions of the CT sequences that contribute most to the classification, we present two representative cases—one of bacterial pneumonia and one of viral pneumonia—using qualitative visualizations generated by Grad-CAM. The results are shown in Fig. 9. In each case, the slices containing prominent lesions are highlighted with red boxes, from which six representative slices are selected for visualization. As shown, the Grad-CAM heatmaps effectively highlight the lesion areas, clearly indicating the regions most relevant to the model’s decision-making process.

Table 4.

Comparison of MTGNet with other multi-modal pneumonia classification methods.

Method	Accuracy	Precision	Recall	F1 score
MUTU	0.844	0.840	0.840	0.839
DeCovNet	0.888	0.883	0.883	0.883
AD3D-MIL	0.900	0.898	0.898	0.900
TandemNet2	0.906	0.905	0.903	0.903
SGPNet	0.919	0.918	0.916	0.917
MTGNet	0.944	0.950	0.941	0.944

Fig. 8.

Confusion matrices of different multi-modal pneumonia classification methods.

Fig. 9.

Visualization results of two cases.

Ablation study of MTGNet

We conducted ablation experiments to validate the effect of main modules on pneumonia classification. The ablation experiment results are listed in Table 5. As the baseline, the SGPNet is a sequential graph pooling network to encode CT sequences for unimodal classification. Modal transfer (MT) generates representations of simulated text from CT sequence features, a crucial step for multi-modal training and single-modal testing. Cross-modal attention (CMA) integrates simulated textual features into visual information, thereby fusing multi-modal features. Compared to SGPNet, the MT and CMA modules contribute to an accuracy increase of 0.3%, a precision increase of 0.5%, a recall increase of 0.4%, and an F1 score increase of 0.3%. The results demonstrate that incorporating complementary information from CT descriptions can more directly guide the learning of CT sequences, and multi-modal text guidance significantly enhances the performance of pneumonia classification. In addition, multi-modal contrastive learning (CL) maximizes the similarity between similar sample pairs in different modalities and minimizes the similarity between dissimilar samples, enhancing representation learning. It further improves accuracy by 2.2%, precision by 2.7%, recall by 2.1%, and F1 score by 2.4%. The multi-modal contrastive loss optimizes the distribution of samples of different categories in the feature space, facilitating the learning of both visual and textual semantics, and thereby effectively improving the performance. These ablation results prove that the proposed modules are effective in improving performance.

Table 5.

Ablation analysis of different modules in MTGNet.

SGPNet	MT	CMA	CL	Accuracy	Precision	Recall	F1 score
				0.919	0.918	0.916	0.917
				0.918	0.919	0.918	0.918
				0.922	0.923	0.920	0.920
				0.944	0.950	0.941	0.944

Hyperparameter tuning of the number of adjacent slices for graph construction in the SGPNet

In the proposed SGPNet, the construction of the graph—particularly the design of the adjacency matrix—is critical to the model’s performance. The adjacency matrix A is constructed such that each slice is connected to k preceding and k succeeding slices. To determine the optimal value of k, we evaluated the performance of SGPNet across various k values. As shown in Fig. 10, the best performance was achieved when . The choice of k plays a pivotal role: smaller values may fail to capture sufficient contextual continuity, whereas larger values may incorporate irrelevant features. The experimental results suggest that SGPNet effectively captures contextual dependencies without introducing noise, particularly when .

Fig. 10.

Experimental results for different values of .

This study has some limitations. Firstly, Single-center data may limit generalization, so future work could be focused on incorporating more CT sequence datasets from multiple sources to improve the generalization and robustness of the proposed model. Secondly, the temporal dynamics of lesion variation were scarcely modeled, which may be informative for the classification of pneumonia. Thirdly, it would be interesting to apply or even develop a computationally efficient multi-modal large language model (MLLM) to enhance pneumonia classification accuracy.

Conclusion

Pneumonia is a common and severe respiratory disease, typically caused by bacterial, viral, or fungal infections. The traditional method of diagnosing pneumonia is based primarily on the experience of physicians, which is highly subjective and has low diagnostic efficiency. Therefore, AI-assisted diagnosis of pneumonia has been widely studied. We proposed a multi-modal text-guided network called MTGNet for chest sequence pneumonia classification using coarse-grained labels of CT sequences. The combination of the Sequential Graph Pooling Network (SGPNet) for encoding temporal relationships from CT sequences and the modal transfer module with a cross-attention mechanism for integrating multi-modal information, as well as the multi-modal contrastive learning module for facilitating the representation learning of both visual and textual semantics, successfully enhanced pneumonia classification. The experimental results validated the superiority of the proposed model over traditional sequence-based and multi-modal pneumonia classification methods.

Author contributions

Fujiao Ju conceived the experiments, Guangyi Huang and Hao Cui conducted the experiments, Guangyi Huang and Yujuan Feng analysed the results. All authors reviewed the manuscript.

Data availibility

The datasets generated during the current study are not publicly available, but can be obtained from the corresponding author on reasonable request.

Declarations

Competing interests

The authors declare no competing interests.