Detection of COVID-19, pneumonia, and tuberculosis from radiographs using AI-driven knowledge distillation

Abstract

Chest radiography is an essential diagnostic tool for respiratory diseases such as COVID-19, pneumonia, and tuberculosis because it accurately depicts the structures of the chest. However, accurate detection of these diseases from radiographs is a complex task that requires the availability of medical imaging equipment and trained personnel. Conventional deep learning models offer a viable automated solution for this task. However, the high complexity of these models often poses a significant obstacle to their practical deployment within automated medical applications, including mobile apps, web apps, and cloud-based platforms. This study addresses and resolves this dilemma by reducing the complexity of neural networks using knowledge distillation techniques (KDT). The proposed technique trains a neural network on an extensive collection of chest X-ray images and propagates the knowledge to a smaller network capable of real-time detection. To create a comprehensive dataset, we have integrated three popular chest radiograph datasets with chest radiographs for COVID-19, pneumonia, and tuberculosis. Our experiments show that this knowledge distillation approach outperforms conventional deep learning methods in terms of computational complexity and performance for real-time respiratory disease detection. Specifically, our system achieves an impressive average accuracy of 0.97, precision of 0.94, and recall of 0.97.

1. Introduction

Respiratory illnesses, such as COVID-19, pneumonia, and tuberculosis (TB), are significant global health hazards, accounting for substantial morbidity and mortality worldwide [1], [2]. Therefore, early and accurate diagnosis of these diseases is essential for reducing the spread of disease and improving patient consequences. Chest X-ray imaging is a frequently utilized modality in diagnosing respiratory diseases. Still, it might be challenging for radiologists to diagnose the condition based on X-ray images independently, and misdiagnosis may occur. Hence, an independent research domain of Medical Imaging is explicitly working on this problem to diagnose these diseases in real-time [3], [4].

Deep learning techniques have shown promising results in automatically detecting respiratory diseases from pulmonary radiology images [5], [6], [7]. Convolutional neural networks (CNNs) have been applied to this task with high accuracy [8], [9]. However, they are often complex and require large amounts of annotated data to train. This may limit their applicability in low-resource circumstances and make them impractical for deployment in practical clinical settings like mobile applications, web applications, or cloud-based systems. Therefore, the Knowledge distillation technique might be practical for low-resource environments.

In deep learning, knowledge distillation is a method that allows the transfer of knowledge from a large and intricate model (known as the teacher) to a smaller and less complex model (referred to as the student) [10]. This is typically achieved by training the student model to emulate the predictions made by the teacher model. The consequent student model is smaller, faster, and easier to deploy than the teacher model, making it an optimistic approach for deploying deep learning models in real-world clinical environments like mobile applications, web applications, or cloud-based systems.

Transfer learning is also widely used in similar types of image classification problems [11]. However, knowledge distillation might be more valuable than transfer learning for its simplicity. Knowledge distillation trains a smaller model to imitate a larger, pre-trained model. The idea is to forward the experience from the larger model to the smaller one so that the smaller model may achieve similar performance with fewer parameters. On the other hand, transfer learning is a technique that involves adapting a pre-trained model for a new task rather than building a new model from the ground up with a large number of parameters [12]. The idea is to leverage the knowledge learned by the model on a related task to improve its performance on the new task [13]. Knowledge distillation is preferred when computer resources are limited since it reduces computational complexity. The computational requirements can be significant in chest X-ray processing, which often involves high-resolution images and large data sets. However, knowledge distillation can distill or compress the model, resulting in a lighter and more computationally intensive version. This makes it suitable for use on platforms with limited computational resources, such as mobile devices, edge devices, or resource-constrained healthcare facilities. Using knowledge distillation, we may balance accurate disease detection and efficient use of computational resources.

This research strives to explore, using knowledge distillation methods in neural networks, to detect respiratory diseases, including COVID-19, pneumonia, and TB, from chest scans. In addition, the study concentrates on the comparison of various deep learning strategies and their effect on model accuracy and computational efficiency. The leading aim is to design a model that accurately detects respiratory disorders from chest X-rays while adequately being small, quick, and efficient for deployment in real-world medical environments. The research work makes the following contributions:

• •Developing a lightweight Deep Learning-based approach to diagnose respiratory diseases (COVID-19, pneumonia, and tuberculosis) from chest X-ray images.
• •Apply knowledge distillation techniques to transfer knowledge from an extensive, pre-trained network to a shorter network, resulting in higher accuracy and lower computational cost.
• •Reduce the complexity of the deep neural networks (DNN) model to make a faster, less complex, and more efficient medical diagnosis system for respiratory disease.
• •Evaluate the effectiveness of the presented technique on various annotated large-scale X-ray images and compare the results with existing advanced methods.

Our research work is organized as follows. Section 2 reviews the current state of the art in automatically detecting respiratory diseases from chest X-ray images, including applying various deep-learning techniques. Our methodology, including the design of the teacher and student models and the different neural networks used, is described in Section 3. In Section 4, we demonstrate the performance of the models developed in this research using a combination of accuracy and computational efficiency metrics, including model size. Section 5 briefly discusses the results, future research directions, potential applications of the models developed in this research, and the limitations of the current approach. Finally, Section 6 presents the conclusion.

2. Related work

Chest X-rays (CXR) are widely used in medical diagnosis, including respiratory disease detection such as COVID-19, pneumonia, and tuberculosis [14]. However, traditional methods for diagnosing these diseases involve manual interpretation by trained radiologists, which is time-consuming and prone to subjectivity. In recent years, computer-aided diagnosis (CAD) systems have been proposed to solve this problem [15], [16], [17]. This section reviews the existing literature on detecting respiratory diseases from chest X-rays from a Medical Imaging perspective.

One of the earliest studies in this field was performed by Lodwick et al. (1963) [18]. Their studies have shown that the detection accuracy for chest illness is enhanced with an X-ray computer-aided design method as an assistant. The authors demonstrate the notion of transforming the graphical images on roentgenograms into numerical series. These can be exploited and assessed by the digital computer and notify the impacts of utilizing this system to specify the importance of specific radiographic discoveries in lung cancer.

Classic machine-learning algorithms have been widely applied to computer-aided design systems for a long time. Specifically, the SVM-based ANN algorithm [19], k-NN algorithm [20], and DT algorithm [21] are primarily used. However, these algorithms do not provide the expected accuracy required for deployment in medical devices [22]. In recent years, deep learning techniques, specifically convolutional neural networks, have been applied to analyze chest X-ray images to detect respiratory diseases [23], [24].

Bhandari et al. [25] introduced a lightweight CNN architecture for respiratory disease diagnosis in chest CXR images. The authors obtained an average test accuracy of 94.3% and a validation accuracy of 94.5% for 10-fold cross-validation. In addition, the study provides a remarkably intuitive understanding of architecture findings in CXR images using interpretable artificial intelligence (XAI) algorithms. Similarly, Mahbub et al. [9] provided a deep neural network architecture with fewer layers, parameters, and epochs and obtained significant updated results in accuracy and performance. Furthermore, Malik et al. proposed an end-to-end CNN architecture combining Residual Neural Network and Dilated Convolution (CDC_Net) to classify various infectious diseases [26]. The authors obtained significant results in terms of accuracy (99.39%), recall (98.13%), precision (99.42%), and F1-score (98.26%). The paper's author [27] distinguishes lung cancer and COVID-19 using chest CT images. It employs a deep ensemble neural network, achieving a remarkable 96.30% accuracy, 96.39% sensitivity, and 98.44% precision across three datasets.

Among deep learning architectures, transfer learning is quite popular in medical imaging [28]. Mamalakis et al. proposed a deep transfer learning pipeline named DenResCov-19 by joining DenseNet-121 and the ResNet-50 networks with an extra layer with CNN block, which delivers superior performance to detect patients with COVID-19, pneumonia, TB, or health based on CXR images [5]. In addition, the authors applied the Monte Carlo cross-validation procedure for multi-class classification. Similarly, Mehta et al. introduced a Conditional Generative Adversarial Network (cGAN) with a fine-tuned deep transfer learning model to classify chest X-rays into six categories: COVID-Mild, COVID-Medium, COVID-Severe, Normal, Pneumonia, and Tuberculosis [29]. The model obtained a training and validation accuracy of 98.20% and 94.21%, respectively. The author of the paper [30] introduces the Truncated Inception Net, a Convolutional Neural Network excelling in distinguishing COVID-19 from Pneumonia, Tuberculosis, and healthy cases, outperforming current AI tools. Emphasizes the critical role of sensitivity, precision, and F1 score metrics in model evaluation.

With the outbreak of the COVID-19 pandemic, several studies have focused on detecting COVID-19 from chest X-ray images [31]. For example, Kaya et al. proposed a deep transfer learning model based on MobileNetV2 to detect COVID-19 disease [32]. The authors gained average accuracy rates of 95.62%, 96.10%, and 97.61% for 3-class cases with fivefold cross-validation and decreased 81.92% of total fine-tuning operations. Similarly, Malik et al. introduced a novel clinical diagnostic image investigation method to differentiate COVID-19 from four separate chest infections, including lung cancer, tuberculosis, pneumonia, and pneumothorax, named decision-making-based federated learning network (DMFL_Net) [6]. This method acquires an accuracy of 98.45% and exceeds other methods in categorizing COVID-19 from four chest diseases.

Despite the success of these deep learning models, they have limitations in terms of computational complexity and memory requirements. Limitations of existing deep-learning models for real-time respiratory disease detection in clinical environments are as follows.

• •Lack of large and diverse annotated datasets: One of the significant challenges in this field is the limited availability of large and diverse annotated datasets of chest X-ray images for training deep learning models. This can lead to biased models towards specific populations or imaging protocols and may not generalize well to new ones.
• •Lack of standardization in image acquisition: Variability in the acquisition of chest X-rays, including differences in patient positioning, image resolution, and image acquisition parameters, can make it difficult for models to generalize across different imaging studies.
• •Adversarial attacks: Deep learning models are vulnerable to adversarial attacks, where small perturbations to the input data can result in incorrect model outputs. This can be particularly problematic in medical image analysis, where incorrect diagnoses can seriously affect patient health.
• •High false positive rates: Chest X-rays can be challenging to interpret, and current models may produce high rates of false positives. This can result in incorrect diagnoses and inappropriate treatment, seriously harming patient health.
• •Integration with existing clinical workflow: Finally, there is a need for integrating deep learning models for diagnosing respiratory diseases with the existing clinical workflow, including integrating the model outputs with electronic health records and providing clear and concise results for clinicians. However, due to the vast number of parameters of deep learning architectures and computational complexity, it is very challenging to integrate.

We investigate knowledge distillation techniques to optimize deep learning models to overcome these limitations. The following sections demonstrate the potential of this technique for diagnosing COVID-19, pneumonia, and tuberculosis.

3. Methodology

In this section, we describe the framework used to detect respiratory diseases from chest X-ray images using knowledge distillation techniques in neural networks. The following steps were followed in this process:

3.1. Data pre-processing

The first step in the process was to collect many chest X-ray images from patients with the disease and healthy individuals. The data collection and statistics description is explained in the next section 4.1.

To prepare the data for training the model, the images were pre-processed to ensure they were the same size and grayscale. This was done to decrease the complexity of the model and enhance its computational efficiency. Besides, the pixel values of the images were normalized to minimize the effects of lighting variations. To increase the size of the training dataset and reduce overfitting, we employed deep learning-based augmentation on the images. It involves applying random rotation, translation, flip and zoom to images. This helped the model learn from different perspectives of the same image, thereby generalizing its performance to the validation set [33]. Finally, the pre-processed and augmented datasets were divided into training and validation sets. The training set was used to train the model, while the validation set was used to tune its hyperparameters and test the performance. A standard split ratio of 80-20 is used for training and validation.

Data pre-processing techniques and deep learning-based enhancements in this study helped improve the model's performance in detecting respiratory disease from chest X-ray images.

3.2. Proposed framework

This section explains the three foundations of the framework - teacher model training, distillation algorithm, and student architecture. Fig. 1 demonstrates the complete architecture.

Figure 1.

Proposed teacher-student framework for knowledge distillation. The teacher model with many parameters passes on its knowledge to the student model with significantly fewer parameters. The student model is trained to mimic the behavior of the teacher model in identifying respiratory diseases from chest X-ray images.

3.2.1. Teacher model training

The role of the teacher model is to create soft targets in the form of class probabilities, which serve as guidelines for the student model. The student model then learns to simulate the predictions of the teacher model, effectively distilling the knowledge it contains. The rest of the sub-section explains the deep neural network architecture and the types of knowledge transferred to the student model.

• 1.Deep Convolutional Neural Network (DCNN): A deep neural network is used to train the teacher model. The model consists of an input layer, multiple convolution layers, activation layers, batch normalization layers, and max-pooling layers repeatedly. Also, it has a flattened layer and two dense layers. Fig. 2 presents the deep neural network architecture.
• 2.Types of Knowledge: Knowledge transfer from the teacher network can be exemplified differently depending on the distillation method employed. To enhance the accuracy of our disease detection model, we incorporate all three types of knowledge transfer, which are explained below:
  Response-Based Knowledge: Response-based knowledge distillation involves transferring the output probabilities of the teacher network to the student network. We perform this by minimizing the cross-entropy loss between the teacher and student network probabilities. The cross-entropy loss is defined in Eq. (1).

  $$L_{response}=-{\lambda }_{response}\sum _i{t}_i\log (s_i)$$ (1)

  Where $t_i$ is the ground truth probability distribution for class i, $s_i$ is the predicted probability distribution for class i, and ${\lambda }_{response}$ is a hyperparameter that controls the weight of the response-based loss.

  Feature-Based Knowledge: Feature-based knowledge distillation involves transferring the high-level features learned by the teacher network to the student network. These features are obtained from intermediate layers of the teacher network and provide valuable guidance to the student network. We transfer feature-based knowledge by minimizing the mean squared error between the feature maps produced by the teacher network and the corresponding feature maps produced by the student network. This can be formulated in Eq. (2).

  $$L_{feature}={\lambda }_{feature}||F_{teacher}-F_{student}|{|}^2$$ (2)

  Where $F_{teacher}$ and $F_{student}$ are the feature maps produced by the teacher and student networks, respectively, and ${\lambda }_{feature}$ is a hyperparameter that controls the relative importance of the feature-based loss.

  Relation-Based Knowledge: Relation-based knowledge distillation involves transferring the spatial relationships between different parts of the input image. First, we computed the Gram matrix, representing the correlations between different channels in the feature maps. The Gram matrix can be computed using Eq (3).

  $$G=F^{T}F$$ (3)

  where F is the feature map and G is the corresponding Gram matrix. Then, the relationship between different parts of the input image is transferred by minimizing the mean squared error between the Gram matrices produced by the teacher and student networks are represented in Eq. (4).

  $$L_{relation}={\lambda }_{relation}⁎||G_{teacher}-G_{student}|{|}^2$$ (4)

  Where $G_{teacher}$ and $G_{student}$ are the Gram matrices produced by the teacher and student networks, respectively, and ${\lambda }_{relation}$ is a hyperparameter that controls the relative importance of the relation-based loss.

Figure 2.

Deep convolutional neural network architecture that applied as a teacher model. The architecture consists of multiple convolutional layers, pooling, and fully connected layers.

3.2.2. Student model training

The student model is a simple shallow neural network with a single Conv2D layer followed by LeakyReLU activation, MaxPooling, Flatten, and Dense layers. The Conv2D layer extracts features from the input chest X-ray image, and the LeakyReLU activation introduces non-linearity in the network. The MaxPooling layer reduces the spatial dimensions of the feature maps and helps the network to be translation-invariant. The Flatten layer converts the 2D feature maps into a 1D feature vector, fed to the Dense layer for classification.

The architecture of the student model is intentionally kept shallow to reduce the computational complexity and memory requirements, which is suitable for deployment on resource-constrained devices. The student model is designed to learn from the teacher model through knowledge distillation, where the student model learns from the high-level knowledge of the teacher model to improve its classification accuracy. The student model is trained on the dataset using the cross-entropy loss function and the Adam optimizer. During training, the student model is guided by the teacher model's predictions and learns to mimic the teacher model's behavior. Fig. 3 presents the shallow neural network (SNN) architecture.

Figure 3.

Shallow neural network architecture used as a student model.

3.2.3. Knowledge transfer approach

The modified pipeline for the classification task can be explained through the following Algorithm 1:

Algorithm 1.

Knowledge Distillation Framework for respiratory disease classification using a teacher-student architecture and soft target cross-entropy loss.

In this algorithm, we first initialize the student network with random weights and set the learning rate for the optimizer. Then, we train the student network for a fixed number of epochs. In each epoch, we loop over the training data in batches and compute the logits (pre-softmax outputs) for both the teacher and student networks. We then apply the softmax function to these logits and use the resulting probabilities to compute the distillation loss and the classification loss. The distillation loss measures the difference between the teacher's and student's softmax probabilities, while the classification loss measures the error in predicting the true labels. The total loss is a weighted sum of these two losses, where the distillation loss is weighted by a temperature parameter T (usually set to 1). Finally, we update the student network weights using the optimizer based on the total loss. The resulting trained student network is returned as the output.

The knowledge distillation algorithm described here is an offline method. This means the training data is assumed to be fixed and available before training the student network. The algorithm loops over the training data in batches and updates the student network weights based on the total loss computed from the distillation and classification losses.

4. Experimental analysis

This section presents the empirical evaluation of the proposed method for detecting respiratory diseases from chest X-ray images. We have conducted several experiments using different datasets and variations of the knowledge distillation technique. This section details the experimental setup, including the dataset and the implementation environment, and explains the evaluation metrics used to measure the performance of the models. The experimental results are analyzed in detail to provide insights into the effectiveness of the proposed method, the impact of different factors on its performance, and its practical implications.

4.1. Data collection

The dataset used in this study was collected from publicly available sources, including the following:

• •COVID-19 Chest X-ray Dataset: This dataset was sourced from Kaggle and contains chest X-ray images of patients with COVID-19 [34], [35]. The dataset comprises 3616 images of COVID-19-positive cases and 10192 images of healthy patients.
• •Chest X-Ray Images (Pneumonia): This dataset was sourced from Kaggle and contains chest X-ray images of patients with pneumonia [36]. The dataset consists of 4265 images of pneumonia cases and 1575 images of healthy patients.
• •Tuberculosis Chest X-ray Dataset: This dataset was sourced from Kaggle and contains chest X-ray images of patients with tuberculosis [37]. The dataset consists of 3700 images of tuberculosis cases and 3500 images of healthy patients.

COVID-19, pneumonia, and tuberculosis (TB) are all respiratory diseases that can be observed through X-ray imaging. However, they have distinct characteristics and patterns that can help differentiate them. Table 1 gives an overview of the differences between COVID-19, pneumonia, and tuberculosis (TB) in X-ray imaging.

Table 1.

Differences between COVID-19, Pneumonia, and Tuberculosis in X-ray Imaging.

Characteristic	COVID-19	Pneumonia	Tuberculosis (TB)
Location of Opacities	Bilateral, peripheral, and central opacities	Patchy infiltrates, location varies	Predominantly upper lung lobes
Opacity Appearance	Ground-glass opacities (GGO), patchy, hazy	Patchy opacities, may appear as consolidations	Granulomas, possible cavities in advanced cases
Air Bronchograms	May be absent	Often present as visible air-filled bronchi	Absent
Crazy Paving Pattern	Can be observed in some cases	Not typically associated with pneumonia	Not typically associated with TB
Distribution of Lesions	Random and variable throughout the lungs	Variable but usually scattered	Often upper lung zones, but can occur anywhere
Other Findings	Consolidation in severe cases	May cause pleural effusion	Cavities, fibrosis in advanced cases
Clinical Context	Often associated with a viral pandemic	Various causes including infections	Mycobacterium tuberculosis infection

This study's dataset was organized into three sets to facilitate various experiments. First, the dataset was divided into a two-class classification set, denoted as RDD2, which included chest X-ray images of patients with COVID-19 and healthy individuals. The second classification set, RDD3, was created to support three-class classification and included chest X-ray images of patients with COVID-19, pneumonia, and healthy individuals. Finally, RDD4 was established, a four-class classification set containing chest X-ray images of patients with COVID-19, pneumonia, tuberculosis, and healthy individuals. Details of the dataset and the distribution of the different categories can be found in Table 2. All chest X-ray images were preprocessed by resizing to $250⁎250$ pixels and normalization. Referring to Table 2, it is evident that the tuberculosis class has a notably smaller amount of data, potentially leading to a class imbalance issue within the dataset. To address this concern, we implemented data augmentation techniques to augment the tuberculosis class. These techniques included popular methods such as random flipping, zooming, rotation, and shifting.

Table 2.

Dataset details with varying classes for respiratory disease classification. (Open source).

Dataset	Covid-19	Pneumonia	Tuberculosis	Normal	Total
RDD2	3616	–	–	3616	7232
RDD3	3616	4265	–	4250	12131
RDD4	3616	4265	700	4502	13083

4.2. Experimental setup

The neural network models were implemented using Keras and TensorFlow deep learning frameworks. The experiments were conducted in a virtual environment running on a system with an RTX3070 GPU. In addition to the RTX3070 GPU, the virtual environment used for the experiments included a Ryzen 7 processor, 16 GB of RAM, and a 1 TB solid-state drive. The operating system used was Ubuntu 20.04 LTS, and the software environment included Python 3.8, Keras 2.6.0, and TensorFlow 2.6.0. The virtual environment was created using the Anaconda package manager to ensure the reproducibility of the experimental results.

To accelerate the training of the neural network models, the CUDA toolkit and cuDNN library were installed and configured to work with the GPU. The batch size and learning rate were chosen based on the results of the hyperparameter tuning experiments. We used the Adam optimizer with a learning rate of 0.0001 and a batch size of 128 for all the experiments. The models were trained for 25 epochs for each experiment.

Overall, the experimental setup provided a powerful and efficient platform for training the neural network models and evaluating the performance of the proposed approach for detecting respiratory diseases from chest X-ray images.

4.3. Evaluation metrics

The performance of the proposed approach was evaluated using several widely used evaluation metrics, including accuracy, precision, and recall. These metrics were chosen because they are commonly used in the literature for evaluating the performance of deep learning models in medical imaging applications.

Accuracy is defined as the ratio of correctly classified samples to the total number of samples. Precision is defined as the ratio of true positive predictions to the total number of positive predictions. The recall is the ratio of true positive predictions to the total number of positive samples.

4.4. Performance analysis

In this section, we present and analyze the results of our experiments using the proposed knowledge distillation technique to detect COVID-19, Pneumonia, and Tuberculosis from Chest X-ray images.

Table 3 summarizes the results obtained by our student models trained using knowledge distillation with different configurations. As shown, the best-performing student model is obtained by transferring all three types of knowledge, i.e., feature-based, response-based, and relation-based, with a temperature of 2.8 and a lambda value of 0.5. This model achieves a validation accuracy of 0.96, precision of 0.94, and recall of 0.97 for the RDD4 dataset. Table 4, Table 5, Table 6 present the class-wise results of our experiments. These tables allow us to assess our model's performance across various class labels by examining class-specific accuracy, precision, recall, and F1 scores. The following part presents a detailed analysis of our proposed method's performance on different dataset configurations based on the accuracy and loss curve.

Table 3.

Accuracy, Precision, and Recall scores for three different dataset configurations.

Dataset	DCNN-Teacher model					KDT-Student model
–	Tr. Acc.	Val. Acc.	Precision	Recall	F-1 Score	Tr. Acc.	Val. Acc.	Precision	Recall	F-1 Score
RDD2	0.99	0.97	0.98	0.99	0.98	0.98	0.97	0.95	0.98	0.96
RDD3	0.99	0.96	0.97	0.98	0.97	0.98	0.96	0.94	0.97	0.95
RDD4	0.98	0.95	0.97	0.97	0.97	0.97	0.96	0.94	0.97	0.95

Table 4.

Class-wise result of the RDD2 dataset configurations.

	DCNN-Teacher model			KDT-Student model
	Precision	Recall	f1-score	Precision	Recall	f1-score
COVID-19	0.97	0.99	0.97	0.93	0.98	0.95
Normal	0.99	0.99	0.98	0.97	0.98	0.97
Accuracy			0.97		0.97
macro avg	0.98	0.99	0.98	0.95	0.98	0.96
weighted avg	0.98	0.98	0.98	0.96	0.98	0.97

Table 5.

Class-wise result of the RDD3 dataset configurations.

	DCNN-Teacher model			KDT-Student model
	Precision	Recall	f1-score	Precision	Recall	f1-score
COVID-19	0.97	0.98	0.97	0.95	0.95	0.95
Pneumonia	0.96	0.97	0.97	0.94	0.97	0.96
Normal	0.98	0.99	0.98	0.97	0.95	0.96
Accuracy			0.96		0.96
macro avg	0.97	0.98	0.97	0.94	0.97	0.95
weighted avg	0.97	0.98	0.98	0.95	0.96	0.96

Table 6.

Class-wise result of the RDD4 dataset configurations.

	DCNN-Teacher model			KDT-Student model
	Precision	Recall	f1-score	Precision	Recall	f1-score
COVID-19	0.96	0.94	0.95	0.95	0.97	0.96
Pneumonia	0.93	0.95	0.94	0.94	0.94	0.94
Tuberculosis	0.94	0.94	0.94	0.93	0.98	0.95
Normal	0.96	0.98	0.97	0.98	0.98	0.98
Accuracy			0.95		0.96
macro avg	0.97	0.97	0.97	0.94	0.97	0.95
weighted avg	0.97	0.97	0.97	0.94	0.98	0.96

4.4.1. Binary classification

First, we have applied the framework to the binary classification of COVID-19 and normal cases from chest X-ray images.

Fig. 4a illustrates the performance of our model during the training process. Here, the x-axis represents the number of epochs, and the y-axis represents the accuracy. The blue line represents the training accuracy, which improves from 0.53 to 0.98 over 25 epochs, while the red line represents the validation accuracy, which improves from 0.51 to 0.97. Our model achieves high accuracy on both the training and validation sets, indicating that it has learned to classify the images accurately. Fig. 4b shows the training and validation loss over 25 epochs. The blue line represents the training loss, while the red line represents the validation loss. The model achieves a minimum training loss of 0.03 and a minimum validation loss of 0.05. The two curves closely follow each other, with the validation loss slightly higher than the training loss, indicating a reasonable fit between the model and the data.

Figure 4.

Illustration of training and validation accuracy and training and validation loss curves for the student and teacher methods trained using knowledge distillation on the RDD2 dataset.

In addition, Fig. 4c illustrates the performance of the teacher model during the training process. The training accuracy improves from 0.58 to 0.99, while the validation accuracy improves from 0.49 to 0.97. Fig. 4d shows the training and validation loss. The model achieves a minimum training loss of 0.02 and a minimum validation loss of 0.03.

Overall, Fig. 4 demonstrates the effectiveness of our framework for binary classification of COVID-19 and normal cases from chest X-ray images. The high accuracy and low loss on the training and validation sets suggest that our model has learned to distinguish between the two classes with high precision even though the number of parameters is extremely low. Despite the huge difference in several parameters, our proposed distillation framework allows the models to perform similarly.

4.4.2. Multi-class (3) classification

Similarly, we continued the study with three classes with the RDD3 dataset.

Fig. 5a illustrates the performance of the student model during the training process. The training accuracy improves from 0.53 to 0.98, while the validation accuracy improves from 0.35 to 0.96. Fig. 5b shows the training and validation loss. The model achieves a minimum training loss of 0.02 and a minimum validation loss of 0.03. Fig. 5c illustrates the performance of the teacher model during the training process. The training accuracy improves from 0.51 to 0.99, while the validation accuracy improves from 0.41 to 0.96. Fig. 5d shows the training and validation loss. The model achieves a minimum training loss of 0.03 and a minimum validation loss of 0.04.

Figure 5.

Illustration of training and validation accuracy and training and validation loss curves for the student and teacher methods trained using knowledge distillation on the RDD3 dataset.

4.4.3. Multi-class (4) classification

Finally, we have tested the proposed framework with four classes using the RDD4 set. Fig. 6a illustrates that training accuracy improves from 0.35 to 0.97, while the validation accuracy improves from 0.17 to 0.96. Fig. 6b shows the training and validation loss. The model achieves a minimum training loss of 0.04 and a minimum validation loss of 0.05. Fig. 6c illustrates the performance of the teacher model during the training process. The training accuracy improves from 0.51 to 0.98, while the validation accuracy improves from 0.41 to 0.95. Fig. 6d shows the training and validation loss. The model achieves a minimum training loss of 0.03 and a minimum validation loss of 0.04.

Figure 6.

Illustration of training and validation accuracy and training and validation loss curves for the student and teacher methods trained using knowledge distillation on the RDD4 dataset.

Our findings suggest that the distillation framework effectively compresses larger models without sacrificing performance. This has important implications for practical applications where computational resources are limited. By using our distillation framework, models with significantly fewer parameters may be trained to perform similarly to larger models, allowing for faster inference times and reduced computational costs.

4.5. Number of parameters

Table 7 illustrates the parameter count of the Teacher model, which amounts to approximately 13.8 million after undergoing several layers, including convolution, activation, batch normalization, max pooling, dropout, flattening, and dense layers. In contrast, the Student model has a significantly reduced parameter count of only 1 million [Table 8]. The observed substantial decrease in parameters demonstrates the effectiveness of the proposed knowledge distillation technique for disease classification from X-ray images.

Table 7.

DCNN architecture summary.

Model: “teacher”
Layer (type)	Output Shape	Param #
conv2d (Conv2D)	(None, 250,250,32)	896
activation (Activation)	(None, 250,250,32)	0
batch_normalization (BatchNormalization)	(None, 250,250,32)	128
max_pooling2d (MaxPooling2D)	(None, 83,83,32)	0
dropout (Dropout)	(None, 83,83,32)	0
conv2d_1 (Conv2D)	(None, 83,83,64)	18496
activation_1 (Activation)	(None, 83,83,64)	0
batch_normalization_1 (BatchNormalization)	(None, 83,83,64)	256
max_pooling2d_1 (MaxPooling2D)	(None, 41,41,64)	0
dropout_1 (Dropout)	(None, 41,41,64)	0
flatten (Flatten)	(None, 107584)	0
dense (Dense)	(None, 128)	13770880
activation_2 (Activation)	(None, 128)	0
batch_normalization_2 (BatchNormalization)	(None, 128)	512
dropout_2 (Dropout)	(None, 128)	0
dense_1 (Dense)	(None, 4)	516
Total params: 13,791, 684 Trainable params: 13, 791, 236 Non-trainable params: 448

Table 8.

SNN architecture summary.

Model: “student”
Layer (type)	Output Shape	Param #
conv2d_2 (Conv2D)	(None, 125,125,16)	448
leaky_re_lu (LeakyReLu)	(None, 125,125,16)	0
max_pooling2d_2 (MaxPooling2D)	(None, 125,125,16)	0
flatten_1 (Flatten)	(None, 250000)	0
dense_2 (Dense)	(None, 4)	1000004
Total params: 1, 000, 452 Trainable params: 1, 000, 452 Non-trainable params: 0

4.6. Sensitivity analysis

To evaluate the sensitivity of our proposed method to different hyperparameters, we have performed a sensitivity analysis by varying the temperature and lambda values. Table 9 shows the results obtained by our student model with varying temperature and lambda values. As shown, the best performance is achieved with a temperature of 2.8 and a lambda value of 0.5, which is consistent with our previous findings. In our evaluation, we began by setting the T value to 1.5 and the λ value to 0.1, which yielded a favorable score. Subsequently, as we increased T, the model's performance improved. However, at higher T values (more than 2.8), the accuracy began to decrease. Through our experimentation, we identified the optimal combination of T and λ to be 2.5 and 0.5, respectively, which produced the model's highest accuracy.

Table 9.

Performance of the student model with varying temperature and lambda values.

Hyperparameters		Accuracy	Precision	Recall
T	λ	(%)	(%)	(%)
1.5	0.1	86.45	87.22	87.84
1.5	0.5	88.31	88.15	89.91
1.5	1.0	87.75	87.65	87.14
2.0	0.1	90.09	89.28	91.25
2.0	0.5	92.85	91.99	92.27
2.0	1.0	89.94	88.92	90.51
2.5	0.1	94.05	93.67	95.19
2.5	0.5	95.21	94.18	96.01
2.5	1.0	93.97	91.05	94.38
2.8	0.1	94.05	93.67	96.19
2.8	0.5	96.08	94.48	97.76
2.8	1.0	93.97	92.05	94.38
3.0	0.1	93.05	92.67	94.19
3.0	0.5	94.21	93.18	94.01
3.0	1.0	92.97	90.05	93.38

4.7. Statistical results analysis

In our statistical analysis, we utilized the Adjusted $R^2$ test on the confusion matrix for each fold, as defined in Eq. (5), and the outcomes are presented in Table 10, Table 11. The adjusted $R^2$ value increases when a new term enhances the model's performance beyond what would be considered random chance. Conversely, it decreases when a predictor contributes less to the model than expected. This indicates that, with a 94.54% variable, the model can effectively classify the four distinct groups in most cases [25].

$$\text{Adjusted }{\text{R}}^2=1-\frac{(1-R^2)\cdot (n-1)}{n-k-1}$$ (5)

Table 10.

Adjusted R² Test Results for the 5-Fold Experiment on Different Dataset Configurations Applied to the DCNN Model. ‘k1,’ ‘k2,’ through ‘k5’ represent the five folds corresponding to various dataset configurations.

Dataset	K1	K2	K3	K4	K5	Avg (μ ± σ)
RDD2	0.995	0.994	0.997	0.994	0.996	0.995 ± 0.002
RDD3	0.993	0.996	0.991	0.994	0.993	0.993 ± 0.002
RDD4	0.993	0.991	0.994	0.993	0.992	0.992 ± 0.002

Table 11.

Adjusted R² Test Results for the 5-Fold Experiment on Different Dataset Configurations Applied to the SNN Model. ‘k1,’ ‘k2,’ through ‘k5’ represent the five folds corresponding to various dataset configurations.

Dataset	K1	K2	K3	K4	K5	Avg (μ ± σ)
RDD2	0.997	0.993	0.996	0.991	0.994	0.994 ± 0.002
RDD3	0.995	0.992	0.993	0.993	0.991	0.992 ± 0.002
RDD4	0.991	0.994	0.991	0.993	0.992	0.992 ± 0.006

In the equation, “Adjusted $R^2$” represents the adjusted coefficient of determination, “$R^2$” is the unadjusted coefficient of determination, “n” is the sample size, and “k” is the number of predictors in a regression model. The adjusted R-squared for the model in Table 10, Table 11, suggests that the model's performance is very good and consistent.

4.8. Comparison with existing methods

In conclusion, we evaluated the performance of our proposed framework against state-of-the-art methodologies. The results, presented in Table 12, indicate that our proposed framework outperforms all existing architectures regarding both performance and computational cost for two, three, and four-class disease classification. Our approach achieves higher accuracy while using fewer computational resources, indicating its potential for practical implementation in real-world clinical settings.

Table 12.

Comparison of the performance of our proposed method with other state-of-the-art techniques on various datasets, based on accuracy and computational cost.

Ref.	Number of Classes	Number of images	Methods	Accuracy	Parameters (in Millions)
Liu et al. [38]	2	TB = 4248, Normal = 453	Transfer learning with (AlexNet and GoogLeNet)	85.68	AlexNet = 61, GoogleNet = 7
Waheed et al. [39]	2	Covid19 = 403, Normal = 721	COVIDGAN	95.00	–
Heidari et al. [40]	3	Covid19 = 445, Pneumonia = 5179, Normal = 2880	VGG16	94.5	Pre-trained VGG16
Rahimzadeh et al. [41]	3	Covid19 = 180, Pneumonia = 6054, Normal = 8851	Ensemble of Xception and ResNet50	91.40	XceptiotNet = 22, ResNet = 11
Ucar et al. [42]	3	Covid19 = 76, Pneumonia = 4290, Normal = 1583	Deep-SqueezeNet	98.30	–
Ozturk et al. [43]	3	Covid19 = 130, Pneumonia = 500, Normal = 500	Custom CNN	87.20	CNN = 1.17
Khan et al. [44]	4	PneumoniaB = 330, PneumoniaV = 327, Covid19 = 284, Normal = 310	CNN-based CoroNet	89.60	CNN = 33.97
Qaqos et al. [45]	4	COVID-19 = 576, Pneumonia = 4273, TB = 155, Normal = 1583	Custom CNN	94.53	CNN = 34.73
Sitaula et al. [46]	4	PneumoniaB = 330, PneumoniaV = 327, Covid-19 = 284, Normal = 310	Attention-based VGG	87.49	VGG-16 = 18, VGG-19 = 21.2
Bhandari et al. [25]	4	Covid19 = 576, Pneumonia = 4273, TB = 700, Normal = 1583	Custom CNN	95.94	CNN = 3.7
Al-Timemy et al. [47]	5	COVID-19 = 435, PneumoniaB = 439, PneumoniaV = 439, TB = 434, Normal = 439	Transfer learning with Resnet18	91.60	Resnet18 = 11
Das et al. [48]	2	TB = 400, Normal = 420	Pre-trained deep learning model	76.05	–
Das et al. [49]	2	TB = 400, Normal = 420	Inception Net V3	91.7	InceptionNet V3=25
Mukherjee et al. [50]	2	COVID-19 = 168, Non COVID-19 = 168	DNN	96.28	DNN = 0.22
Makkar et al. [51]	3	COVID-19 = 2358, Pneumonia = 4273, Healthy = 1583	Federated learning with NN	89.9	–
Kamal et al. [52]	3	COVID-19 = 1200, Pneumonia = 1345, Normal = 1341	Spike Neural Network (SNN)	96	–
Proposed technique Kabir et al.	2	Covid19 = 3616, Normal = 3616	Knowledge distillation in NN	97.56	KDT = 1.0
	3	Covid19 = 3616, Pneumonia = 4265, Normal = 4250		96.34
	4	Covid19 = 3616, Pneumonia = 4265, TB = 700, Normal = 4502		96.08

5. Discussion and future research directions

Using chest radiographs to detect respiratory diseases such as COVID-19, pneumonia, and tuberculosis is critical for early diagnosis and treatment of these diseases. In recent years, advances in deep learning and computer vision have enabled the use of neural networks for image classification tasks. However, these networks often require large amounts of data and computational resources, making them difficult to use in practice. The proposed Knowledge distillation framework addresses this issue by allowing knowledge transfer from a large, complex model to a smaller, simpler one. The main contribution of this study is to develop a knowledge distillation framework for detecting respiratory diseases from chest X-ray images. Our framework is based on a pre-trained deep neural teacher network. This network generates soft targets, which are then used to train a smaller student network. The student network is designed to have a simpler architecture, requiring fewer computational resources than the teacher network.

The performance of our framework is evaluated by combining popular CXR datasets collected from various open sources. The dataset contains images of patients diagnosed with COVID-19, pneumonia, tuberculosis, and healthy individuals. The images were annotated with labels indicating the presence or absence of the diseases. To conduct diverse experiments, our dataset is partitioned into three distinct sets, namely RDD2, RDD3, and RDD4. This division allowed us to investigate the performance of our model as the number of class labels and image data increased. Our results show that the knowledge distillation framework can accurately detect respiratory diseases from chest X-ray images. The student network achieved an overall accuracy of 96.08%, comparable to the teacher network's accuracy (95.02%). The results demonstrate the effectiveness of the knowledge distillation technique for detecting respiratory diseases from chest X-ray images.

It is worth noting that our framework has several advantages compared to traditional deep neural network approaches. First, using the student network reduces the computational requirements, making it possible to deploy the model on resource-constrained devices, such as mobile phones or edge devices. Second, the framework might be easily adapted to different datasets by fine-tuning the teacher network on a new dataset and using the resulting soft targets to train the student network. In addition, the proposed framework mitigates most of the limitations of existing deep learning approaches explained in Section 2. For example:

• •Improved performance with smaller datasets: The study showed that the proposed framework could transfer the knowledge from a large, pre-trained model to a smaller, more computationally efficient model, allowing for improved performance even with smaller, less diverse datasets.
• •Increased robustness to adversarial attacks: The proposed framework can also help to improve the robustness of deep learning models to adversarial attacks by transferring the robustness of a pre-trained model to a smaller student model.
• •Reduce false positive rates: Section 4 explained that the proposed framework increased the performance and reduced the false positive rate. We have seen that the recall value is very high, which lowers the false positive rate.
• •Improved interpretability: Deep learning approaches are often complex and challenging to interpret, hindering their adoption in real-world environments. On the other hand, the student network generated by the knowledge distillation framework is simpler and more interpretable than the teacher network, making it easier to understand and rely on the results.
• •Improved integration with clinical workflows: Another challenge faced by deep learning approaches is the computational requirements. Deep neural networks are computationally expensive models requiring significant computational resources, such as GPUs or TPUs. This makes deploying these models in real-world environments challenging, particularly on resource-constrained devices like mobile phones or edge devices. By transferring knowledge from the teacher to the student network, our knowledge distillation framework reduces the computational requirements, making it possible to deploy the model on resource-constrained devices.

In conclusion, this study demonstrates the potential of knowledge distillation techniques for detecting respiratory diseases from chest X-ray images. Furthermore, the results show that our framework can achieve high accuracy while reducing computational requirements compared to traditional deep neural network approaches. These findings have important implications for the deployment of deep learning models in real-world settings, particularly in the context of medical imaging.

In future work, we plan to evaluate our framework's performance on larger datasets and explore the use of other knowledge distillation techniques, such as dynamic distillation and attention transfer. Additionally, we plan to investigate the potential of our framework for detecting other diseases from medical images, such as cardiovascular diseases and cancers. Some possible research directions in this domain are as follows:

• •Incorporating other imaging modalities: Models can be improved by incorporating other imaging modalities, such as CT scans or MRI images, to improve the accuracy of diagnosis.
• •Exploration of different distillation techniques: Further research can be done to explore the effectiveness of different knowledge distillation techniques in detecting respiratory disease from chest X-ray images.
• •Incorporating patient demographics: Demographic information such as age, gender, and medical history can be included in models to improve diagnostic accuracy.
• •Combining with other AI techniques: Finally, The models can be combined with other AI techniques, such as natural language processing, to extract relevant information from medical reports and improve diagnostic accuracy.

6. Conclusions

Using chest X-ray images, this study evaluated the effectiveness of Knowledge Distillation Techniques in neural networks for detecting respiratory diseases such as COVID-19, pneumonia, and tuberculosis. The results demonstrated the proposed model's superiority over traditional deep learning methods, achieving high accuracy in disease detection. These results highlight the potential for further research on applying knowledge distillation techniques in medical imaging and the proposed model's potential to improve patient outcomes through the early detection and treatment of respiratory diseases. Chest radiographs are widely accepted in the medical community as a diagnostic tool for respiratory infections. The proposed model is a valuable tool that may help medical professionals make accurate diagnoses quickly. We strongly believe that this study may have important implications for the medical community and may help develop efficient and accurate disease detection methods.

Ethical approval and consent to participate consent for publication statement

We followed all the ethical consent to prepare this manuscript.

CRediT authorship contribution statement

Md Mohsin Kabir: Conceptualization, Methodology, Data curation, Writing – original draft. M.F. Mridha: Methodology, Investigation, Formal analysis. Ashifur Rahman: Writing – review & editing, Resources, Data curation. Md. Abdul Hamid: Validation, Supervision. Muhammad Mostafa Monowar: Writing – review & editing, Software.

Declaration of Competing Interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Dr. M.F. Mridha reports was provided by American International University Bangladesh. Dr. M.F. Mridha reports a relationship with American International University Bangladesh that includes: employment.

Data availability

The current study's datasets are publicly available sources [28], [29], [30], [31].