Cross-platform privacy-preserving CT image COVID-19 diagnosis based on source-free domain adaptation

Abstract

In the wake of the Coronavirus disease (COVID-19) pandemic, chest computed tomography (CT) has become an invaluable component in the rapid and accurate detection of COVID-19. CT scans traditionally require manual inspections from medical professionals, which is expensive and tedious. With advancements in machine learning, deep neural networks have been applied to classify CT scans for efficient diagnosis. However, three challenges hinder this application of deep learning: (1) Domain shift across CT platforms and human subjects impedes the performance of neural networks in different hospitals. (2) Unsupervised Domain Adaptation (UDA), the traditional method to overcome domain shift, typically requires access to both source and target data. This is not realistic in COVID-19 diagnosis due to the sensitivity of medical data. The privacy of patients must be protected. (3) Data imbalance may exist between easy/hard samples and between data classes which can overwhelm the training of deep networks, causing degenerate models. To overcome these challenges, we propose a Cross-Platform Privacy-Preserving COVID-19 diagnosis network (CP${}^3$Net) that integrates domain adaptation, self-supervised learning, imbalanced label learning, and rotation classifier training into one synergistic framework. We also create a new CT benchmark by combining real-world datasets from multiple medical platforms to facilitate the cross-domain evaluation of our method. Through extensive experiments, we demonstrate that CP${}^3$Net outperforms many popular UDA methods and achieves state-of-the-art results in diagnosing COVID-19 using CT scans.

1. Introduction

Since its emergence in late 2019, COVID-19 has caused widespread pandemonium and devastating impacts on individuals and communities worldwide. The pandemic has caused suffering and has tragically destroyed the lives of many [1], [2], [3], [4]. A reliable and efficient diagnosis process is crucial for minimising the impact of the pandemic. Recent medical studies have highlighted the value of chest computed tomography (CT) for COVID-19 diagnosis, primarily due to its potential for rapid detection. Deep learning models and computer vision techniques such as convolutional neural networks (CNN) can diagnose COVID-19 by classifying CT scans into categories such as COVID-19, pneumonia, and healthy [5], [6], [7], [8], [9], [10].

Current CNN models have achieved state-of-the-art results in image classification, including the classification of CT scans. However, these models only perform well when the training and testing datasets are highly similar, and their performance may suffer when applied to data from different hospitals. Many current approaches fail to consider that different medical centres have different CT machines and patient populations, resulting in dissimilar source and target data distributions. This is known as domain shift and hinders the classification accuracy of CNN models across different datasets and medical platforms. Therefore, we seek a Cross-Platform deep learning diagnosis framework that could effectively transfer knowledge between domains and accurately classify CT images across all hospitals. This would drastically reduce the costs of implementing computer-assisted diagnosis for individual hospitals, hence accelerating COVID-19 diagnosis as shown in Fig. 1.

Fig. 1.

CT images collected from different medical platforms have domain shifts that decrease the performance of A.I. diagnosis models.

In theory, the performance penalties caused by domain shift can be mitigated with supervised transfer learning, where a model pre-trained on a large set of labelled CT scans (i.e. a label-rich source domain) is fine-tuned by additional training on a smaller set of labelled CT images in the target domain. However, it is often impractical and costly to manually label a sufficient quantity of CT scans for effective fine-tuning, making this approach unsuitable for COVID-19 diagnosis. Conversely, unlabelled data is more abundant since it does not require additional labour from medical professionals. As a result, it would be far more practical to harness unlabelled data for model training. The traditional solution to utilise unlabelled target data is unsupervised domain adaptation (UDA) [11], which has been successfully utilised in a wide range of applications, including computer vision [12], natural language processing [13] and wireless sensing [14]. UDA enables a model to transfer knowledge learned from a label-rich source domain to an unlabelled target domain. However, two challenges hinder the application of UDA in COVID-19 diagnosis. The first challenge is data privacy: UDA methods typically require access to both the source and target data. This is problematic because medical data is highly sensitive, and accessing the source data raises privacy concerns. Therefore, we require a source-free domain adaptation technique to protect the privacy of patient data initially used to train the source model. The second challenge is data imbalances, where there is a disproportionate distribution of easy and hard samples or data classes [15]. During training, the model can be overwhelmed by the majority class or easy samples. This can lead to overfitting, thus, poor generalisation and degraded performance.

To address the aforementioned issues, we develop a Cross-Platform, Privacy-Preserving COVID-19 diagnosis network (CP${}^3$Net), which achieves universal diagnosis performance while guaranteeing patient privacy. CP${}^3$Net consists of two primary components: privacy-preserving domain adaptation and imbalance-robust loss function. (1) CP${}^3$Net achieves privacy-preserving UDA by training a source model on source data, then generating a target model from the source model. To do this, the source model classifier is directly copied to become the target classifier, and the source model feature extractor is used to initialise the target model feature extractor. To mitigate domain shift between source and target datasets, we align the source and target feature outputs by training the target feature extractor with Information Maximisation and self-supervised Pseudo Labelling [16]. Rotation classifier training is also used to improve feature learning [17]. (2) CP${}^3$Net mitigates difficulty and class imbalances in CT data by improving the loss function. We choose Focal Loss as it allows our network to place greater training emphasis on the most important samples [18]. The final result is a deep learning framework that is ethical, economical, and effective for CT-based COVID-19 diagnosis across all medical platforms.

The contribution of our study is summarised as follows:

• •We propose a new CNN-based deep learning network which uses unsupervised domain adaptation to classify CT scans from unlabelled target domains, enabling easy and efficient application of computer-assisted COVID-19 diagnosis in all medical centres.
• •We implement a source-free UDA approach which preserves the privacy of the sensitive medical data used to generate the source model.
• •We alleviate the performance detriments from class imbalances and difficulty imbalances in CT data by utilising a unique loss function which places training focus on the most important samples.
• •We construct a COVID-19 benchmark comprised of 3 domains and 6 transfer tasks, and we test our proposed approach against various popular UDA methods. Our evaluation demonstrates the impact of domain shift on diagnosis performance. Our method achieves state-of-the-art results with the highest average accuracy out of all methods tested and a 9.60% higher average accuracy than the source-only method.

2. Related work

2.1. CT image diagnosis

Computed tomography has revolutionised disease diagnosis with its unprecedented availability and speed of operation [19]. It remains today as the most common imaging technology for diagnosing potential health problems [20] such as chronic pulmonary thromboembolism [21], acute pancreatitis [22], and pneumonia [23]. Some recent research has evaluated CT imaging as a method of diagnosing COVID-19 and found a high sensitivity of 67%–100% and specificity of 83-100% [24]. This fact, combined with the widespread availability and speed of CT imaging, makes it a viable COVID-19 diagnosis tool that has already been widely adopted in countries such as China [7].

2.2. Deep learning for CT diagnosis

Deep learning has been applied in various aspects of COVID-19 research, including the analysis of CT scans to diagnose COVID-19 [6]. [25] combined a CNN and Long Short-Term Memory (LSTM) to form a hybrid network and applies it to a dataset consisting of 4575 X-ray scans, achieving 99.4% accuracy. [26] proposed a vanilla CNN architecture termed EMCNet, and tested it on a self-collected dataset of $4600images$ encompassing $2$ classes: COVID-19 and Normal. Their network achieved 96.52% accuracy on the test set. [27] applied a pre-trained ResNet-v2 with group normalisation and weight standardisation to investigate how their results were influenced by hyperparameter settings [28], [29]. They achieved the highest accuracy of 99.2% using their Bit-M model incorporating transfer learning. [5] proposed a deep network architecture, COVIDNet-CT, that uses machine-driven design exploration to accommodate COVID-19 diagnosis using CT scans. [8] developed a novel CNN model called UA-ConvNet, which, along with detecting COVID-19 from Chest X-ray images, estimates the uncertainty in the model’s predictions using the posterior predictive distribution obtained from Monte Carlo dropout. [9] offered an overview of current deep neural networks for image classification and evaluated various popular pretrained CNN architectures on COVID-19 diagnosis. They found that their implementation of AlexNet achieved the best accuracy of 97.6%. [30] combined the Inception CNN architecture with Marine Predators Algorithm to achieve a high classification accuracy of more than $98$% on two testing datasets. Their approach reduced the computational complexity of X-ray image classification. [10] presented a novel feature selection algorithm. They enhanced the cuckoo search algorithm with fractional-order calculus and heavy-tailed distributions to achieve high performance while decreasing computational complexity and increasing convergence rate. [31], [32], [33], [34] present several optimisation algorithms which offer good performance for challenging optimisation tasks. All these works introduce interesting, innovative points and are certainly valuable for deep-learning-based COVID-19 diagnosis. However, these studies fail to address the major obstacle that prevents deep learning algorithms from being widely adopted in COVID-19 diagnosis. Namely, achieving high classification performance across all medical platforms where domain shift is present while simultaneously respecting data privacy. This is the primary challenge we aim to tackle in this paper.

2.3. Domain adaptation

Domain adaptation is a machine learning technique that aims to adapt a model trained on one dataset (the source domain) to perform well on a different dataset (the target domain), particularly when labelled target data is insufficient, and domain shift is present. Over the years, many methods of domain adaptation have been proposed. [35] introduced a CNN architecture incorporating an adaptation layer to minimise the Maximum Mean Discrepancy [36] between source and target representations. They suggest that this maximises domain confusion, leading to similar classification performance in the source and target domains. [37] proposed another divergence-based DA method, where they minimise the difference in second-order statistics between the source and target distribution through a nonlinear transformation [38]. [39] trained a domain discriminator and the feature extractor in an adversarial fashion to generate similar features from source and target data. [40] designed a local maximum mean discrepancy to align subdomain distributions of source and target domains, and [41] used Batch Nuclear-norm Maximisation to maximise the nuclear norm of the batch output matrix.

3. Method

Our method is motivated by the scenario of source-free domain adaptation, where we only have access to the source model and unlabelled target data. The whole algorithm is depicted in Fig. 2. First, the source model minimises Focal Loss, a unique loss function that addresses CT data imbalances by selectively weighting training samples. Next, the source classifier is frozen and transferred to the target model while the source feature extractor initialises the target feature extractor. Finally, source-free UDA is conducted by training the target model with information maximisation, self-supervised pseudo-labelling, and rotation classifier training.

Fig. 2.

An illustration of CP${}^3$Net’s training pipeline. A source model is first trained on labelled data by minimising Focal Loss. The trained parameters are then used to initialise the target feature extractor while the source classifier is frozen and directly transferred as the target classifier. The target feature extractor is then trained by Information Maximisation, self-supervised pseudo-labelling, and RotNet.

3.1. Problem formulation

We aim to classify a given set of CT scans using source-free UDA. We start with a source model $f_s\left({\mathcal{X}}_s\right)\to {\mathcal{Y}}_s$ that predicts the labels ${\left\{y_s^i\right\}}_{i=1}^{n_s},y_s^i\in {\mathcal{Y}}_s$ of $n_s$ CT images ${\left\{x_s^i\right\}}_{i=1}^{n_s},x_s^i\in {\mathcal{X}}_s$ in the source domain ${\mathcal{D}}_s$. Then, given $n_t$ unlabelled CT scans ${\left\{x_t^i\right\}}_{i=1}^{n_t},x_t^i\in {\mathcal{X}}_t$ from the target domain $D_t$, we attempt to find a target model $f_t\left({\mathcal{X}}_t\right)\to {\mathcal{Y}}_t$ that correctly predicts their labels ${\left\{y_t^i\right\}}_{i=1}^{n_t},y_t^i\in {\mathcal{Y}}_t$. Throughout the entire training pipeline, the source domain $\{{\mathcal{X}}_s,{\mathcal{Y}}_s\}$ is only accessed to train the source model initially. During training and inference of the target network, the source data is never accessed, thereby protecting the privacy of the patients who constitute the source domain.

3.2. Imbalance-robust supervised learning in the source domain

CP${}^3$Net first generates the source model $f_s\left({\mathcal{X}}_s\right)\to {\mathcal{Y}}_s$, which is a CNN consisting of a feature extraction module $g_s$ and a classifier module $h_s$. However, we observed that some CT scans may be harder to classify than others (difficulty imbalances), and the availability of CT images varies across different health conditions (class imbalances). These data imbalances may be detrimental to model training based on the traditional cross-entropy (CE) loss since CE weighs all samples equally, potentially leading to biased predictions that prioritise majority classes or easier samples. To address this, we adopt Focal Loss [18] as CP${}^3$Net’s loss function:

$${\mathcal{L}}_{FL}\left(p\right)=\left\{\begin{array}{cc}-ϵ{(1-p)}^{\gamma }log\left(p\right),\hfill & y=1\hfill \\ -(1-ϵ)p^{\gamma }log(1-p),\hfill & otherwise\hfill \end{array}\right)$$ (1)

where $p$ denotes the predicted probability for the ground-truth label, $\gamma \ge 0$ is a focusing parameter that determines how much relative emphasis is placed on poorly classified cases, and $ϵ$ is a class balancing parameter. This loss function differs from the traditional Cross Entropy (CE) loss

$${\mathcal{L}}_{CE}\left(p\right)=\left\{\begin{array}{cc}-log\left(p\right),\hfill & y=1\hfill \\ -log(1-p),\hfill & otherwise\hfill \end{array}\right)$$ (2)

in the addition of the $\left\{\begin{array}{cc}ϵ,\hfill & y=1\hfill \\ (1-ϵ),\hfill & otherwise\hfill \end{array}\right)$ class balancing multiplier and the $\left\{\begin{array}{cc}{(1-p)}^{\gamma },\hfill & y=1\hfill \\ p^{\gamma },\hfill & otherwise\hfill \end{array}\right)$ probability modulator. The class balancing multiplier places the training focus on the correct class, while the probability modulator decreases the loss function’s output for higher predicted probabilities to prevent easy classifications from drowning out the training process. Fig. 3 offers a visual comparison between Focal Loss for various $\gamma$ values and vanilla cross entropy (CE) loss. As we can see, Focal Loss can selectively output relatively higher loss values for difficult samples, while vanilla CE loss can de-emphasise difficult samples and output high losses for confident samples. In effect, Focal Loss allows our model to focus its training efforts on the most important samples, thus improving training efficiency and final classification performance. Focal Loss is minimised using a standard backpropagation procedure with stochastic gradient descent (SGD). Eventually, we obtain a source model that effectively alleviates data imbalances and accurately classifies source data.

Fig. 3.

A visual comparison between Focal Loss for various $\gamma$ values and vanilla cross entropy (CE) loss on the true class ($y=1$).

3.3. Privacy-preserving source-free domain adaptation

Now that we have a well-performing source model, we protect the privacy of patients by not accessing the source domain. Instead, our source model is used to generate our target model. We freeze and transfer the source classifier $h_t=h_s$, and use the source feature extractor to initialise our target model $g_t\leftarrow g_s$. Our challenge then becomes learning a target feature encoder specific to the target domain that produces features which can be accurately classified by the source classifier that now becomes our target classifier.

CP${}^3$Net learns the optimal target feature extractor with Information Maximisation (IM), which has been proven effective in source-free domain adaptation [16]. Specifically, IM consists of the following entropy ${\mathcal{L}}_{ent}$ and divergence ${\mathcal{L}}_{div}$ losses:

$${\mathcal{L}}_{im}={\mathcal{L}}_{ent}\left(\hat{p}\right)+\alpha {\mathcal{L}}_{div}\left(\overline{p}\right)$$ (3)

$${\mathcal{L}}_{ent}\left(\hat{p}\right)=-\hat{p}log\left(\hat{p}\right)$$ (4)

$${\mathcal{L}}_{div}\left(\overline{p}\right)=\overline{p}log\left(\overline{p}\right)$$ (5)

where $\alpha$ is a weight parameter; $\hat{p}=max\left(\delta \left(f_t\left(x_t\right)\right)\right)$ denotes the highest predicted probability for all classes, serving as a substitute for the standard ground-truth probability that is unavailable in this scenario; $\overline{p}={\mathbb{E}}_{x_t\in {\mathcal{X}}_t}\left[\delta \left(f_t\left(x_t\right)\right)\right]$ is the target domain’s mean output embedding; $f_t\left(x_t\right)$ is the model’s output logits, and $\delta$ refers to the softmax function $\delta \left(a_i\right)=\frac{e^{a_i}}{{\sum }_j{e}^{a_j}}$ which normalises these logits into probabilities that sum to $1$. On one hand, minimising ${\mathcal{L}}_{ent}$ polarises $\hat{p}$ to $1$ or $0$, increasing our model confidence in individual predictions; on the other hand, minimising ${\mathcal{L}}_{div}$ makes $\overline{p}$ less certain, creating diverse predictions for all classes in the entire target image domain. In essence, IM makes our target outputs individually certain and globally diverse, thereby clustering our target outputs and aligning them with the unseen source outputs(see Fig. 4).

Fig. 4.

Visualisation of the information maximisation losses ${\mathcal{L}}_{ent}$ and ${\mathcal{L}}_{div}$.

Unfortunately, the source data can sometimes be clustered into the wrong class. [16] argues that this is caused by forcing certainty in uncertain target outputs; for example, IM may force a CT scan whose true label is [1 0 0] with predicted probabilities of [0.4 0.5 0.1] to take on the incorrect output of [0 1 0]. To combat this possibility, we supervise the training of our target feature extractor with self-supervised pseudo labelling on the target data, similar to K-Means clustering and DeepCluster [42]. We first generate an initial centroid $c$ for all classes by the following:

$$\forall k\in \mathbb{K},c_k-=\frac{\sum _{x_t\in {\mathcal{X}}_t}{p}_k{g}_t\left(x_t\right)}{\sum _{x_t\in {\mathcal{X}}_t}{p}_k},$$ (6)

where $k$ indicates the class, and $p_k$ is the $k$th element in the target model’s softmax output.

Intuitively, a centroid represents the geometric centre of all data in a class, so all data should cluster around their respective centroids based on their classes. Consequently, the centroid acts as an indicator of the distribution of classes in a particular dataset, which can be used to obtain pseudo labels for a given data point in a way similar to K-Nearest Neighbours:

$$\widehat{y_t}=arg\underset{k}{min}\left(D\left(g_t\left(x_t\right),c_k\right)\right)$$ (7)

where $\widehat{y_t}$ refers to the pseudo label and $D(g_t\left(x_t\right),c_k)$ gives the cosine distance between the feature outputs $g_t\left(x_t\right)$ and the centroid $c_k$. We then generate a new centroid using our pseudo labels and iterate this process of obtaining pseudo labels from centroids to update new centroids:

$$\forall k\in \mathbb{K},c_k^{\prime }=\frac{\sum _{x_t\in {\mathcal{X}}_t}\mathbb{I}(k=\hat{y})g_t\left(x_t\right)}{\sum _{x_t\in {\mathcal{X}}_t}\mathbb{I}(k={\hat{y}}_t)},$$ (8)

$${\hat{y}}_t=arg\underset{k}{min}\left(d(g_t\left(x_t\right),c_k^{\prime })\right),$$ (9)

where $\mathbb{I}\left(\cdot \right)$ denotes the one-hot vector.

The result is a set of pseudo labels which we can use to construct an entropy loss term ${\mathcal{L}}_{PL}$ to supervise the training of our feature encoder:

$${\mathcal{L}}_{PL}(f_t,{\mathcal{X}}_t,{\hat{\mathcal{Y}}}_t)=-log\left({\delta }_{\hat{k}}\left(f_t\left(x_t\right)\right)\right),$$ (10)

where ${\delta }_{\hat{k}}\left(f_t\left(x_t\right)\right)$ refers to the softmax output for the “correct class” according to our pseudo labels. This updates our optimisation term to

$${\mathcal{L}}_{ent}\left(\hat{p}\right)+\alpha {\mathcal{L}}_{div}\left(\overline{p}\right)+\beta {\mathcal{L}}_{PL}(f_t,{\mathcal{X}}_t,{\hat{\mathcal{Y}}}_t),\Downarrow$$ (11)

$$-\hat{p}log\left(\hat{p}\right)+\alpha \overline{p}log\left(\overline{p}\right)-\beta log\left({\delta }_{\hat{k}}\left(f_t\left(x_t\right)\right)\right),$$ (12)

In addition to pseudo labelling, CP${}^3$Net employs another self-supervised learning strategy inspired by RotNet [17], [43]. The central idea behind this strategy is to learn effective feature representations by training our network to classify the 2D rotation of an input image as either 0°, 90°, 180°, or 270°. Since predicting the absolute rotation of a single image may be challenging, we adapt the rotation classifier to determine the relative rotation between a pair of images instead. In practice, we randomly rotate our input images 0°, 90°, 180°, or 270°to produce a rotated image, then feed the concatenated feature representations of the original image and the rotated image to our rotation classifier $h_c$. From this, we obtain another self-supervised learning loss:

$${\mathcal{L}}_{rot}(g_t,h_c,{\mathcal{X}}_t)=-log\left({\delta }_z\left(h_c\left([g_t\left(x_t\right)\oplus g_t\left(x_t^z\right)]\right)\right)\right),$$ (13)

where $x_t^z$ is our rotated image, $[g_t\left(x_t\right)\oplus g_t\left(x_t^z\right)]$ is the concatenated features of our original and rotated image, and ${\delta }_z\left(h_c\left([g_t\left(x_t\right)\oplus g_t\left(x_t^z\right)]\right)\right)$ denotes the softmax output produced by our rotation classifier for the correct classification, which is equivalent to the predicted probability for the true rotation. This new term updates our optimisation objective to

$${\mathcal{L}}_{ent}\left(\hat{p}\right)+\alpha {\mathcal{L}}_{div}\left(\overline{p}\right)+\beta {\mathcal{L}}_{PL}(f_t,{\mathcal{X}}_t,{\hat{\mathcal{Y}}}_t)+{\mathcal{L}}_{rot}(g_t,h_c,{\mathcal{X}}_t).$$ (14)

3.4. Summary of CP${}^3$Net

Algorithm 1 summarises the training structure of CP${}^3$Net, which offers several advantages over other deep learning methods for COVID-19 diagnosis. Firstly, CP${}^3$Net allows our network to be trained and applied to different datasets, thus opening the opportunity to diagnose disease with deep learning for smaller hospitals with less access to CT data. We achieve this by addressing the domain shift in CT data across different CT platforms and centres with a unique implementation of UDA, reaching significantly better performance than directly applying a source model to target data. Secondly, compared to existing UDA methods for classifying CT data, CP${}^3$Net does not require access to the source data, achieving test-time domain adaptation that protects the privacy of patients who constitute the source data. Thirdly, CP${}^3$Net mitigates easy-hard data imbalances and class imbalances, improving the network’s specificity of training and, thus, its performance in CT classification.

4. Experiment

We evaluate CP${}^3$Net by comparing it against various other popular unsupervised domain adaptation methods on a lung CT scan dataset consisting of 3 classes — healthy, pneumonia, and COVID-19. We find that CP${}^3$Net outperforms many other methods and generally offers good performance in diagnosing COVID-19.

4.1. Setup

Our COVID-19 benchmark was constructed using various online datasets and consists of CT scans of healthy patients, pneumonia patients, and COVID-19 patients. The benchmark is split into 3 domains labelled domain A, domain B, and domain C. These separate domains allow us to simulate the data distribution differences in actual CT data. Furthermore, these 3 domains vary significantly in size, which mirrors the heterogeneity of real CT datasets. Fig. 5 demonstrates several CT scans from each domain and each category. We can observe apparent visual differences between CT images from the different domains, such as variations in contrast, brightness, and dimensions of the image, thus demonstrating the presence of domain shift that can hinder network performance.

Fig. 5.

Visualisation of CT scans from three different domains with three categories. There are visible differences between the CT scans from different domains, visually indicating the presence of domain shift.

Domain A consists of 312 CT scans released by the University of Montreal and compiled by P. Raikote [44]. 132 of these scans are COVID-19, 90 are pneumonia, and 90 are healthy. Domain B has 2016 images collected from various online sources and compiled by D. Deshpande [45]. 69 of these images are COVID-19, 617 are pneumonia, and 1330 are healthy. Domain C is the largest one, with 6432 images from another set of online sources: 576 are COVID-19, 4273 are pneumonia, and 1583 are healthy [46]. Our COVID-19 benchmark has 8752 total CT scans: 777 COVID-19, 4980 pneumonia, and 3003 healthy. These statistics are summarised in Table 1. In addition, we have calculated the normalised per-channel mean and std pixel values for each domain which we can use as a rudimentary similarity metric. With these three data domains, we create six transfer tasks which entail all possible permutations of source and target domains: $A\to B$, $B\to A$, $A\to C$, $C\to A$, $B\to C$, and $C\to B$. We expect relatively poor performance on more difficult tasks, such as $A\to B$ and $B\to A$, where the two domains have limited training samples and a relatively significant difference in mean pixel values. By the same token, we predict relatively high performance on transfer tasks such as $C\to B$ since the smaller differences in pixel mean values indicate a lesser degree of domain shift.

Table 1.

Statistics of the dataset.

	A	B	C
COVID-19	132	69	576
Pneumonia	90	617	4273
Healthy	90	1330	1583
Total	312	2016	6432
Mean $\mu$	0.4948, 0.4948, 0.4952	0.4882, 0.4883, 0.4883	0.4917, 0.4917, 0.4919
Std $\sigma$	0.2815, 0.2815, 0.2816	0.2482, 0.2482, 0.2482	0.2301, 0.2301, 0.2302

We first train and test a neural network on each target domain. This scenario mitigates domain shift since the training and testing data both come from the same sample. Baseline performance is then established by testing source-only networks on every other target domain, allowing us to investigate the performance penalty caused by domain shift. We then test our method against the following popular UDA methods: DAN [36], DeepCoral [37], DANN [39], DSAN [40], and BNM [41].

To guarantee a fair comparison between all the methods, we used the same data, trained the models for the same number of iterations on the same hardware, and adopted the same backbone architecture. Furthermore, we experimentally determined the optimal parameters for our method and used the optimal hyperparameters determined by the original authors for all other methods. We also adopted the same image transformations across all methods: resize to 224 × 224 for testing data, and resize to 256 × 256 followed by 224 × 224 random crop for training data. Lastly, all experimental trials were repeated three times to ensure reliable and comparable performance data.

4.2. Implementation details

4.2.1. Network architecture

For all our tests, we adopt a pre-trained 50-layer deep neural network ResNet-50 as our backbone [47]. As with [16], [17], [48] we adopt a $256$ dimensional bottleneck layer into the model architecture. We use a 32 GB NVIDIA V100 GPU to train and test all our networks, and our code is implemented using PyTorch v1.9.1.

4.2.2. Hyperparameters

We train our network and minimise the loss function with the standard procedure of SGD (stochastic gradient descent [49]) with Nesterov momentum of 0.9 [50] and weight decay of 0.001. Our initial learning rate $l{r}_0=0.01$, and it decays with $lr=l{r}_0{(1+{\gamma }_{lr}p)}^{\upsilon }$, where $p$ is used to measure the linear progress of training as a fraction from $0$ to $1$, and is calculated by the current training iteration divided by the total iteration. We use ${\gamma }_{lr}=10$ and $\upsilon =-0.75$, so our learning rate $lr=0.01{(1+10p)}^{-0.75}$. The models were trained for 20,000 iterations, at which point the models’ performance ceased to improve with additional training. Focal Loss used the parameters $\gamma =2,ϵ=1$. Our loss balancing parameters were set to $\alpha =0.3,\beta =0.6$.

4.3. Computational complexity

To determine the complexity of our method, we measured our model to have 5.19G FLOPs and 23.51M trainable parameters. As we did not modify the ResNet backbone architecture, the complexity and memory of CP${}^3$Net are identical to a standard ResNet architecture. The training time of our method is greater than the other methods due to the optimisation process of various losses during domain adaptation. However, this will not significantly detriment the model’s applicability in real-world situations, as our model does not introduce extra computation during the inference process; therefore, diagnosis via CT classification will be equally efficient.

4.4. Overall results

To begin with, we evaluated various traditional classification techniques on the six transfer tasks we constructed earlier, which encompass all permutations of source and target domains, and the results are shown in Table 2. We use the percentage of correct classifications as our accuracy metric. Some techniques are able to achieve high classification performance on particular transfer tasks such as $C\to B$, $B\to C$, and $C\to A$, suggesting that these domains have highly similar feature spaces, which is expected as CT images are relatively standardised. Nevertheless, we find that these traditional techniques perform rather poorly on average. Therefore, there is a need for deep learning techniques to achieve higher classification accuracy.

Table 2.

Classification accuracy of various traditional techniques.

Method	$A\to B$	$A\to C$	$B\to A$	$B\to C$	$C\to A$	$C\to B$	Average
SVM (rbf kernel)	41.74	58.47	65.38	76.77	83.65	87.35	68.89
Gaussian naive bayes	33.33	31.44	74.36	74.36	65.06	78.27	59.47
Random forest	37.25	38.77	80.45	89.79	90.71	97.02	72.33
QDA	49.36	48.17	42.31	8.96	57.05	64.19	45.01
AdaBoost	31.60	56.58	64.42	79.37	76.60	80.85	64.90
Average	38.66	46.69	65.38	65.85	74.61	81.54	62.12

We next test the deep CNN-based UDA methods on our CT dataset. Table 3 shows all of our performance data. The results show that deep learning techniques are far more effective at CT classification than traditional algorithms, outperforming traditional algorithms in almost all cases. We expect the UDA techniques to perform even better if the source training datasets were larger. Further, we can deduce the difficulty of these domain transfer tasks and, thus, the similarity between different domains. Comparing the average results for the UDA tasks, we again see that the domain adaptations from $C\to B$, $B\to C$, and $C\to A$ are the easiest, with average accuracies of 89.69, 88.80, and 84.33 respectively. These transfer tasks have relatively low differences in mean pixel values, as shown in Table 1, thus supporting our hypothesis that domain shift hinders classification performance. Despite the similarity in mean pixel values, $A\to C$ is an anomaly with a poor average accuracy of 66.97. We hypothesise that the poor domain adaptation performance is caused by the small size of domain A, which leads to insufficient training data and a higher probability of difficulty and class imbalances. In this non-ideal situation, CP${}^3$Net outperforms the other UDA methods, suggesting that our imbalance-robust training framework is effective.

Table 3.

Evaluation results on cross-domain CT COVID-19 diagnosis.

Method	$A\to B$	$A\to C$	$B\to A$	$B\to C$	$C\to A$	$C\to B$	Average
Source only [47]	60.19	62.03	45.21	90.03	80.82	96.47	72.46
DAN [51]	60.24	66.07	48.29	88.39	84.59	96.01	73.93
CORAL [38]	57.91	58.93	45.89	90.70	82.19	94.92	71.76
DANN [39]	59.25	63.26	68.49	85.35	82.88	97.10	76.06
DSAN [40]	52.05	50.33	58.90	94.41	80.82	96.63	72.19
BNM [41]	72.63	66.16	37.33	92.21	70.21	96.27	72.47
Ours	71.54	85.81	86.30	87.74	96.23	64.75	82.06
Average	61.22	66.97	59.68	88.80	84.33	89.69	75.10
Target only	93.78	95.80	98.48	95.80	98.48	93.78	96.02

Additionally, we test a basic ResNet-50 network for each target domain to investigate the base accuracy of deep learning on COVID-19 classification. We find that it is possible to achieve a high average accuracy of 96.02% despite the simplicity of our network, which is likely due to the fact that domain shift is not a relevant issue when the training and testing data are highly similar. This scenario is analogous to many previous studies that applied deep learning to COVID-19 identification. The dramatic drop in performance between training on the target dataset and the domain adaptation tasks suggests that domain shift is a significant barrier in using neural networks to diagnose COVID-19, therefore justifying the necessity for our research.

4.5. Ablation study

To examine the contribution of incorporating Focal Loss into CP${}^3$Net, we conduct an ablation study for Focal Loss. In tests without Focal Loss, a traditional entropy loss is used for the optimisation tasks; the entropy loss is replaced by Focal Loss for Focal Loss tests. All other parameters of CP${}^3$Net were constant across the tests. Table 4 demonstrates these results. It is observed that the inclusion of Focal Loss makes a positive impact of on average 2.07% on CP${}^3$Net’s performance.

Table 4.

Ablation Study of imbalance-robust learning.

Method	$A\to B$	$A\to C$	$B\to A$	$B\to C$	$C\to A$	$C\to B$	Average
CP${}^3$Net w.o. ${\mathcal{L}}_{FL}$	55.94	83.13	86.99	81.56	96.92	75.38	79.99
CP${}^3$Net (full)	71.54	85.81	86.30	87.74	96.23	64.75	82.06

5. Discussion

Overall, CP${}^3$Net offers several advantages and disadvantages compared to other deep-learning-based COVID-19 diagnosis frameworks.

Advantages:

• •We address practical, application-level challenges in implementing computer-assisted COVID-19 diagnosis.
• •Our model uses unsupervised domain adaptation to achieve consistently high classification accuracy across different medical datasets without requiring labelled target data, thus enabling fast and economical application of the model in all hospitals.
• •Our source-free UDA method protects the privacy of patients. No source data will be accessed after the generation of the source model, meaning that hospitals can successfully utilise a pretrained source model without privacy concerns.
• •Our enhanced loss function mitigates the negative impacts of CT data imbalances, making our training process more effective and efficient.

Disadvantages:

• •A large quantity of labelled data is still needed to generate an effective source model initially.
• •Our model has a considerable computational cost in the training process. However, this is not overly disadvantageous because there is no huge computational cost during the inference process once the model is deployed.
• •Our deep CNN-based approach offers little insight into how COVID-19 can be identified from CT scans, such as the particular features that indicate positive cases.

6. Conclusion

This paper identified a research gap in applying deep neural networks to diagnose COVID-19 when domain shifts are prevalent. We then devised an original UDA method (CP${}^3$Net) to combat the performance penalties that result from domain shifts between medical platforms while preserving the privacy of highly sensitive medical data. Our method freezes the source model’s classifier and trains the target model’s feature extractor with information maximisation and pseudo-labelling such that the target features match the source features, allowing the source classifier to transfer its knowledge to the target domain effectively. Additionally, we enhance the loss function by replacing traditional entropy loss with Focal Loss, which allows the model’s training process to prioritise the most significant data samples. Our method was pitched against 5 other popular UDA methods on our custom COVID-19 CT dataset, and our method achieved the highest average accuracy on the 6 UDA tasks.

In future research, it would be interesting to expand our model’s medical-specific functionalities. For example, estimating the confidence of predictions could better inform medical professionals in determining the status of the disease. Additionally, our method should be evaluated in other scenarios where labelled data is not readily available, domain shift is present, and privacy is required.

CRediT authorship contribution statement

Yuanyi Feng: Conceptualization, Methodology, Experiment. Yuemei Luo: Methodology, Paper revision. Jianfei Yang: Conceptualization, Visualization, Investigation, Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

The authors do not have permission to share data.