Adversarial selective domain adaptation with feature cluster for skin cancer diagnosis

Abstract

Medical imaging approaches widely employ deep neural networks for the investigation and diagnosis of different skin disorders. However, recent studies suggest that even a proficient model based on deep learning might struggle with generalization when applied to datasets from disparate cohorts due to domain shift phenomena. Meanwhile, there are usually need many well-labelled images utilized for the training process to attain a stronger level of performance. In order to alleviate the domain shift and the necessity for adequate training data, we introduce a novel method termed as adversarial selective domain adaption with feature cluster (ASDA). It achieves effective performance improvement of model when the target dataset is smaller than the source dataset. Specifically, we generate a set of feature clusters for each sample in the target domain to alleviate the demand for data. Subsequently, a conditional domain adversarial network is used to mitigate domain shift. Finally, due to consistency issues between feature clusters and samples, we propose a method of selective minmax entropy to maintain consistency. Our method diverges from typical domain adaption approaches that solely target reducing the domain gap. Instead, we address both the discrepancy between domains and the problem of limited data in the target dataset simultaneously. Extensive experiments have been undertaken on datasets pertaining to skin cancer, that confirms ASDA’s efficacy in skin cancer diagnosis for dermatoscopic and clinic image.

Introduction

The majority of the human body is covered by skin, which protects the internal organs from external harmful elements such as heat, dust, ultraviolet rays, and contaminated water¹. However, direct contact with the skin can result in various skin conditions that can affect people of any age, including rosacea, eczema, moles, and cancer. Along with these diseases, skin cancer is becoming an increasingly significant hazard². Tumor cells are divided into malignant and benign classes. Malignant cells excessively spread and expand, whereas benign cells are simple cells that cannot spread or expand. Because cancer cells are malignant, treatment options including surgery, radiation, or chemotherapy are used. Tumors show various distributions based on age and gender range, and cancers reveal various symptoms depending on the organ they harm and the species. Therefore, early diagnosis is crucial in the treatment of cancer patients, including skin cancer patients³.

Melanoma, also known as malignant melanoma, is a recurrently occurring cancer in men and women and is a dangerous skin cancer⁴. Dermoscopy is commonly used as a noninvasive skin imaging method for the detection of melanoma. Without any additional observation, the accuracy for identifying melanoma is higher by utilizing dermoscopy⁵. However, the diagnostic accuracy depends on the dermatologist’s professional and experience skills. Although dermatologists can diagnose melanoma, the analysis outcomes from various doctors may be different⁶. Additionally, the best diagnosis is provided by a biopsy, which is done during surgery, and the patient may find it unpleasant⁷. It is crucial to make an accurate and timely diagnosis of cancer with the assistance of several clinicians and various necessary tools. Therefore, some computer assistance tools are used to aid the experts in this field.

The advancement of Convolutional Neural Networks (CNNs) has enabled AI-assisted diagnostic systems to exhibit expert-level performance in classifying skin cancers, a condition frequently diagnosed visually⁸. The considerable potential of these systems can aid in teledermatology as both a diagnostic tool and decision support system, thereby improving dermatological access in rural regions with limited medical resources⁹. However, substantial challenges emerge due to the limited availability of labeled skin disease images essential for constructing precise diagnostic models, a common issue encountered in medical imaging analysis. Efforts have been undertaken to tackle the challenge of data scarcity in skin lesion analysis¹⁰, for instance, one research approach involves transferring models trained on a source domain to a target domain using fine-tuning, several methods have been developed to utilize CNNs pre-trained on ImageNet effectively for addressing medical image analysis challenges. It can reduce the demand for labeled data, improve the generalization ability of the model, accelerate the training speed of the model, and reduce the risk of overfitting, another approach utilizes Generative Adversarial Networks (GANs) to augment the training dataset by generating synthetic images that encompass a broad range of skin colors and conditions. Further studies also investigate the use of GANs as an augmentation technique to introduce diversity into skin lesion training samples¹¹. They fine-tune a pre-trained CNN-based classification network for skin disease diagnosis by incorporating the generated images alongside the original training target set as input.However, even though these generated images are visually indistinguishable from real examples, their inclusion in the training set results in only marginal improvements over the baseline and mainly benefits rare cases, often at the cost of accuracy for common classes. Utilizing GANs for augmentation has shown minimal enhancements while requiring significant computational resources. Deep neural networks often demand extensive labeled datasets featuring diverse visual variations for effective training¹². However, this approach may not be viable for clinical applications due to the diverse acquisition conditions inherent in clinical practice (e.g., varying equipment setups), along with the high cost associated with annotating data for every different domain (e.g., a dataset from different modalities). So, we need suitable method to solve the problem of domain shift and limited data.

Recent studies have endeavored to tackle the challenges posed by limited target labels and domain distribution shifts¹³. Among these approaches, the primary strategies involve reducing the statistical distance between domains and leveraging adversarial learning to address domain shift. Additionally, some methods incorporate techniques from semi-supervised learning, such as the use of pseudo-labels. However, these methods can exhibit instability due to they often use noisy pseudo-labels to train model or on minimizing entropy of condition¹⁴, which can lead to potentially miscalibrated predictions¹⁵. See Fig 1 (top). Moreover, it is challenging to achieve satisfactory results when data is scarce.

Fig. 1.

Top: Conventional methods can exhibit instability due to they often use noisy pseudo-labels to train model or on minimizing entropy of condition, which can lead to potentially miscalibrated predictions. Botton: Our method generates a set of feature clusters for each sample in the target domain to alleviate the problem of insufficient data, while using the selective minmax entropy to correct incorrect pseudo labels.

To tackle the aforementioned challenges, we introduce a novel method named adversarial selective domain adaptation with feature cluster (ASDA) for unsupervised skin cancer diagnosis of transfer learning(see Fig 1, bottom). ASDA strives to learn features that are invariant across domains by minimizing domain discrepancy and to address the issue of accumulated errors from inconsistent pseudo label of feature clusters by selective minimax entropy. First, ASDA produces a feature cluster for each target sample by a random image transformations to mitigate the data scarcity problem within the unlabeled target domain. Then we take advantage of the feature cluster expansion domain adversarial training module for the extract domain-invariant features in an end-to-end manner by minimising domain discrepancy. However, in the case where the distributions of the source domain and target domain are disjointed (which is common for high-dimensional data like images), there is no assurance of correct labeling of target images and feature cluster labels even with a perfect distributional matching of representations. Therefore, a selective minimax entropy module is used for solving the above problem. In contrast to the majority of current transfer learning methods that solely strive to augmentation technique to diversify skin lesion training samples, our method focuses on mitigating the domain gap of extensive source and limited target domain and relieve noisy pseudo-labels. We identify the reliable and unreliable image based on the model prediction consistency on the augmented image sets. We then use the selective minimax entropy strategy to remold the model predictions, improving the effectiveness of transferring knowledge between source and target domain.

The key contributions of our work are as follows:

• We introduce a novel method termed adversarial selective domain adaptation with feature cluster (ASDA) method for automated diagnosis of skin cancer, that comprising feature cluster expansion domain adversarial training module and selective minimax entropy module. It achieves effective performance improvement of model when the target dataset is smaller than the source dataset. The skin cancer diagnosis scenario encounters the challenge of domain distribution shift of limited data and noisy pseudo labels can benefit from the proposed ASDA.

• A new learning strategy based on feature cluster expansion adversarial training is introduced to extract features for each target sample to promote dispersed representations of target data in the feature space, our method addresses both the problem of domain disparity and the limited data within the target domain. This leads to improved domain alignment, particularly when dealing with a target dataset that is markedly smaller in scale compared to the source dataset.

• We propose a selective minimax entropy strategy to remold the model predictions. We identify the reliable and unreliable feature cluster image based on the model prediction consistency on the augmented image sets. Specifically, we consider those augmented images with consistent predictions for origin image to be reliable and therefore minimize their prediction entropy to increase the model confidence score; meanwhile, those augmented images with inconsistent predictions for origin image may be unreliable and therefore we maximize their prediction entropy to decrease their model confidence score.

Related works

Deep learning for diagnosis task of skin cancer

Over the past few years, a multitude of methods have been introduced for the classification of skin diseases. The diagnosis of dermoscopy lesions has witnessed extensive usage of the Convolutional Neural Network (CNN). In particular, Shahin M, Chen F F, Hosseinzadeh A, et al.¹⁶ employ a collection of 16 distinct convolutional neural network models, leveraging deep learning techniques, to analyze more than 45,000 high-quality images from the HAM10000 dataset, covering seven categories of skin diseases. The outcomes indicate that the majority of these deployed models achieved an accuracy rate of up to 99%. In¹⁷, Wan Y, Cheng Y, Shao M. introduce a multi-scale long attention network (MSLANet) designed for the classification of skin lesions in dermoscopy images. MSLANet comprises three long attention networks (LANet) and attains a rank-1 average AUC of 93.7% on the ISIC 2017 dataset and an AUC of 92.4% on the SIIM-ISIC 2020 dataset. Moreover, Wang Y, Wang Y, Cai J, et al.¹⁸ propose a paradigm for representing relational features inside a single instance and incorporating it into existing research on knowledge distillation (KD). A self-supervised approach is employed to train a dual relational knowledge distillation architecture. Furthermore, the utilization of weighted softer outputs is employed to enhance the student model’s ability to capture more comprehensive knowledge from the instructor model. The experimental findings indicate that the streamlined MobileNetV2 model can get a classification accuracy of up to 85% for 8 distinct skin conditions, while simultaneously minimizing parameters and computational demands. Wang L, Zhang L, Shu X, et al.¹⁹ introduce a deep learning methodology designed to improve both the consistency within a certain class and the ability to distinguish between different classes in the automatic classification of skin lesions. The approach proposed by the researchers has strong generalizability and has the capacity to dynamically highlight more distinct locations within the skin lesion. In addition, Razzak I, Naz S.²⁰ present a multistage unit-wise deep dense residual network featuring transition and additional supervision blocks, which enforce shorter connections and lead to improved feature representation. To effectively combine skin disease data from different sites, Fu X, Bi L, Kumar A, et al.²¹ tackle these challenges by introducing a graph-based intercategory and intermodality network (GIIN) consisting of two modules. He X, Wang Y, Zhao S, et al.²² propose a novel approach called the co-attention fusion network (CAFNet), which utilizes two branches to extract characteristics from dermoscopy and clinical images. Additionally, a hyper-branch is incorporated to enhance and integrate these features throughout all phases of the network. However, several challenges must be overcome before the successful implementation of deep learning in medical image analysis, one such challenge is the requirement for large volumes of annotated datasets to train deep learning models²³.

Transfer learning for diagnosis task of skin cancer

In this context, transfer learning has emerged as a solution to mitigate the challenges posed by limited labeled data and the difficulty faced by deep learning methods in achieving high performance with small datasets that differ from the training dataset. An intuitive approach is to repurpose pre-trained models for related domains²⁴. For instance, to address the issue of limited target samples for fine-tuning a model, Gu Y, Ge Z, Bonnington C P, et al.²⁵ employed a fully supervised deep convolutional neural network classifier that had been pre-trained on the ImageNet dataset. They conducted an investigation on a two-step progressive transfer learning technique, wherein the network was fine-tuned using two separate datasets pertaining to skin diseases. Recently, Pérez E, Ventura S.²⁶ presents a novel and enhanced methodology for Progressive Growing of Adversarial Networks (PGAN) that leverages residual learning techniques. The utilization of this approach is strongly advised for the purpose of enhancing the training process of deep networks. It involves selecting samples to generate synthesized images, which are then combined with original samples from the target domain to form the training set. This approach effectively addresses data scarcity and imbalance issues. Moreover, Balaha H M, Hassan A E S.²⁷ utilized the meta-heuristic SpaSA optimizer to optimize hyperparameters, employing eight pre-trained CNN models. Anupama C S S, Yonbawi S, Moses G J, et al.²⁸ propose a novel Sand Cat Swarm Optimization with Deep Transfer Learning (SCSODTL) technique for skin cancer detection and classification(SCC).

The methodology of pre-training and fine-tuning has significantly advanced the current state-of-the-art in many machine learning difficulties and applications. Deep networks that have been pre-trained can be easily customized to suit certain jobs, even in situations where there is a scarcity of labeled data. However, in numerous practical scenarios, labeled training data is unavailable, necessitating the transfer of an adapting a deep network from a labeled source domain to an unlabeled target domain is available²⁹. Moreover, for GAN-based data augmentation, positive outcomes are observed primarily on out-of-distribution test sets. Given the costs and potential risks associated with GAN usage, these findings advocate for caution in adopting them for medical applications³⁰.

In order to attain effective domain adaptation for application of skin cancer diagnosis in real-world scenarios, we investigate the challenge posed by limited unlabeled data and domain shift in the target dataset. Our proposed ASDA method tackles these two issues. On the one hand, we use adversarial domain adaptation with feature cluster to solve the challenge associated with the unlabeled and limited target dataset and reduce the distribution shift. On the other hand, a selective entropy strategy is utilized to relieve the problem of noisy pseudo-labels.

Method

Overview of the model

An overview of the proposed ASDA method is shown in Figure 2. It concentrates on adversarial learning based on feature cluster expansion and feature cluster expansion of predictive consistency simultaneously. To solve the problem of domain shift and limited data, we employ domain-adversarial training with feature cluster to minimize the cross-domain discrepancy. To relieve the problem of error accumulation arising from inconsistent feature cluster of target images, we propose a new strategy to remold model confidence on the target domain which is showed in Figure 3. We tackle the challenge of unsupervised domain adaptation for skin disease diagnosis, which involves transferring an adequately model trained on extensive source domain of label to limited target domain of no label. In addition to addressing the covariate shift across domains with limited data of target domain, we specifically concentrate on the practical scenario of addressing the issue of inconsistent feature cluster labels, and present a selective minmax entropy method that results in robust domain alignment under the condition.

Fig. 2.

An overview of the proposed ASDA framework. We generate a set of feature clusters for each target domain image and train them together with the source domain image for adversarial training. Subsequently, the consistency of the target domain and corresponding feature cluster images is maintained through the selective minimax entropy module. We train the model in an end-to-end fashion.

Fig. 3.

An overview of the selective minmax entropy module. The target domain image and its corresponding feature cluster image are obtained with corresponding pseudo labels argmax((x)) through a model. The feature cluster image corresponding to each sample is checked for consistency, and the label of the most type of image is selected and compared with the pseudo label of the original sample. If it is consistent, the entropy of the feature cluster image is minimized; otherwise, the entropy of the feature cluster image is maximized.

Notation

In this context, let and represent the source and target domains, respectively. We denote and as input and output spaces, where model will learns a type of CNN mapping parameterized by . In the traditional UDA setup, the case where we have access to labelled data points, we are given a labelled source domain dataset denoted as where is the ith source domain image, is the corresponding label to ith source domain image. Furthermore, we possess an unlabeled dataset pertaining to the target domain , but we can not get label of target domain dataset. As per the UDA protocol, it is assumed that all data points originating from the source and destination domains possess an identical set of classes, . For a image x, we denote that the ultimate probabilistic result generated by the model is . For each target domain image , a pseudolabel is estimated .

Network model

In the first stage, we train CNN to classify labeled source examples correctly. To obtain discriminative features, an entropy minimization of labeled source is used to train the model, this step is crucial. we follow the practices of existing work on UDA^31–33, and include the following standard cross-entropy loss in the training loss. The objective is as follows:

1

Adversarial domain adaption with feature cluster expansion of random image augmentation

In the second stage, the feature representation is denoted as f and the classification prediction from the classifier is denoted as h. Next, given a target image , we apply a series of random image transformations³⁴, such as fast data augmentation by a reduced search space, to generate N augmented images based on random transformations:

2

Subsequently, introduced a discriminator to distinguish between the real data distribution and the generated data distribution. Inspired by GAN³⁵, we utilize a domain discriminator to differentiate features generated by distinct data domains. The domain discriminator’s accuracy refers to the degree of disparity in the distribution of edges between two data domains, the objective of the feature generator f is to deceive the domain discriminator D and minimize the disparity in edge distribution. However, adversarial training aligns marginal feature distributions, but this approach may be inadequate when the joint distributions of features and labels change between domains³⁶. Moreover, in multi-class classification, the feature distribution is often multimodal. Therefore, even if the discriminator is completely confused, it does not ensure that the two feature distributions are similar. This defect may also arise when the joint distributions of features and labels change between domains. To address these two issues, Conditional Domain Adversarial Network (CDAN)³⁷ conditions features x on classifier predictions and introduces multilinear map instead of x as the input to domain discriminator D. Meanwhile, we train the model using both the original images and their corresponding augmented feature cluster images:

3

Discriminative representations are promoted by minimizing the cross-entropy loss of source domain. Meanwhile, transferable representations are encouraged by reducing the domain adversarial loss between source domain and target domain.

4

where is the cross-entropy loss, is a hyper-parameter that trades off source error and domain adversary. Meanwhile, and .

Through the above methods, we address the problem of limited training data for target domain and the problem of domain distribution shift, CNN model can learn sufficiently good representations and reduce the differences in data distribution at the representation level. However, the possibility that the generated feature cluster image may have different pseudo labels from the original target domain image, adversarial training may have a negative impact on the model. Because in reality, we cannot guarantee that the generated feature cluster image will have the same pseudo labels as the original image and as the model trains against the original image of the target domain and the feature cluster image, it may not necessarily have a positive effect on the consistency of the model with the feature cluster images and original image.

Selective minmax entropy based on feature cluster consistency

Therefore, a method called SMME was proposed to address this issue. Frist, the output class distributions of the model are computed for the original samples. For the image , the class distribution of can be represented as , where p represents a vector that represents the probability of classes C. Subsequently, a pseudo-label is estimated for the feature , denoted as .

Based on a sets of augmented images, we also estimate its pseudo label, , where . Then we will perform statistical analysis on the augmented image pseudo labels, get the most number of identical pseudo labels from augmented images in class C and compare them with the pseudo labels from the original image . If , we consider the image as “reliable”. Similarly, If , we mark it as “unreliable”.

After we get the “reliable” and “unreliable” images, for a image marked as reliable, we enhance model confidence by minimizing predictive entropy³⁸ concerning one of its reliable augmented versions. On the contrary, when identifying such an image through predictive inconsistency between pseudo label of original image and feature cluster, we decrease model confidence by maximizing predictive entropy³⁹ concerning one of its dependable augmented versions.

Unlike other methods, we consider two scenarios of labels and choose appropriate methods to optimize the model, rather than only selecting samples with high confidence for training⁴⁰. Our selective minimax entropy objective is given by:

5

where , for reliable images of feature cluster, we minimize the entropy of its consistent versions rather than the entropy of the original image. This strategy aids in mitigating overfitting and promotes consistency between the feature cluster and the original image.

Overview, these loss functions form our complete objective we optimize is given by:

6

where and are non-tuned hyper-parameters which trade-off the magnitude of the contributions between two modules for training.

Algorithm 1.

Optimization.

Datasets

We select two datasets related to skin diseases, namely HAM10000⁴¹ and the seven-point checklist⁴² dataset, to validate our proposed methods. The HAM dataset, obtained from the ISIC archive, stands as the most extensive publicly accessible skin dataset and is a sophisticated and easily accessible resource for digital dermatoscopy. Seven-point checklist dataset comprises multimodal skin cancer cases collected from both dermatoscopic and clinical environments. Each case in this dataset includes a pair of images captured by clinical and dermoscopy, both reflecting the identical disease of a patient. These two datasets encompass samples from diverse disease and cohort distributions, making them ideal for studying domain shift in dermatology. Details regarding the disease distribution of these datasets are provided in Table 1 and Table 2. Table 1 offers descriptions of the two datasets, while Table 2 presents the sample sizes for each disease, with numbers in parentheses indicating percentages.

Table 1.

Dataset description.

Dataset	Abbrev.	Samples	Type
HAM10000	HAM	10015	Dermoscopic
Derm7pt-Derm	d7pt-d	1011	Dermoscopic
Derm7pt-Clinic	d7pt-c	1011	Clinical

Table 2.

Sample size and class distribution for each dataset.

Diagnostic classes	Abbrev.	HAM	d7pt-d	d7pt-c
Basal cell carcinoma	BCC	514(0.0513)	42(0.0426)	42(0.0426)
Benign keratosis	BKL	1099(0.1097)	69(0.0669)	69(0.0669)
Dermatofibroma	DF	115(0.0114)	20(0.0203)	20(0.0203)
Melanoma	MEL	1113(0.1111)	252(0.2553)	252(0.2553)
Nevus	NV	6705(0.6695)	575(0.5826)	575(0.5826)
Vascular lesion	VASC	142(0.0142)	142(0.0142)	142(0.0142)
Actinic keratosis	AK	327(0.0327)

HAM10000: The dataset consists of 10,015 dermatoscopic images that have been classified into 7 distinct categories. These categories include 5 benign categories and 2 malignant categories, specifically melanoma and basal cell carcinoma. The aforementioned images were gathered over a span of two decades from the countries of Australia and Austria. The dataset from Australia exclusively comprises digital images, while the dataset from Austria encompasses a combination of digital dermatoscopic images and non-digital diapositives. The non-digital diapositives were digitized by scanning techniques and human correction strategies.

seven-point checklist: A total of 1,011 instances with multimodal skin cancer are included in this dataset. Figure 4 demonstrates that each case in the dataset consists of a pair of clinical and dermoscopy photos, which depict the same lesion of a patient. Additionally, each image is accompanied by labels indicating the diagnosis (DIAG). The DIAG diagnosis system encompasses six distinct kinds of skin diseases, namely basal cell carcinoma (BCC), benign keratosis (BKL), dermatofibroma (DF), melanoma (MEL), nevus (NEV), and vascular lesion (VASC). The dataset’s data distribution is displayed in Table 2, revealing its imbalanced distribution across multiple categories in various classification tasks, hence increasing the complexity of these tasks.

Fig. 4.

Image samples from two distinct modalities are included in the HAM10000 and seven-point checklist datasets. The HAM10000 dataset comprises entirely dermoscopic images, whereas the seven-point checklist dataset contains both dermoscopic and clinical images. There is a noticeable shift in the domain between these two datasets as a result of variations in the capturing settings.

Experimental settings

The approaches that were suggested were executed using the PyTorch library on a GPU with an NVIDIA 4090. In this work, we use N = 3 random transformations: The method of RandAugment executes a series of N picture transformations that preserve labels, which are randomly selected from a pool of 14 transforms and transformation severity M = 2.0 to construct the feature cluster. The optimal values for N and M are contingent upon the scale of the data in the target domain and the computational resources available. Usually, a higher value for N is chosen to augment training data when dealing with a target dataset that has a restricted sample size. However, it is worth noting that the maintenance of consistency in feature cluster of randomly augmented original sample can present certain difficulties if N is excessively large and it would require significantly more memory for model training. Meanwhile, It is also necessary to set a suitable M to avoid generating a large number of inconsistent feature cluster images. Excessive maximum entropy training can have adverse effects on model training. ASDA utilizes the widely-used convolutional layers from ResNet50⁴³ as its backbone architecture. The weight parameters of the feature extractor in our model are initialized with the pre-trained backbone from ImageNet⁴⁴. Before training, all images underwent normalization to a uniform scale by subtracting the mean (Mn) and dividing by the standard deviation (Std). Additionally, images were augmented through horizontal flipping and random cropping to dimensions of . During testing, the images underwent resizing to and center-cropping to . Following this, prior to inputting the data into the networks, data standardization was performed using a zero mean and unit variance. The learning rate was initially set at 0.01 and subsequently decreased by a factor of 0.75 for each epoch, resulting in a total of 200 epochs of training. PyTorch was utilized to implement the algorithm, employing the SGD optimizer and momentum. The parameter for weight decay was assigned a value of 1e-3, while the momentum was assigned a value of 0.9. All competing algorithms across both datasets used a training batch size of 8. The optimal values for and are setting 1.0 and 1.0.

Evaluation metrics

For testing evaluation, we adopt the model that attains the best accuracy (ACC) score on the target domain.

In parameter-based transfer learning and domain adaption synthesis, we employed four evaluation metrics to assess the effectiveness of binary and multi-class classification. The criteria considered in this study are overall accuracy (ACC), sensitivity (SEN), specificity (SPC), and AUC (Area Under the ROC Curve) score. The acronym ACC represents the comprehensive rate of accurately identified samples and can be applied to both binary and multi-class classification. On the other hand, SEN, SPC, and AUC were strictly used for jobs involving binary classification. In the context of binary classification, melanoma and cancer conditions were classified as positive, whereas benign and non-cancer diseases were classified as negative. The SEN metric quantifies the ratio of accurately identified positive samples to the total number of positive samples, whereas the SPC metric quantifies the ratio of accurately identified negative samples to the total number of negative samples. The aforementioned three conditions are articulated as:

7

8

9

where , , N, , represent the true positives, true negatives, total testing samples, positively classified samples, and negatively classified samples, respectively. It’s important to note that sensitivity (SEN) and specificity (SPC) vary with different classification thresholds. Hence, the Area Under the Curve (AUC) metric is frequently employed for comprehensive measurements.

Result

Dermatoscopic image classification for skin cancer diagnosis

In this study, we conduct a comparative analysis between ASDA and various image classification domain adaptation methods, specifically Deep Adaptation Neural Network⁴⁵, Joint Adaptation Network (JAN)⁴⁶, Conditional Domain Adversarial Network (CDAN), Maximum Classifier Discrepancy (MCD)⁴⁷, Margin Disparity Discrepancy (MDD)⁴⁸, Fixmatch⁴⁹, Masked Image Consistency(MIC)⁵⁰ and Explicitly Class-specific Boundaries(ECB)⁵¹. In this experimental setup, we utilize the HAM dataset as the source dataset and the seven-point checklist dataset as the target dataset. The evaluation of performance is conducted using ResNet50 backbones.

In the first experiment, we extracted BCC, NV, and MEL from two datasets, with HAM as the source domain and Derm7pt-Derm as the target domain. Due to the high degree of malignancy of melanoma, we also evaluated the results of melanin being classified as a separate category. Table 3 reports the statistical results of these methods, from which we can find that:

Table 3.

Performance of ASDA compared to peer domain adaption methods on the d7pt-d dataset.

		Cancer vs Non-cancer1			Melanoma vs Benign2
Method	ACC	SEN	SPC	AUC	SEN	SPC	AUC
DANN	0.792	0.5238	0.9565	0.7961	0.4325	0.9660	0.7788
CDAN	0.774	0.4796	0.9461	0.8116	0.4206	0.9481	0.7826
MCD	0.796	0.5544	0.9530	0.8313	0.4921	0.9481	0.7734
MDD	0.789	0.5204	0.9496	0.8140	0.4524	0.9514	0.7829
JAN	0.801	0.5850	0.9409	0.8430	0.4921	0.9514	0.8154
MIC	0.762	0.5327	0.9334	0.7867	0.4243	0.9544	0.7744
ECB	0.788	0.5253	0.9661	0.8247	0.4367	0.9577	0.7963
Fixmatch	0.716	0.2041	0.9948	0.7313	0.1468	0.9935	0.6931
ASDA	0.814*	0.5476	0.9670	0.8778*	0.5079	0.9643	0.8595*

¹ The result of representing basal cell carcinoma and melanoma as positive.

² The result of representing melanoma as positive.

^* The symbol indicates that the value of the proposed method is significantly different from all other methods at a 5% level by Wilcoxon’s rank sum test⁵².

• Except for Fixmatch, despite the lack of labels in the target dataset, both ASDA and most peer methods achieve an AUC score exceeding 75%. Due to the lack of domain alignment in Fixmatch, which only utilizes knowledge from the source domain to select pseudo labels, the performance of the model is poor. These methods have evidenced the good performance of unsupervised domain adaptation in skin cancer diagnosis.

• The methods of statistical matching(i.e., JAN) are more effective in transferring knowledge than the methods of adversarial learning (i.e., DANN, CDAN, MCD, MDD and ECB). And due to MIC’s consideration of mask consistency, it may be difficult to meet this requirement during domain adaptation, resulting in poor performance. For the adversarial learning-based methods, there is not much difference in performance between these methods. Perhaps it is because there class balance gap is too large. So there are no apparent benefits to reducing the Margin Disparity Discrepancy between source and target domain.

• ASDA surpasses all comparable methods. In particular, it enhances the AUC for melanoma and cancer scores of the method that performed second-best by 4.41% and 3.48%, respectively, using ResNet50 as the backbone. Additionally, it boosts the ACC score of the method that performed second-best by 1.3% with ResNet50 as the backbone. The findings indicate that the domain adaption approach we have presented has efficacy in the automated diagnosis of skin cancer within triple categorization scenarios.

Clinical image classification for skin cancer diagnosis

We evaluate the effectiveness of ASDA and methods related to domain adaption in clinical image classification by comparing their performance on the Derm7pt-Clinic. Similarly, under the condition of ResNet50 as the backbone, HAM10000 dataset is used as the source domain, while Derm7pt-Clinic dataset serves as the target domain. Table 4 presents these results of these methods, from which we can find that:

Table 4.

Performance of ASDA compared to peer domain adaption methods on the d7pt-c dataset.

		Cancer vs Non-cancer			Melanoma vs Benign
Method	ACC	SEN	SPC	AUC	SEN	SPC	AUC
DANN	0.712	0.3639	0.9217	0.7169	0.3016	0.9449	0.7003
CDAN	0.718	0.2959	0.9513	0.7068	0.2817	0.9514	0.6936
MCD	0.713	0.2551	0.9617	0.6709	0.2619	0.9498	0.6822
MDD	0.712	0.3333	0.9322	0.6819	0.2817	0.9498	0.6818
JAN	0.708	0.3197	0.9252	0.6679	0.2659	0.9481	0.7001
MIC	0.709	0.3034	0.9220	0.6623	0.2456	0.9279	0.6927
ECB	0.717	0.3249	0.9232	0.7066	0.3137	0.9056	0.7057
Fixmatch	0.670	0.0374	0.9726	0.6175	0.0079	0.9984	0.6376
ASDA	0.723*	0.3469	0.9339	0.7167	0.3373	0.9254	0.7180*

• Although the gap between dermatoscopy and clinical imaging has become significant, the methods of unsupervised domain adaptation and ASDA have a ACC score higher than 70%. The results illustrate the efficacy of unsupervised domain adaptation in clinical skin cancer diagnosis.

• Although JAN is superior to adversarial based methods under dermatoscopy conditions, in cases where there is a significant gap in two domain, it may be due to the greater influence of the maximum and minimum values, resulting in poor effectiveness.

• Even DANN has higher accuracy (0.02%) than ASDA on AUC for cancer and no-cancer, it has a lower AUC(1.77%) than ASDA for melanoma. This implies that our method has a greater ability to detect skin cancer from clinical data.

Ablation study

We first verify the effectiveness of the key components of the proposed algorithm. Our baseline algorithm uses only CDAN for distribution adaptation. As shown in Figure 5, the adversarial training with feature cluster and selective minimax entropy result in a 2.2% and 1.8% ACC improvement, respectively, indicating that these two modules are useful for cross-modality skin cancer image classification. In addition, our method attains the optimal classification performance by combining these two components. We further explore the effectiveness of various elements of the selective entropy. As shown in Table 5, when using feature clusters for adversarial training, 79.6% of ACC was achieved. Subsequently, we minimized the entropy of reliable feature cluster images, but only improved by 0.6 percentage points,but with entropy maximization for unreliable ones, and it can be seen that the performance is 1.6% higher than entropy minimization of all images. This suggests the significant to remold the prediction confidence for unreliable images.

Fig. 5.

Comparison of ASDA with two baselines in terms of ACC(%) and AUC(%).

Table 5.

Ablations study for Selective minmax entropy.

Entmin	Entmax	Acc(%)
None	None	79.6
All images	None	79.8
None	Unreliable images	79.9
Reliable images	None	80.2
Reliable images	Unreliable images	81.4

Visualisation of feature distribution for different domains

In order to evaluate the efficacy for method that we are proposed for feature distribution learning, we utilize the t-SNE method⁵³ to map the data distribution of images from various domains onto a 2D plane in the high-level feature space. In this experiment, we collect all samples from each class in both the target domain and the source domain. Figure 6 illustrates the outcomes of the distribution of samples from both the source domain and the target domain. Figure 6(a) presents the visual distribution of the data obtained without domain adaptation, while Figure 6(b) illustrates the visual feature distribution of the data obtained utilizing ASDA.

Fig. 6.

Visualization of feature distribution through the t-SNE method. The colors cyan, yellow, and magenta represent the nevus samples, basal cell carcinoma samples, and melanoma samples in the source domain, respectively. The colors green, blue, and red correspond to the nevus cases, basal cell carcinoma cases, and melanoma cases, respectively, in the target domain.

From Figure 6(a), it is apparent that the feature distribution of the two domains are largely separated into distinct groups, suggesting a gap in feature distribution between the domains. However, the feature distribution of target samples across different categories are blended, indicating the absence of a clear classification boundary that could precisely discriminate categories from the source and target domain. Especially for two types of skin cancer, the effectiveness is relatively poor.

From Figure 6(b) and Figure 6(c), they respectively show the effects of using conditional adversarial domain adaptation and feature clusters. It can be seen that due to the lack of target domain data and incorrect pseudo labels, the effect of only using conditional adversarial domain adaptation is poor. However, the classification effect has been greatly improved after using feature cluster, especially for the classification of two types of skin cancer.

Differently, in Figure 6(d), the melanoma and basal cell carcinoma samples from the source domain (marked with magenta and yellow points) are intertwined with melanoma and basal cell carcinoma samples from the target domain (marked with red and blue points). Similarly, the nevus samples from both domains exhibit overlap, signifying effective mitigation of the domain gap. Meanwhile, by correcting incorrect pseudo labels, the classification performance of the model has been improved.

Computational Complexity

In our experiments, for ASDA with the ResNet-50 backbone, the network training for 200 epochs takes about 1.21 hours on 24GB NVIDIA GeForce RTX 4090 GPU. Besides, ASDA has a fast inference speed, which takes about 0.025 seconds per image pair. The fast training and inference speeds suggest that ASDA has the potential to be applied to real clinical workflows.

Statement

All methods were carried out in accordance with relevant guidelines and regulations.

All experimental protocols were approved by Chongqing University of Technology review board.

The informed consent was obtained from all subjects and/or their legal guardian(s).

Conclusion

A new unsupervised domain adaptation method is propsed by us for cross-domain skin cancer diagnosis. It resolves the problem of limited data in the specific field by generating a group of feature cluster for each individual sample in the target domain. In addition, it reduces the disparity between the source and target domains by transferring learned knowledge between the high-quality source samples and the limited target samples with their feature clusters. In order to improve the model’s capacity to generalize, we have developed the selective entropy module to ensure the consistency of feature cluster. Specifically, we compare the pseudo label among original target image and feature clusters images, if there is consistency in predictions between the original sample and its corresponding feature cluster, we minimize the entropy of the feature cluster image, if inconsistency, we maximize the entropy of the feature cluster image. The experimental results demonstrate that ASDA achieves an AUC score of 85.95% for melanoma diagnosis in dermatoscope imaging of skin cancer and an AUC score of 87.78%. The results presented here clearly show the effectiveness of our suggested ASDA method in improving the automated diagnosis and screening of skin cancer.

Limitations and prospect

When compared to other UDA methods, the spatial overhead of the feature clusters generated by the proposed method during training is reasonable. Regarding potential negative impacts, our method shares the same limitations as any other UDA algorithm: while they are generally effective in reducing the domain gap, this relies on the assumption that the domains share the same class data. Therefore, we hope that future work can achieve effective domain adaptation across different classes of data and, where possible, correct incorrect pseudo-labels caused by domain shift while conserving space.

Author contributions

Qiyu Gou conceived and conducted the experiments, Qiyu Gou and Guanxun Cui analysed the results, Qiyu Gou wrote the manuscript, and all authors read and approved the final version of the manuscript.

Data availability

Data openly available in a public repository. The data that support the findings of this study are openly available in HAM10000 at https://challenge.isic-archive.com. The data that support the findings of this study are openly available in seven-point checklis at https://derm.cs.sfu.ca/Welcome.html.

Declarations

Competing interests

The authors declare no competing interests.

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.