Skin cancer segmentation and recognition from dermoscopy images: a novel framework based on improved DeepLabV3+ and network-level fused deep architectures

Graphical abstract

Highlights

• •Design a DeepLabV3+ architecture with customized ResNet20 as a backbone model for skin lesion segmentation.
• •The purpose of ResNet20 is to repeat filters and depth for better learning on smaller and larger skin lesions.
• •Proposed a new MAKNet100 architecture for skin lesion classification.
• •Bayesian Optimization (BO) is utilized for the hyperparameters initialization.
• •Proposed model is interpreted using an explainable AI (XAI) technique named GradCAM Visualization.

Abstract

Introduction

Skin lesion segmentation and classification is an active research area in medical imaging for the large number of reported deaths in the recent years. Early diagnosis of skin cancer is essential to decrease the death rate and increase life expectancy.

Objectives

Several artificial intelligence (AI) based techniques have been introduced in the literature for the diagnosing skin cancer; however, due to challenge of imbalanced datasets, irregular lesion shape, presence of lesions on boundary regions, and selection of inappropriate model selection, the performance of AI model is highly impacted. Therefore, in this work our main objective is to propose a fully automated deep framework for skin lesion segmentation and classification with more efficient and effective way.

Method

This work proposes a novel framework for segmenting and classifying skin lesions using improved ResNet20-DeepLabV3+ and MAKNet100 deep models. In the segmentation task, a ResNet20 architecture is designed as a backbone of DeepLabV3+. In the designed ResNet20 architecture, a few grouped convolutional layers are added with smaller filter sizes to extract more insight information. A new MAKNet100 model is proposed in the classification task based on the network-level fusion of two custom models. The proposed network has few parameters and can extract more information about the lesion images. The proposed model is trained and further analyzed using the GradCAM explainable artificial technique (XAI) as a black box interpretation. Features are extracted from the self-attention layer and passed to classifiers for the final classification.

Results

The experimental process of the proposed framework is performed on HAM10000, ISIC-2018, ISIC-2019, and ISBI-2020 datasets with an accuracy of 90.5 %, 88.9 %, 84.5 %, and 96.35 % respectively and the highest obtained dice score on ISIC-2018 and HAM10000 is 94.63 and 96.69 % respectively.

Conclusion

The proposed framework obtained improved accuracy and precision rates for skin lesion segmentation and classification on these datasets. Moreover, the ablation study and comparison with existing techniques show the proposed framework’s dominance.

Introduction

Skin cancer is the most prevalent type of malignancy among people [1]. Because of the increasing loss of the ozone layer and worldwide air pollution, skin cancer affects more people than all other forms of Cancer combined [1]. Melanoma is more likely to result in death than other types of skin cancer [2]. Melanocytes create varying amounts and types of melanin based on individual characteristics. It not only colours our skin but also defends against UV radiation from the sun [3]. Human skin cancer exhibits a higher level of aggressiveness and prevalence [4]. It's caused by abnormal cell growth. The condition is attributed to the proliferation of aberrant cells. These cells undergo mitosis and undergo replication [5]. The prevalence of skin cancer has experienced a substantial rise in recent years [6].

Basal and squamous cell carcinoma are the predominant forms of non-melanoma skin cancer [7]. According to the World Health Organization (WHO), skin cancer accounts for approximately one-third of all forms of Cancer [8]. Annually, there is a rise in the number of skin cancer diagnoses and corresponding mortality [9]. Presently, 3 million incidences of non-melanoma skin cancer and 132,000 cases of melanoma are found each year in the world [10]. According to the WHO, 9500 individuals in the United States (US) alone are diagnosed with skin cancer each day, where the death numbers are two [11]. The average yearly cost to treat such illnesses is USD 3.3 and 4.8, respectively [12]. According to the data, the number of aggressive melanoma cases has climbed by 47 % during the previous ten years [13]. Approximately 100,000 new cases of melanoma are identified in Europe each year. Conversely, Australia has 15,229 recorded incidences of melanoma annually [14,15].

Improving the early and precise detection of skin cancers, especially melanoma, is crucial for improving patient survival and lowering death rates. This motivates the development of skin cancer segmentation and classification systems. With state-of-the-art artificial intelligence and medical imaging developments, these systems can provide accurate and effective diagnosis assistance, even in areas with limited dermatological specialists. The proposed technique reduces the number of unnecessary surgeries and treatments, which not only reduces the clinical burden but also maximizes healthcare resources.

Differentiating benign lesions from skin malignancies can be challenging [16]. Specialists in skin cancer use hand-held dermoscopy and visual examination to assess the skin lesions in their patients [17,18]. They could take dermoscopic (microscopic and macroscopic) images [16]. Dermoscopy is a noninvasive imaging technique using a light magnifying instrument and immersion fluid to examine the skin's surface [19]. The most popular dermatology imaging technique increased malignant patients' diagnostic accuracy by 50 %. However, using human vision to identify melanoma in skin lesion images may be arbitrary, inaccurate, and dependent on doctors with expertise [20]. The biopsy procedure has been used to examine skin malignancies for several decades. It is the most basic approach accessible, although its dependability is questioned [21]. To overcome the obstacles in melanoma identification, physicians typically use diagnostic techniques such as the ABCDE rule [22], three-point checklist [23], seven-point checklist [24], and Menzies [25] rule to determine the kind of skin lesions [26]. ABCDE represents asymmetry, border, color, diameter, and evolution [27]. A: The asymmetry attribute determines whether the color, shape, and borders of the two sides of the skin lesion match. Melanoma is likely to look asymmetrical in this instance [28]. B: Border indicates whether or not a skin lesion has smooth, distinct margins. The margins of melanoma will probably be sharp, fuzzy, and irregular [29]. C: Color attribute. A melanoma's hue varies from place to place and is frequently tan, brown, red, or black in various shades [30]. D: Diameter attribute. It gauges the skin lesion's approximate diameter. A melanoma often has a diameter larger than 6 mm or the size of an eraser pencil [31].

Dermatologists can employ CAD systems to detect skin lesions more rapidly and precisely. Computer-aided diagnosis (CAD) systems are necessary to support medical practitioners during the diagnostic process [32]. Most current CAD systems for dermoscopic images involve several steps, including image preprocessing, segmentation, feature extraction, feature scaling [33], and classification [34,35]. The significant variance in skin lesion images' texture, size, color, and location makes this segmentation step a challenging process [36]. Furthermore, the low contrast in the images prevents us from distinguishing between neighbouring tissues [36]. Other features, including blood vessels, ebony frames, ruler lines, air bubbles, and color illumination, make lesion segmentation more complex [37].

Deep learning show a huge success in the recent years for the classification and detection of medical diseases [[38], [39], [40]]. Recently, substantial improvements in skin lesion identification and classification have been achieved by using Convolutional Deep Neural Networks (CDNN) techniques [41]. The CNN allows lower-quality images to minimize the estimations and variables in the network. However, several important characteristics may be lost due to changes in filter and stride size [42]. Several CDNN techniques have been introduced in the literature to segment and classify skin lesions [43]. The deep learning models employed for skin lesion segmentation are based on the training data (original and ground truth (GT) images) [44]. In the segmentation process, the most challenging issue is the availability of GT images that are utilized to train a model for the segmentation process, such as U-NET, SegNet, and DeepLabV3. These automated techniques help dermatologists, as a second opinion, diagnose skin lesions [29]. The U-Net is the standard medical image segmentation architecture that can capture fine features and contextual information [45]. The DeepLab employed atrous convolution (or dilated convolution) to gather multi-scale contextual details of an image. A framework of Mask R-CNN that combines object identification with instance segmentation to reliably segment individual lesions and provide bounding boxes [46].

In the classification phase, traditional machine learning techniques focused on extracting handcrafted features; however, such features are insufficient to accurately classify skin lesions' complex structures. The handcrafted features include histograms of oriented gradients (HOG) [47], local binary patterns (LBP), and point features [48]. The DL techniques significantly impact the classification tasks, especially in medical imaging. The DL architecture successfully accurately classified complex pattern skin lesion types such as melanoma, nevus, and basal cell carcinoma. Features are usually extracted from the fully connected layers; however, in recent works, the global average pooling layer has been employed as feature extraction. In DL, several pre-trained models have been used for skin lesion classification, such as ResNet, Densenet, and EfficientNet. The individual model's performance is always challenging in achieving the target accuracy or precision rates. The researchers tried to resolve this challenge using features level fusion; however, this step increased the computational time and added some redundant information. In addition, interpreting these models is another challenge, as each model is a black box. The primary challenge is the ability to accurately segment and classify skin lesions is complicated by unbalanced datasets, abnormal shape lesions, low contrast, and the limitations of traditional and single deep learning models.

To consider all these challenges, we proposed a novel framework that includes two architectures: Modified LabV3+ for the segmentation of skin lesions and a fused architecture based on bottleneck and dense blocks for lesion classification. Our significant contributions to this work are as follows:

• •Design a DeepLabV3 architecture with ResNet20 as a backbone model for skin lesion segmentation. The purpose of ResNet20, instead of other models, is to repeat filters and depth for better learning on smaller and larger skin lesions.
• •Proposed a new deep-learning architecture named MAKNet100 based on a bottleneck and self-attention mechanism with 100 hidden layers for skin lesion classification. The proposed model consists of smaller learning parameters and minimum loss.
• •The proposed classification model is trained using optimized hyperparameters such as Bayesian Optimization (BO), and the trained model is further interpreted using an explainable AI (XAI) technique named GradCAM Visualization.
• •The proposed segmentation architecture is tested on the HAM10000 dataset, whereas the classification architecture is tested on HAM10000, ISIC2018, ISIC2019, and ISIC2020. Also, a detailed ablation study is conducted to validate the proposed model's accuracy.

Literature review

Deep learning techniques

Several computerized techniques have been introduced in the literature for skin lesion segmentation and classification [49,50]. Hatice et al. [51] created a quick and exact architecture for small-to-large-sized medical images. They concentrated on often detecting and segmenting the skin lesion. They presented an InSiNet deep learning-based CNN for detecting benign and malignant tumors. The computational speed and accuracy of the presented approach were compared to existing ML algorithms such as ResNet152V2, GoogleNet, EfficientNetB0, and DenseNet-201, and they showed improved accuracy. Bindhu et al. [52] developed an innovative technique for segmenting and obtaining the specific area affected in skin cancer images—the technology known as MFO-Fuzzy U-Net was introduced. The preprocessed images were sent as input to the Fuzzy U-net. The images were divided into segments and enhanced with MFO to expand the precision level. The performance of the provided technique was assessed on the ISIC 2017 dataset, achieving an accuracy of 97.57 %. Muhammad et al. [53] explained the challenges associated with computerized skin lesion detection from dermoscopic images, including artefacts (such as hairs), variability, variations in lesion form, and the extraction of irrelevant features. These challenges hinder the process of dividing and categorizing. The recommendation was to employ an optimized color feature for segmenting lesions and utilize deep neural networks for classifying skin lesions. The approach was evaluated on the ISBI 2016, 2017, and 2018 series datasets and an average accuracy of 90 % was obtained. Vardasca et al. [54] expert systems have been used in various contexts to support medical decision-making since the 1980 s. These technologies were more important than ever because of the high standards of medical treatment and the scarcity of human resources. One of the diseases that was growing more quickly was skin cancer, which was made more difficult by the shortage of dermatologists in the majority of the regions affected by it. Mhathesh et al. [55] presented a 3d convolutional neural network for recognizing light sheer microscopy images. The utilized dataset was categorized into two classes: bacterial and non-bacterial. The authors achieved the highest accuracy than the traditional techniques.

The DCNN model presented by Shahin et al. [55] performed better in classification rate than other transfer learning methods. The suggested strategy aims to differentiate benign from malignant skin lesions using the sigmoid output activation layer instead of the binary classification. Using the HAM10000 dataset, the presented technique was evaluated, and an accuracy of 93.16 % was obtained. Amina et al. [56] discussed the difficulties in extracting duplicate or irrelevant features, poor contrast lesions, and unbalanced datasets. They presented a hybrid deep-learning method with dual filtering to improve contrast for accurate lesion area detection feature extraction through TL, fusion, and classification for these challenges. Lesion segmentation was performed using the ISIC 2017 dataset, whereas classification was performed using the HAM10000 dataset. The accuracy was 95.6 % and 96.7 % for each dataset, respectively. Waqas et al. [57] presented a 38-layer deep-learning model for the skin lesion classification. The method is based on the filter size, depth, and hidden layers. The softmax classifier was utilized for the final classification. Two datasets, such as HAM10000 and ISIC2019, are employed and show better accuracy of 93.06 %. Rania et al. [58] introduced an improved multi-input color U-net model for accurately segmenting skin lesions based on their semantic properties. Three variants of the U-Net model have been introduced: Single Input Colour U-Net, Dual Input Colour U-Net, and Triple Input Colour U-Net. The feature maps obtained from each encoder subnetwork were inputted into the decoder subnetwork to create a segmented image map. Subsequently, the contributions were combined using a channel-wise attention module. In addition, the suggested CU-Net models performed better because of the creation of a composite loss function. Saleh et al. [59] developed an Explainable Artificial Intelligence (XAI) framework for early Warning detection of Myocardial Infarction (MI) based on DL and interpretation methods. Five pre-trained CNN such as InceptionV3, InceptionResNetV2, ResNet50V2, VGG16, Xception models were utilized along with Grad-CAM to generate saliency maps. The study used the Emidec dataset for MI from the MICCAI 2020 challenge which contains DE-MRI data for 150 patients; 100 cases are publicly available and labeled indications of normal and abnormal conditions. The images were augmented using CLAHE. Different data augmentation techniques were used to increase images due to class imbalance. The model evaluation was conducted using patients level split (70 % train, 30 % test). Among all models evaluated, InceptionResNetV2 had the highest accuracy, 83.33 % and F-score 88.10 %, and ResNet50V2 provided the most consistent and reliable focus regions associated with Grad-CAM visualizations. The study was limited to relatively small datasets and did not engage an external validation. And in spite of the variety of models that were evaluated, the models' predictions were not fused, nor were the clinical data utilized in the model training to minimize computational and access overload. AbuAlkebash et al. [60] developed an end-to-end AI framework meant to detect melanoma and other skin lesions using dermoscopic images. The authors focused on YOLOv8, the latest object detection version of YOLO, and used this as part of the deployed framework. The authors trained and evaluated the YOLOv8 model on the PH2 and ISIC2018 benchmark datasets. The authors used image pre-processing and augmentation techniques such as brightness modification, gaussian blur, and image rotation to support training process. YOLOv8 was also fine-tuned specifically for skin lesion detection, yielding mAP@0.5 scores of 99.50 % on PH2, and 97.30 % on ISIC2018. Comparing to previous studies in other public datasets, the results indicated a significant advancement over studies in earlier versions such as YOLOv4, YOLOv5, and YOLOv3. The authors used YOLOv8′s anchor free detection capabilities to structure image annotations within the experiment's design. Limitations of this study included the authors only focused on detection and excluded segmentation and classification, no interpretability features were provided, and while reasonable, the evaluation across datasets was limited to YOLOv8 without systematic consideration of generalizability across broader lesion types. Farea et al. [61] proposed a hybrid deep learning architecture for skin cancer classification reassessing the classification of skin cancer using and utilizing hybrid design reconciling multiple public datasets, and optimization. The study pooled over 27,000 images from seven publicly available datasets, including ISIC, PH2, Dermquest, MED-NODE and other(s), for nine types of skin cancer. The hybrid framework consisted of utilization of CNN and transfer learning with hyperparameter tuning using the Artificial Bee Colony optimization algorithm for hyperparameter tuning to mitigate local minima arising from weight initialization. As such prior to implementing the CNN, additional preprocessing included morphological filtering to remove hair, resizing images, segmentation via Otsu thresholding, and data augmentation through rotation, flipping, SMOTE to balance distributions within classes and aid in classification. Overall the hybrid model reported a classification accuracy of 93.04 %, 93 % precision, 92 % recall, and 93.12 F1-score, thus it was found to outperform typical CNN architecture. With all positives noted with the application of ABC, the limitations with the hybrid method included the computational processing high costs, stemming from the complexity of the model. Ranpreet et al. [62] presented an automated segmentation technique to improve diagnostic precision and speed. They presented a CNN framework that includes atrous convolutions, which have been helpful in semantic segmentation. Convolutional, batch normalization, and leakyReLU layers have been incorporated into the custom-built network, which also has tuned hyperparameters for improved performance. On the ISIC 2016, ISIC 2017, and ISIC 2018 datasets, the presented method obtained improved accuracy of 90.4 %, 81.8 %, and 89.1 %, respectively. Rafael et al. [63] utilized a melanoma segmentation technique that integrates U-net and LinkNet deep learning networks. In addition, they evaluate the networks' ability to acquire knowledge about disease characteristics or particular attributes of datasets using dataset combination experiments. They performed experiments utilizing three accessible datasets, ISIC 2018, DermIS, and PH2, and obtained better accuracy on LinkNet. There are several more works in the literature that performed skin lesion classification, such as the explainable deep AI technique [64], custom and pre-trained deep learning networks [65], intelligent fusion network [66], end-to-end DL framework [67], LsNet that works on limited samples [68], and triple fused CNN architecture [49]. These methods focused on the original data and pre-trained networks. However, these methods have several issues, such as augmentation, choice of CNN model, selection of hidden layers, parameters for the training, hyperparameter selection, and model interpretation. Considering these gaps, we proposed a new framework based on two novel lesion segmentation and multiclass lesion classification architectures in this work. Eliwa et al. [39] presented an automatic classification of monkeypox using the CNN and improve GWO optimization. The approach included two experiments: one contained a CNN application with the Monkeypox raw data, and the other contained data preprocessing, feature selection, GWO hyper parameter tuning. The Monkeypox dataset was obtained from Kaggle which contained skin lesion data. In the experimentation without data preprocessing, the CNN neural network provided poor results 68.82 % accuracy; 61.48 % AUC. However, the fully optimized CNN model, called “MonkeyNet” achieved 98.91 % accuracy using data augmentation. The authors acknowledged limitations to the study such as a relatively small dataset and limited generalizability across different clinical environments. Eliwa et al. [35] presented framework of a combination of Microsoft's Azure service and blockchain fields to classify and provide remote consultation of lung and colon cancer in a secure way. The approach integrated Microsoft Azure Services for preprocessing, anonymization, and cloud storage capabilities along with a permissioned blockchain for access control. They used the LC25000 dataset, which is a publicly available dataset containing histopathological images from 25,000 specimens and covering five classes of cancer. The authors evaluated multiple deep-learning models systematically which yielded 100 % accuracy. There were several limitations in the current approach such as risks involved with potential for data re-identification, scalability challenges regarding blockchain architecture, concerns related to the complexity of the technology integrations, and a bias regarding the dataset based on limited cancer variations. Elmessery et al. [38] described the semantic segmentation of strawberry leaf diseases based on the SegFormer architecture using three Mix Transformer encoders (MiT-B0, MiT-B3 and MiT-B5). The dataset was obtained from Kaggle and consisted of 2,450 raw images and 4,574 augmented images, covering three types of strawberry leaf diseases including leaf spot, gray mold, and powdery mildew. The authors assessed the models using the mean IoU for segmentation and Mean Pixel Accuracy (MPA) for final predictions. MiT-B3 and MiT-B5 outperformed the MiT-B0 model, yet MiT-B5 was capable of achieving the highest quality segmentation and the greatest efficiency. MiT-B5 was tested with respect to real-world deployment, while SegFormer required far fewer FLOP points compared to previous state-of-techniques (SETR, DeepLabV3). However, the study acknowledged some limitations including the geographic and environmental limitations of the dataset, sensitivity of strawberry leaf disease to weather conditions, and that cross-crop validation should be conducted further. El-Sayed et al. [15] described a method of detecting edges based the method of entropy for grayscale images, in an improved entropy-based method of detecting edges. The method worked by dividing the image into four equal sections, and determining an optimal local threshold for those areas which reduced computation time and eventually leads to better edge detection. They used a collection of standard grayscale images to evaluate the performance. The experimental results indicated that the methodology employed was superior in that they produced sharper edge maps, in comparison to the other method. They stated that the proposed method was effective in the worst and best case but required less CPU time. The limitation of this work was, it is constrained to grayscale images and the method usedfour separate regions in divide the input image.

From the recent studies, there are some research gaps such as, the segmentation models described in the literature are inaccurate, as they are either shallow architectures to limit the models' ability to learn features of the discriminator − fine features of the lesions. Secondly, classification performance is curbed significantly for pre-trained deep learning algorithms use of complex multi-class dermoscopy dataset because these models are unable to become adapted to those relevant patterns or patterns within the domain. Additionally, most of the current models are black boxes with minimal explainability features, marred by a limited route to trusting the model with clinical decisions, and limited insight into the decision process.

Datasets

In this article, we utilized two datasets for the segmentation and four for the classification tasks. For the segmentation, we employed the HAM10000 and ISIC2018 datasets, as the ground truths of these datasets are available. For the classification purpose, we utilized HAM10000, ISIC2018, ISIC2019, and ISIC2020 datasets, respectively.

HAM10000: The HAM10000 [69] “Human Against Machine with 10,000 training images” [69], which is accessible to the public via the ISIC repository, is among the most extensive database. In total, 10,015 dermoscopy images were utilized to identify pigmented skin lesions [70]. The dataset comprises seven distinct classes, each represented by a specific number of images: melanocytic nevus (6705 images), actinic keratosis (327 images), dermatofibroma (115 images), basal cell carcinoma (514 images), vascular lesions (115 images), benign keratosis (1099 images), and melanoma (1113 images) [71]. The 54 % of the images in the collection depict male skin lesions, while 45 % depict female skin lesions [72]. Numerous skin lesion images with low inter-class variation and high intra-class variation comprise this difficult-to-classify dataset, thereby increasing the likelihood of misclassification. Fig. 1 illustrates a few sample images of this dataset.

Fig. 2.

Sample images of skin lesion dataset.

ISIC-2018: The ISIC-2018 [73] dataset, assembled by the International Skin Imaging Collaboration, is an extensive collection of dermoscopy images comprising more than 12,500 instances [74]. Three distinct operations are executed on the dataset: segmentation of lesions, identification of attributes, and classification of illnesses. To perform the classification task, this dataset comprises more than 10,000 images that belong to seven distinct classes. Fig. 2 illustrates a few sample images [74].

ISIC-2019: The ISIC 2019 [75] dataset comprised the HAM10000 dataset. It comprises 25,328 dermatoscopic images captured in various locations, each undergoing a unique preparation process. Images of actinic keratosis (AKIEC), basal cell carcinoma (BCC), benign keratosis (BKL), squamous cell carcinoma (SCC), dermatofibroma (DF), melanoma (MEL), melanocytic nevus (NV), and vascular lesion (VASC) are included in the collection [57]. This dataset comprises an additional class, increasing the total to eight compared to HAM10000. Images were obtained using various equipment from multiple locations and over several years [76]. They were varying between 450 $\times$ 600 and 1024 $\times$ 1024 pixels in resolution. A few sample images of this dataset are shown in Fig. 2.

ISIC-2020: The ISIC-2020 [77] data collection consists of 33,126 dermoscopic training images depicting various benign and malignant skin diseases obtained from a sample size of over 2,000 individuals. This dataset is highly unbalanced, with only 584 images in the melanoma class and 32,542 images in the benign class [78]. A few sample images are illustrated in Fig. 2.

Proposed methodology

The proposed skin lesion segmentation and classification framework is presented in this section with detailed mathematical formulation and visual results. HAM10000 and ISIC-2018 datasets have been employed for the segmentation, whereas HAM10000, ISIC-2018, ISIC2019, and ISIC-2020 have been used for classification. In the segmentation phase, the DeepLabV3+ architecture is modified with the Resnet20 model and trained on the ground truth images. After that, the modified model is evaluated and computed using the Jaccard index, Jaccard distance, and dice.

In the classification phase, we proposed a novel MAKNet100 and trained on the selected datasets. The hyperparameters are initialized using BO, and the trained model is utilized further for testing. In the testing process, features are extracted from the self-attention layer and classified using neural network classifiers. The proposed MAKNet100 is further interpreted using an explainable AI technique such as GradCAM visualization. Fig. 3 shows the proposed framework of skin lesion segmentation and classification.

Fig. 3.

Proposed architectural diagram of skin lesion segmentation and classification.

Preprocessing

In the preprocessing phase, we performed the dataset augmentation, as the selected datasets are highly imbalanced, as shown in Table 1. In this table, the original images for the training purposes of each dataset are given, and it is observed that the nevi (NV) class images for the HAM10000 datasets are 6705, whereas the VASC class includes 142 images. Based on this problem, we performed well-known operations such as LR flip, UD flip, and rotate 90. A few samples of this step are seen in Fig. 4. After applying these operations, the imbalance datasets are converted into balance classes with approximately 50,000 images for HAM10000, 45,000 for ISIC-2018, 65,000 for ISIC-2019, and 50,000 for the ISIC-2020 dataset, respectively.

Table 1.

All skin cancer dataset descriptions. A 70:30 approach is conducted.

Dataset	AKIEC	BKL	BCC	DF	MEL	NV	VASC	SCC	Benign	Malignant	Total
HAM10000 Original	327	514	1099	115	1113	6705	142	−	−	−	10,015
HAM10000 Original training and testing	Training Images: 7010 Testing Images: 3005
HAM10000 Augment	6561	6666	7589	7359	7200	7162	7364	−	−	−	49,901
HAM10000 Augment training and testing	Training Images: 34,930 Testing Images: 14,971
ISIC-2018 Original	327	514	1099	115	1113	6705	142	−	−	−	10,015
ISIC-2018 Original training and testing	Training Images: 7010 Testing Images: 3005
ISIC-2018 Augment	5818	5808	6075	5806	6145	6704	6024	−	−	−	42,380
Training and testing images	Training Images: 29,666 Testing Images: 12,714
ISIC-2019 Original	867	3323	2624	239	4522	12,875	253	628	−	−	25,331
Training and testing images	Training Images: 17,323 Testing Images: 8008
ISIC-2019 Augment	7964	8052	8036	8028	8108	8237	8134	8312	−	−	64,871
Training and testing images	Training Images: 45,410 Testing Images: 19,461
ISIC-2020 Original	−	−	−	−	−	−	−	−	32,542	584	33,126
ISIC-2020 Original training and testing images	Training Images: 23,188 Testing Images: 9938
ISIC-2020 Augment	−	−	−	−	−	−	−	−	32,542	15,344	47,886
ISIC-2020 Augment training and testing images	Training Images: 33,520 Testing Images: 14,366

Fig. 4.

Data augmentation images.

Improved DeepLabV3+ model

DeepLabV3+ is an improved version of the DeepLabV3 architecture. It incorporates a decoding module that is more efficient and effective, resulting in improved performance in semantic segmentation and fine-tuning of feature information. Typically, the foundation of DeepLabV3 is a CNN that has already been trained with backbone networks such as Xception, ResNet, and MobileNet. These networks are usually responsible for retrieving high-level information from an input image.

A crucial part of DeepLabV3 is dilated convolutional and Atrous Spatial Pyramid Pooling (ASPP), which enables the model to efficiently incorporate context information such as an image's local and global aspects. This module obtains high-level semantic information by employing three simultaneous 3 × 3 convolutions with 6, 12, and 18 dilation rates and a worldwide average pooling layer. The mathematical formulation of the dilated convolutional and ASPP is:

$${\mathrm{O}}_{[\mathrm{k}]}=\sum _{\mathrm{f}=1}^{\mathrm{F}}\mathrm{z}\left[\mathrm{k}+{\mathrm{\alpha }}_{\mathrm{r}}.\mathrm{f}\right].{\mathrm{\omega }}_{[\mathrm{k}]}$$ (1)

Where $\mathrm{z}[\mathrm{k}]$ is the input feature map, ${\mathrm{O}}_{[\mathrm{k}]}$ is the resultant feature map, ${\mathrm{\alpha }}_{\mathrm{r}}$ is the dilation factor, $\mathrm{F}$ is the filter size, ${\mathrm{\omega }}_{[\mathrm{k}]}$ is the filter and ${\mathrm{O}}_{[\mathrm{k}]}$ is the resultant feature map. The ASPP utilized multiple parallel dilation convolutions with various dilation factors to capture and concatenate the multi-scale feature maps. The ASPP is formulated as:

$${\phi }_{\mathrm{A}\mathrm{S}\mathrm{P}\mathrm{P}}\left(\mathrm{z}\right)=\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}({\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}_{{\mathrm{\alpha }}_{\mathrm{r}1}}\left(\mathrm{z}\right),{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}_{{\mathrm{\alpha }}_{\mathrm{r}2}}\left(\mathrm{z}\right),{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}_{{\mathrm{\alpha }}_{\mathrm{r}3}}\left(\mathrm{z}\right),\cdots ,{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}_{{\mathrm{\alpha }}_{\mathrm{r}\mathrm{n}}}\left(\mathrm{z}\right)$$ (2)

The encoder is utilized as a backbone of the network that extracts the features and is defined as:

$${\tau }_{{\phi }_{\mathrm{A}\mathrm{S}\mathrm{P}\mathrm{P}}}={\phi }_{\mathrm{A}\mathrm{S}\mathrm{P}\mathrm{P}}({\mathrm{\psi }}_{\mathrm{F}})$$

Where ${\mathrm{\psi }}_{\mathrm{F}}$ extracted feature map 3.

The decoder component uses 1 × 1 convolutions to downscale the low-level features from the input layer of the backbone network, which is then combined with the high-level information from the encoder.

Bilinear up-sampling is used to precisely modify the bounds of the target objects after several 3 × 3 convolutions are performed to extract spatial information from the feature maps. This procedure ultimately yields the segmentation results. The decoder is defined as:

$${\mathrm{\psi }}_{{\mathrm{F}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}}}=\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}({\tau }_{{\phi }_{\mathrm{A}\mathrm{S}\mathrm{P}\mathrm{P}}},{\mathrm{\psi }}_{\mathrm{l}\mathrm{o}\mathrm{w}})$$ (4)

$${\mathrm{\psi }}_{\mathrm{u}\mathrm{p}}=\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}({\mathrm{\psi }}_{{\mathrm{F}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}}})$$ (5)

Where ${\mathrm{\psi }}_{\mathrm{l}\mathrm{o}\mathrm{w}}$ presented the low-level feature map achieved from the encoder. $\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}$ reduces the number of depth. The final segmentation is obtained by employing the convolutional layer and up-sampling to match the input dimension. The up-sample operation is defined as:

$${\mathrm{Y}}_{\mathrm{S}}=\mathrm{u}\mathrm{p}\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}(\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}\left({\mathrm{\psi }}_{\mathrm{u}\mathrm{p}}\right))$$ (6)

Proposed ResNet-20 model

In this work, we proposed a ResNet-20 model as a backbone network for extracting essential feature maps of an input image. An extension of the DeepLabvV3+ framework is based on dual-path, dual-feature pyramid structure, and multi-scale feature fusion. The feature maps are based on the backbone network, such as the proposed ResNet20. The proposed ResNet20 presents a residual structure with identity mapping, which permits gradient propagation smoothly from shallow to deep layers. Training intense neural networks (DNN) with this residual architecture improves their capacity to extract features and capture the restraints and qualities of the input data. The properties of various scales can be better retained for the final portion of the network by employing residual mechanisms.

The proposed ResNet20 architecture extensively uses the Residual Network (ResNet) architecture, which incorporates residual connections to mitigate the vanishing gradient problem in profound neural networks. Firstly, we resize the images of selected dataset images from 650 × 450 dimension to 240 × 240. After that, the 20 residual blocks were added, as shown in Fig. 5. The input image is connected with the last centre-cropped layer. The initial layer of the neural network, known as the input layer, is followed by a convolutional layer. The first convolutional layer has a stride of 2, a depth of 64, and a filter size 7 × 7. Subsequently, the Batch Normalization (BN) and ReLU layers were added.

Fig. 5.

Proposed ResNet20 model as a backbone of DeepLabV3+ for lesion segmentation.

Furthermore, a max pooling layer with a kernel size 3 × 3 has been added. Subsequently, a new convolutional layer is added with a filter size 3 × 3, a stride of 1, and a depth of 64. Subsequently, the BN and ReLU activation layers have been added. A new series block of filter size 3 × 3, stride 1, and depth size 64 has been added. After that, BN and ReLU layers were added that follow the convolutional layer with the same filter size, depth, and stride. This block connects to an additional layer. When an extra layer is added to the ReLU layer after inserting a new block containing a convolutional layer, the BN, ReLU, with attributes similar to those of the preceding series block, this block is connected to a new additional layer that is associated with ReLU.

After that, three blocks were added in a parallel fashion. The first block consists of a convolutional layer of size 3 × 3, stride value of 2, and 128 depth. The BN and ReLU layers are included, following a convolutional layer of size 3 × 3 and 128 depth size. The second parallel block started with a BN layer followed by a convolutional layer with a 1 × 1 filter size, stride value of 2, and 128 depth.

Another parallel block includes a 1 × 1 convolutional layer with stride one, depth 48, and BN and ReLU activation layers. The third block is then linked to the Depth concatenation and centre-cropped layers. An additional layer joins all of these parallel blocks. Add five blocks in the series. The first block has a convolutional layer with a 3 × 3 filter size, stride of 1, depth of 128, and ReLU activation layer. This block is connected to an additional layer. Based on these blocks, the next four blocks are added in a manner identical to the previous blocks. The maximum size of depth is 256 to 512. All of these blocks are interconnected through an additional layer. Last, a transposed convolutional layer followed the centre cropped and softmax layers. Finally, a classification layer is added for the final lesion segmentation.

Transposed convolutional layer: The transposed convolutional layer improves the spatial resolution of feature maps. This is helpful for semantic segmentation and image creation tasks. The transposed convolutional layer upsamples a feature map $\mathrm{F}$ using a transposed convolutional filter $\mathrm{T}$ to produce an output feature map $\mathrm{R}$ of a skin lesion image. Mathematically, the output $\mathrm{R}$ is computed as follows:

$$\mathrm{R}\left(\mathrm{m},\mathrm{n},\mathrm{l}\right)=\sum _{\mathrm{x}=0}^{\mathrm{A}-1}\sum _{\mathrm{y}=0}^{\mathrm{A}-1}\sum _{\mathrm{z}=0}^{\mathrm{D}-1}\mathrm{F}\left(\mathrm{m}+\mathrm{x},\mathrm{n}+\mathrm{y},\mathrm{z}\right).\mathrm{T}(\mathrm{x},\mathrm{y},\mathrm{l},\mathrm{z})$$ (7)

Where, $(\mathrm{m},\mathrm{n})$ denotes the output feature map of spatial coordinates, $\mathrm{l}$ denotes the output channel, $\mathrm{A}$ denotes filter size, and $\mathrm{D}$ represents the number of channels in the input feature map.

Depth Concatenation Layer: The depth concatenation layer combines feature maps from many layers, allowing data from various scales or levels of abstraction to be integrated. The depth concatenation layer improves the computational efficiency of semantic segmentation, such as skin cancer, along with other tasks related to computer vision, by allowing multi-scale data to be fused.

Center Crop Layer: By removing the surrounding regions, the centre crop layer takes a core region from an input feature map. If the input feature map has dimensions, then mathematically $\times \mathcal{H}\times \complement$, the centre crop operation selects an area of size ${\mathrm{w}}_{\mathrm{i}}\times {\mathrm{h}}_{\mathrm{i}}\times \mathrm{c}$ from the centre, where ${\mathrm{w}}_{\mathrm{i}}$ and ${\mathrm{h}}_{\mathrm{i}}$ are the desired width and height of the cropped region.

Training Segmentation Model: After the design of improved DeepLabV3+ architecture, the training has been conducted by employing original and ground truth images of HAM10000 and ISIC-2018 datasets. The number of epochs in the training process was 100, and the initial learning rate was 0.0015. The initial learning rate is selected using the Bayesian Optimization algorithm. The obtained training accuracy of the proposed model is 90.5 % on the HAM10000 dataset and 94.62 % on the ISIC2018 dataset, respectively. After training, the model is tested on test set images using both datasets. Several parameters are computed, such as JaccardIndex, JaccardDistance, and Dice. Each tested image is converted into binary and mapped with ground truth images. A few sample visual results are shown in Fig. 6.

Fig. 6.

Proposed DeepLabV3+ visual results.

Proposed MAKNet100 classification model

We proposed a novel MAKNet100 architecture for multiclass skin lesion classification in this work. The proposed architecture is specifically designed for skin lesion classification. As shown in Fig. 1 and Fig. 3, the skin lesion images contain complex patterns and high inter- and intra-class similarity; therefore, it is essential to design a model accurately to classify the skin lesion into a correct class. The aim to design a deep 100-layer network is grounded in the requirement to learn the complex and subtle representations found in dermoscopic images which exhibit high inter-class similarities and high intra-class variations. In deep architecture, the bottleneck and dense blocks build new hierarchical and multi-scale features available for learning using residual connections and feature reuse. Also, a deeper network enhances the model’s ability to learn an abstract and discriminative representation desired for multi-class classification.

Fig. 1.

Sample images that show the complex patterns for the segmentation and classification.

The proposed MAKNet100 model consists of two individual architectures, the Bottleneck and Dense-3 network, fused using a depth concatenation layer. The bottleneck network consists of three parallel residual blocks, whereas the Dense-3 contains three dense blocks, each including a single residual block. The resultant layer of each network is fused using a depth concatenation layer that is further followed by a self-attention module. The detailed architecture of the proposed MAKNet100 is shown in Fig. 7. The proposed architecture consists of a total of 100 layers.

Fig. 7.

Proposed MAKNet100 architecture for skin lesion classification.

Bottleneck Network: The input of the proposed architecture is 240*240*3, which is passed to a convolutional layer of filter size 3*3, depth size 32, and stride 2. A ReLu activation layer is added after the first convolutional layer. The bottleneck network incorporates three parallel residual blocks, each consisting of a convolutional layer, BN layer, and reLU activation layer, which are connected to the predecessor ReLU layer. After each convolutional layer in this block, BN and ReLU layers are appended. At the start, a convolutional layer of depth size 64, filter size 1*1, and stride one is added. In each block, the stride remains constant, such as one. In the second convolutional layer, the filter size is increased by 3*3, whereas the depth is continuous, such as 64. In the next convolutional layer, a 1*1 filter size is opted, whereas the depth is 32. An identical residual block is incorporated into the parallel side, as shown in Fig. 7. After that, a skip connection is added and connected with parallel blocks into a single addition layer. Hence, a residual bottleneck network consists of a parallel block with two paths and one skip connection.

After the first residual block, we added a convolutional layer with a 3*3 filter size, a depth value 64, and stride 2. The ReLu activation layer is added after convolutional. The second parallel residual block is added just like the first one except for a change in depth value. The depth value of the first and second convolutional layers is 128, whereas the third convolutional contains 64 depth size. The filter size and stride value are added the same way as the previous block, such as stride one, and filter sizes are 1*1, 3*3, and 1*1, respectively. The same residual block is added to the parallel side to improve the learning of a model for skin cancer images. After that, both parallel blocks and a skip connection are combined using an additional layer.

After the second residual parallel block, a convolutional layer of filter size 3*3, stride 2, and depth value of 128 is added. Like the second residual block, the depth is updated in this block with 256 and 128, respectively. In addition, the parallel block is also executed on the same parameters, such as a depth size of 256 for the first convolutional, 256 depth for the second, and 128 depth for the third convolutional, respectively. The last convolutional layer of the presented Bottleneck network contains 512 depth size, 3*3 filter, and stride value of 2.

The second network is also based on the Dense Blocks. In this network, three dense blocks are added, each consisting of a single residual block. Each residual block consists of a convolutional layer, a BN layer, and a ReLU activation layer. A batch normalisation layer is added after each convolutional layer in this block. In the first Dense Block, a convolutional layer of depth size 64, filter size 1*1, and stride one is added. The depth size of the second convolutional is also 64 of filter size 3*3 and stride one. In the third convolutional layer, a depth size of 32 is added of filter size 1*1 and stride one. The stride is considered the same in each block of the entire block. After that, a skip connection is added and connected with the depth concatenation layer (seen in Fig. 7).

After the first Dense Block, a convolutional layer of depth size 64 and stride value two is added. In the next Dense Block, three convolutional layers have been included. The output is passed in the depth concatenation layer with a skip connection. The first convolutional layer depth size is 128 and stride one, whereas the filter size is 1*1. The second convolutional layer contains a 3*3 filter size, depth of 128, and stride one. The third convolutional layer includes a 1*1 filter size and depth of 64. After that, the output of this layer is connected with the depth concatenation layer along with a skip connection. On the outside of the second Dense Block, another convolutional layer of depth size 128, filter size 3*3, and stride two is added.

Moreover, a ReLU activation layer is added after each convolutional layer of this network. The third Dense Block consists of three convolutional layers of 256, 128, and 256 depth sizes. Also, the stride is set as one, and the filter size of each layer is 1*1, 3*3, and 1*1, respectively. This dense block joins the depth concatenate layer with one path and skips the connection. The current block is followed by convolutional and BN layers of filter size 3*3 and depth size 256. In addition, another convolutional layer is added, with depth size 512, filter size 3*3, and stride value of two.

After that, a depth concatenation layer was added, and the output of bottleneck and dense block networks was passed to this layer for fusion. After the fusion of both networks, two global average pooling layers have been added, followed by a flattened layer. A self-attention layer is added after the flattening layer that extracts the most dependent and essential features.

The self-attention (SAN) module in CNN design enhances network performance (efficiency) by focusing on the most important aspect of a skin image. SAN extracts local features of the input vector, such as the global average pooling layer. Consider the global average pooling layer features, represented by $\mathrm{\varphi }\in {\acute{\mathrm{R}}}^{D\times C}$, where $D$ represents the total number of channels. A $1\times 1$ convolutional operation was performed on $\mathrm{\varphi }$, yielding three 1D matrices.

$(\mathrm{Ғ},\mathrm{G}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{H}$) as illustrated in Fig. 8. Mathematically, these terms are defined as:

$$\mathrm{Ғ}\left(\mathrm{\varphi }\right)=S_{\mathrm{Ғ}}\mathrm{\varphi },\mathrm{G}\left(\mathrm{\varphi }\right)=S\mathrm{\varphi }$$ (8)

$$\mathrm{H}\left(\mathrm{\varphi }\right)=S_{\mathrm{Ғ}}\mathrm{\varphi }$$ (9)

$$S_{\mathrm{Ғ}},S_{\mathrm{G}},S_{\mathrm{H}}\mathrm{∊}{\acute{\mathrm{R}}}^{{\mathrm{D}}^{\ast }\times \mathrm{D}}$$ (10)

A sequence of softmax operations is used to merge the feature maps of $\mathrm{Ғ}\left(\mathrm{\varphi }\right)$ and $\mathrm{G}\left(\mathrm{\varphi }\right)$, whereas the softmax operation is formulated as follows:

$${\mathfrak{R}}_{\mathrm{x}\mathrm{y}}=\frac{{\mathrm{e}\mathrm{x}\mathrm{p}(S}_{\mathrm{x}\mathrm{y}})}{{\sum }_{\mathrm{i}=1}^{\mathrm{N}}{\mathrm{e}\mathrm{x}\mathrm{p}(S}_{\mathrm{x}\mathrm{y}})}$$ (11)

$$S_{\mathrm{x}\mathrm{y}}=\mathrm{Ғ}{{(\mathrm{\varphi }}_{\mathrm{x}})}^{\mathrm{t}}\mathrm{Ғ}({\mathrm{\varphi }}_{\mathrm{y}})$$ (12)

The self-attention layer's output is specified in Eq. (13), which is then used to extract features from the map.

$${\mathbb{O}}_{\mathrm{y}}=(\sum _{\mathrm{x}=\mathrm{n}}^{\mathrm{N}}{\mathfrak{R}}_{\mathrm{x}\mathrm{y}}\mathrm{b}({\mathrm{\varphi }}_{\mathrm{x}})$$ (13)

Where ${\mathbb{O}}_{\mathrm{y}}$ denotes the attention map feature matrix passed to fully connected and softmax layers for the classification task.

Fig. 8.

Feature extraction using self-attention module.

Proposed model training

After designing a proposed fusion network for the classification task, training on the selected datasets is the next important phase. In the training phase, the first step is the training data ratio, which is 70 % in this work, whereas 30 % is used for validation. The second step is to initialize the hyperparameters of the designed model for smooth training. For this purpose, the researchers usually employed manual hyperparameter values such as learning rate, optimizer, momentum, and regularization factor. Manual initialization is not an accurate solution; therefore, we implemented the Bayesian Optimization (BO) [68] algorithm to select hyperparameter values. Through BO, the initial range is given to the hyperparameters. For example, we are given the range of (0, 1) for the learning rate. Based on the range, the BO returned the best value, as shown in Fig. 9. The trained model is finally saved and further utilized for testing purposes. The ranges of the hyperaparameters are shown in Table 2.

Fig. 9.

Process of Bayesian Optimization for the hyperparameter tuning.

Table 2.

Description of hyperparameters parameters employed in BO.

Hyperparameters	Ranges
Learning Rate	0.01–0.9998
Momentum	0.1–0.99
Batch Size	8–64
Epochs	1–150
L2-Regularization Factor	0.01–0.8

Proposed model testing

In the testing process of the proposed model, the testing images are passed to the proposed MAKNet100, and features are extracted from the self-attention layer. The extracted features are compared with the trained model and obtained predicted output such as lesion type. Fig. 10 shows the visual process of the proposed testing task. The classification process uses several methods, such as a narrow neural network and a few more. In addition, the proposed MAKNet100 is interpreted using the GradCAM visualization technique. Through GradCAM visualization, the self-attention features are mapped on the original image that, in output, shows the highlighted lesion region.

The entire segmentation and classification work is also defined in algorithm 1. In this algorithm, all steps are defined.

Algorithm 1
Segmentation Steps
Input $\leftarrow$ Original training images and corresponding ground truth images
Output $\leftarrow$ Segmented image and region localization
Step 1: Design modified DeeepLab V3++
-Backbone $\leftarrow$ Resnetv20 Model
-Epochs $\leftarrow$ 100
-Initial Learning Rate $\leftarrow$ 0.0015
Step 2: Training Segmentation Model
Step 3: Employed Trained model for Segmentation Output
Step 4: Obtained visual and numerical results
Classification Steps
Input $\leftarrow$ Training labeled data
Output $\leftarrow$ Predicted images with labeled output
Step 1: Proposed model $\leftarrow$ MAKNet
Step 2: Proposed model hyperparameters $\leftarrow$ BO based selection
-Loss function $\leftarrow$ Cross Entropy
Step 3: Training MAKNet model on training set images
Step 4: Testing proposed model
Step 5: Predicted images + Numerical Results

Results and discussion

In this section, the results of the proposed framework are presented. This section includes experimental setup, segmentation results, classification results, discussion, and comparison with state-of-the-art techniques (SOTA).

Experimental setup

The experimental process of the proposed segmentation and classification tasks is presented in this section. Several hyperparameters are employed to train a model during the segmentation process, as given in Section 3.3.1. The maximum epochs performed for each dataset are 100. The testing phase of the segmentation task is performed on 500 images of each dataset, and performance measures such as the Jaccard Index, Distance, and Dice are computed. In the classification task, 30 % of testing images are utilized, and features are extracted. The extracted features are passed to the classifiers for the classification results. The entire experimental process is performed on MATLAB2023b using a Personal Computer Corei7 with 128 GB of RAM and a 12 GB graphics card of 3060RTX. The hyperameters and their values for reprodubility are presented in the table 3.

Proposed segmentation results

The numerical and visual segmentation results are discussed in this section. As mentioned, 500 images were picked from each dataset (test images picked before training), and segmentation was performed. A few numerical results are given in Table 3. In this table, it is shown that the dice value has reached 96 %. The average dice value of the proposed architecture on the HAM10000 dataset is 94.69 %.

Table 3.

Description of hyperparameters parameters employed in BO.

Hyperparameters	Values
Learning Rate	0.0015
Learning Decay	0.0001
No. of Epochs	100
Momentum	0.61
Batch Size	32

Similarly, the ISIC-2018 employs 500 images for testing purposes. The validation error rate was 0.0538, and the obtained validation accuracy was 94.62 % during training with the ISIC2018 dataset. During the testing phase, an average dice value of 94.36 was achieved. Table 3 presents the numerical results of the HAM10000 and ISIC-2018 datasets. Moreover, the visual results are shown in Fig. 11. Overall, the results of this task are improved using the presented method; however, there are a few limitations, such as lesions on the boundary area.

Fig. 10.

Visual process of the proposed classification model testing for the final prediction.

Proposed classification results

The classification results of the proposed MAKNet100 are presented in this section using four datasets: HAM10000, ISIC2018, 2019, and 2020. The classification performance is assessed through several measures such as classification accuracy (ACC), Sensitivity rate (SEN), Precision rate (PRE), Time, FNR, and F1-score. Five distinct classifiers are utilized to conduct classification in the results section that was Narrow Neural Networks (N³), Medium Neural Networks (M−N²), Wide Neural Networks (W-N²), Bi-layered Neural Networks (B-N²), and Tri-Neural Neural Networks (T-N²). The following tests have been carried out to assess the suggested architecture.

• •Experiment # 1: Classification of proposed MAKNet100 on imbalance datasets such as HAM10000, ISIC-2018, ISIC-2019 and ISIC-2020.
• •Experiment # 2: Classification of proposed MAKNet100 on augmented datasets.

Experiment # 1: The experimental results are presented in Table 4 for all datasets using the proposed MAkNet100 model without an augmented dataset. (a) The results for the HAM10000 dataset demonstrate that the WN² classifier attained the highest accuracy of 75.3 %. The execution time for this classifier is also noted as 174.11 (seconds). The minimum noted time is 109.52 (sec) for BN². Additional performance metrics of WN² comprised the F1-score of 75.86 %, SEN of 76.25 %, PER of 75.48 %, and FNR of 23.75 %, respectively. The confusion matrix in Fig. 12 illustrates the correct predicted values of classifier WN². The diagonal values correspond to the following: 78.2 % for AKIEC, 72.4 % for BCC, 58.9 % for BKL, 82.3 % for DF, 65.3 % for MEL, 76.7 % for NV, and 97.3 % for VASC.

Table 4.

Segmentation results of HAM10000 and ISIC-2018 using improved DeepLabV3+ with Resnet20 model.

(a). HAM10000 Dataset Segmentation Results
Jaccard Index	Jaccard Distance	Dice
0.9195	0.0805	0.9581
0.9092	0.0908	0.9524
0.9413	0.0587	0.9698
0.8704	0.1296	0.9307
0.8368	0.6132	0.9112
0.9384	0.0616	0.9682
0.9349	0.0651	0.9664
0.9024	0.0976	0.9487
0.9085	0.0915	0.9521
0.8382	0.1618	0.9120
(b). ISIC-2018 Dataset Segmentation Results
Jaccard Index	Jaccard Distance	Dice
0.9546	0.0454	0.9768
0.9101	0.0899	0.9530
0.8467	0.1533	0.9170
0.8992	0.1008	0.9469
0.9556	0.0444	0.9773
0.8589	0.1411	0.9241
0.9238	0.0762	0.9604
0.9714	0.0286	0.9855
0.9419	0.0581	0.9701
0.9352	0.0648	0.9665

Fig. 11.

Segmented images of HAM10000 and (b) ISIC-2018 using implemented DeepLabV3+ model.

(b) The results of the ISIC-2018 dataset are presented in this part using the proposed MAKNet100. The WN² classifier attained an accuracy of 69.8 %, whereas the execution time of 184.125 (seconds). The minimum time is noted for TN². The other computed measures and values of the WN² classifier, such as F1-score of 37.481 %, SEN of 36.24 %, PER of 38.81 %, and FNR of 63.74 %, respectively. The confusion matrix is presented in Fig. 12, which illustrates the correct predicted values of classifier WN².

Table 4(c) presents the results of the proposed model on the original ISIC-2019 dataset, and the WN² classifier attained the highest accuracy of 64.5 %. The time for this classifier is increased due to an increase in the number of testing images. The other performance metrics and their values of this classifier include SEN of 42.26 %, PER of 30.881 %, FNR of 57.74 %, and F1-score is 40.46 %, respectively. The confusion matrix in Fig. 12 illustrates the correct predicted values of classifier WN². Through these values, the computed measures can be verified.

Table 4(d) presents the classification results of the ISIC-2020 dataset using the proposed MAKNet100 model. In this table, the WN² classifier achieved an accuracy of 97.2 %, whereas the sensitivity rate of 52.05 %, PER of 52.7 %, FNR of 47.95 %, and F1-score of 52.37 %, respectively. As seen by the confusion matrix shown in Fig. 12, the predicted value of the Melanoma class is low due to a highly imbalanced issue. Fig. 13 shows the area under the curve utilized to verify each class's performance. In this figure, AUC is computed for each dataset separately and noted values are 0.9208 (HAM10000), 0.8257 (ISIC2018), 0.789 (ISIC2019), and 0.5718 (ISBI2020). Based on these values, it is observed that the imbalance issue impacts the classification accuracy. Therefore, we performed data augmentation (discussed in the proposed methodology) for the dataset imbalance issue.

Fig. 12.

Confusion matrix of Wide Neural Network (WN²) classifier on imbalance datasets.

Experiment # 2: The selected datasets have been balanced using data augmentation techniques such as horizontal, vertical, and rotate90. Results are presented in Table 5. Table 5(a) presents the classification results of the augmented HAM10000 dataset using the proposed MAKNet100. In this table, the WN² classifier achieved an accuracy of 90.5 % within a runtime of 239.3 (seconds). The other calculated performance measures are SEN of 90.55 %, PER of 91.58 %, FNR of 63.76 %, and F1-score of 91.06 %, respectively. As seen by the confusion matrix shown in Fig. 14, the diagonal values represent the correct predicted values of classifier WN^2, that is, 94.9 % for AKIEC, 89.7 % for BCC, 81.6 % for BKL, 98.6 % for DF, 88.8 % for MEL, 80.8 % for NV and 99.5 % for VASC. Comparing the results of augmented HAM10000 with the original HAM10000 dataset (seen in Table 5 (a)) shows that the results are improved for all classifiers after the augmentation process. After the augmentation process, a 15 % improvement occurred in the accuracy.

Table 5.

Classification results of proposed MAKNet100 on selected datasets (Imbalanced datasets).

(a): Classification results of HAM10000 Dataset (Original Dataset)
Classifiers	SEN (%)	PER (%)	ACC (%)					Time (sec)			FNR			F1-score (%)	MCC
N3	65.25	64.8	65					119.6			34.75			65.02422	0.545
MN2	70.185	69.9	69.8					141.59			29.815			70.04221	0.6157
WN2	76.25	75.48	75.3					174.11			23.75			75.86305	0.7162
BN2	65.214	64.65	64.9					109.52			34.786			64.93078	0.6387
TN2	65.285	64.74	6.0					185.91			34.715			65.01136	0.6398
(b): Classification results of ISIC-2018 Dataset (Original Dataset)
Classifiers	SEN (%)	PER (%)		ACC (%)			Time (sec)					FNR		F1-score (%)	MCC
N3	29.6	80.8		66.2			134.51					70.4		43.32754	0.2414
MN2	34.67	37		68.9			151.439					65.33		35.79713	0.3287
WN2	36.24	38.81		69.8			184.125					63.76		37.481	0.4065
BN2	32.2	32.2		66.1			191.12					67.8		32.2	0.3805
TN2	29.44	29.25		65.3			109.53					70.56		29.34469	−0.3257
(c): Classification results of ISIC-2019 Dataset (Original Dataset)
Classifiers	SEN (%)	PER (%)	ACC (%)		Time (sec)				FNR				F1-score (%)		MCC
N3	27.40	80.8	61.9		141.02				72.6				40.92274		0.2658
MN2	38.425	37	61.1		141.09				61.57				37.69904		0.3037
WN2	42.26	38.81	64.5		199.75				57.74				40.46159		0.4708
BN2	35.51	32.2	62.1		185.05				64.49				33.7741		0.3571
TN2	34.6	29.25	62.1		195.45				65.4				31.70086		−0.2882
(d). Classification results of ISIC-2020 Dataset (Original Dataset)
Classifiers	SEN (%)	PER (%)	ACC (%)			Time (sec)				FNR				F1-score (%)	MCC
N3	47.65	53.25	97.4			243.77				52.35				50.2946	0.4004
MN2	52.35	53.1	97.2			189.769				47.65				52.72233	0.5108
WN2	52.05	52.7	97.2			148.03				47.95				52.37298	0.9088
BN2	51.3	51.75	97.1			212.65				48.7				51.52402	0.5045
TN2	53.3	53.9	97.1			264.9				46.7				53.59832	0.5164

Fig. 13.

Area under the curve of wide neural network (WN²) classifier on imbalance datasets.

Table 5(b) presents the results of the augmented ISIC-2018 dataset using MAKNet100. The WN² classifier attained an accuracy of 88.9 % within a runtime of 278.6 (seconds). The rest of the calculated measures and their values, such as SEN of 89.12 %, a precision rate PER of 88.81 %, a false negative rate of 10.88 %, and an F1-score of 88.96 %, respectively. The performance of WN² is further confirmed by a confusion matrix presented in Fig. 14. In this figure, it is observed that the prediction performance is improved than the performance of the original ISIC-2018 dataset. After the augmentation, the correct prediction rate of 95.2 % for AKIEC, 91.0 % for BCC, 76.5 % for BKL, 99.6 % for DF, 79.3 % for MEL, 82.5 % for NV, and 99.8 % for VASC. Compared to the results in Table 5(b), the accuracy and precision rates are significantly improved.

Table 5(c) presents the results of the augmented ISIC-2019 dataset using the proposed MAKNet100. The WN² classifier attained the best accuracy of 84.5 %, whereas the execution time was 238.9 (sec). The other performance metrics calculated for this classifier include a Sensitivity of 84.5 %, a Precision of 84.21 %, a False Negative Rate (FNR) of 15.5 %, and an F1-score of 84.35 %, respectively. The performance of the WN² classifier can be verified using a confusion matrix, given in Fig. 14. In this figure, it is noted that the prediction rate of the proposed method is improved compared to the classification performed on the original ISIC-2019 dataset. In this figure, the correct prediction rate of 89.8 % for AKIEC, 79.3 % for BCC, 70.9 % for BKL, 99.8 % for DF, 65.8 % for MEL, 77.1 % for NV, 93.4 % for SCC, 16.6 % for SCC, and 99.9 % for VASC, respectively. Similarly, Table 5(d) presented the results of the augmented ISIC-2020 dataset using the proposed MAKNet100 and obtained a maximum accuracy of 96.4 % using WN². Fig. 14 shows the confusion matrix of this classifier. Based on this figure, it is noted that the prediction rate of the melanoma class is significantly improved. The correct prediction rate for this dataset is 94.8 % of the benign cases and 98.3 % of the malignant cases. Fig. 15 shows the area under the curve of each dataset, and obtained values are 0.9696 for the HAM10000 dataset, 0.9671 for the ISIC-2018 dataset, 0.9615 for ISIC-2019 dataset, and 0.9836 for ISBI-2020 dataset, respectively. Based on these values, it is observed that the overall performance of the proposed architecture has been improved after performing data augmentation.

Fig. 14.

Confusion matrix of Wide Neural Network (WN²) classifier after the augmentation process.

Discussion

A brief discussion of the entire proposed framework is added under this section, which includes the analysis of segmentation results, interpretation of proposed MAKNet100, comparison of proposed MAKNet100 with pre-trained models, and comparison with recent SOTA techniques. The proposed framework is shown in Fig. 3, which consists of two different tasks: lesion segmentation and classification. The improved DeepLabv3+ model is employed along with ResNet20 architecture. The ResNet20 architecture is shown in Fig. 5. The visual results of this method are illustrated in Fig. 6, Fig. 10. Moreover, the numerical results are presented in Table 2. This table shows that the average dice value is above 94 % for both selected datasets. Moreover, one of the limitations is that the model may not reliably detect tumors situated on the edges of dermoscopic images, as the edges have much less discriminative features, and ambiguity regarding context can lead to less reliable detection. Our model was tested against several public datasets, but its capability in real-life clinical environments with a variety of imaging conditions and different devices has not been determined. Another limitation is the amount of computation needed to train MAKNet100 because it consisted of many deep layers, which may limit deployment on restricted-resource environments.

In the next phase, a classification model is proposed named MAKNet100 based on the fusion of two networks and a self-attention module. Fig. 7 shows the proposed MAKNet100 and the numerical results in Table 3, Table 4. In these tables, results are discussed in two different experiments. The results are presented in the first experiment without the augmentation process, as shown in Table 6. The results noted in this table are not as good as those of the recently published works. Also, the dataset seems highly imbalanced, showing the high probability of the maximum number of image classes. Therefore, we performed the augmentation process; the results are in Table 6. In this table, the accuracy and precision rates are improved to the results in Table 6. The wide neural network classifier shows the best performance of all the classifiers. Also, the confusion matrices are illustrated in Fig. 11, Fig. 13, which can be utilized to confirm the reported performance measures of this classifier for each dataset (see Table 6 and Figure 15 (ROC Curves)).

Table 6.

Classification results of proposed MAKNet100 using selected augmented datasets (Augmented images).

Classifiers	SEN (%)	PER (%)	ACC (%)	Time (sec)	FNR	F1-score (%)	MCC
(a) Proposed classification results on HAM10000 Dataset (Augment Dataset)
N3	78.72	78.614	78.8	235.6	70.4	78.6669	0.7658
MN2	85.7	85.45	85.7	268.6	65.33	85.5748	0.6037
WN2	90.55	91.58	90.5	239.3	63.76	91.0620	0.6708
BN2	79.17	79.08	79.2	257.8	67.8	79.1249	0.6571
TN2	78.5	78.514	78.5	252.6	70.56	78.507	0.6882
(b) Proposed classification results on ISIC-2018 Dataset (Augment classes)
Classifiers	SEN (%)	PER (%)	ACC (%)	Time (sec)	FNR	F1-score (%)	MCC
N3	75.54	75.014	75.9	225.93	24.46	75.27608	0.7658
MN2	82.84	82.28	82.6	252.98	17.16	82.55905	0.8037
WN2	89.12	88.81	88.9	278.6	10.88	88.96473	0.4708
BN2	75.81	75.34	75.6	241.8	24.19	75.57427	0.6571
TN2	75.05	74.88	74.8	262.2	24.95	74.9649	0.7882
(c) Proposed classification results on ISIC-2019 Dataset (Augment Dataset)
Classifiers	SEN (%)	PER (%)	ACC (%)	Time (sec)	FNR	F1-score (%)	MCC
N3	65.725	64.725	65.9	208.5	34.275	65.22117	0.6581
MN2	74.475	74.02	74.6	302.4	25.525	74.2468	0.7037
WN2	84.5	84.21	84.5	238.9	15.5	84.35475	0.8708
BN2	64.98	64.337	65.1	267.3	35.02	64.6569	0.6571
TN2	65.16	64.675	65.3	302.5	34.84	64.91659	0.6882
(d) Proposed classification results on ISIC-2020 Dataset (Augment Dataset)
Classifiers	SEN (%)	PER (%)	ACC (%)	Time (sec)	FNR	F1-score (%)	MCC
N3	95.3	95.15	95.2	365.87	4.7	95.22494	0.9658
MN2	96.05	95.9	95.9	175.18	3.95	95.97494	0.9037
WN2	96.55	96.35	96.4	233.24	3.5	96.4499	0.9708
BN2	95.3	95.15	95.2	644.02	4.7	95.22494	0.9571
TN2	95.05	94.9	95.0	1023.5	4.95	94.97494	0.9882

Table 7.

Comparison of the proposed tasks with SOTA.

Segmentation Task
Paper	Year	Dataset	Average DICE (%)
[51]	2022	ISIC 2018	94.52
[63]	2021	ISIC 2018	89.93
[79]	2023	HAM10000	87.2
[80]	2022	HAM10000	87.0
Proposed Method (Segmentation)	2024	ISIC-2018	94.63
		HAM10000	94.69
Classification Task
[81]	2024	HAM10000	85.94
[82]	2023	HAM10000	75.90
[83]	2022	ISIC-2018	83.2–85.8
[84]	2021	ISIC-2018	76.6
[85]	2022	ISIC-2020	92.63
[86]	2022	ISIC-2020	91.61
Proposed Method (Classification)	2024	HAM10000	90.5
		ISIC-2018	88.9
		ISIC-2019	84.5
		ISIC-2020	96.35

This work employs an explainable AI (XAI) technique named GradCAM to interpret the proposed classification model. The testing images are passed to a trained model that further concedes the extracted features to the GradCAM method and visualizes the image's important part (lesion part. A few sample images are shown in Fig. 16. In this figure, a false negative rate is observed, which causes the lesion to be visualized incorrectly.

Fig. 15.

Area under the curve of Wide Neural Network (WN2) classifier after the augmentation process.

After the visualization process, the segmentation and classification performance is compared with several pre-trained models. In the segmentation phase, the backbone of DeepLabV3+ is changed with ResNet18, ResNet50, and ResNet101. The accuracy of the training obtained is plotted in Fig. 17. In this figure, it is noted that the ResNet18 model achieved an accuracy of 86.5 %, ResNet50 obtained 87.4 %, and ResNet101 obtained an accuracy of 86.1 % using the HAM10000 dataset. Similarly, the values are plotted for ISIC-2018 and show the proposed backbone obtained improved training accuracy.

Fig. 16.

GradCAM visualization results for model interpretation.

We implemented several networks on the selected datasets to compare the proposed MAKNet100 with several pre-trained models and noted the training accuracy. The training accuracy of the proposed MAKNet100 on the HAM10000 dataset is 92.46, ISIC-2018 is 91.60, ISIC-2019 is 88.52, and ISBI-2020 is 97.40 %, respectively. The comparison of these accuracies with other models is illustrated in Fig. 18. Based on this figure, it is concluded that the proposed MAKNet100 shows better accuracy.

Table 7 compares the proposed tasks (segmentation and classification) with several state-of-the-art techniques. In this table, the ISIC-2018 and HAM10000 dataset dice value is given for the segmentation task by researchers such as [51,63,79], and [80] and achieved average dice values of 94.52, 89.93, 87.2, and 87.0 %, respectively. The proposed method achieved an average dice value of 94.63 for ISIC-2018 and 94.69 % for the HAM10000 dataset, showing improved performance. In addition, the method presented in [43] used a backbone model such as InSiNet, designed due to a shortage of ground truth images. This model performed well for fewer images; however, the developed model is not tested on the larger dataset such as HAM10000. Our proposed segmentation backbone network has less number of parameters and improved accuracy on the larger and smaller datasets for the segmentation task.

Moreover, the classification task is also compared with several techniques, such as [81,82,83,84,85], and [86]. In these methods, they obtained an accuracy of 85.94, 75.90, 85.8, 76.6, 92.63, and 91.61 %, respectively, for HAM10000, ISIC-2018, ISIC-2019, and ISIC-2020. The proposed method obtained improved classification accuracy of 90.5, 88.9, 84.5, and 96.35 %, respectively. Overall, the proposed method indicated an improvement in accuracy of above 4 % for HAM10000, a 3 % improvement in ISIC-2018, and 5 % for the ISIC-2020 dataset, respectively.

In the final experiment, Kruskal-Wallis test is utilized in this analysis; it is a non-parametric statistical test that is being used to determine if statistically significant differences exist between the medians of three or more independent groups. In our case, the groups are the five classifiers (N3, MN2, WN2, BN2, TN2) and the test gauges whether or not the distribution of accuracy is sufficiently different among them. Fig. 19 shows the distribution of accuracy values for five classifiers tested against four datasets. From this plot, we observe that WN2 has the highest median accuracy and less interquartile range than the classifiers, so it generally has a better performance, although the data varied less. There are greater interquartile ranges and lower medians for N3, BN2, and TN2, indicating greater variability and worse performance. While these differences visually support a conclusion that WN2 performs better on average, the results of the Kruskal-Wallis test (H = 4.24, p = 0.374) indicate that observed differences in accuracy between classifiers are not significant (see Fig. 19).

Fig. 17.

Comparison of proposed segmentation model results based on change in Backbone network.

Fig. 18.

Comparison of proposed MAKNet100 classification model performance with several pre-trained networks.

Conclusion

Skin disease is the most frequently diagnosed disease in humans nowadays. The disease can be diagnosed at an early stage and treated better. The proposed framework performed two tasks: lesion segmentation and classification. The segmentation task is based on the improved DeepLabV3+, where a ResNet20 model is designed as a backbone. Changes in the backbone, such as ResNet18 and ResNet50, degrade the accuracy by up to 4 %.

Fig. 19.

Comparison of all the classifier across different datasets using Kruskal Wallis test.

Further, the improved DeepLabV3+ with ResNet20 gives more accurate segmentation results. In the classification task, the imbalanced issue is resolved by data augmentation and performed training of the proposed MAKNet100. The proposed model obtained improved training, validation, and testing accuracy compared to the model trained on the original imbalanced datasets. Furthermore, the XAI technique such as GradCAM is employed and finds the visualization of lesion part than the healthy region. Overall, we conclude with the following points:

• ▪Adding grouped convolutional layers in ResNet20 architecture improved the segmentation accuracy compared to the pre-trained models such as ResNet50 and 101.
• ▪The proposed MAKNet100 classification model is more efficient than pre-trained networks based on fewer training parameters and improved accuracy.
• ▪Initialization of hyperparameters using BO instead of manual selection significantly impacts improved training accuracy.
• ▪Visualization of the proposed MAKNet100 using GradCAM shows the effectiveness of the selected datasets.

In future, we will expand the framework by including models Swin Transformer architectures to enhance both segmentation and classification accuracy. These models may be anticipated as a means of further aligning global dependencies and long-range contextual information in dermoscopic images. In addition, we will employ domain adaptation techniques to improve generalization across datasets with different imaging protocols.

Compliance with ethics requirements

1) This material is the authors' own original work, which has not been previously published elsewhere.

2) The paper is not currently being considered for publication elsewhere.

3) The paper reflects the authors' own research and analysis in a truthful and complete manner.

4) The paper properly credits the meaningful contributions of co-authors and co-researchers.

5) The results are appropriately placed in the context of prior and existing research.

6) All sources used are properly disclosed (correct citation). Literally copying of text must be indicated as such by using quotation marks and giving proper reference.

7) All authors have been personally and actively involved in substantial work leading to the paper, and will take public responsibility for its content.

Declaration of competing interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Data availability

The datasets of this work are publically available for research purposes on the ISIC website: HAM10000: https://datasetninja.com/skin-cancer-ham10000; ISIC2018: https://challenge.isic-archive.com/data/#2018; ISIC2019: https://challenge.isic-archive.com/data/#2019, and ISIC2020: https://challenge.isic-archive.com/data/#2020. All these datasets are available for the research purposes.