DermNet: integrative CNN-ViT architecture for bias mitigation in dermatological diagnostics using advanced unsupervised lesion segmentation

Abstract

In this paper, we propose a method for reducing the bias in skin disease identification for people of color with the aid of lesion only zero shot unsupervised approach that is then passed to the classifier Dermnet comprising of a hybrid Vision Transformer and Convolutional Neural Network, achieving robust validation accuracy of approximately 81%. Our Segmentation without training with labeled data as is the case with traditional U-Net has achieved an IOU of 90% across all skin colors in segmenting the lesion from skin effectively eradicating the impact of skin in the classification of disease.

Introduction

Skin diseases pose a significant challenge in the field of medicine, affecting millions of people, not only affecting their physical health but also their mental health. Studies show that people with skin diseases such as eczema and psoriasis experience higher levels of anxiety, as noted in Ref. 1. Unfortunately, skin disease identification is often neglected. Numerous studies conducted in the developing world indicate that about 80% of cases go undiagnosed^2,3 due to a lack of immediate reporting and early diagnosis, which are essential for effective treatment. This highlights the critical need for the timely identification of skin diseases. During this investigation, the Pak Italian Burn Centre expressed concern about severe skin conditions that affect patients and complicate treatment, as noted by a plastic surgeon. They emphasised that the timely identification of skin diseases can prevent severe skin conditions. Furthermore, some skin diseases are difficult to differentiate, underscoring the importance of prompt identification and diagnosis for effective treatment. Traditionally, skin disease identification requires visiting an expert dermatologist, where the process of booking, scheduling, and long wait times leads to significant costs and burdens for both patients and dermatologists. Dermatologists often need scheduled appointments and specialized equipment to identify the disease before beginning treatment, making it a time-consuming task. Additionally, the limited availability of dermatologists causes further delays in timely diagnosis.

The global shortage of dermatologists has been an issue since the early 2000s^4–6, with demand increasing alongside the high incidence of skin diseases. There is a severe deficit of dermatologists and an uneven distribution of services, especially in developing countries and remote areas, where there is an urgent need for more medical facilities, professional consultations, and clinical assistance. The world is rapidly evolving with digitization, and technology has already started transforming the dermatological field, as AI now enables rapid diagnoses^7–9, leading to more diverse and accessible treatment approaches, as depicted in Ref. 1. With the advent of AI, studies on skin disease identification using ML have become more common; however, limitations and gaps still exist within current systems. These limitations often result in inaccurate diagnoses of skin diseases, sometimes leading to severe conditions. The main issues with existing systems include a lack of diversity and inefficient outcomes in skin disease identification, resulting in biased and inaccurate diagnoses for brown and dark skin tones. A significant factor in these inaccuracies is the lack of diversity in datasets, which leads to biased results. Additionally, human experience in medical imagery often surpasses that of fundamental image analysis due to the unpredictable nature of human physiology.

This study aims to bridge the limitations and gap between humans and machines by utilizing reliable data annotated by expert dermatologists. These dermatologist-annotated data ensure transparency and reliability within the system, supporting a diverse range of skin tones and promoting inclusivity and diversity by eliminating biases in skin disease identification. Our working prototype uses advanced technology in computer vision, medical imagery, and deep learning, along with the latest techniques in image segmentation and traditional methods. This integration ensures technological advancement while retaining the human touch with dermatologist-annotated datasets, making it reliable and more diverse. It supports the reduction of bias and enables the timely diagnosis of skin diseases, preventing progression toward severe skin conditions and treatment difficulties. We are committed to advancing AI and ML technology with the latest techniques. Our goal is to provide a pipeline for the identification of skin diseases, serving as a state-of-the-art solution for the timely identification of skin diseases across diverse skin tones with higher accuracy. The study finds that most skin disease datasets lack sufficiently diverse labeled data, which prevents optimal suggestions for people of color. This lack of human validation in skin disease datasets hinders the credibility and scalability of current classifiers, which, despite high metrics, tend to underperform in cases involving diverse skin tones. Furthermore, data collection poses a challenge, as most skin disease data are collected from hospitals, and many underdeveloped countries tend to overlook data collection when treating diseases. The existing approach makes it much harder to develop datasets representing various regions. Therefore, the study aims to separate skin lesions from surrounding skin, ensuring consistent disease images across different skin colors.

This approach has introduced a significant hindrance: a supervised approach is challenging to implement in this case because most datasets, though annotated, lack provided segmentation masks even in non-diversified datasets. Manually generating segmentation masks is both time-consuming and costly, as well as prone to errors. Therefore, this study employs an unsupervised approach to ensure that existing resources are used optimally without compromising image diversity. The study tests and evaluates multiple methods to segment skin lesions from skin diseases, aiming to find an optimal balance for a segmentation pipeline effective across a variety of skin diseases and skin tones. The study utilizes various computer vision techniques and also employs Segment Anything, a highly generalized model for unsupervised segmentation, to achieve optimal segmentation. Through our research on skin diseases at the Pak Italian Burn Centre, we have gathered substantial evidence of the challenges patients face, as many are unaware they have a skin disease and are constrained by the high costs, scheduling issues, and long wait times for proper checkups. This situation has led to numerous cases progressing from minor to major due to patient neglect. We have developed an application prototype that offers an end-to-end solution to address this issue. This application is not a replacement for a dermatologist but acts as an assistant, providing simplified classification results with increased reliability in separating the disease from the skin.

The framework initiates with the user uploading their image, which the system segments and classifies while attempting to mitigate biases across skin tones due to its diverse dataset. The system then classifies the disease further. Dermatologists can use this information to verify the condition and provide feedback. This data is also stored, enabling further collection of labeled data from diverse skin tones to ensure the model continually evolves and adapts to the needs of different users. Figure 1 shows three distinct cases. Figure 1a shows a minor case of eczema that can be treated promptly with appropriate medication, compared to the severe case of eczema shown in Fig. 1b and c. Our study aims to bridge the gap by providing individuals with tools to identify diseases promptly, thus preventing disease spread and worsening. Figure 2 illustrates the abstract structure and basic concept of the developed system, which integrates basic computer vision techniques with state-of-the-art models. The second dataset we utilized is a Stanford-based dataset called Diverse Dermatology Images (DDI), collected with diverse skin tones and annotated by dermatologists with the appropriate skin tones and disease labels. This dataset and our analysis show how top-performing algorithms lack the variance necessary for use across diverse skin tones, thereby emphasizing the need for diversity¹⁰. This deficiency adds an 8-12% difference in current systems’ performance. It worsens with the fact that available datasets, such as Dermnet Ham1000, under-represent diverse images, thus embedding biases in dataset fundamentals and leading models to favor white-pigmented skin images. The main contributions of our work are given below:

1. Zero-shot lesion-only segmentation: Our model fuses SAM (ViT-H) with HSV/YCbCr transforms, Otsu, and adaptive thresholding computer vision techniques to isolate lesions without mask supervision, robust across skin tones.

2. Bias mitigation via lesion-only training in classifier: We train the classifier exclusively on lesion crops, removing the noise induced due to background of the skin.

3. Efficient classification with Dermnet: Our model works with only two convolutional layers, without having the complex backbones such as ResNet leading to a smaller model of only 2.51 million parameters and only 10 MB in size whilst still having the highest accuracy of previously trained models.

Fig. 1.

(a) Eczema patient hands from the Pak Italian Burn Centre that shows the disease is not very visible on the skin surface. (b) Eczema patient’s leg skin disease image from Pak Italian Burn Centre that shows the skin disease in a severe condition due to not being diagnosed promptly. (c) Eczema involves both limbs.

Fig. 2.

Proposed pipeline that starts with the skin-con dataset and uses segmentation before it goes into the classifier (DermNet).

The rest of sections are organized as follows. Section "Related work" discusses detailed literature review skin dataset and existing challenges. Section "Methodology" describes the components of the proposed approach including dataset specifications and our proposed skin lesion segmentation masks technique. The experimental setup are explained sequentially in Sect. "Experimental setup". Section "Results and discussions" presents results and discussions. Finally, Section "Conclusion & future work" concludes the paper and provides further research directions.

Fig. 11.

Segmented skin lesion - bitwise AND applied on masked image. This is the actual representation of the bitwise AND skin image that shows how well the implemented technique of segmentation works, with each area of skin where the disease is present being properly highlighted and distinguished.

Related work

A considerable amount of research has been conducted on skin datasets such as the ones in Refs. ^11,12 along with the evaluation of different neural networks and deep learning frameworks on those datasets like the one study conducted on Fitzpatrick 17k dataset¹³, which are abundant in colored images, with the vast majority comprising different skin diseases. Our approach addresses the end to end pipeline of the process focuses on challenges in data limitations to the bias in background skin and the large models required for classification of diseases.

Data Limitations and Synthetic Image Generation: A dataset annotated by a specialized dermatologist in the field of medicine is challenging to find therefore, this led to us using the combination of the two datasets respectively that were merged in the package of SkinCon¹⁴ This dataset has been used in our research paper and has been manually labeled and updated with the 114 current diseases from Fitzpatrick and 85 diseases from the Diverse Dermatology Images Dataset. This dataset is further studied using ITA analysis and compared with the Fitzpatrick skin formulas to identify skin types based on the Fitzpatrick scale. This dataset is further annotated by both the algorithm and expert labels, thus ensuring the comparison and making sure that the experts fundamentally check the data. This dataset is regarded as a SkinCon dataset that comprises the data annotations in the form of multiple features extracted by the expert dermatologists in the field in numeric form that are mapped on one by the images’ Metadata of both the Fitzpatrick images, along with the DDI dataset images, whose experimentations are mentioned in Fig. 3. These annotations are pretty diverse, claiming to encompass over 48 numeric-based features from both datasets. This diversity is leveraged by summing up the two datasets and merging them through the use of these features. This data is further evaluated with the feedback, and fine-grained analysis is performed over it for the annotations. Each category has at least 22 annotated images on what to expect from the dataset, and that can be further exploited towards collecting more data and fine analysis of the algorithm, even with that smaller amount of data, supervised learning techniques can perform quite well considering the number of features, ensuring the transparency.

Fig. 3.

Experimentation on Diverse Dermatology Images (DDI) of the different algorithms in the SkinCon dataset. The results differ in accuracy as the darker skins (FST V-VI) show inferior results as compared to their lighter skins (FST I-II) counterparts.

The reason is the problems with the collection of actual data in underdeveloped countries, where skin treatment is still done manually. The data collection is next to nothing, in the case where it would be great to synthesize a few images based on the limited dataset, the developments in the field of Generative Adversarial Networks provide this opportunity to build on existing images, by synthesizing the exact size of the photos to produce brand new images instead of simply augmenting the previous ones. The following paper¹⁵ presents a novel approach to synthesizing medical images that are equivalent to the original photos, as evaluated by both an average human and dermatologists, and are modifiable by skin tone. However, the results are lagging in this step. The proposed architecture is based on the pix2pix architecture, and in the human Turing test, it performs exceptionally well. Additionally, in evaluation metrics, it performs equally well or better than the baselines in rare cases. Another end-to-end system, LesionAid, developed and deployed in Ref. 16, features the same analysis for skin cancer detection. This system can be further evolved into skin disease identification, utilizing the Vision transformer after GAN augmentation¹⁷ to explore predictive analysis in depth.

Image segmentation: The image segmentation field has progressed leaps and bounds in terms of separating and detecting skin lesions and skin parts of the region. To do that, the systems have shown resiliency even with simple pixel-based computing segmentations. The problems that have arisen are the facts that this model doesn’t work uniformly with all types of skins and their varying contrast and can instead perform quite poorly for the brown skins especially without the change in parameters manually of the kind of skin¹⁸, one method is to change the parameters dynamically before detecting or classifying the skin tones in various categories such as the ones proposed in the Ref. ¹⁹ Fitzpatrick scale as seen in Fig. 4, but the resultant segmentation varies and depends significantly on the type of the model used for identifying the skin tone of the skin which in itself can vary greatly as well as fine tuning the parameters, one other problem of whose is the fact that we need relatively high quality images for the segmentation to perform well due to inherent degeneration of the picture in the process. This process is both time-consuming and unreliable for user input, as we can’t predict the type of image the user will show to the system. The other model that aims to curb the effects of this particular case by using the game theory zero-sum model²⁰, which treats each pixel as a binary classification which is shown as the conflicted area with each skin detector acting as strategy and then each pixel is calculated for Nash’s equilibrium reducing the false positives For the hand detection images dataset, the ROC curve shows auspicious results.

Fig. 4.

Scale of Fitzpatrick skin tones. The Individual Typology Angle (ITA) qualitatively measures skin tone. ITA categorizes skin types into six categories, from very light to dark skin. The higher the ITA, the lighter the skin: very light (), light ( to ), intermediate ( to ), tan ( to ), brown ( to ), and dark (). An ITA corresponds to darker skin phototypes.

The methods mentioned above are an extension of the computer vision techniques and color gradients used in Ref. 21 to provide a more computer vision-based approach for segmentation instead of deep learning models that make use of the HSV and YCbCr color schemes for segmentation. We look at the old yet revolutionary architecture of CNN U-NET based²². This work has shown potential in separating the spores and can be applied to a variety of images with almost similar results. Still, since much of this is quite old, we have gone on to study some more recent versions of similar work that have been applied to the works of U-NET and combined with the recent works in the fields of attention. Therefore, the new UNET is the attention-based neural network, and in this case, especially gated axial attention U-NET that further improves performance²³. Furthermore, we have investigated two state-of-the-art models, one of which is STEGO, which is based on unsupervised image segmentation that focuses on contrastive images, by distilling multiple similar photos together, it’s able to semantically perform the segmentation by checking the identical features from the other pictures thus being able to segment the data better, we feel this could be used to segment the models even on small training data due to the contrastive learning it’s easier to train because all diseases of similar types are grouped and have identical patterns making them an ideal choice for the segmentation²⁴.

The limitations of these systems is they either require intensive training requiring segmentation masks or they use a subset of the computer vision techniques such as a dedicated model named MRCNN proposed in Ref. 25. While being extremely performative and adept at finding details, requires a training with annotated dataset along with a training loop of 100 Epochs, which in itself is limited in case of diversity in skin diseases. Another study²⁶ claiming a breakthrough in terms of accuracy and IOU score that combines the deep learning with the Ant Colony Optimization still requires intensive training with the predefined masks and annotated dataset. Our segmentation technique reduces the dependency on the masks and instead takes into account the zero-shot capabilities of segment anything that is further optimized by the computer vision techniques and is experimented on a range of skin tones to ensure that diverse skins along with a range of diseases can be catered to without the need of densely annotated datasets while saving the resources in terms of training leading to the model being generalized enough to cater towards other diseases with tweaking as well as a range of skin tones. Lastly, we look at the state-of-the-art model released by the Meta that has already been trained on 11 million images, and 1.1 billion masks of a variety of types that can generate quite accurate masks based on the segmentation without requiring additional training provided the thresholds are pretty accurate, the exports in the ONNEX types are also quite reasonable in terms of further loading and collecting the meta types²⁷

Existing classifiers: There have been various ways where this system has been implemented albeit on a smaller scale, the system has a lot of pre-trained models as well, but we note their outdated eras and realize the fact that much more sophisticated models in today’s world are making strides in the other areas of deep learning, this presents us the opportunity to build on and improve the system that is already in place more transparently and reliably. The basic architecture that was the end-to-end system was built in Ref. 28 that combines the skin disease identification system with an end-to-end system, while the basics of the system work the technology used from 2021 is already outdated with a simple SVM for classification and seven diseases with 10-14 images per dataset with a pre-trained model and then using the YcBcR image masks

This system utilizes a block of explainable AI (XAI)²⁹ to describe the true positives and provide system descriptions. This system is another end-to-end system deployed on the web, similar to those mentioned above, with the key difference being its use of state-of-the-art technology and its focus on skin cancer detection. While the existing systems use the CNNs and ViT separately (See Table 1 for abbreviations), several systems have found competitive results combining both of them through stacking as well as bagging³⁰. In the following novel architecture³¹, the CNNs are combined with the Vision Transformers for multi-focus image fusion. This being able to combine the local region features of CNN with the Global features of ViTs makes the edges detectable as well as localization of the region. The new ensemble architecture of CNN and SVM proposed in the Ref. 16 shows effective results along with a variety of ensemble models most notable as the Cswan transformer and EfficientNet While also proposing the other forms such as CvT. The following paper³⁰ examines the performance of the pre-trained models available and their performance based on this. Table 3 provides a comparison of the various state-of-the-art models that have been previously used in multiple networks and concludes that most trained models can only attain an accuracy as high as 76% when it comes to identification performance. The paper further explains that most models still struggle to perform on less popular diseases, thereby vindicating the claim that the model’s performance is insufficient for the needs of unpopular diseases in developed countries, rather than those in developing countries.

Table 1.

Table of acronyms.

Acronym	Full form of acronyms
ViT	Vision Transformer
CNN	Convolutional Neural Networks
DDI	Diverse Dermatology Images
SAM	Segment Anything Model
HSV	Hue, Saturation, Value
GAN	Generative Adversarial Networks
SVM	Support Vector Machine
Ham1000	Human and Machine generated 1000 images
YCrCb	Luma (Y), Chrominance-red (Cr), and Chrominance-blue (Cb)
STEGO	Self-supervised Transformer with Energy-based Graph Optimization

Table 3.

Performances of the pre-trained networks for skin lesion classification.

Network name	Imbalanced data	Balanced data
	Accuracy [%]	Accuracy [%]
NasnetMobile	66.60	73.24
GoogleNet	68.06	72.20
MobileNetv2	67.43	71.99
ShuffleNet	69.73	71.16
EfficientNetb0	65.97	73.44
ResNet-18	72.03	74.27
EfficientNetb3	69.73	75.88

Another study³² proposes a Hybrid CNN-ViT model for the efficient detection of the seizure and notes the superior performance of the hybrid CNN-ViT model denoting the accuracy of over 94% and sensitivity of over 99%. They also faced the same problem of capturing the local and long-range correlations in EEG signals with traditional methods leading to the hybrid model with exceptional results. Two further works of Hybrid CNN-ViT models are discussed for medical images classification for Tuberculosis chest xrays^33,34, while being highly accurate require a pre-trained backbone like Resnet and EfficientNetV2L as part of their hybrid architecture leading to the model being large in size and slow in training. Our proposed Dermnet trains on just two convolutional layers leading to an efficient model of less than 10MBs in size and being effective enough to compete and outperform other models on a range of disease in terms of accuracy.

Existing hardware-based solutions: There are aplenty hardware based research on the medical issues as the one proposed in Ref. 35 focuses on active learning using deep reinforcement learning instead of CNNs mentioning the same limitations of dataset scarcity in the case of using CNNs with another study³⁶ proposing an optimization of the deep learning to further improve accuracy. Furthermore, there are models that leans towards the detection of the disease before time by focusing on the spatio-temporal modeling of the disease like the one proposed in Ref. 37. Another study³⁸ focuses on the novel aspects of removing irregularly sampled time series data, they perform the proposed study on two healthcare datasets showing competitive results. Further studies, shown on using the memristor focusing on household products for medical decisions and capturing temporal data^39–41 can result in intelligent households that can capture the instances of the temporal data leading to early detection and even prevention methods. Our core goal with this study is to extend it to the point where we can use IOT devices to capture not only the skin parts but also the direct and indirect effects of the skin diseases that might be causing other problems like dermatomyositis can cause inflammation to the lung muscles to cause shortness of breath or to use skin disease and temporal data alongside skin diseases to identify the broader illnesses such as auto-immune disorders. Whilst we are working on this the study also aims to address the inherent bias in the future studies to ensure that the systems and devices work equally well for all the patients.

Methodology

We devised a bias-mitigated skin disease classification pipeline by integrating an unsupervised zero-shot segmenter with a hybrid CNN–Vision Transformer (DermNet) architecture. SkinCon Dataset (A combination of Fitzpatrick17k and Diverse Dermatology Images), annotated by dermatologists, was combined and augmented to ensure skin tone diversity. As most datasets lacked segmentation masks, we implemented an unsupervised segmentation process using Meta’s Segment Anything Model (ViT-H), optimized with HSV and YCrCb colour space transformations, Otsu thresholding, and adaptive contouring to isolate lesions from surrounding skin regardless of pigmentation and colour of skin. The segmented lesion regions were then fed into DermNet, which extracted local features via CNN layers and global context via ViT (Dermnet), improving generalization and reducing skin tone bias. Experiments demonstrated that segmentation before classification significantly improved accuracy and reduced variance across skin tones, achieving  81% validation accuracy with an enhanced zero-shot segmentation pipeline that works consistently for lighter and darker skin tones, leading to a reduction of skin-based bias in the ending classifier (DermNet). While the existing studies catering towards skin diseases either cater towards majorly white skin datasets leading to bias or training based segmentation like U-Net having limitation for the segmentation masks availability for diverse range of skins or pre-trained backbones for CNN leading to huge models. The existing studies are focused on one of the problems of either dataset, segmentation, or classification. Our study’s main novelty lies in the complete end to end pipeline to reduce bias by the utilization of diverse dataset, zero-shot segmentation achieving a mean IOU of 90% of skin lesion and an efficient classifier (Dermnet) that combines the CNN and Vision Transformer in a way that CNN doesn’t rely on pre-trained backbones whilst achieving a competitive accuracy of 81%

Dataset specifications

The dataset used in the study is referred to as the skin-con dataset, which is a combination of the Diverse Dermatology Images and Fitzpatrick dataset, consisting of 174 distinct diseases labelled for various skin tones. But some diseases only have a single image corresponding to those diseases; therefore, we adjusted our threshold to ensure that the disease should have at least 20 pictures related to it to be considered in the dataset, leading us to a dataset consisting of 122 distinct diseases, with each disease having at least 20 images associated with it. We optimized our segmentation to ensure that all lesions are segmented from all types of skin, regardless of the disease and skin tone, so that only the infected area is fed to the DermNet. The images have varying resolution, quality, and aspect ratio, which are then pre-processed to ensure uniformity before going to the DermNet, which accepts the images at resolution. The total images used after pre-processing and removing the diseases whose number of images is below 20 for training and testing is 3643, out of which 70% constitutes the training dataset and 15% pertains to the test and validation datasets each. The skin tones are segregated as per the ITA number of the Fitzpatrick scale to be: For the white skin tones: 1340 For the brown skin tones: 1154 For the black skin tones: 1149.

The entire dataset undergoes augmentation before training, including rotation and brightness adjustments, to ensure consistency across training sessions, reduce model bias, and balance the dataset with the detailed parameters in the Table 2.

Table 2.

Augmentation of dataset parameters.

Category	Parameters
Geometry	Rotate (0–), Flip (H/V), Scale (0.9–1.1), Shift (±10%)
Photometric	Brightness/Contrast (±10%), Gamma (0.9–1.1), Noise, Blur ()
Boundary	Dilate/Erode (±1 px), Edge-ring blur (3–5 px, )

Skin lesion segmentation masks technique

The dataset lacks segmentation masks i.e. masks that help separating the lesions from the background skin, as is the case with most datasets of these sizes (Table 3). We aimed to generate segmentation masks using computer vision. Since the process can vary, we adapted thresholds for different tasks and conducted various experiments. For our base Image to judge the experimentation graphically, we use the image that has the most varying amounts of problems. The skin image in Fig. 5 is chosen as the base image due to its specific challenges, such as the dark shadow and uneven distribution of the disease on the skin. The skin is of brownish pigment taken from the Indian subcontinent, where eczema is rampant. The disease is recognizable at early stages, but this can only be seen by examining the various spots within the marked lines. The high-quality image also poses a potential limitation: the skin tissue structure might be misinterpreted as part of the disease, even if it is merely a skin issue. First, we apply multiple variants of unsupervised segmentation algorithms to train and work on them without labels or available segmentation masks, which enables us to provide the images with various algorithms. The comparison of the of segmentation algorithms is stated in Table 4 and is shown below:

Fig. 5.

Base image is taken from a diverse dataset. It shows evidence for an uneven distribution of disease, having varying brightness and contrast, as well as shadows that demonstrate the complexity of image segmentation.

Table 4.

Results and names of unsupervised segmentation algorithms.

Algorithm	Dataset	Mean IOU
STEGO	Fitzpatrick	71%
LeoPart ViT-B/B	Fitzpatrick	84%
SAM (VIT-H)	Fitzpatrick	91%
STEGO	DDI	70%
LeoPart ViT-B/B	DDI	87%
SAM (VIT-H)	DDI	90%

Our aim is to binarize the image i.e. convert the image into black and white image that ends up being the segmentation mask for lesion on skin. For that we use the SAM to perform semantic segmentation and then use YCbCr Image for detecting the contours of the lesion and in parallel using HSV Space with OTSU’s algorithm for generating the segmentation mask of the disease. These two binarized images (Cr Channel and OTSU binarized image) are then combined using the bitwise AND leading to the segmentation masks that segments the lesion from skin as well as detects the skin area if present inside the skin area for diseases that are non-uniform and might have small scale regions of lesion alongside the healthy skin improving reliability for different types of diseases. While the LeoPart ViT-B/B pre-trained increased the IOU on the DDI dataset, the SAM held well in either of the cases, with the highest IOU as evident in the Fig. 6; therefore, we are going to use the SAM for the initial part of our unsupervised segmentation to highlight diseases. The most fundamental aspect of using SAM was that we did not need to fine-tune it; the model was sufficiently generalized from the start, allowing for auspicious results with minimal parameter tuning and threshold adjustments. The parameters are defined in Table 5. Afterwards, the image is changed from the RGB to the HSV and YCbCr systems. HSV Systems were implemented to show the separation of infected and normal skin in a flattened highlight, as shown in Fig. 7. Further, we perform the binarization of the Skin images by applying Otsu’s algorithm on the HSV Spaced Image.

Fig. 6.

SAM ViT-H results. The highest IOU is achieved using the state-of-the-art SAM model after configuring the appropriate layers and settings. The VIT-H model is employed in the proposed method.

Table 5.

Parameters for segment anything VIT-H.

Parameter	Value
points_per_side	32
points_per_batch	256
pred_iou_thresh	0.90
stability_score_thresh	0.92
crop_n_layers	1
crop_n_points_downscale_factor	2
min_mask_region_area	100

Fig. 7.

Comparison between SAM-based segmentation and HSV color space representation. (a) The SAM ViT-H model produces a segmentation map. (b) The HSV representation enhances contrast, highlighting the disease based area of skin and uniforming the background intensity.

According to our experiments, Otsu’s algorithm works best on this type of HSV-spaced image with prominent backgrounds and foregrounds. Therefore, apply Otsu’s algorithm to generate a segmentation mask of the image and contouring on Skin images with YCrCb after HSV to have better and more accurate results of the classification models. Now that we have a mask to make the figure more precise, the image is split into the YCbCr channels, which provide us with accurate edge detection of each infected part. In our results, the best approach is to utilize the luminescence data. Now, as per our experimentation, we have molded the image from basic SAM to the YCbCr Image, which has paved the way for adaptive thresholding. In this case, the luminescence has a minimal impact on the system and can now adapt according to the threshold to find the system’s contours, as evident in Fig. 8 where Y, Cr, and Cb channels are being illustrated in Fig. 8a–c respectively. Now, applying the adaptive thresholding on each of the channels, we get a strong result of the contours of each disease. resulting in strong contour detection, these contours further help with the problem of even more detailing and reducing the noise of the image, as each contour will make it easier for us to identify the specific disease. These contours are helpful in the use-case where the skin has specific tiny details, this is a simple method of contours that can further help in reducing the size of the image, the images made by contouring can then be bitwise and with the images created by adaptive thresholding of the image leading to binarized image i.e as evident in the segmentation mask in Fig. 10.

Fig. 8.

YCrCb color space decomposition of skin lesion images. The input image is separated into Y (luminance), Cr (red chrominance), and Cb (blue chrominance) channels to analyze channel-specific feature representations.

Fig. 10.

Otsu-Cr based segmentation mask. This skin disease image shows the segmentation mask generated by the Bitwise Operation between OTSU algorithm and Cr Channel, which is to be used to segment the lesion of disease from skin itself.

Applying the adaptive thresholding to each of the channels to properly form the contours and evaluate the contour of each channel, and using the image on actual skin by combining the contours with the segmentation mask. As shown in Fig. 9 the binarized channels of YCrCb channels are depicted with Fig. 9a–c showing binarization of Y, Cr, and Cb channels respectively. They have been used in conjunction with the OTSU based image using bitwise AND first to account for the details inside the local region of lesion themselves, improving the reliability and understanding the details inside the lesion itself (Fig. Fig. 10). These have been proven effective techniques for varying skin diseases and for the denoising of the skin image. Now we would take the picture and combine it with the contours and the segmentation mask using the AND operation to retain the infected area of the disease and reduce the skin flesh area, a bit with the least threshing, even though if the particular method is not as visible here with the contours, it is more visible where the boundaries are a little less specific. The results show that the YCbCr approach is best if we want to find contours and finer granular details, as each channel leaves a particular room for detail in the contouring. The best results for a segmentation mask are obtained using the HSV colour space, which provides the best balance between information and optimal segmentation of lesions as evident in Fig. 11, which shows only the affected part of the skin, despite the disease being spread non-uniformly across the entire skin. Similarly, we experimented with the same leisoning techniques on the images that are high up on the ITA Scale, i.e, darker skin-tones. Figure 16 shows the comparison of the segmentation mask of the acne on a darker skin tone with varying brightness and spatial density of the disease itself. The original image in Fig. 16a is completely and accurately segmented as visible in Fig. 16b.

Fig. 9.

Binarized YCrCb channel representations of the skin disease dataset. Among the three channels, the Cr channel highlights lesion boundaries and localized disease regions more distinctly than the Y and Cb channels.

Fig. 16.

Combined results of each model. (a) shows the demo image of acne on a DDI image, while (b) depicts the segmentation mask returned by the server in 1 minute 16 seconds.

DermNet

The proposed architecture, illustrated in Fig. 12, combines a Convolutional Neural Network (CNN) with a Vision Transformer (ViT) encoder and leverages the strengths of both algorithms for devising a classifier. The CNN part, consisting of two convolution layers, is particularly adept at extracting local features such as boundaries, shapes, and edges from the input images (segmented images that only contain the affected part of the skin). These feature maps are then divided into patches and passed to the ViT, where they undergo linear projection and positional encoding. The ViT applies multi-head self-attention to these encoded patches, enabling the model to capture global contextual relationships across the entire image. This design allows us to capture the complex disease patterns, especially in cases where relevant features are non-uniform and dispersed. Finally, a classifier head is added where the final output is normalized after the dropout layer for prevention of overfitting, followed by a dense activation layer that produces the logits of probabilities of the diseases (Tables 6 and 7).

Fig. 12.

Detailed diagram of DermNet for classification of diseases.

Table 6.

Results of all experiments of pre-segmented classification.

Model name	Approx epochs	Training accuracy	Validation accuracy
Basic CNN	80	0.45	0.23
Basic ViT	70	0.47	0.12 (Highly Fluctuating)
DermNet	100	0.85	0.56

Table 7.

Results of all experiments of segmented classification.

Model name	Approx epochs	Training accuracy	Validation accuracy
Basic CNN	50	0.71	0.50
Basic ViT	60	0.76	0.55
DermNet	60	0.812	0.826

For example, in dermatological conditions such as acne, the contours of affected regions are often irregular and distributed over large, non-uniform skin patches. A pure CNN may struggle in such cases, as its locality bias can result in misclassification when lesions are not evenly spread. The inclusion of the ViT addresses this limitation by modelling long-range dependencies between distant patches, thus enhancing classification accuracy for diseases with subtle or spatially distributed features. In our implementation, we first segment the images and binarise them into the disease-only areas. This preprocessing step standardizes the visual input and allows us to evaluate the probability scores output by the classifier more effectively by reducing the excess skin as input. The evaluation results, as summarized in Table 8 return the top 3 diseases it can find, with the probability scores, demonstrating the effectiveness of this hybrid CNN–ViT (DermNet) approach in capturing both local and global feature representations.

Table 8.

Condition Percentages by Skin Type.

Skin type	Acne (%)	Dermatomyositis (%)	Pediculosis lids (%)
White	98.47	0.56	0.97
Black	99.99	0.0016	0.0008
Brown	99.98	0.0211	0.0008

Firstly, we have given the model the entire RGB images as shown after the pre-processing and normalization. After this, we demonstrated the model using skin-masked pictures that consist of only the infected area in RGB and the skin behind it in black, which means that we have omitted the entire RGB image and replaced it with the masked image. A detailed high-level diagram of our model is shown in Fig. 12.

Without applying segmentation, the validation accuracy dwindles around 0.50. It takes around 100 epochs to reach the point after which model starts being overfit making it non-generalized model as evident by the training graph in Fig. 13, this accuracy is provided that entails that the model might be over-fitting and the despite the overall high accuracy, the validation accuracy is fluctuating and seems likely to decrease should the model be allowed to train more. With the application of the segmentation masks our accuracy has risen to 90% with our validation accuracy of 81% this shows that the data has been generalized as evident by the graph in Fig. 14, and the model has been pretty specific in the generation following the masked images, highlighting the fact that we have made our algorithm much more effective, this achievement highlights the importance of performing only on the segmented data that separates the disease from the skin, this has allowed us to reduce bias all the while improving the performance of our algorithm that has been the key reason for us to move further and focus more on making sure that the data that goes into the classifier is only the data that’s needed to identify the disease instead of sending the skin lesions that go unused or in several cases counterproductive.

Fig. 13.

Accuracy for pre-segmentation DermNet (Training without segmenting images) leading to an over-fitted model.

Fig. 14.

Graph of model accuracy after segmentation. Model validation accuracy substantially despite more number of epochs, leading to better accuracy and generalization of the model.

Experimental setup

We have experimented with all our models using Colab and Kaggle as their backend to make standardised use of the publicly available and accessible GPU. The GPU we used is named P100. The architectures and considerations of both platforms are listed below in Tables 9 and 10,  respectively. Post that we have made it into a prototype app that uses an Expo foundation as shown in Fig. 15, this app allows the user to upload the image and get the results displayed elegantly in a chart, as we can see the results in Fig. 15b. The picture we have used to demonstrate here is a picture of disease as shown in Fig. 15a. For classification, the server returns the probability of the top 3 classes in less than 20 seconds.

Table 9.

Google colab architecture used and configurations.

Component	Description	Considerations
Compute & Environment	Google Colab with P100 GPU,  12GB RAM,  100GB Disk, Intel(R) Xeon(R) CPU	Data need to be reloaded if the session expires.
Deployment	Flask for API creation + Ngrok for tunnelling	Ngrok tunnels are temporary and
Accessing API	Postman	Ensuring proper format of data.
Version Control	GitHub	Save the dataset
Monitoring	Ngrok Web Interface + Logging in Flask	Monitor traffic and errors by connection endpoints

Table 10.

Kaggle architecture used and configurations.

Component	Description	Considerations
Compute & Environment	Kaggle Notebooks with P100 GPU,  16GB RAM,  5GB Disk, 2-core Intel CPU	Storage space is more limited compared to Colab.
Deployment	Flask for API creation + Ngrok for tunneling	Like Colab,
Accessing API	Postman	Test endpoints thoroughly, considering Kaggle’s specific environment constraints.
Version Control	Kaggle Notebook Versioning + GitHub	Use GitHub for more complex version control beyond Kaggle’s automatic versioning.
Monitoring	Ngrok Web Interface + Logging in Flask	can’t use TensorBoard for monitoring.

Fig. 15.

Results from the actual application when given a photo of the disease. The top 3 are shown and displayed with their probabilities.

Results and discussions

The results are competitive for both the segmentation as well as the dermnet with the segmentation IOU results acheieving a mean IOU of 90% on DDI dataset which is already a densely diverse dataset leading to a generalized approach for the segmentation of skin-lesion as in the Table 4.

Furthermore, the Dermnet also achieves competitive accuracy of 81% among it’s competitors leading to an efficient and effective model for various skin tones as described in Table  11 and Fig. 16.

Table 11.

Results of all experiments of segmented classification.

Model name	Size (MB)	Parameters (million)	Input size	Accuracy (%)
EfficientNetb0	20	5.3		73.24
ResNet-18	44	11.7		72.20
MobileNetv2	13	3.5		71.99
GoogleNet	27	7.0		71.16
NasnetMobile	20	5.3		73.44
ShuffleNet	5.4	1.4		74.27
EfficientNetb3	75	12		75.88
ResNet-18	44	11.7		74.27
DermNet	9.87	2.51		81.02

Segmentation results on skin tones

The unsupervised approach towards segmentation worked best when Incorporated with the HSV image and the segmentation algorithm is Segment Anything by Meta, This pipeline appears to be the most optimal for most skins as the Segment Anything successfully assigns the classes to different patterns of the skin lesions while providing a single class to the skin while also being robust to changes in brightness of the image thus paving way for effective segmentation. The results of acne on 3 types of generic skins, namely brown, black, and white, are studied in the results of the segmentation. The images are all taken generically from the web in order to properly test the generalization of the model, except for dedicated models. The white skin results are referred to in Fig.  17 with the image in Fig. 17a and its segment mask in Fig. 17b. The black skin results are referred to in Fig. 18 with the image in Fig. 18a and its segment mask in Fig. 18b.

Fig. 17.

Segmentation results for acne on White Skin.

Fig. 18.

Segmentation results for acne on Black Skin.

The brown skin results are referred in Fig. 19 with the image in Fig. 19a and it’s segment mask in Fig. 19b

Fig. 19.

Segmentation results on brown skin.

DermNet results

The results represent the key findings by achieving state-of-the-art accuracy of 81% and reducing bias in classification through segmented images that only contain the infected area; we believe this work will contribute to diversity in skin disease identification systems. This zero-shot segmentation approach could potentially lead to advances in technology and techniques in the digital era of dermatology. We have achieved a fairly comparable accuracy across the board as evidenced by the existing pre-trained networks in Table 3, and have increased efficiency and effectiveness across all skin tones through our segmentation process. Through our work, we have noticed that after applying the segmentation masks, our data accuracy has increased many times. This improvement comes from the use of a diversified and dermatologist-verified data set.

As evident in Table 11, our model has performed better than the pre-trained models with the same network set. Not only that our model has been more efficient, and much quicker to train than most of these models, although the pre-processing of our system takes a significant portion of time, with the generation of the segmentation masks we have concluded that most of the time is saved given the proper segmented images that has increased our accuracy by a significant measure, these segmentation masks have helped generalize the disease on the majority of the photos, ergo requiring less training data and a less complex model. Our model of the CNN-based transformer that feeds the local features into the ViT has been small but adequate. This model can be scaled to include more convolutions and more patches. The pipeline of pre-processing has not only helped improve the accuracy but has also been a step towards generalizing the model for most of the skin tones. The segmentation process has helped us to bring the non-diversified data and diversified data into the same net, which has led to a more generalized pipeline and a pipeline that can work for the majority of the skin tones effectively as evidenced by the difference in accuracies pre-segmentation Table 6 and accuracies post-segmentation Table 7.

Conclusion & future work

In this study, we have concluded that it is indeed possible to reduce the bias in the existing systems having effectively turning most, if not all, skin-diseased datasets into segmentation masks, which in turn has made the data less diverse as once if the disease is separated from the skin, the skin tones effectively do not matter. We have also defined a classifier that has withstood the effectiveness and efficiency test and has performed exceptionally well and in many cases better than the existing systems, all while consuming fewer resources because the input to that classifier is the segmented image that, after the extraction of global and local features, has made the process more accurate and robust. This approach also serves as a framework for future systems that require reliability and robustness. The segmentation approach employed shows that the traditional methods can work alongside the state-of-the-art algorithms to provide effective results and a comprehensive approach to do that in the case of skin disease segmentation. The study also finds the challenges in the current systems and how the lack of adequate data is a considerable challenge, with the ethical considerations of collecting such data that should require the consent of the patient, along with the fact that the data collected needs to be verified and labeled by an actual dermatologist or a professional in this regard. The study also finds the financial impact of making such systems a reality where the data is diverse and verified, and the study notes the limitations of computer vision that, at the current is not a replacement for the human but can assist an actual dermatologist in speeding up the process towards it.

This paper has laid the groundwork for the segmentation pipeline that works for most of types of skin tones. In the future, work can be done on optimizing the pipelines for more types of pigments. The model is currently limited to only the skin images, which can be further optimized to remove the background of the picture, which can lead to its commercial implementation. The accuracy of the model while improving is still a significant step before it can be accurate enough for an average dermatologist to drive its accuracy. The model can be scaled to inherit more diseases with ample resources. The timing for pre-processing and prediction can be reduced to improve the efficiency of the model. A single Deep Learning model can be trained to remove the segmentation pipeline but train it to learn segmenting that is generalized enough that it does not need a tremendous amount of preprocessing. Using this technique, a large number of data sets can be diversified enough to be used in the future. As the unsupervised algorithms continue to grow and become more generalized with the additional data. The pipeline can be updated and incorporated with bigger disease datasets, leading to a constantly evolving and improving system that includes the core features of diversity, ethical concerns, and integrity of the data used.

Author contributions

H. Imran, M. Shahid, and M. Aazam formulated the idea and designed the research; H. Imran and M. Shahid performed the simulations; M. Aazam, R. Sajid, M.A. Adnan, A. Ali, and K. Naeem analyzed the results; M. Aazam, R. Sajid, A. Ali, and K. Naeem supervised the work; H. Imran and M. Shahid wrote the original manuscript; M. Aazam, A. Ali, and K. Naeem revised the manuscript. All authors have read and agreed to this version of the manuscript.

Funding

Open Access funding is provided by Carnegie Mellon University Qatar.

Data availability

This research utilized two publicly available datasets. The Diverse Dermatology Images (DDI) dataset is publicly accessible via Stanford AIMI at https://ddi-dataset. github. io/, following registration according to the Stanford University Dataset Research Use Agreement. The Fitzpatrick17k dataset is publicly available through the following sources: - Annotation: https://skincon-dataset.github.io/files/annotations_fitzpatrick17k.csv. - Dataset: https://github.com/mattgroh/fitzpatrick17k/blob/main/fitzpatrick17k.csv. Additionally, direct access and explicit permission to use the Fitzpatrick17k dataset for research purposes were provided by Prof. Matt Groh via email communication, available upon request.

Declarations

Competing interests

The authors declare no competing interests.

Ethical approval

This study utilized publicly available datasets, namely the Fitzpatrick17k dataset and the SkinCon dataset. Both datasets were collected and published by their original authors with appropriate ethical approvals and participant consents. The Fitzpatrick17k dataset contains de-identified images intended for non-commercial research purposes. The SkinCon dataset is accessible to researchers who register and agree to abide by ethical usage terms. All images used in this study are anonymized to prevent the identification of individuals. Therefore, no additional institutional ethics approval was required for the current study.

Consent for publication

The datasets utilized in this research are publicly available, and consent for publication was obtained by the original creators at the time of data release. All data presented are anonymized, and no personally identifiable information is included in this publication.