Robustly detecting mpox and non-mpox using a deep learning framework based on image inpainting

Abstract

Due to the lack of efficient mpox diagnostic technology, mpox cases continue to increase. Recently, the great potential of deep learning models in detecting mpox and non-mpox has been proven. However, existing methods are susceptible to interference from various noises in real-world settings, require diverse non-mpox images, and fail to detect abnormal input, which makes them unsuitable for practical deployment and application. To address these challenges, we proposed a novel strategy based on image inpainting called “Mask, Inpainting, and Measure” (MIM). In MIM’s pipeline, a generative adversarial network learns feature representations of mpox images by inpainting the masked mpox images. On this basis, MIM measure the similarity between the inpainted image and the original image to detect mpox and non-mpox. Compared with multi-class classification models, MIM can handle unknown categories and abnormal inputs more effectively. We used the recognized mpox dataset (MSLD) and a dataset containing 18 categories of non-mpox skin diseases to verify the effectiveness and robustness of MIM. Experimental results show that the average AUROC of MIM achieves 0.8237. In addition, external clinical testing further demonstrates the robustness of MIM. Importantly, we developed a free smartphone app to help the public and healthcare professionals detect mpox more conveniently.

Introduction

Mpox (Monkeypox), is a zoonotic viral disease caused by the mpox virus¹. This disease is transmitted through respiratory droplets, direct contact, or contaminated objects² and results in symptoms such as fever, rash, and swollen lymph nodes in patients³. Among various symptoms, the rash is a characteristic feature of mpox patients⁴ and visually resembles chickenpox and measles⁵. Statistics show that the fatality rate of mpox varies between 0% and 11% in the general population, with a greater rate among small children⁶. Initially, mpox was primarily endemic to remote regions of west and central African countries⁷. However, since May 2022, mpox has continuously swept through many countries. As of September 19, 2023, 115 countries have reported 90,439 cases of mpox⁸. On August 14, 2024, World Health Organization declares mpox outbreak a public health emergency of international concern⁹. These facts demonstrate the mpox outbreak is posing significant threats to global public health and highlight the urgency of addressing this evolving threat. Therefore, there is an urgent need for effective, easy-to-use, and cost-effective monkeypox detection methods.

Like other infectious diseases, timely and rapid detection of mpox patients is key to control outbreaks¹⁰. Especially in this fast-paced era with frequent interpersonal interactions, the spread of infectious diseases can quickly become global¹¹. Therefore, routine monitoring and management of the mpox outbreak are indispensable, further underscoring the real need for more efficient detection techniques¹². Utilizing Polymerase Chain Reaction (PCR) to detect the virus’s DNA from a patient’s rash is the preferred laboratory method for mpox diagnosis. However, the PCR requires advanced specialized equipment and experienced personnel and can be costly. Such costs are often prohibitive for resource-limited areas with higher mpox incidence rates¹³. Moreover, existing detection techniques are ill-suited for mass screening during outbreaks.

The rapidly developing deep learning technology has pointed out the direction for developing efficient and convenient mpox detection techniques. Many studies have successfully used advanced deep learning techniques for the detection and classification of skin disease images. Specifically, researchers have optimized the efficiency and quality of skin disease classification through deep learning, and helped dermatologists and general practitioners analyze data faster and improve the accuracy of diagnosis. For example, by utilizing additional residual block based fine-tuned deep models information fusion and best feature selection, MSRNet achieves 85.4% and 98.8% accuracy on ISIC2018 dataset and ISIC2019 dataset, respectively¹⁴. Besides, the proposed D2LFS2Net achieves an accuracy of 94.4% on multi-class skin lesion diagnosis using deep learning and variance-controlled marine predator optimization¹⁵. Since the outbreak of mpox, more and more researchers have begun to pay attention to this disease and have successfully employed various deep learning models and rash images to perform efficient mpox detection^16,17. However, our review suggests that these studies have yet to profoundly consider models’ detection capabilities in real-world settings. Specifically, the first step in the typical workflow of developing these models is to collect images of mpox. Researchers then aggregate various non-mpox images and train a binary-class classification model^18–29 to distinguish between mpox and non-mpox. Alternatively, researchers subdivide non-mpox data and train multi-class classification models to differentiate between mpox, chickenpox, measles, normal skin, and other skin diseases^30–35. While these models have shown preliminary efficacy, they have the following drawbacks. Firstly, during classification tasks, to quickly optimize parameters, models tend to find a “shortcut”³⁶, which might lead to inadequate learning of critical features in the training set and susceptibility to noise^37,38. Secondly, the detection performance of the classification model relies not only on numerous mpox images but also on a diverse set of non-mpox images. However, procuring comprehensive medical images remains a formidable challenge. Besides, estimating the number and categories that non-mpox images should contain is difficult. Lastly and most crucially, classification models do not refuse to detect when confronted with an input not belonging to known categories. In other words, classification models always expect that the input image belongs to the training set categories. However, in the real world, the number of patients with non-mpox rashes vastly outnumbers those with mpox, and their rashes often do not belong to the model’s training set category. Classification models cannot refuse to detect these patients and will unthinkingly judge the model input as one of the classes in the training set, which greatly reduces the safety and usefulness of these models.

Based on the above challenges, we believe that, in comparison with supervised learning, adversarial learning strategies can help models learn more robust and generalizable features with limited data. Therefore, this work aims to develop a robust monkeypox detection framework based on adversarial learning. Specifically, we utilized masks to destroy the original mpox images, forcing the generative adversarial network to learn the image’s feature representation through inpainting. However, several major challenges arise in implementing this strategy: (1) How to build a generative adversarial network suitable for the mpox dataset? (2) How to determine a suitable masking strategy? (3) How to determine a suitable inpainting strategy? (4) How to detect mpox and non-mpox through the inpainted images? Faced with these challenges, we conducted an in-depth analysis of mpox detection task and defined this detection task as a One-class Novelty Detection Task in computer vision. In this scenario, mpox data is treated as normal, and all non-mpox data from the real world is treated as abnormal. On this basis, we proposed a novel detection method called “Mask, Inpainting, and Measure” (MIM).

We summarized our contributions as follows: (1) We proposed a novel detection method called “Mask, Inpainting, and Measure” (MIM). The core of MIM is that a Generative Adversarial Network (GAN) models normal images via image inpainting and a masking strategy. Since the model is trained only on mpox images, it cannot effectively repair anomalous images during inference. Thus, new images can be classified as mpox or non-mpox based on the similarity between the restored and original images. This method cleverly detects mpox versus non-mpox without needing any non-mpox datasets. Importantly, MIM is not easily disturbed by various anomalous data in the real-world environment. (2) To our knowledge, this is the first time that generative models have been used for detecting mpox versus non-mpox. To enhance the detection performance of MIM, we compared various image similarity calculation methods. We found that using both the structural similarity index measure and the histogram Bhattacharyya distance resulted in the best detection performance of MIM. (3) By comparing MIM with existing classification models, we demonstrated the deep learning framework based generative adversarial network with image inpainting is more suitable for detecting mpox and non-pox. Besides, we also revealed the limitations of classification models. (4) We further validated MIM using non-mpox images collected in clinical settings and developed a smartphone application to provide a more convenient and efficient testing service for the public in areas affected by mpox and resource-limited settings.

Materials and methods

Novelty detection

Novelty detection (ND) is a classical and important research problem in machine learning with a wide range of applications, including medical diagnosis, fraud detection, and autonomous driving³⁹. It aims to detect any test samples that do not belong to any training category⁴⁰. In this task, a sample similar to the training data can be termed a “Normal sample” or “In-distribution sample (ID sample).” Otherwise, it is called an “Anomaly sample” or “Out-of-distribution sample (OOD sample)”^39,40. Based on the number of training classes, ND contains two different settings. The first is one-class novelty detection (one-class ND), which means only one class exists in the training set. The second is multi-class novelty detection (multi-class ND), which means multiple classes exist in the training set⁴¹.

Our research falls under one-class novelty detection. Traditionally, One-class ND is viewed as a representation-learning problem⁴². In this task, the model solely attempts to capture the distribution of normal samples, eventually detecting unknown anomalies or new concepts. Earlier, related algorithms largely depended on Support Vector Machines (SVM). With the advent of deep learning techniques in recent years, researchers have shifted focus toward learning a representative latent space of in-distribution data⁴³. As a potent unsupervised representation learning method, autoencoder has demonstrated immense potential in modeling images⁴⁴. Following this, autoencoder-based approaches have taken a dominant position in the field, such as Denoising Autoencoders (DAE)⁴⁵, Variational Autoencoders (VAE)⁴⁶, and Adversarial Autoencoders (AAE)⁴⁷. Specifically, to detect out-of-distribution images, an autoencoder is first trained on in-distribution samples. At test time, the reconstruction error of a sample serves as an anomaly score. Ultimately, the classification of an image is determined based on this anomaly score. This idea stems from an intuition: since the model only learns the distribution of normal samples, the reconstruction error for normal samples will be lower than that of anomalous samples.

Data source

Based on the requirements of the novelty detection task, we need to collect mpox images as in-distribution data for model training and gather various types of non-mpox images as anomaly data to verify the effectiveness of the proposed method. The data used in this work comes from four sources. The first part is MSLD^48,49. It is a recognized and expert-verified mpox dataset. It contains images of mpox, chickenpox, hand-foot-mouth disease (HFMD), measles, and normal skin. In practical settings, patients or doctors should take close-up front-facing photographs of skin lesions and submit them to the visual skin disease detection system. Therefore, we exclude samples that do not meet the actual shooting angle and distance, have complicated backgrounds, are repetitive, and are blurred. Figure 1a illustrates our requirements for image quality. Figure 1b reports the number of images in different categories.

Fig. 1.

Requirements for image quality and the distribution of data used in this work. (a) Images with green borders represent qualified images, while images with red borders represent unqualified images; (b) The number of images in each category included in the data from the first part to the third part; c: The number of images in each category included in the fourth part.

The second part is the ISIC 2018 dataset^50,51. This dataset contains seven types of skin diseases: Melanoma, Melanocytic nevus, Basal cell carcinoma, Actinic keratosis / Bowen’s disease (intraepithelial carcinoma), Benign keratosis (solar lentigo / seborrheic keratosis / lichen planus-like keratosis), Dermatofibroma, and Vascular lesion. The corresponding abbreviations for these seven diseases are MEL, NV, BCC, AKIEC, BKL, DF, and VASC. In this study, we randomly selected 100 images from each category. The third part is the Dermnet dataset⁵² from the Kaggle website, which includes 23 types of skin diseases. We selected eight skin diseases (Bacterial Infection, Bullous, Eczema, HPV, Urticaria, Vascular Tumor, Vasculitis and Viral Infection) from it that have features similar to mpox. We filtered out 100 images from each category based on the image quality. Figure 1b reports the number of images for each type of skin disease from the first to the third part of the data. Figure 2 displays samples of skin diseases from the first part to the third part.

Fig. 2.

Samples of skin diseases included in first part to third part data. a-b: Mpox; c-d: Chickenpox; e-f: Measles; g-h: Hand, foot and mouth disease; i: Bullous; j: Viral infection; k: Bacterial infection; l: Urticaria; m: HPV; n: Eczema; o: AKIEC; p: BKL; q: DF; r: MEL; s: Vasculitis; t: Vascular tumor; u: BCC; v: NV; w: VASC.

The fourth part is the clinical dataset. The Department of Dermatology at the Second Affiliated Hospital of Guangzhou Medical University provided us with skin lesion images of 146 patients. The clinical data used in this study has been approved by the ethics committee of the Second Affiliated Hospital of Guangzhou Medical University (reference number: 2024-YS-ks-13). The Helsinki Declaration’s tenets were scrupulously followed throughout the study to respect the rights, privacy, and anonymity of the subjects. Due to the retrospective nature of the study and the use of de-identified data, the need of informed consent was waived by the ethics committee of the Second Affiliated Hospital of Guangzhou Medical University.

Additionally, we collected 40 high-resolution images of normal skin from team members using smartphones based on a reasonable shooting angle. Figure 1c displays the number of images corresponding to each type of skin disease in the clinical data. Figure 3 shows some rash samples provided by the hospital and normal skin samples collected by ourselves.

Fig. 3.

Skin disease samples provide by the hospital and normal skin samples collected by ourselves. (a) Chickenpox; (b) Bullous; (c) Herpes zoster; (d) Urticaria; (e) Eczema; (f) Vasculitis; (g,h) Normal skin.

After collection and screening, this study obtained 220 publicly available mpox rash images, 1690 publicly available non-mpox rash images, and 100 privately collected non-mpox rash images.

Mask, inpainting and measure

This paper proposes “Mask, Inpainting, and Measure” (MIM) to detect mpox and non-mpox. The core idea of MIM is from the autoencoder-based reconstruction method. Specifically, MIM first uses a mask to destroy the image’s spatial structure and semantic information and then uses the Generative Adversarial Network to inpaint the image. Finally, MIM employs an image similarity metric to measure the similarity between the original and repaired images. This idea stems from the following intuition: Since the GAN only repairs masked mpox images during training, it only learns the contextual and semantic information of the mpox images. When a non-mpox image is input into the network, the network cannot repair it well, resulting in an image dissimilar to the original. In the end, the similarity between the original image and the repaired image is used as an anomaly score, and the category of the image is determined through a threshold. In the following sections, we will individually introduce the core details of MIM.

Mask

In computer vision, a “mask” typically refers to a binary image where certain regions are marked as 1 and others as 0. Recent literature has demonstrated that masked image modeling can significantly improve a model’s capability to learn data representations^53,54. The principle behind this strategy is to obscure a portion of the input image and have the deep network predict the masked signal based on visible cues. This idea originated from masked signal modeling, which can force the network to predict the obscured parts, thereby aiding the model in learning a more robust and richer data representation.

In this work, we utilize a rectangular white mask. Without considering the image channels, we define the following notation. I denotes an image of size H×W (H and W represent the image’s height and width, respectively). I(h, w) denotes the pixel value at each position of I. After normalizing I, I(h, w) lies in the range [0,1]. Next, we use M to denote a mask of size H×W and choose the coordinate range of the masked area, i.e., h₁ ~ h₂(0 ≤ h₁ ≤ h₂≤H) and w₁ ~ w₂(0 ≤ w₁ ≤ w₂≤W), and the pixel values of each position in M can be expressed using the following formula:

Last, we use I′ to denote the masked image and the following formula is established:

where ☉ denotes the Hadamard product. In terms of how masks are used, MIM uses a random mask strategy in the training pipeline (Fig. 4a), that is, the position of the mask is not fixed, which can improve the robustness of the generator. In the inference pipeline, MIM adopts a center masking strategy (Fig. 4b).

Fig. 4.

The workflow of MIM. Here, PW-Conv means Point-wise Convolution and CNN means a Convolutional Neural Network. (a) MIM’s training pipeline; (b) MIM’s inference pipeline.

Inpainting

In the MIM’s training pipeline (Fig. 4a), a generative model learns the key representations of training data (in-distribution data) by predicting the masked region of an input image. In this work, we adopted the generative adversarial network as the generative model of MIM⁵⁵. We utilized an autoencoder and a simple CNN as the generator and discriminator, respectively. The output of the discriminator will be a part of the overall loss of the generator, thereby effectively improving the generator’s robustness and generalization. Figure 4 illustrates the detailed process of the MIM. Specifically, the generator adopts a downsampling and upsampling structure based on multiple convolutional layers, which effectively captures the fine-grained features and global information of the mpox image by gradually extracting and restoring image features. In addition, Batch Normalization is used to stabilize the training process, accelerate model convergence, and improve the quality of generated images. The decoder uses deconvolution layers and ReLU activation functions to repair image details. On the other hand, the discriminator effectively extracts feature representations of the input image through multiple convolutional layers and Instance Normalization, improving its ability to distinguish between real and generated images. Numerous LeakyReLU activation functions are embedded into the encoder and discriminator to avoid “neuron death” and improve the effective transmission of gradient information. Overall, the simple architecture design reduces the risk of overfitting of MIM on small datasets and ensures efficient deployment. Tables 1 and 2 show the internal structure of the generator and discriminator in detail, respectively.

Table 1.

The internal structure of generator.

Block	Type	Input channels	Output channels	Kernel size	Stride	Padding	Normalization	Activation
1	Conv2d	3	64	4	2	1	None	LeakyReLU
2	Conv2d	64	64	4	2	1	BatchNorm2d	LeakyReLU
3	Conv2d	64	128	4	2	1	BatchNorm2d	LeakyReLU
4	Conv2d	128	256	4	2	1	BatchNorm2d	LeakyReLU
5	Conv2d	256	512	4	2	1	BatchNorm2d	LeakyReLU
6	Conv2d	512	4000	1	1	0	None	None
7	ConvTranspose2d	4000	512	4	2	1	BatchNorm2d	ReLU
8	ConvTranspose2d	512	256	4	2	1	BatchNorm2d	ReLU
9	ConvTranspose2d	256	128	4	2	1	BatchNorm2d	ReLU
10	ConvTranspose2d	128	64	4	2	1	BatchNorm2d	ReLU
11	Conv2d	64	3	3	1	1	None	None

Table 2.

The internal structure of discriminator.

Block	Type	Input channels	Output channels	Kernel size	Stride	Padding	Normalization	Activation
1	Conv2d	3	64	3	2	1	None	LeakyReLU
2	Conv2d	64	128	3	2	1	InstanceNorm2d	LeakyReLU
3	Conv2d	128	256	3	2	1	InstanceNorm2d	LeakyReLU
4	Conv2d	256	512	3	1	1	InstanceNorm2d	LeakyReLU
5	Conv2d	512	1	3	1	1	None	None

We used a weighted loss function to optimize the model parameters of the autoencoder. Specifically, the loss function of the generator comes from two parts: pixel-level loss and adversarial loss. We formalize the calculation process of loss function: and denotes generator and discriminator, respectively. Given a distribution of interest D_in, we use I ~ D_in to denote the MIM’s input. After using a mask, I_masked denotes the masked area in I and I′ denotes the masked image, respectively. Then, the output of the , the masked area that needs to be predicted, is denoted by . Last, the pixel-level loss of the generator can be defined as follows:

f_MAE in the above formula represents the mean absolute value error between real masked area and fake masked area, and its formula is as follows:

where x₁ and x₂ respectively denotes images with the same shape. In addition, C, H, and W represent the number of channels, height, and width of the image, respectively. (c, h, w) represents the position of each pixel in the image.

The formula of the adversarial loss of generator is as follows:

Here represents a matrix whose elements are all 0, and its shape and size are consistent with the output of the discriminator. f_MSE represents mean square error, and its formula is as follows:

Finally, the total loss of the generator is calculated using a weighted sum:

where α_G and β_G represent weight factors respectively. We set them to 0.01 and 0.99 respectively.

For the discriminator, we also used a weighted loss function. Specifically, the loss of the discriminator must consider the loss values ​​from both real masked area and fake masked area predicted by the generator. The loss function of fake masked area is as follows:

For real masked area, the loss function is:

Here represents a matrix whose elements are all 1, and its shape and size are consistent with the output of the discriminator. Based on the above, the total loss of the discriminator is:

where α_D and β_D represent weight factors respectively. We set them to 0.5 and 0.5 respectively.

Measure

According to the novelty class detection process, we use the similarity between the original and inpainted images as an anomaly score. The similarity value determines whether the image belongs to mpox or non-mpox. In this work, we chose five metrics to calculate similarity.

Mean Absolute Error (MAE). Mean Absolute Error is a metric to measure the average of absolute errors between two sets of values. When comparing two images, it measures the average pixel absolute difference between the two images. The smaller the mean absolute error, the higher the similarity. Based on the formula for calculating the mean absolute error, the similarity can be calculated using the following formula:

This formula shows that the smaller the mean absolute error, the higher the similarity between images.

Mean Squared Error (MSE). The Mean Squared Error is a metric that calculates the average of the squared differences between the corresponding pixels of two images. Like the Mean Absolute Error (MAE), it measures the difference between two sets of data, but by squaring the differences, it gives more weight to more significant errors, making it more sensitive to outliers. Based on the formula for calculating the mean squared error, the similarity can be calculated using the following formula:

This formula shows that the smaller the mean square error, the higher the similarity between images. Structural Similarity Index Measure (SSIM)⁵⁶. Structural Similarity Index Measure is a metric used to measure the similarity between two images. It provides a more comprehensive and accurate assessment of perceived quality differences between images compared to traditional metrics such as Mean Squared Error (MSE) or Mean Absolute Error (MAE). The main idea behind SSIM is to consider changes in structural information, luminance, and contrast, which are believed to be highly important in the human perception of image quality. When SSIM returns a value of 1, it indicates that the two images being compared are identical in terms of structural, luminance, and contrast characteristics. A value of 0 or near 0 indicates that the images have significant differences in these characteristics. The calculation formula of SSIM is as follows:

In this formula, µ_x1 and µ_x2 represent the mean of images x₁ and x₂, respectively. σ_x1 and σ_x2 represent the variance of images x₁ and x₂, respectively. σ_x1 × 2 represents the covariance of x₁ and x₂. c₁ and c₂ are two constants to avoid division by zero. c₁ and c₂ are defined as follows: , where K₁ is 0.01, K₂ is 0.03 and L is the dynamic range of pixel values.

Histogram Bhattacharyya Distance (HBD). A histogram is a tool used to represent the distribution of pixel intensities in an image. Through the histogram, we can understand attributes such as an image’s brightness and color distribution. The Bhattacharyya distance is used to gauge the similarity between two probability distributions. When we apply the Bhattacharyya distance to the histograms of two images, it can measure the similarity between these two images in terms of color and brightness distribution, especially in terms of overall color distribution. Based on the principle of Bhattacharyya distance, we define the similarity as follows:

where x₁ and x₂ respectively represent the histograms of the two images and ∑ is over all bins of the histograms.

In addition to the above four calculation methods, we intuitively propose a joint metric that simultaneously considers the HBD and SSIM between images. The formula of joint metric is as follows:

In the formula, λ₁ and λ₂ represents weights, and we set them to 0.5 and 0.5 respectively.

To thoroughly and objectively evaluate the performance and potential of MIM, we followed the rules of previous work^39,40,42 and used the area under the receiver operating characteristic curve (AUROC) as a threshold-free evaluation metric.

Results and discussion

Implementation details

We conducted the entire experiment on a computer with an Nvidia GeForce RTX 4090 graphics card and Ubuntu 20.04 system. We used the Adam optimizer to optimize the parameters of each model in the MIM’s training pipeline. In the Adam optimizer, β₁ and β₂ were set to 0.9 and 0.999, respectively. We set the initial learning rate to 0.001, the batch size to 128, and the number of epochs to 300. Besides, the weight decay of Adam was set to 0.0001. MIM’s input size and mask size were set to 224 × 224 × 3 and 112 × 112 × 3, respectively. Notably, we employed five-fold cross-validation to thoroughly evaluate the potential of MIM. Specifically, there are 176 mpox images and 44 mpox images in each dataset. One thing to note is that we did not use any data augmentation strategies on the training set. However, when calculating the AUROC of MIM, we generated 220 new images by augmenting (horizontal flipping + vertical flipping + saturation change) the mpox images in each test set. The purpose of this is two-fold. First, mpox images in practical settings are relatively rare, so we tried to explore the representation-learning ability of MIM in a small-scale dataset. Second, there is a sizeable quantitative gap between the mpox images in the test set and the non-mpox images from out-of-distribution, which may cause AUROC not fully to reflect MIM’s performance. Finally, we utilized 220 normal mpox images and 1690 abnormal non-mpox images to calculate AUROC. The average AUROC was used to evaluate MIM’s final performance. Table 3 lists the detailed data usage.

Table 3.

Distribution of data used to train and test MIM. ID and OOD mean In-distribution and out-of-distribution, respectively.

	Train (ID)	Test (ID)	Real test (ID)	Real test (OOD)
Fold1	176 mpox images	44 mpox images	(44) → (220) mpox Images	1690 non-mpox images
Fold2	176 mpox images	44 mpox images	(44) → (220) mpox Images
Fold3	176 mpox images	44 mpox images	(44) → (220) mpox Images
Fold4	176 mpox images	44 mpox images	(44) → (220) mpox Images
Fold5	176 mpox images	44 mpox images	(44) → (220) mpox Images

The best similarity metric

We first explored the impact of various similarity metrics for MIM. Table 4 reports the AUROC calculated from different metrics. Among all the metrics we investigated, the metric we proposed exhibited the best performance, with an average AUROC of 0.8237(± 0.0101) and up to 0.8401 in the best case. Additionally, the values for SSIM and HD were 0.7582(± 0.0455) and 0.7979(± 0.0243) respectively. The performance of pixel-difference-based metrics, MAE and MSE, was relatively poor, with values of 0.7214(± 0.0312) and 0.6984(± 0.0286), respectively.

Table 4.

Differences in effects of different image similarity calculation methods.

	SSIM + HBD	SSIM	HBD	MAE	MSE
Fold1	0.8123	0.6757	0.8395	0.6832	0.6685
Fold2	0.8401	0.7660	0.8082	0.6935	0.6646
Fold3	0.8167	0.7500	0.7876	0.7285	0.6994
Fold4	0.8190	0.7989	0.7685	0.7302	0.7230
Fold5	0.8303	0.8004	0.7855	0.7715	0.7364
Mean (± Std)	0.8237 (± 0.0101)	0.7582 (± 0.0455)	0.7979 (± 0.0243)	0.7214 (± 0.0312)	0.6984 (± 0.0286)

We offer the following explanations regarding the differences in these results: When analyzing human rash images involving structure, texture, and color distribution characteristics, it is often not enough to measure the similarity between images only by the difference in pixel values. During the evaluation process, SSIM and HBD consider more of the images’ visual and structural properties, making them more sensitive to the rich texture and color information in rash images. Therefore, compared to pixel-difference-based metrics like MAE and MSE, SSIM and HBD demonstrated superior results. Additionally, since our metric evaluates image similarity from multiple dimensions, this means that even if a metric performs poorly or is affected by noise under certain conditions, the other metric might still provide meaningful feedback for the evaluation. Therefore, our metric can help MIM achieve optimal performance.

In order to visually feel the effectiveness of the MIM method, we show some inpainted samples in Fig. 5. In Fig. 5, we can intuitively see that MIM can better repair the masked mpox image (in-distribution data). Whether in terms of texture, color, or image content, the repair effect of the in-distribution image looks more natural and closer to the original image. On the other hand, for non-mpox images (out-of-distribution data), the repair effect produced by MIM looks very jarring. For example, there is an obvious dissimilar color or a mismatch in image content between the repaired and original images. The visual effect display further proves the rationality of the MIM method.

Fig. 5.

Mpox image inpainted by MIM and non-mpox image inpainted by MIM. The first, second and third columns in the figure represent the original image, the masked image and the repaired image respectively.

MIM is better than classification models

One of the motivations for proposing MIM was our belief that the existing classification models were insufficient to detect mpox and non-mpox in real-world settings. Therefore, it is essential to compare MIM with classification models. We selected three representative architectures used for image classification: ResNet (ResNet50)⁵⁷, MobileNetV3 (MobileNetV3 Large)⁵⁸, and Swin Transformer (Swin Transformer Base)⁵⁹.

Following the experimental setup from related works, we first combined measles, chickenpox, and hand-foot-mouth disease data to create the in-distribution non-mpox data (190 images). Then, we took the remaining fifteen types of skin diseases as the out-of-distribution non-mpox data (1,500 images). Just like with MIM, we used a five-fold cross-validation to train the three classification models above. At this time, the model’s in-distribution data will contain four categories. We equally used the average AUROC to assess the potential of classification models in real-world settings. When testing the classification models, in addition to using in-distribution mpox data and non-mpox data, we also tested the models with 1,500 out-of-distribution non-mpox images (anomalous data). Table 5 displays the detailed data usage of classification models. Furthermore, the training parameters for the classification models, such as the number of epochs, optimizer choice, and learning rate, were consistent with what was used for MIM.

Table 5.

Distribution of data used to train and evaluate classification models.

	Train(ID)	Test(ID)	Real Test
Fold1	176 mpox images 152 non-mpox images	44 mpox images 38 non-mpox images	(44)→(220) mpox Images (ID) 38 non-mpox images (ID) + 1500 non-mpox Images(OOD)
Fold2	176 mpox images 152 non-mpox images	44 mpox images 38 non-mpox images	(44)→(220) mpox Images (ID) 38 non-mpox images (ID) + 1500 non-mpox Images(OOD)
Fold3	176 mpox images 152 non-mpox images	44 mpox images 38 non-mpox images	(44)→(220) mpox Images (ID) 38 non-mpox images (ID) + 1500 non-mpox Images(OOD)
Fold4	176 mpox images 152 non-mpox images	44 mpox images 38 non-mpox images	(44)→(220) mpox Images (ID) 38 non-mpox images (ID) + 1500 non-mpox Images(OOD)
Fold5	176 mpox images 152 non-mpox images	44 mpox images 38 non-mpox images	(44)→(220) mpox Images (ID) 38 non-mpox images (ID) + 1500 non-mpox Images(OOD)

Table 6 reports the average AUROC for the three classification models. The Swin Transformer achieved the best results among these three models, with an AUROC of 0.6306. ResNet50 and MobileNetV3 reached 0.6221 and 0.5755, respectively. Since the worst AUROC for the three models is only slightly above 0.5, it can be inferred that, when faced with a large amount of out-of-distribution non-mpox images and augmented mpox images, the classification models are essentially making random predictions. The detection results also showed that these classification models are particularly easy to recognize non-mpox images as mpox. While thoroughly detecting mpox patients can help control the spread of the virus, the model should also recognize its detection performance on non-mpox patients. In real life, the number of non-mpox patients vastly outnumbers mpox patients and a detection method with a high false-positive rate is unacceptable in clinical settings.

Table 6.

AUROC of various classification models.

Model name	Fold1	Fold2	Fold3	Fold4	Fold5	Mean (± Std)
MobileNetV3 large	0.5745	0.5309	0.4955	0.6645	0.6123	0.5755 (± 0.0594)
ResNet50	0.6342	0.6113	0.5135	0.6239	0.7278	0.6221 (± 0.0682)
Swin transformer base	0.7073	0.6036	0.5422	0.5821	0.7178	0.6306 (± 0.0698)

Besides, although binary-class classification models do judge some out-of-distribution non-mpox images as non-mpox, this result is difficult for users to understand and convince, as the input to a classification model should belong to in-distribution categories. Compared to classification models, our proposed MIM models the images at the pixel-level. MIM only uses mpox data to train the model, but its average AUROC outperformed Swin Transformer, ResNet, and MobileNetV3 by 0.1931, 0.2016, and 0.2482, respectively, which means MIM is more robust and efficient. Additionally, as MIM only models the mpox images, as MIM only models the mpox images, it can adequately detect non-mpox patients without being influenced by out-of-distribution data. To visually grasp how MIM is more effective than classification models, we plotted the best and worst AUROC of various models in Fig. 6.

Fig. 6.

The best AUROC and the worst AUROC of MIM and classification models.

Ablation experiment

To further explore the impact of each module in the MIM framework on detection performance, we conducted rigorous ablation experiments under consistent experimental settings. Specifically, we analyze the impact of three main factors: the discriminator, mask position, and mask size. (1) We compared the performance of MIM with and without the discriminator; (2) We compared the performance of MIM using masks of different sizes; (3) We compared the impact of mask position (random mask vs. fixed mask) on MIM performance. Table 7 reports the detailed performance changes of MIM under different settings. The experimental results show that when the discriminator is removed from MIM, the average AUROC is significantly reduced, demonstrating that the adversarial learning strategy is crucial for the restoration effect. In contrast, when a fixed mask is used, the AUROC of the upper-right mask achieves SOTA, while the AUROC of the upper-left mask is the worst. We believe this phenomenon may be due to the relative position of the lesion in the rash image. When considering only the mask size, the AUROC of MIM increases roughly with the increase in mask size. In general, the adversarial learning strategy, random masks, and a larger mask size contribute to building the best MIM framework.

Table 7.

The performance comparison of MIM under different settings.

	Discriminator	Mask position	Mask size	AUROC
MIM-1	×	Random	112 × 112	0.6452 (± 0.0207)
MIM-2	√	Center	112 × 112	0.7328 (± 0.0114)
MIM-3	√	Upper left	112 × 112	0.6699 (± 0.0226)
MIM-4	√	Upper right	112 × 112	0.7597 (± 0.0265)
MIM-5	√	Lower left	112 × 112	0.7194 (± 0.0177)
MIM-6	√	Lower right	112 × 112	0.6777 (± 0.0252)
MIM-7	√	Random	14 × 14	0.7732 (± 0.0113)
MIM-8	√	Random	28 × 28	0.7630 (± 0.0239)
MIM-9	√	Random	56 × 56	0.7839 (± 0.0117)
MIM (ours)	√	Random	112 × 112	0.8237 (± 0.0101)

Clinical validation and application development

Validating MIM through clinical images provides stronger evidence. We used Gaussian noise to augment the original mpox images in the test set and then obtained 200 new images. This was done to explore the anti-interference ability of MIM and keep the number of mpox images consistent with clinical images. Subsequently, MIM sequentially loaded model weights from fold1 to fold5 and predicted the 200 mpox images and 186 non-mpox images. The results showed that MIM’s average AUROC reached 0.871.

Furthermore, these images were also used to test the Swin Transformer, but its average AUROC was only 0.583. We took a closer look at the classification model’s prediction results. Consistent with the findings in Sect. 3.3, the Swin Transformer is prone to generating incorrect detection results. This result also confirms that the classification model cannot fully learn the data representation and is, therefore, susceptible to interferences from noise and out-of-distribution data.

The clinical validation results preliminarily validated that the MIM method can effectively identify mpox and non-mpox data in an open-world setting. Simultaneously, it further revealed the shortcomings of classification models. Lastly, based on our previous work, we updated our application (Fig. 7) to provide a more convenient and efficient testing service for people in areas affected by mpox and resource-limited settings^16,17. Based on clinical validation results, we set the anomaly score threshold to 0.8526 in our application. If MIM calculates a score greater than 0.852 for an image, the app will output “Mpox”; otherwise, it will output “Non-Mpox”.

Fig. 7.

The application page of Mpox-AISM V3.

Comparison with related work

Table 8 reports the limitations of previous work and the progress of this work. Since the experimental settings and dataset usage strategies of these works are different, here we focus on discussing the application capabilities of the models in real-world settings. In previous work, most chose a binary-class (BC) classification model to detect mpox and non-mpox. The remaining work uses multi-class (MC) classification models to detect mpox, chickenpox, measles, and normal skin. Although these models have achieved good indicator results, all these classification models require large amounts of non-mpox data. Even more fatal is that these classification models always expect the model input to belong to one of the categories in the training set. However, in actual settings, the model more often faces data from out-of-distribution and cannot refuse to detect these data. It is important to reiterate that classification models are easily affected by noise and cannot adequately learn image representations. We have already demonstrated the shortcomings of classification models in the previous section.

Table 8.

Limitations of related work and strengths of the proposed approach. “BC” and “MC” represents binary-class classification model and multi-class classification model.

Paper	Algorithm name	Model type	The use of non-Mpox data	Refuse to detect anomaly data	Robustness in the real-world settings	Clinical Validation
18	PoxNet22	BC	✓	×	Low	×
19	VGG19 MobileNetV2	BC BC	✓ ✓	× ×	Low Low	× ×
20	DenseNet201	BC	✓	×	Low	×
21	MobileNetV3-s	BC	✓	×	Low	×
22	Monkey-CAD	BC	✓	×	Low	×
23	MobileNetV2	BC	✓	×	Low	×
24	ResNet18	BC	✓	×	Low	×
25	Customed CNN	BC	✓	×	Low	×
26	Vision Transformer	BC	✓	×	Low	×
27	MobileNet Inception V3	BC BC	✓ ✓	× ×	Low Low	× ×
28	ResNet34	BC	✓	×	Low	✓
29	Xception ResNet101	BC MC	✓ ✓	× ×	Low Low	× ×
30	Xception DenseNet169	MC MC	✓ ✓	× ×	Low Low	× ×
31	MonkeyNet	MC	✓	×	Low	×
32	CNN + LSTM	MC	✓	×	Low	×
33	ResNet18	MC	✓	×	Low	×
34	InceptionV3	MC	✓	×	Low	×
35	VGG16 AlexNet	MC MC	✓ ✓	× ×	Low Low	× ×
Ours	MIM	Generative	×	✓	High	✓

A model that can be applied in the real world should be able to detect mpox and all non-mpox cases. Our proposed MIM precisely fits this property and is more efficient and robust than traditional classification models. From the model’s functionality perspective, MIM is similar to a binary-class classification model. Although it cannot further classify non-mpox like a multi-class classification model, it can detect all non-mpox images in the real environment, regardless of the category. The core concept of MIM is more in line with daily life because, during outbreaks, the public or doctors hope to quickly determine whether the rash belongs to mpox through artificial intelligence-based mpox detection tools without caring about the specific categories of non- mpox. From an algorithmic perspective, MIM only treats mpox as normal data and uses generative models such as generative adversarial networks and mask repair tasks to model normal data. MIM cleverly detects mpox and non-mpox without using various non-mpox data. Particularly, MIM also avoids the interference of various abnormal data, greatly improving the security and robustness of the model. In order to verify MIM more scientifically, we also performed clinical validation, while most previous work did not perform this validation step. In conclusion, compared with related work, our work has made more rigorous progress through generative models and image inpainting for the first time.

Conclusion

In response to the public health challenges posed by the mpox outbreak, this work proposes a deep learning-based framework for mpox detection, named ‘Mask, Inpainting, and Measure’ (MIM). While the traditional PCR test is accurate, its high cost and reliance on specialized equipment and personnel make it difficult to be widely used in resource-limited areas. Although existing deep learning classification models have shown preliminary results in mpox detection, they perform poorly in handling real-world noise interference and cannot process abnormal inputs and unknown categories, limiting their safety and effectiveness in practical applications. In comparison with existing methods, the proposed MIM is simple yet effective. MIM treats mpox as normal data and all non-mpox data as anomalies, effectively detecting mpox and non-mpox cases. Moreover, MIM eliminates the need for non-mpox data and is not affected by abnormal inputs. MIM may provide new solutions for other binary classification tasks in medical imaging. For example, MIM may prove useful in detecting lesions in endoscopic or ultrasound images. Extensive experiments show that MIM can achieve efficient and robust mpox detection with only a small number of mpox images. Validation on external clinical images further demonstrates the generalizability of MIM. Furthermore, the developed smartphone application, based on the MIM method, provides a convenient and efficient mpox detection tool for the public and medical professionals, greatly improving the accessibility of mpox detection services.

To gain a deep understanding of the contribution of each component in the MIM framework to detection performance, we conducted a detailed ablation study and found that the mask position and size can significantly affect overall detection performance, playing a crucial role in the MIM framework. By comparing various similarity measurement methods, we also found that combining the structural similarity index measure (SSIM) and histogram Bhattacharyya distance can significantly improve the accuracy and robustness of MIM. Our experimental results demonstrate that, compared to traditional multi-class classification models, the MIM framework based on generative adversarial networks exhibits higher security and reliability when dealing with unknown categories and abnormal inputs. These findings not only verify the effectiveness and robustness of the MIM framework but also highlight its advantages in handling various complex data in real-world settings.

We acknowledge that this work has some limitations. For example, we have yet to explore the impact of different types of generative models on the detection performance of MIM, model complexity, and model response speed. Moreover, the validation and optimization of our model necessitate a more expansive dataset comprising both mpox and non-mpox images. With the recent increase in mpox patients, we are strategizing collaborations with specialized medical institutions to procure a more diverse image dataset. We also plan to conduct human-machine comparative studies to benchmark our model’s performance against expert evaluations, thereby elevating its reliability and adaptability. Future research focuses on developing lightweight generative detection models. Given its lean architecture, such a model could be seamlessly integrated offline in densely populated areas, such as airports and subways, becoming indispensable tools for large-scale surveillance during outbreaks.

As deep learning technology continues to develop, diffusion models and large vision-language models (such as CLIP) may offer new solutions for mpox detection technology research. Diffusion models generate high-quality and detailed images by gradually adding and removing noise, which could further enhance the performance of the MIM framework in image inpainting and feature extraction. In contrast, large vision-language models like CLIP can provide richer semantic information for cross-modal tasks by combining visual and textual data. In the future, it may be integrated with the MIM framework to achieve more accurate and robust mpox detection. Recently, Vision Mamba has shown great potential in medical image analysis, as it can effectively model long-distance dependencies while maintaining linear complexity. Therefore, it is also worth considering using Vision Mamba blocks to build generators and discriminators, thereby enhancing the MIM framework’s ability to extract global and local features from mpox images. Additionally, the integration of these cutting-edge technologies is expected to further improve the model’s generalization ability and detection accuracy, facilitating the practical deployment and application of the MIM framework in real-world settings.

Author contributions

Y.B.Y., Y.J.C., Z.Z.L., and X.M.M. designed this research. Y.B.Y., Y.J.C., Z.Z.L., X.M.M, D.L., R.K.N. and H.H.L. participated in various experimental parts of the project implementation, analyzed the experimental results, and wrote the manuscript. Y.B.Y. and Y.J.C. revised the manuscript. Y.B.Y. and Y.J.C. contributed equally to this work. All authors reviewed and approved the final manuscript.

Data availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

Declarations

Competing interests

The authors declare no competing interests.