AIDMAN: An AI-based object detection system for malaria diagnosis from smartphone thin-blood-smear images

Summary

Malaria is a significant public health concern, with ∼95% of cases occurring in Africa, but accurate and timely diagnosis is problematic in remote and low-income areas. Here, we developed an artificial intelligence-based object detection system for malaria diagnosis (AIDMAN). In this system, the YOLOv5 model is used to detect cells in a thin blood smear. An attentional aligner model (AAM) is then applied for cellular classification that consists of multi-scale features, a local context aligner, and multi-scale attention. Finally, a convolutional neural network classifier is applied for diagnosis using blood-smear images, reducing interference caused by false positive cells. The results demonstrate that AIDMAN handles interference well, with a diagnostic accuracy of 98.62% for cells and 97% for blood-smear images. The prospective clinical validation accuracy of 98.44% is comparable to that of microscopists. AIDMAN shows clinically acceptable detection of malaria parasites and could aid malaria diagnosis, especially in areas lacking experienced parasitologists and equipment.

Graphical abstract

Highlights

• •An AI-based object detection system has been developed to detect malaria parasites
• •The YOLOv5 and Transformer models were combined for detection and classification
• •A CNN classifier is used to reduce interference from false positive cells
• •The clinical validation accuracy is comparable to that of microscopists

The bigger picture

Malaria is among the most severe threats to human health worldwide. There were 228 million cases worldwide in 2021, of which ∼95% occurred in Africa. Early and precise diagnosis can reduce transmission and prevent deaths. Accordingly, accurate and affordable systems for rapid detection of malaria parasites are urgently required. Several artificial intelligence systems that employ deep learning models to detect malaria parasites have been reported, but clinical diagnosis remains challenging. Here, we developed an artificial intelligence-based object detection system for malaria diagnosis. This system employs a deep learning algorithm for detection of Plasmodia in thin-blood-smear images with a prospective clinical validation accuracy of 98.44%, comparable to that of microscopists. Our model shows clinically acceptable malaria parasite detection and could aid in malaria diagnosis in resource-limited regions, especially areas lacking experienced parasitologists and equipment.

Accurate and timely diagnosis of malaria is problematic in remote and low-income areas. Liu et al. develop an artificial intelligence-based object detection system for malaria diagnosis (AIDMAN). In this system, the YOLOv5 model and the Transformer model were combined to perform an entire process from image analysis to malaria diagnosis. A heatmap of the most characteristic cells is generated to reduce interference caused by false positive cells. AIDMAN shows clinically acceptable detection of malaria parasites and could aid in malaria diagnosis.

Introduction

Malaria is among the most severe threats to human health worldwide. Approximately 200 million new cases of malaria and 400,000 malaria-related deaths are reported globally each year.^1,2,3,4,5,6 For instance, there were 247 million cases of malaria and 619,000 deaths worldwide in 2021, of which ∼95% of cases occurred in Africa.^1,2 Early and precise diagnosis of malaria can reduce transmission and prevent deaths. Accordingly, accurate and affordable systems for rapid detection of malaria parasites in blood samples are urgently required.

Malaria is conventionally diagnosed by microscopy examination of blood smears, rapid diagnostic tests (RDTs), or polymerase chain reaction (PCR)-based methods, all of which are indispensable for current malaria control strategies.^{7,8,9,10,11,12,13,14,15} The cost and complexity of PCR-based methods have prevented their widespread use in areas where malaria is common.^16,17,18,19 Malaria RDTs are effective diagnostic tools as they do not require trained experts or complex PCR equipment, and they can diagnose within 15–30 min. However, according to the World Health Organization (WHO) and others, malaria RDTs suffer from several shortcomings including low specificity, inability to quantify parasite density, sensitivity to heat and humidity, and higher costs compared with microscopy-based methods.^15,20,21 In particular, histidine-rich protein 2 (HRP2) antigen-based RDTs are most effective for detection of Plasmodium falciparum, but the issue of false negative RDT results due to target gene deletions remains unresolved.^22,23

The use of microscopy images of blood smears to detect malaria parasites is inexpensive, rapid, and universal. Although it can be limited by poor image resolution and misidentification of impurities, traditional microscopy examination of blood smears is currently the gold standard for malaria diagnosis worldwide. Nevertheless, the requirements for experienced and skilled microscopists as well as effective quality control and quality assurance systems remain a crucial challenge for malaria diagnosis, especially in Africa.^24,25 Several artificial intelligence (AI) systems that employ deep learning models to detect malaria parasites have been reported recently.^{26,27,28,29,30,31,32,33,34,35,36,37} One group trained a model based on convolutional neural networks (CNNs) using 1,034 infected-cell images and 1,531 non-infected-cell images obtained from the University of Alabama.²⁶ This model achieved a diagnostic accuracy of 95%. Liang et al. employed 27,000 blood-cell images at three different magnifications to train a model that detected Plasmodia with an accuracy of 98.08%.²⁸ Khan et al. achieved 97.98% accuracy using a CNN that employs split-transform-merge and channel squeezing-boosting principles.³⁴ However, these methods typically suffer from two limitations. Firstly, the datasets used are not universal, and they contain an insufficient number of blood cells and few morphological types of malaria parasites.^26,29 The most commonly used dataset is the National Institute of Health (NIH) Malaria Dataset maintained by the National Library of Medicine. Unlike clinical images of blood cells typically collected in Africa, those in the NIH dataset have few overlapping cells and dye impurities. Secondly, some models focus on individual red blood cells (RBCs) for detection rather than complete thin-blood-smear images, which limits their clinical application.^29,30,31 There have also been attempts to detect malaria parasites using blood-smear images.^38,39 For example, Loh et al. used the Mask R-CNN deep learning model to segment, classify, and count Plasmodium falciparum-infected RBCs,³⁸ while Koirala et al. developed a custom deep learning architecture (YOLO-mp) to detect malaria parasites in thick-blood-smear microscopy images that achieved an average accuracy of 94.07%.³⁹

Herein, we developed an AI-based object detection system for malaria diagnosis (AIDMAN; Figure 1). This system employs a deep learning algorithm for detection of Plasmodia in thin-blood-smear images to facilitate malaria diagnosis. The YOLOv5 object detection model⁴⁰ and the Transformer model^41,42 were combined to perform an entire process from image analysis to malaria diagnosis. For each blood-smear image, a heatmap of the most characteristic cells is generated and used for diagnosis, reducing interference caused by false positive cells. Finally, the system was verified by application to clinically diagnosed patients.

Figure 1.

Schematic of AIDMAN developed in this study

(A–D) (A–D) consists of four modules: (A) data collection, (B) detection of image sections, (C) classification of malaria parasites in the patches, and (D) malaria diagnosis of blood-smear images.

(E) Network structure diagram of AAM for cellular classification.

(F) Module structure diagram for the local context aligner. P and N represent the positive and negative samples, respectively.

Results

Malaria detection of patches

We took 150 images from SmartMalariaNET and obtained 35,489 patches by performing data splitting at the image level. Table S1 shows distribution details for the datasets. Some samples of infected and uninfected images are shown in Figure S1. Compared with the counting results for manual detection by trained microscopists, YOLOv5 performed well for cell detection using the dataset, with an average precision value of 90.8% (Figure 2A illustrates the cell detection process by YOLOv5). The dataset for classification of cells contains 5,654 patches from 1,822 thin-blood-smear images. Three trained microscopists classified the patches. Patches with malaria parasites were defined as positive samples, while those without malaria parasites were defined as negative samples. The dataset includes images containing impurities and aperture effects. Table S2 summarizes the cell classification datasets. The datasets were randomly divided into a training set of 3,393 patches, a verification set of 1,131 patches, and a testing set of 1,130 patches.

Figure 2.

Performance of AIDMAN

(A) Detection of cells for patches in the image taken by a smartphone (top) and the image enlarged by a smartphone (bottom). The original, label, and result images predicted using YOLOv5 are shown.

(B) ROC used to evaluate AAM performance for classifying the presence or absence of malaria parasites in cells.

(C) Generation of datasets for the stratification of blood smears. After AAM, a score for each patch is obtained; the 25 patches with the highest scores are sorted according to score; a heatmap is obtained using the AAM and reassembled into a larger map.

(D) ROC used to evaluate CNN performance for classification of blood smears to diagnose malaria.

A local context aligner that uses the encoding process of Transformer was introduced into the attentional aligner model (AAM), and the multi-head attention mechanism allows the network to capture more features. Therefore, we optimized the model by changing the number of heads. With head number = 2, all indicators for the model were highest, as shown in Table 1. In the AAM feature extractor, features from different scales make different contributions to the identification of malaria. We explored the optimal feature extractor by varying the number of scales from 1 to 5. No more than five different scales were tested because the size of the input (64 × 64 pixels) restricts the features of the scale. When the feature extractor was operated at five different scales (head number = 2), interference was reduced, and more parasites were correctly identified, with all indicators for the model at their highest values, as shown in Table 1. The accuracy, precision, sensitivity, specificity, F1 score, and area under the receiver operating characteristic curve (AUC) values were 98.62%, 98.62%, 98.62%, 98.62%, 98.62%, and 99.92%, respectively. The receiver operating curve (ROC) for the AAM is shown in Figure 2B.

Table 1.

Quantitative classification results

Classification of cells
Heads	ACC	PRE	SENS	SPE	F1	AUC
2	98.62	98.62	98.62	98.62	98.62	99.92
4	93.16	93.22	93.16	93.18	93.16	98.72
8	95.55	95.60	95.55	95.56	95.55	99.43
Scales	ACC	PRE	SENS	SPE	F1	AUC
1	97.77	97.37	98.22	97.32	97.79	99.85
2	98.31	98.31	98.31	98.31	98.31	99.91
3	98.20	98.20	98.20	98.20	98.20	99.89
4	98.59	98.59	98.59	98.59	98.59	99.87
5	98.62	98.62	98.62	98.62	98.62	99.92
Malaria diagnosis of thin-blood-smear images
Patches	ACC	PRE	SENS	SPE	F1	AUC
1	89.00	89.38	89.00	88.633	89.10	94.70
4	92.00	91.96	92.00	88.75	92.94	97.64
9	94.00	94.00	94.00	92.63	94.00	97.28
16	94.00	93.99	94.00	91.20	93.95	98.31
25	97.00	97.13	97.00	94.18	96.96	98.84
36	91.00	91.34	91.00	91.09	91.09	97.77

Malaria diagnosis from thin-blood-smear images

Although the AAM performed well for malaria detection from single patches, it was still prone to false positives in overall diagnosis because there are far more negative cells than positive cells in one thin-blood-smear image, which leads to misdiagnosis. The accuracy of the AAM in malaria diagnosis of whole images from thin blood smears was 87.10%, with a false positive rate of 19.02%.

We then explored diagnosis of the overall thin-blood-smear image using a dichotomous CNN model based on the AAM. According to patch scores, a heatmap was generated from the top patches (1, 4, 9, 16, 25, 36), and these heatmaps were stitched into a new image, as shown in Figure 2C. Table S3 shows the distribution of the datasets used to classify the blood-smear images. The dataset for diagnosis of thin blood smears contains 496 images comprising a training set of 297 images, a verification set of 99 images, and a testing set of 100 images. As shown in Table 1, weightings from AAM were used to mosaic the heatmaps for the 25 patches with the highest scores to obtain a heatmap for each blood-smear image. For classification blood-smear images, the accuracy, precision, sensitivity, specificity, F1 score, and AUC values were 97%, 97.13%, 97%, 94.18%, 96.96%, and 98.84%, respectively. The ROC curve of the CNN is shown in Figure 2D.

In addition, we compared the diagnostic performance of images reconstructed from 25 patches in the original thin-blood-smear image or from the generated heatmaps. The indicator values of the reconstructed heatmap as input to the CNN model were higher than those of the original blood smear (accuracy, precision, sensitivity, specificity, F1 score, and AUC values were 93%, 93.02%, 93%, 89.26%, 92.92%, and 92.47%, respectively).

Prospective clinical validation of AIDMAN

Clinical validation of AIDMAN was based on 64 patients with thin blood smears at the Sierra Leone-China Friendship Hospital whose images were not previously used to develop the model. Images of thin blood smears were acquired using different microscopes at multiple magnifications. Microscopy diagnosis of thin blood smears was performed by three experts, each with more than 5 years of experience. Detection accuracies of microscopy examination, malaria RDTs, and AIDMAN were compared, and the results are shown in Table 2 and Table S4. Microscopists accurately identified 34 infected blood smears (positive) and 30 non-infected blood smears (negative). Malaria RDTs returned 32 positive and 2 negative results for the 34 positive samples (Figures 3A and S2) and 29 negative and 1 positive results for the 30 negative samples (Figures 3B and S3). AIDMAN yielded 33 positive and 1 negative results for the 34 positive samples (Figure 3A) and correct results for all 30 negative samples (Figure 3B). Thus, the detection accuracy of AIDMAN (98.44%) was comparable with that of microscopists.

Table 2.

Comparison of the accuracies of different malaria diagnosis methods

Detection method		No. patients		Total	Accuracy, %
		Positive	Negative
Microscopic examination	positive	34	0	34	100
	negative	0	30	30
RDTs	positive	32	1	33	95.31
	negative	2	29	31
AIDMAN	positive	33	0	33	98.44
	negative	1	30	31

Figure 3.

Three expert microscopists diagnosed images of infected and non-infected thin blood smears to validate the results of RDTs and AIDMAN

Discussion

In this study, we developed an AI-based object detection system capable of rapid and simple malaria diagnosis in remote and low-income areas. The framework was designed, from dataset selection to algorithm development, with clinical applicability in mind and to contribute to the field of AI-aided malaria diagnosis. Specifically, YOLOv5 was combined with Transformer to achieve an all-in-one operation from cell segmentation to malaria diagnosis. As a result, AIDMAN handles interference such as image background artifacts, cell overlap, dye impurities, and aperture effects extremely well, achieving accuracies of 98.62% for cells and 97% for blood-smear images. For malaria diagnosis using thin-blood-smear images, AIDMAN was evaluated in terms of accuracy, precision, sensitivity, specificity, F1 score, and AUC, achieving values of 97%, 97.13%, 97%, 94.18%, 96.96%, and 98.84%, respectively. Additionally, the performance of the AI algorithm was compared with that of malaria RDTs and microscopy examination in a clinical context. In prospective clinical validation, AIDMAN achieved a diagnostic accuracy of 98.44%, similar to that of microscopists. Thus, it may be extremely valuable for the diagnosis of malaria in Africa.

Previous research on the detection of malaria parasites by machine learning relied on expensive slide microscopes to scan blood smears and used images acquired from expensive digital cameras to train the deep learning model.^43,44,45,46 Clearly, this is not a practical approach for low-resource and/or remote settings, such as those encountered in West Africa. However, integration of smartphones with AI models for malaria diagnosis presents a potential solution.⁴⁵ AI-aided diagnosis solutions reduce reliance on microscopists and may thus be more suitable for successful deployment in the field. We are currently attempting to develop AIDMAN for smartphones. This involves training the model and predictions on mobile devices or offline training of the model and then importing it to mobile devices for prediction. Several AI systems for smartphone-based malaria parasite detection have been reported.^47,48,49 For instance, Yang et al. implemented a deep learning application for smartphones to detect malaria parasites in thick-smear images,⁴⁷ and Yu et al. developed an Android application for detecting malaria parasites in blood smears, providing greater flexibility than traditional parasite detection methods.⁴⁹ Automated malaria parasite detection on smartphones is a promising alternative to traditional malaria diagnosis, especially in resource-limited areas. Finally, malaria diagnosis might be just one element of a suite of diagnostic web tests that could be run on this type of system. Several other applications could be performed simultaneously using the same images, for instance, cell counting or detection of other hemoparasite-caused conditions such as trypanosomiasis, babesiosis, and microfilaria.^{50,51,52,53,54}

The present study has some limitations. Our datasets were established using data from a limited number of patients; hence, the generalizability of the algorithm remains to be determined. In addition, AIDMAN only detects malaria parasites in thin-blood-smear images in which malaria parasites maintain their original morphology, making it difficult to identify different species and life cycle stages. Thick blood smears involve a larger volume of blood and a larger number of malaria parasites per blood volume than thin blood smears; hence, malaria detection using thick blood smears must also be considered as a means for clinical diagnosis.^55,56 In future studies, we will endeavor to obtain more blood smears from different countries and clinical settings to introduce variability into our dataset and thereby test the generalizability of AIDMAN. Regarding best practice, we recommend additional tuning and validation prior to deployment in any new setting, followed by pilot field validation. In addition, the clinical workflow design needs to be improved through consultation with local health workers and subsequent pilot field validation. Furthermore, once deployed, the performance of the algorithm should be tracked and optimized by periodically testing it against results obtained by microscopists and then recalibrating the model.

Experimental procedures

Resource availability

Lead contact

Further information and resources should be directed to the lead contact, Yue Teng (yueteng@sklpb.org).

Materials availability

This study did not generate new unique reagents.

Method details

Our proposed AIDMAN system for automatic diagnosis of malaria is schematized in Figures 1A–1F. The process comprises modules for dataset collection, detection of image sections, classification of cells, and diagnosis of blood-smear images. For the first module, a smartphone camera was placed on the eyepiece of a microscope and used to obtain blood-smear images, as shown in Figure 1A. For the second module, shown in Figure 1B, the inputs were these smartphone images, which showed either infected or non-infected RBCs. After pre-processing, sections of images clearly showing individual or overlapping RBCs were identified and saved as “patches.” As shown in Figure 1C, these patches were inputs for the third module, which assigned each a score reflecting the probability of parasites in patches. Finally, in the fourth module, the input was a heatmap image constructed from the 25 patches with the highest score, and the output was a decision on whether the entire blood-smear image contained malaria parasites, as shown in Figure 1D.

AAM

The AAM was developed on the Keras platform based on Transformer and U-Net architectures.^41,57,58 It comprises a feature extractor, local context aligner, and multi-scale attention modules, as shown in Figure 1E. The custom AAM performed well at extracting information about malaria parasites from images and distinguishing parasites from dye impurity and aperture effects.

Feature extractors

Different feature layers handle different types of information; higher feature layers are more concerned with global information, while lower feature layers are more concerned with local features. The malaria parasite is identified by a purple ring inside the cell with one or two dark purple nuclei inside the ring. In our system, the feature extractor operates at different scales. For lower scales, fine nuclei can be extracted; for intermediate scales, purple rings can be extracted; and for higher scales, cell features can be extracted. We then utilized up-sampling in U-Net to make the images from the different feature layers the same size.⁵⁷

$$X=\left[X_1,X_2,X_3,X_4,X_5\right]$$ (Equation 1)

Local context aligner

Using the encoding process of Transformer (Figure 1F), we used high-scale features to aggregate low-scale features and thereby distinguish malaria parasites and impurity artifacts. Before multi-head attention, position coding is required. Here, we improved the position-coding method according to the characteristics of malaria parasites (i.e., their ring structure). As well as horizontal position coding, vertical encoding (grid position encoding) was also performed. For each position (i, j), we calculated position encoding using the following formulae:

$$P_{i,j}=\left[P_i^{\left(H\right)},P_j^{\left(V\right)}\right]$$ (Equation 2)

$$P_i^{\left(H\right)}=\{\begin{array}{c}\sin (0.5\ast i/\left({\mathrm{10,000}}^{\frac{i}{d}}\right),\mathrm{i}\mathrm{f}i\mathrm{i}\mathrm{s}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}\\ \cos (0.5\ast \left(i-1\right)/\left({\mathrm{10,000}}^{\frac{i-1}{d}}\right),\mathrm{i}\mathrm{f}i\mathrm{i}\mathrm{s}\mathrm{o}\mathrm{d}\mathrm{d}\end{array}$$ (Equation 3)

$$P_j^{\left(V\right)}=\{\begin{array}{c}\sin (0.5\ast j/\left({\mathrm{10,000}}^{\frac{j}{d}}\right),\mathrm{i}\mathrm{f}j\mathrm{i}\mathrm{s}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}\\ \cos (0.5\ast \left(j-1\right)/\left({\mathrm{10,000}}^{\frac{j-1}{d}}\right),\mathrm{i}\mathrm{f}j\mathrm{i}\mathrm{s}\mathrm{o}\mathrm{d}\mathrm{d}\end{array}$$ (Equation 4)

where $P_i^{\left(H\right)}$ and $P_j^{\left(V\right)}$ represent the horizontal and vertical position encodings, respectively; $d$ represents the dimension; and $i$ and $j$ denote the position in the image. A new variable is obtained by summing the features obtained from the feature extraction module and the grid position encoding as follows:

$$\overline{X}=X+P\text{.}$$ (Equation 5)

For the local context aligner, the input to the multi-head attention layer consisted of features at two different scales and included the grid position encoding. For multi-head attention, the input was divided into multiple heads to form multiple subspaces, which allowed the model to process information concerned with the differences between malaria parasites and impurities. The lower-dimensional receptive field feature was used to project keys (K) and values (V), and the higher-dimensional receptive field feature was used to project queries (Q). We used different linear maps for queries, keys, and values, which have dimensions of $d_q$, $d_k$, and $d_v$, respectively, h times. We then performed the self-attention process in parallel on each of the projected versions of queries, keys, and values, yielding $d_{model}$-dimensional output values. These were concatenated and once again projected, resulting in the final values. The multi-head attention mechanism allows the model to consider information from different representation subspaces in different positions.

$$MultiHead\left(Q,K,V\right)=Concat\left({head}_1,\dots ,{head}_h\right)W^o\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}{head}_i=\mathrm{s}\mathrm{e}\mathrm{l}\mathrm{f}-\mathrm{a}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left({QW}_i^Q,{KW}_i^K,{VW}_i^V\right)$$ (Equation 6)

$$K=V={\overline{X}}_i$$ (Equation 7)

$$Q=\{\begin{array}{c}{\overline{X}}_{i+1}\mathrm{i}\mathrm{f}i<5\\ {\overline{X}}_0\mathrm{i}\mathrm{f}i=5\end{array}$$ (Equation 8)

where $W_i^Q\in R^{d_{model}\times d_q}$, $W_i^K\in R^{d_{model}\times d_k}$, $W_i^V\in R^{d_{model}\times d_v}$, and $W^O\in R^{{hd}_v\times d_{model}}$.

In our model, $h$ = 2. We used two parallel attention heads. For each head, the formula $d_k=d_v=\frac{d_{model}}{h}=64$ was used. Because of the reduction in the size of each head, the total computational cost was similar to that for a single attention head for all dimensions. For multi-head attention, self-attention was used, and the input was composed of query Q, key K, and value V. The weight of values was obtained using the following formula^42,59,60,61:

$$\mathrm{s}\mathrm{e}\mathrm{l}\mathrm{f}-\mathrm{a}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left(Q,K,V\right)=softmax\left(\frac{{QK}^T}{\sqrt{d_k}}\right)V\text{.}$$ (Equation 9)

Multi-scale attention

Multi-scale features play different roles in malaria diagnosis. Hence, we developed a multi-scale attention module to strengthen the discriminating features and aggregate the extracted features, formulated as follows⁶¹:

$$Y=multi-head\left(Q,K,V\right)\text{,}$$ (Equation 10)

$$f\left(Y_i\right)=\mathit{tan}h\left(W·Y_i+B\right)·U,\mathrm{a}\mathrm{n}\mathrm{d}$$ (Equation 11)

$$Att\left(Y_i\right)=\frac{\mathit{exp}\left(f\left(Y_i\right)\right)}{{\sum }_{j=1}^L\mathit{exp}\left(f\left(Y_i\right)\right)}\text{,}$$ (Equation 12)

where Y denotes the output of the alignment by the local context aligner; W and U are trainable parameters; L is the number of features at different scales; and $Att\left(Y_i\right)$ serves as the feature map attention. Using the above formula, we calculated the multi-scale attention weight value. Since cells have a variety of shapes and sizes, malaria parasites can occupy many different places within a cell. Therefore, we assigned different attention weights to the characteristics of different malaria parasites and impurities. Thus, when the inputs were different patches, different features at the same location would have different attention weights according to Equation 8. Multi-scale attention mechanisms allow analysis to be performed from a global perspective alongside an in-depth and detailed analysis of certain smaller components. Thus, the final classification results are based on aggregated features from multiple scales.

Generation of patches

A patch is a sub-image of individual RBCs (or overlapping cells that cannot be distinguished into individual cells) detected in a thin-blood-smear image by a target detection model, and a thin-blood-smear image usually contains multiple patches. These patches may show malaria parasites or impurities. Herein, we use the YOLOv5 target detection model to generate patches,⁴⁰ the input for which is a blood-smear image taken using a smartphone, and the output is the patch showing a cell image. YOLOv5 can detect cells of different colors, shapes, and sizes. The images in Figure 2A are original views taken by a smartphone and enlarged views of cell regions, illustrating how YOLOv5 detects cell patches. The green circles in Figure 2A represent the detection of intact cells. Even for unlabeled parts of images during the annotation process, YOLOv5 can still detect cells for the prediction. The yellow circles in Figure 2A represent failure to detect incomplete cells.

Each blood smear can generate 250−350 patches showing cells, with fluctuations in the number of cells identified depending on how the blood smears were performed. Here, we segment patches from a blood smear to reduce the impact of differences relating to cells, image background, and dye impurities. From these sub-images, a smartphone-acquired patch dataset was created and manually annotated by a majority vote of three experts, each with >5 years of experience in microscopy diagnosis of malaria. Some negative samples did contain other objects, such as staining and impurity artifacts. Cells in a blood smear have different sizes, hence the pixel sizes of patches vary, and patches were subsequently resized to 64 × 64 pixels to meet the input requirements of the different classification algorithms employed.

Malaria diagnosis of thin-blood-smear images

Using the above steps, we could determine whether a malaria parasite was present in a particular patch. However, due to its sensitivity to the influence of impurities, such as dye and aperture effects, the AAM was prone to false positive diagnosis of thin-blood-smear images. Therefore, we added this step to reduce the false positive rate. A dataset was created for the classification of entire blood smears. Figure 2C shows part of the data build process for the stratification of blood smears using the 25 top patches as an example. First, to determine whether a blood smear contains malaria parasites, it was divided into multiple patches. Second, each patch was fed into the trained AAM and scored. Third, patches were sorted according to scores, a heatmap was generated from the highest-scoring patches, and these heatmaps were stitched into a 320 × 320 × 3 image as shown in Figure 2C. By repeating the above steps, multiple combined heatmaps were generated, and the categories to which these heatmaps belonged were identified. Finally, a dichotomous CNN model was trained with the data generated above, and the model was used to complete sample diagnosis.

Performance optimization and evaluation

Once the model is trained, new test samples can be processed quickly. The average processing time for each blood smear was 1 s using a workstation comprising an NVIDIA RTX 3090 GPU (24 GB), 64 GB RAM, and an Intel Core i9-10900K CPU (3.70 GHz). Six widely used metrics, accuracy (ACC), sensitivity (SENS), precision (PRE), specificity (SPE), F1 score (F1), and the AUC, were employed to evaluate the performance of the system. We utilized average precision (AP) to determine image section metrics as follows:

$$\mathrm{A}\mathrm{C}\mathrm{C}=\frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}$$ (Equation 13)

$$\mathrm{S}\mathrm{E}\mathrm{N}\mathrm{S}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}$$ (Equation 14)

$$\mathrm{P}\mathrm{R}\mathrm{E}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}$$ (Equation 15)

$$\mathrm{S}\mathrm{P}\mathrm{E}=\frac{TN}{\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{P}}$$ (Equation 16)

$$\mathrm{P}\mathrm{R}\mathrm{E}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}$$ (Equation 17)

$$\mathrm{F}1=2\times \frac{\mathrm{P}\mathrm{R}\mathrm{E}\times \mathrm{S}\mathrm{E}\mathrm{N}\mathrm{S}}{\mathrm{P}\mathrm{R}\mathrm{E}+\mathrm{S}\mathrm{E}\mathrm{N}\mathrm{S}}$$ (Equation 18)

$$\mathrm{A}\mathrm{U}\mathrm{C}=\frac{{\sum }_{i\in positiveclass}{rank}_i-\frac{M\left(1+M\right)}{2}}{M\times N}$$ (Equation 19)

where P and N denote the number of positive and negative samples, respectively; TP is the number of true positives; FP is the number of false positives; TN is the number of true negatives; FN is the number of false negatives; and M and N are the number of positive and negative samples, respectively.

Dataset used for performance evaluation

We used SmartMalariaNET (Figshare: https://doi.org/10.6084/m9.figshare.22679839) to evaluate the performance of AIDMAN.⁶² SmartMalariaNET contains Giemsa-stained thin-blood-smear slides taken by smartphones at the Sierra Leone-China Friendship Hospital and the Rokupa Government Hospital. The overall dataset contains three datasets that are used to train and test models for detection of patches showing cells, cell classification, and malaria diagnosis from thin-blood-smear images. In the present study, we used 1,822 thin-blood-smear images from 140 patients. Thin-blood-smear images were taken by smartphone cameras attached to the eyepiece of a microscope, which does not require directly obtaining relevant information from patients or subjects. Therefore, this work does not compromise personal privacy, and it satisfies the conditions for exemption from informed consent and ethical review.

Published: August 3, 2023

Data and code availability

• Datasets for AIDMAN steps 1–3 have been deposited to Figshare: [https://doi.org/10.6084/m9.figshare.22679839].⁶³

• The SmartMalariaNET dataset has been deposited to Figshare: [https://doi.org/10.6084/m9.figshare.22679860].⁶²

• The source code underlying this article has been deposited to Zenodo: [https://doi.org/10.5281/zenodo.7972493].⁶⁴