Artificial intelligence and its applications in digital hematopathology

Abstract

The advent of whole-slide imaging, faster image data generation, and cheaper forms of data storage have made it easier for pathologists to manipulate digital slide images and interpret more detailed biological processes in conjunction with clinical samples. In parallel, with continuous breakthroughs in object detection, image feature extraction, image classification and image segmentation, artificial intelligence (AI) is becoming the most beneficial technology for high-throughput analysis of image data in various biomedical imaging disciplines. Integrating digital images into biological workflows, advanced algorithms, and computer vision techniques expands the biologist’s horizons beyond the microscope slide. Here, we introduce recent developments in AI applied to microscopy in hematopathology. We give an overview of its concepts and present its applications in normal or abnormal hematopoietic cells identification. We discuss how AI shows great potential to push the limits of microscopy and enhance the resolution, signal and information content of acquired data. Its shortcomings are discussed, as well as future directions for the field.

1. INTRODUCTION

Digital pathology (DP), which has emerged in recent years, aims to digitize pathological slides with a full-slide scanner, and finally to the analysis of these digitized whole-section images. DP based on whole-slide imaging (WSI) is able to obtain a large number of pathological images with histological features at high resolution.¹ Typically, these scanners can produce 20× to 40× digital slices for tissue diagnosis at extremely fast scan rates.² However, compared with other solid tissues, blood pathological smears require higher magnification scans in order to obtain more accurate information on pathological features. In practice, scanning at 60× or 100× magnifications is definitely required for peripheral blood or bone marrow (BM) smears. To effectively obtain useful information in digitized images, histopathological image analysis algorithms emerge as the times require, most of which are based on artificial intelligence (AI). These computer-aided diagnosis methods provide pathologists with a powerful tool that is likely to outperform traditional microscopy in hematopathology.³

AI is a branch of computer science that attempts to understand the nature of intelligence and produce a new type of intelligent machine that can respond in a manner similar to human intelligence.⁴ The lack of sizable datasets and insufficient computing power are traditionally considered 2 major factors limiting the development of AI. However, with the advances of image technology and other technologies, massive amounts of image data continue to accumulate. And with the continuous improvement of graphical processing units (GPU) computing power, the capabilities of AI continue to become prominent in many areas, such as image processing, language recognition, and natural language processing.

AI tools developed by commercial companies were greatly increased, lots of them were used in solid tumor anatomical pathology include Lunit, Ibex, aetherAI, DeepBio, PathAI, Paige.AI. Some were used in hematological diseases. For example, DeepFlow is the world’s first flow cytometry AI cloud diagnosis system developed by DeepAnalysis Intelligence. Its accuracy rate of diagnosis of acute leukemia is as high as 95%, which is about 100 times faster than that of doctors.

Although AI is slated to benefit many areas of clinical health sciences, our review is focus on the use in digital hematopathology, including the identification of normal hematopoietic cells, the classification of heterogeneous blood cells, the morphological identification of acute leukemia, and the diagnosis of clonal hematopoietic stem cell disorders. Our review aims to provide clinicians and researchers with tools to understand AI and its applications in field of hematology. Besides, it also provides guidance for the design and evaluation of machine learning studies.

2. CONCEPTION OF AI

Machine learning is a subfield of AI that is broadly defined as the ability of machines to mimic human intelligent behavior. Machine learning systems are used to perform complex tasks in a manner similar to how humans solve problems, it attempts to extract meaningful results from complex data structures through computation. Supervised, unsupervised, and reinforcement are 3 subcategories of machine learning model. These models contain a variety of methods, such as regression, support vector machines (SVM), decision trees, and neural networks.^5,6 There are 3 main types of machine-learning application in medicine: intelligent diagnosis and treatment, medical image intelligent recognition, medical robots. Intelligent diagnosis and treatment is the application of AI technology to disease diagnosis and treatment. Computers can help doctors to carry out statistics on pathology and physical examination reports. Through technologies such as big data and in-depth mining, they can analyze and mine the medical data of patients, and automatically identify the clinical indicators of patients. For medical image intelligent recognition, by learning a lot of medical images, AI can help doctors locate the lesion area and reduce the problem of missed diagnosis and misdiagnosis. Robots are widely used in the medical field, such as intelligent prosthetics, exoskeletons and auxiliary equipment to repair damaged human bodies, and healthcare robots to assist medical staff in their work.

Deep learning is an emerging algorithm in machine learning, emphasizing the simulation of the human brain, using continuous network layers to analyze the internal relationship of data and learn data features. Deep neural network is composed of multilayer interconnected nodes (Fig. 1). Each layer is based on the previous layer to improve and optimize prediction or classification. This calculation process through the network is called forward propagation. The input and output layers of a deep neural network are named visible layers. The input layer integrates the model data preprocessing steps, and the output layer completes the final prediction or classification. Deep learning models such as convolutional neural network (CNN), recurrent neural network (RNN), Long Short-Term Memory (LSTM), which is mainly used in computer vision and image classification applications, can detect hidden features and patterns in images, so as to achieve tasks such as target detection or recognition.^7–13 These models have achieved superior results in the field of medical imaging, realizing aide diagnosis, risk stratification, and treatment planning. Besides, with the growing adoption of WSI, large amounts of digitized slide images have been generated. It ensures that there are enough data to feed deep learning models to learn to classify tissues and cells. And with the improvement of the performance of computer hardware equipment, especially the enhancement of GPU computing power, the computer can process the increasing picture size and process multiple images in parallel.¹⁴ In hematopathology, deep learning shows hope in completing existing tasks and making more effective use of existing data than traditional statistics (Fig. 2; Tables 1 and 2).

Figure 1.

Overview of processing pipeline of a convolutional neural network for the image classification.

Figure 2.

Overview of implemented AI applications in hematology. AI = artificial intelligence.

Table 1.

AI application in hematologic myeloid malignancies.

Study	Publication year	Disease	Digital image type	Model	Training set size	Valuation set size	Performance evaluation
Agaian et al15	2014	AML	Microscopic images of blood smear	SVM	80	NA	98% accuracy
Reta et al16	2015	AML, ALL	Microscopic images of blood smear	K-NN, RF, SL, SVM, RC	295 (ALL), 338 (AML)	34 (ALL), 29 (AML)	95% accuracy
Kazemi et al17	2016	AML	Microscopic images of blood smear	SVM	297	33	87% accuracy
Martinez-Martinez et al18	2016	MM	CT images	SVM, K-NN	93	23	0.996 AUC
Bigorra et al19	2017	Peripheral Blood Cells	Microscopic images of peripheral blood smear	SVM	696	220	85% accuracy
Xu et al20	2018	MM	PET and CT images	V-Net, W-Net, RF, K-NN, SVM	4000	1333	The combination of PET and CT for V-Net: 99.51% specificity
Matek et al21	2019	AML	Microscopic images of peripheral blood smear	ResNeXt CNN	14,692	3308	0.99 AUC
Boldu et al22	2019	AML, ALL	Blood smear with staining	Naive Bayes, K-NN, RF, SVM	4394	1098	85.8% accuracy
Matek et al23	2021	Hematologic malignancies	Bone marrow microscopic cytological images	CNN	171,374	34,274	Above 90% in precision and sensitivity

AI = artificial intelligence, ALL = acute lymphoblastic leukemia, AML = acute myeloid leukemia, CNN = convolutional neural network, CT = computed tomography, K-NN = k-Nearest Neighbor, MM = multiple myeloma, PET = positron emission computed tomography, RC = Random Committee, RF = random forest, SL = simple logistic, SVM = Support Vector Machines.

Table 2.

AI application in hematologic lymphoid malignancies.

Study	Publication year	Disease	Digital image type	Model	Training set size	Valuation set size	Performance evaluation
Putzu et al24	2014	ALL	Microscopic images	SVM, K-NN, NB, DT	267	NA	93.63% accuracy
Neoh et al25	2015	ALL	Microscopic images	Multi-layer perceptron, SVM	180	18	96.72% accuracy
Alferez et al26	2015	FL, HCL, CLL, MCL	Microscopic images	SVM	4389	439	97.67% accuracy
MoradiAmin et al27	2016	ALL	Microscopic images	SVM	5625	625	96.76% accuracy, 94.23% precision
Shafique et al28	2018	ALL	Microscopic images	AlexNet	221	147	96.06% accuracy
Rehman et al29	2018	ALL	Microscopic images	NB, SVM, CNN	264	66	97.78% accuracy
El Achi et al30	2019	Lymphoma	Microscopic images	CNN	1856	464	95% accuracy
Sahlol et al31	2020	ALL	Microscopic image	VGGNet	8737	2184	97.1% accuracy
Mohlman et al32	2020	Diffuse large B-cell lymphoma	H&E staining images	CNN	8796	2022	0.92 AUC
Syrykh et al33	2020	FL	Microscopic images	Bayesian neural network	160,000	80,000	0.99 AUC

AI = artificial intelligence, ALL = acute lymphoblastic leukemia, CLL = chronic lymphocytic leukemia, CNN = convolutional neural network, DT = Decision Trees, FL = follicular lymphoma, HCL = hairy cell leukemia, K-NN = K-Nearest Neighbor, MCL= mantle cell leukemia, NB = Naïve Bayesian, RF = Random Forest, SVM = Support Vector Machines.

3. NORMAL HEMATOPOIETIC CELLS IDENTIFICATION WITH AI

Segmentation and accurate identification of blood cells are considered as an essential step that helps to extract features to diagnose benign and malignant hematologic diseases. The manual counting of hematopoietic cells in microscopic images is extremely tedious, time-consuming, and subjective. Therefore, automatic hematopoietic cell classification technologies have been extensively developed. In 2014, Alomari et al³⁴ proposed a method that uses an iterative structured circle detection algorithm for the segmentation and counting of human white blood cells (WBCs) and red blood cells (RBCs). The image separation of WBCs from RBCs was performed by thresholding, and automatically counting the cells was performed for each image based on modified circle detection. The average accuracy of the method was 95.3% and 98.4% for RBCs and WBCs.³⁴ The first employment of deep learning methodologies for WBCs identification was carried out by Shahin et al³⁵ in 2017. They developed an identification system for WBCs in blood smear images based on deep CNNs. The overall accuracy of the system was up to 96.1%.³⁵ Whereafter, Wang et al³⁶ applied 2 remarkable object detection approaches, Single Shot Multibox Detector and An Incremental Improvement Version of You Only Look Once, to identify leukocyte. The train set consisting of 14,700 annotated images were used to generated the model, and the test sets were used to evaluate the models, which consists of 1120 annotated images and 7868 labeled single object images corresponding to 11 categories of peripheral leukocytes. Their model achieved a best mean average precision of 93.10% and mean accuracy of 90.09%.³⁶ Although the above method achieved satisfactory performances, they were not compared with traditional image processing. The first comparison work between traditional image processing and deep learning approaches were performed by Hegde et al³⁷ in 2019, which focus on the classification of WBCs. The authors obtained the highest accuracy (99%) for the CNN network compared with traditional image processing methods on the test sets.³⁷ It showed that machine learning methods have significant advantages compared with traditional image processing in identification of hematopoietic cell. Yet often, cell populations are still studied using manual gating based on imaging flow cytometry, which could be expensive and potentially confounding. Hence it would be advantageous to replace manual gating with an automated process, such as recognition cell type in stain-free images. According to this principle, Lippeveld et al³⁸ developed an automated and stain-free approach to classify WBC types through comparing 2 deep learning and 2 classical machine learning approach methods. Noteworthy, they found that the deep learning approaches do not outperform the approaches based on manually engineered features.³⁸ It is worth noting that various methods developed using a deep CNN to classify the microscopic images of WBC. Still, those methods suffer from the problem of domain shift—severe performance degradation when they are tested on data (target) obtained in a setting different from that of the training (source). Recently, this problem was solved using unsupervised domain adaptation (UDA) techniques by Pandey et al.³⁹ They described that given a test image from the target data, its “nearest clone” could be obtained from the source data used as a proxy in the classifier.³⁹

The above applications of CNNs focused on classifying the current cell type/state from the image. However, recent studies have shown that even after the original shape is lost, the current and/or past cell shape will affect the future cell state. Buggenthin et al⁴⁰ presented a deep neural network that prospectively predicts lineage choice in differentiating primary hematopoietic progenitors using time-lapse microscopy image patches from brightfield microscopy and cellular movement. In their work, they innovatively linked two biological microscope modes: bright field and fluorescence. The authors recorded moving images of thousands of cells that proliferated eight times, and then combined CNN and RNN architecture to screen the images to find the correlation between lineage selection (by fluorescence microscope) and cell morphology (from bright field images). Not only is this method more efficient in identifying stem cell differentiation, but it also eliminates the need to use fluorescently labeled samples.⁴⁰ Such capability of CNN should be particularly valuable in predicting critical areas, such as the study of the mechanism of cell differentiation or cell movement.

4. AI APPLICATIONS IN DISORDERS OF MEGAKARYOCYTES, PLATELETS, AND ERYTHROCYTE

Platelets, an essential component of blood, are produced by megakaryocytes in the BM and play an important role in hemostasis and various thrombotic diseases. Deficiencies in platelet function include a large and varied set of bleeding disorders ranging in severity from organ tissue damage to fatality.⁴¹ However, most are diagnosed as prone to bruises and bleeding of skin and mucosa, or excessive bleeding after injury or surgery. Hence, having an objective and effective way for megakaryocytes’ shape quantification and classification will lead to better insights and an eventual better prognosis of platelet function defects. Identifying abnormal megakaryocytes under pathological conditions has become the key to diagnosing these blood diseases. For instance, Philadelphia-negative myeloproliferative neoplasms (MPNs) are a group of disorders in which acquired mutations in hematopoietic stem cells. The 3 most common Philadelphia-negative MPNs (essential thrombocythemia [ET], polycythemia vera [PV], and primary myelofibrosis [PMF]) have overlapping clinical and laboratory features that can make their distinction challenging, particularly at early disease time points. Besides, the assessment of megakaryocytes of their cytological and topographic features is the central to the histological interpretation of bone marrow trephines (BMTs) from suspected MPN patients.^42,43 Sirinukunwattana et al⁴⁴ developed a machine learning approach for the automated identification, quantitative analysis, and abstract representation of megakaryocyte features on digital images of routinely prepared hematoxylin and eosin-stained sections. In detail, the method predicted the locations of megakaryocytes on a sample using a deep neural network called Single Shot Multibox Detector, then image segmentation was required to partition the images into different regions which contain the megakaryocyte cells using U-Net. Finally, the methods used a type of neural network (autoencoder) to learn feature representation which is a numerical vector encoding the megakaryocyte cytomorphology. This method achieved a high predictive accuracy with area under the curve of 0.95.⁴⁴

There are many blood diseases involving significant changes in the shape and size of RBCs in the human circulation, such as sickle cell disease (SCD). These morphological changes reflect the subtle changes at the protein level, and ultimately affect the shape and size of cells.⁴⁵ Especially in SCD, in addition to sickle shape, there are many shape types, which are directly related to sickle hemoglobin polymerization in erythrocytes. Xu et al⁴⁶ developed a new computational framework based on deep convolutional networks in order to classify efficiently the heterogeneous shapes encountered in the sickle blood, and the accuracy of their trained CNN model can obtain a high recall of 93.8% but a relatively modest precision of 85.0%.⁴⁶ The above studies showed that it is feasible and should be further studied to apply machine learning methods to obtain objective descriptions of blood cell morphology.

5. AI APPLICATIONS IN LEUKEMIA

Leukemia is a cancer that originates in the BM. The clinical manifestation is the overproduction of immature leukocytes that are originally used to replenish normal blood cells.⁴⁷ Morphological identification has become a powerful tool for hematologists to identify family diseases of acute leukemia.^48,49 Acute myeloid leukemia (AML) is a common subtype of acute leukemia, and its most important clinical feature is the accumulation of myeloid blasts in the BM.^50,51 The first step in making a definitive diagnosis of AML is careful microscopic examination of a stained blood smear or BM aspirate. Kazemi et al¹⁷ developed a new computational framework based on SVM in order to classify AML and its prevalent subtypes. The blood microscopic images can be automatically classified by their method through preprocessing, segmentation, postprocessing, feature extraction, and classification. The results of the proposed method showed that sensitivity, specificity, and accuracy were 95%, 98%, and 96%, respectively.¹⁷ Besides, Shafique et al²⁸ developed a framework for automated detection acute lymphoblastic leukemia (ALL) and classification of its subtypes by a deep CNN. In contrary to the training from scratch, the authors deployed pre-trained AlexNet which was fine-tuned on their training data set. Finally, the framework achieved a sensitivity of 100%, specificity of 98.11%, and accuracy of 99.50% for ALL detection; and it also achieved a good performance for ALL subtype classification with the sensitivity of 96.74%, specificity of 99.03%, and accuracy of 96.06%.²⁸ After this study, Rehman et al²⁹ also proposed a deep learning technique using the Alexnet model for the classification of ALL into its subtypes and normal condition. Performance was evaluated by comparing the results of Alexnet, Naïve Bayesian, K-Nearest Neighbor, and SVM, and the Alexnet method achieved the best accuracy of 97.78%.²⁹ The realization of the above leukocyte classification work is mainly based on feature extraction from images. However, for lymphocytic leukemia, the morphological changes of the cells were smaller compared to AML. To solve this problem, Matek et al²¹ compiled an annotated image dataset of over 18,000 WBCs, used it to train a CNN for leukocyte classification, and evaluated the network’s performance by comparing to inter- and intra-expert variability. For the most common physiological leukocyte classes as well as for myeloblasts, the network attained a precision and sensitivity above 90%.²¹ Sequentially, Matek et al²¹ compiled an annotated image dataset that is the largest expert-annotated pool of BM cytology images available in the literature. It contains 171,374 microscopic cytological images taken from BM smears from 945 patients diagnosed with a variety of hematological diseases. Then they trained high-quality CNN classifiers of leukocyte cytomorphology that identify a wide range of diagnostically relevant cell species with high precision and recall. The results showed that their CNNs model outcompete previous feature-based methods.²³

6. AI APPLICATIONS IN CLONAL HEMATOPOIETIC STEM CELL DISORDERS

Hematopoietic stem cell diseases mainly include aplastic anemia, myeloproliferative diseases, myelodysplastic syndromes (MDSs), and paroxysmal nocturnal hemoglobinuria.^52,53 It is clinically characterized by an overproduction of platelets, RBCs, and neutrophils. MDS is a hematopoietic tumor characterized by abnormal and ineffective hematopoietic function. Its diagnosis is mainly based on morphological manifestations. Kimura et al⁵⁴ developed an automated diagnostic support system for MDS by combining an automated blood cell image-recognition system using a deep learning system powered by CNNs with a decision-making system using extreme gradient boosting (XGBoost). Their method was trained using datasets consisting of 695,030 blood cell images taken from 3261 peripheral blood smears including hematopoietic malignancies. The trained model can simultaneously classify 17 blood cell types and 97 morphological features of such cells with >93.5% sensitivity and >96.0% specificity.⁵⁴

7. CONCLUSION

It is a key step in the diagnosis of blood system diseases that reliable identification of malignant blood cells, such as AML, SCD. While well-trained human examiners are necessary to complete microscopic morphological examination of blood cells, this is bound to be tedious, time-consuming and difficult to standardize. The ability to perform high-throughput morphological classification utilizing AI of blood cells, such as RBCs, WBCs, and megakaryocytes, opens up possible new avenues for the classification and identification of highly heterogeneous cell populations, such as blood diseases. Besides, it has been proved that it is difficult to provide a sufficient number of labeled images for training the deep learning model, because the time cost of experts providing basic fact annotations needs to be considered. The number of training set images or the cost of expert time becomes a major factor affecting model training. Of course, there are some common shortcomings of AI. The model used in AI requires a large number of parameters, such as network topology, initial values of weights and thresholds. In addition, the learning process cannot be observed, and the output results are difficult to explain, which will affect the reliability and acceptability of the results. Although there are many challenges in the application of AI to diagnosis hematopathology, it still greatly promotes and standardizes the diagnostic process to complement and assist human activities in this field.

ACKNOWLEDGMENTS

This work was supported by grants from the National Key Research and Development Project of China [2021YFC2500300, 2019YFA0801800]; National Natural Science Foundation of China [82070109, 62002153]; Guangdong Basic and Applied Basic Research Foundation [2022A1515011253, 2021A1515110653, 2019A1515010784]; China Postdoctoral Science Foundation [2020M682785].