Multi-modality artificial intelligence in digital pathology

Abstract

In common medical procedures, the time-consuming and expensive nature of obtaining test results plagues doctors and patients. Digital pathology research allows using computational technologies to manage data, presenting an opportunity to improve the efficiency of diagnosis and treatment. Artificial intelligence (AI) has a great advantage in the data analytics phase. Extensive research has shown that AI algorithms can produce more up-to-date and standardized conclusions for whole slide images. In conjunction with the development of high-throughput sequencing technologies, algorithms can integrate and analyze data from multiple modalities to explore the correspondence between morphological features and gene expression. This review investigates using the most popular image data, hematoxylin–eosin stained tissue slide images, to find a strategic solution for the imbalance of healthcare resources. The article focuses on the role that the development of deep learning technology has in assisting doctors’ work and discusses the opportunities and challenges of AI.

Introduction

In the era of the precision medicine, the demand for predictive analytics by doctors has increased. Multiple types of medical data, such as digital pathology slides, immunohistochemistry (IHC) images, spatial transcriptome sequencing and medical history records, are used for big data analysis. Unlashing the power of medical big data can be succinctly summarized as providing the most accurate treatments to the right patients at the right time [1]. Compared to molecular exams, pathology slides are widely accepted in clinics with their affordability and low technical requirements. The pathologist is still the gold standard in the diagnosis of many diseases (such as cancer) in tissue slides. Hematoxylin–eosin stained (H&E) tissue slides show multiple cytoplasmic, nuclear and extracellular matrix features, making them an easily accessible and informative source of data [2–4]. However, while the pathologists can flexibly adapt to the high morphological and technical variability of histologic slides, they face the problem of limited objectivity due to cognitive and visual traps. Recently, digital pathology has been focusing on quantitative data management of information generated from digitized specimen slides [5]. The data analysis procedures mainly include scanning and stitching traditional glass sections into whole slide images (WSIs), saving the data, then completing steps such as medical diagnosis or machine learning analysis by viewing the WSIs. This process generates a large amount of digital accessible data that can be easily shared among researchers worldwide, which not only enriches the learning resources for medical purposes, but also attracts a large number of computer scientists and creates a revolution in the field of digital pathology.

Artificial intelligence (AI)-based digital pathology as an emerging area has shown great promise to increase both the accuracy and availability of high-quality health care in many medical fields. Due to the rise in the amount of data, the increase in computing power, and the emergence of new algorithms for machine learning, the concept of ‘AI’ has been widely used in the fields of face recognition and autonomous driving since 2012 [6–8]. Deep learning algorithms have rapidly become the preferred method for analyzing medical images [9]. AI-based digital pathology not only facilitates a more efficient pathology workflow, but also provides a more comprehensive and personalized view, enabling pathologists to address the progress of complex diseases. In addition, a single stained section reveals only little information to the naked eye, and preparing multiple sections for staining and examination is more costly [10]. These difficulties provide opportunities to use AI-assisted systems to give doctors a reduced workload and improved accuracy of diagnosis and treatment.

Computational techniques in digital pathology

AI is a branch of computer science that attempts to understand the nature of intelligence and construct complex machines that possess the same characteristics as human intelligence [11]. Humans initially used machine learning to implement AI, designing features by hand to extract patterns from raw data. Traditional algorithms include Logistic Regression [12], Naive Bayes [13], Decision Tree [14] and Support Vector Machines (SVM) [15]. However, it has gradually been found that the performance of simple machine learning algorithms depends heavily on the selection and representation of features. To overcome the difficulties of feature design, machine learning has developed deep learning techniques. The model is inspired by human neurons and uses artificial neural networks to simulate the biological functions of the brain. Compared to the simple architecture of traditional machine learning models, deep learning models use a hierarchical structure that allows computers to learn to abstract complex, data-driven logic by constructing simple concepts without significant human intervention in the feature design task. The first layer of the model is the input layer, and model training is completed by fitting ground truth in the last layer through techniques such as back propagation and gradient descent. The performance of the model relies on the test evaluation of a dataset that does not overlap with the training set. Popular models include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), etc. [16, 17]. DNNs have the most basic network architecture, where linear operations are performed one by one between layer neurons, and then an activation function is attached to obtain nonlinear features [18]. CNNs have been frequently used in the field of computer vision. Since the creation of LeNet [19] in 1998, CNNs have become deeper, more refined in their structural design, and gradually more detailed in their research tasks. A series of models, such as ResNet [20], emerged that can surpass the level of human recognition. Usually, CNNs are connected with several fully-connected layers after convolution to map the feature map generated by the convolutional layers into a fixed-length feature vector. Fully convolutional networks (FCNs) replace the fully-connected layers with layers of convolutional layers to achieve pixel-level classification of images. CNNs have been widely used on radiological images, IHC images, and pathological images [21, 22]. Another type of networks commonly used to process sequence data are RNNs, which can capture association information between long-distance samples. RNNs have greater advantages in fields such as speech recognition and text prediction and are often applied in the analysis of RNA and DNA sequences [23, 24].

Recently, the application scenarios of AI on digital pathology have gradually expanded. A generalized workflow (Figure 1) has been widely used on a variety of topics and organs, introducing the latest techniques in classification, segmentation, and detection in the field of computer vision. The main research involves auxiliary functions, focused on image quality enhancement, cell identification, tissue typing, data storing, as well as direct applications, such as diagnosis, patient stratification, prognosis, treatment response, survival prediction and biomarker discovery. Several AI-related studies focus on investigating the interpretability of algorithms [25, 26] or building clinical application platforms [27], making it possible to select and stratify patients for treatment. With genetic sequencing databases becoming more refined, combination of digital pathology images and molecular datasets has led to breakthroughs in molecular biology [28, 29].

Figure 1.

Workflow diagram for doctors, pathologists and computer scientists. Based on a doctor’s request for a pathological examination, the pathologist prepares sections of the patient’s tissue samples with stains, digitizes the sections and annotates them with WSIs information as ground truth. These images are combined with relevant electronic clinical information to serve as a training database for AI models that can be shared worldwide, relying on the Internet. Computer scientists train and test the models based on the ground truth and complete multiple performance testing experiments to evaluate model value. The entire research process can be explored in categories based on medical tasks, disease types, computer models and computer tasks. Mature models can provide assistance to the pathologists in clinical diagnosis and treatment [169].

Methods

We investigate the application of AI to pathology images with center on H&E images. Manual feature extraction requires a priori knowledge from pathologists, usually for specific cancers or tissue types. Deep learning addresses the challenge of extracting image features that cannot be manually understood and finding image features that can be generalized. To investigate these techniques, we performed a manual screening on the US National Library of Medicine PubMed database (PubMed) for highly cited peer-reviewed articles published in English between January 2017 and September 2021. Search terms include medical (WSI, H&E) and AI (AI, deep learning, supervised learning, semi-supervised learning, weakly supervised learning, unsupervised learning, multitask). The results of the search contained 6707 articles. The selection process consisted of three steps: 1) Only research articles were retained, excluding types such as reviews, comments, etc. 2) Based on titles and abstracts, articles with at least one keyword in each of the medical and AI groups were retained, while avoiding purely medical experimental articles or computer technology research articles. 3) Papers that are not related to pathology images are excluded since abbreviations such as ‘AI’, ‘WSI’ and ‘H&E’ are easily confused with keywords such as aluminum. Only AI articles, where H&E images were the main subject of study, were retained. Therefore, a total of 155 articles are discussed in the main body of this paper. Sample articles are discussed for each type of article as a demonstration in Table 1 [28–30].

Table 1.

Sample articles for each research direction

	Topic	Year	Tissue type	Medical task	Computer task	Data type	Input size	Main dataset size	Backbone	Learning method	External dataset validation	Refs.a
Information preparation	Automatic segmentation	2017	Seven organs	Nuclear segmentation	Segmentation	H&E	51 × 51 pixels	30 WSIs	CNN	Supervised learning	No	[41]
	Virtual staining	2020	Kidney tissue sections	Data Preparation	Generation	Autofluorescence images	256 × 256 pixels	12-thin tissue sections	GAN	Unsupervised learning	Yes	[57]
	Prognosis (MSI)	2019	Gastric (stomach) adenocarcinoma and colorectal cancer	Classify MSI versus microsatellite stability	Classification	H&E	256 × 256 μm2	1053 WSIs	CNN	Supervised learning	Yes	[65]
Diagnosis	Diagnosis	2019	Prostate cancer, basal cell carcinoma and breast cancer metastases to axillary lymph nodes	Develop diagnostic decision support systems	Classification	H&E	224 × 224 pixels	44 732 WSIs	CNN + RNN	Weakly supervised learning	Yes	[91]
	Diagnosis	2020	Prostate cancer	Automated Gleason grading of prostate biopsies	Classification	H&E	1024 × 1024 pixels	5759 biopsies	CNN	Supervised learning	Yes	[74]
	Diagnosis	2020	Diffuse large B-cell lymphoma (DLBCL)	Differentiate DLBCL and non-DLBCL	Classification	H&E	Various sizes	4530 tissue sections	CNN	Supervised learning	Yes	[92]
Post-treatment	Outcome	2020	Colorectal cancer	Develop an automatic prognostic biomarker	Segmentation, classification	H&E	224 × 224 pixels	12 226 119 tiles	CNN	Weakly supervised learning	Yes	[102]
	Prognosis	2021	Gastric cancer	Improve tumor node metastasis staging system	Segmentation, classification	H&E	700 × 700 pixels, 768 × 768 pixels	9366 WSIs	CNN	Supervised learning	Yes	[113]
Images with nucleic acid sequences	Tumor/normal or subtype and gene mutation	2018	Lung cancer	Detection of cancer types or gene mutations	Classification	H&E and gene mutations	512 x 512 pixels	1634 WSIs	CNN	Supervised learning	No	[133]
	Tumor/normal or subtype and gene mutation	2020	19 cancer types	Reveal conserved spatial behaviors across tumors	Classification	H&E	512 × 512 pixels	27 815 WSIs	CNN	Supervised learning	Yes	[134]
Multiple digital pathology images	Patient stratification	2021	Esophageal adenocarcinoma	Triage-driven diagnosis of Barrett’s esophagus	Classification	H&E and IHC	400 × 400 pixels	4662 WSIs	CNN	Supervised learning	Yes	[152]
	Morphologic assessment	2021	Kidney cortex	Segment kidney histologic structures	Segmentation	H&E, PAS, Silver, and Trichrome	256 × 256 pixels	459 WSIs	CNN	Supervised learning	Yes	[151]
Multiple modal information	Prognosis	2017	Lung adenocarcinoma	Predict the survival outcomes	Classification	H&E, pathology reports, RNA-sequencing, and proteomics data	—	831 WSIs	Random forest	Traditional machine learning	Yes	[157]
Multi-task	Simultaneous segmentation and classification	2021	Hepatocellular carcinoma	Segmentation of HCC lesions	Segmentation, Classification	H&E	256 × 256 pixels	423 images	CNN	Supervised learning	Yes	[159]
	Differential diagnosis	2021	18 primary origins	Predicts origins for cancers of unknown primary	Classification	H&E	256 × 256 pixels	32 537 WSIs	Graph, CNN	Weakly supervised learning	Yes	[163]

CNN, convolutional neural network; RNN, recurrent neural network; GAN, generative adversarial network

^aThe reference numbers are consistent with the text.

Applications in unimodal digital pathology

It is a time-consuming and cumbersome process for pathologists to find a predefined region of interest and conduct with an accurate diagnose under time constraints. Besides, inter- and intra-observer variability for the exact same slide may occur among the experienced pathologists. Many efforts were made to overcome the challenges with traditional pathology methods. Technological advances have recently paved the way for the development of digital pathology-based methods for quantitative assessments, namely WSIs scanning and AI–based models, to assist with extracting information beyond pathologists’ visual perception. Pathology images and corresponding clinical information (such as cancer grading or cancer types) are mostly used for unimodal research [31]. The label for modeling results purely from clinical records without other complex wet experiments. Unimodal research with simple and clear goal is very suitable for AI-based models in clinical applications and translations (Figure 2). However, it is necessary to compare AI-based models’ predictions with the analyses that pathologists produce to reveal potential interpretation discrepancies and accuracy of algorithms. The development and integration of digital pathology and AI-based models provide substantive advantages over traditional methods, enabling spatial analysis while generating highly precise, unbiased and consistent readouts that can be accessed remotely by pathologists.

Figure 2.

Description of a unimodal generalized framework for AI application on digital pathology images. After the raw data collected at the hospital are annotated by pathologists, computer scientists complete the patches cut and enhance the data. The processed data are fed into a model pre-trained by ImageNet to achieve transfer learning of natural images to digital pathology images. The computer scientists train the model to fit the annotations and test the generalization of the model on an extra dataset that is also pre-processed. Finally, the model analysis results are used for statistical analysis as well as visual interpretation to explore their biological significance [169, 204].

Information preparation for digital diagnosis and treatment

H&E images highlight the morphological features of the nucleus and cytoplasm mainly by the pH distinction of the stains. Pathologists need plenty of exercises in their daily work to become proficient at slides analysis [32, 33]. When visualizing H&E images on the computer, there are problems, such as inconsistent staining shades and poor photographic quality due to manual errors. There are also problems in the algorithmic computation stage, where the size of the image is too large for commonly used algorithms. Many studies have been devoted to solving the difficulties associated with image data [34–37]. Therefore, the selecting the part of the image that contains morphological information is an important data pre-processing process.

The use of automated tools to differentiate cells/tissues can reduce manual work [38–43]. In computer vision, the mainstream segmentation network consists of encoder and decoder, which input an image and output a label mask of the same size. Transferring the model to the field of digital pathology, the segmentation of tissues or glands of a single cancer can generally be accomplished only due to the limitation of annotation [44–47]. The processing of pathology images as shown in Figure 2 usually refers to the cut-off approach in natural images, where square images are fed into a neural network. However, the cells are not only arranged in a square order, but also others, as the morphology changes with the type, organ and developmental stage, and some information is always lost when slicing the patches. Cells that are in contact or overlapping make the automatic segmentation task even more difficult [48]. The identification of cell nuclei during the analysis can often be centered on a single pixel, while extending the focus to the tissue requires the aid of the superpixel idea. Each superpixel combines adjacent pixels containing a single tissue type to fit the edges between tissues as irregular polygons [49, 50]. This interactive annotation system developed by Lee et al. [51] returns the less confident parts to the doctors for secondary review, thus optimizing the model in segmenting tumor-infiltrating lymphocytes (TILs), saving significant time in clinical applications.

When multiple special stains (such as periodic acid-Schiff or Masson’s trichrome) are required, the diagnosis and treatment processes become time-consuming and expensive. The stain transformation method in computational staining techniques provides doctors with additional diagnostic information beyond H&E images without laboratory operations [52–56]. With regards to the region-of-interest analysis, virtual staining can directly transform label-free tissue images into high-quality stained digital pathological images. Zhang et al. [57] demonstrated that a single neural network can achieve H&E, Jones’ silver staining and Masson’s trichrome staining of micro-structured on a single autofluorescence image. This method avoids destructing the specimens, completes operations that are not feasible in the laboratory, and achieves staining normalization.

In the grading of cancer, the metastasis of lymph nodes determines the seriousness of the disease and the next step of treatment [58, 59]. Algorithms can follow the process of finding lymph nodes first and then looking for metastatic sites in practice [60]. When expecting to outperform human performance, additional attention is paid to the size difference between micro- and macro-metastases in detection, employing dense scans to avoid missing tiny metastases [61].

Translating the complex sequencing process into simple classification problems in H&E images, as opposed to improving the ability to understand the images, brings new opportunities for oncology patients [62]. For example, consensus molecular subtypes (CMSs) of colorectal cancer are difficult to distinguish even after completing RNA analysis, and optimizing the results of molecular typing is important to guide treatment. As is known, cancer is good at evading immune destruction, but not all patients need immunotherapy or have the condition to complete genetic testing. In studies on immune checkpoint inhibitors (ICIs), tumor mutational burden (TMB) and microsatellite instability (MSI) are shown to be important biomarkers. The researchers assisted in the selection of treatment options by classifying H&E images as high- or low-value divisions [63, 64]. Analysis of TMB values with clinicopathologic features demonstrated that TILs, a morphologic feature that can be observed on pathologic slides, is an independent predictor that correlates tumor morphology with genomic alterations. Most of the MSI-related studies involve gastrointestinal cancers. The access to research data has become more extensive and contributed to the design of more complex deep learning models. With the different models, not only are the performances of the model themselves measured, but also the data source differences, preparation differences, and performance on different subclasses of diseases are investigated [65, 66]. Benefiting from advances in deep learning Ref. [67] referencing the idea of Ref. [65], the first step of the tissue classification task was refined to select mucin and colorectal adenocarcinoma epithelium tiles that are more relevant to MSI for prediction, and the performance was improved.

AI attempts to replace manual diagnosis

Efficient diagnosis of complex diseases can provide an important basis for patient recovery and clinical development, and many AI tools that have been developed to replace manual operations, thus saving time and money [68]. Initially, the diagnostic task was considered as a binary task as either having cancer and being healthy, and high accuracy rates have been achieved in algorithmic experiments with supervised learning [69]. In fact, cancer is a highly heterogeneous disease requiring more detailed subtype diagnosis. Deeper and more structurally complex models were created to accomplish this need [70, 71]. More questions followed and people started to explore whether the models could actually be used in a clinical setting and whether valuable knowledge was actually learned in the process of model prediction.

As shown in Figure 2, the pathologists first draw tumor/normal/other class regions on the WSIs and then slice the WSIs into patches that fit the size of the image model. The researchers train the model to learn the mapping relationship between patches and labels, and in this process, the model achieves abstraction of features. Then, in the testing phase, the same predictions are made for patches of unknown images, which are integrated into the results of slicing. The integration operation can be implemented with statistical analysis, machine learning or neural networks [72–78]. Based on the seriousness of the disease, Yang et al. [73] innovated the two-stage threshold-based tumor-first aggregation method according to the severity of the type. The indicators on public the cancer genome atlas (TCGA) data even exceeds the internal cohort, demonstrating the portability of the model. Across all covered cancers, the Gleason score exacerbates the challenges posed by intraclass similarity and interclass variability between multiple classes [79, 80]. Unlike one label per image, the diagnosis of prostate cancer requires a clear definition of the proportion of each pattern in a single picture [81]. Bulten et al. [56] innovated a semi-automated labeling method, making a model that realizes the simple task of training on the pure Gleason score first, then training it on the complete mixed Gleason sample dataset.

The problem of patch-level annotation also presents an opportunity for multi-instance learning (MIL) to enter the field of medical image analysis. To meet the training requirements of deep learning models, researchers need to prepare a large amount of data and ensure the accuracy of the annotation through the review of multiple pathologists. Weakly supervised learning treats the whole image as a ‘bag’ and each patch as an ‘instance,’ with multiple instances sharing the labels of the bag. If there is a positive instance in the bag, the bag is positive; the bag is only negative if and only if the instances are all negative, so the algorithm usually chooses the top-level patches that have the most discriminative power (the highest probability of being predicted as positive) to infer the slide class in the implementation process [82–90]. The performance of the model even exceeds the supervised learning with pixel-by-pixel annotation in specific cases. Campanella et al. [91] trained CNN models on the oversized real-world dataset (MSK dataset) of prostate, skin, and breast cancers and used a RNN to achieve slide-level determination. When they tested the MIL algorithm trained on the MSK dataset on the filtered CAMELYON16 dataset, the model AUC decreased by 7.15%. However, when the supervised learning algorithm was applied in real-world data, the performance dropped by 20.2%. This suggests that traditional supervised learning methods do not adapt to the variability of data.

Researchers have applied various models to test on data from different sources, [73, 74] aiming to solve AI models generalization problem on low-level data from multiple centers. However, deviations in the sample population with different staining patterns, and even microscope scanning specifications, may bring complex impact on pathological images. Typically, models are trained only on a single hospital data set, and the embedding features captured by the model cannot be transferred directly to another hospital datasets, resulting in low diagnostic accuracy and unable to reach a universal level of clinical application. Li et al. [92] adopted a multi-model approach, using an ensemble of 17 globally optimized transfer deep-learning platform with multiple pretrained CNNs (GOTDP-MP-CNNs) to distinguish between diffuse large B-cell lymphoma (DLBCL) and non-DLBCL on the data of three hospitals. The accuracy rate reached nearly 100%, surpassing individual CNNs and pathologists. In the cross-hospital tests of A and C, the accuracy rate increased from 82.09% to 90.50% only with unifying the image size. This result shows that the AI model can replace manual work on small data sets after avoiding differences in the sample preparation process through data preprocessing.

Despite all these metrics that demonstrate the capabilities of AI, deep learning models have been referred to as black box algorithms, as it is difficult to explain the working mechanism within the model to convince people that it is reliable. Researchers relied on medical knowledge to interpret models in different views, including the ablation studies [77], splitting pre-processing process [74], feature clustering [75] and the activation of class maps [62]. There are also many AI models that have been designed to increase model interpretability by leveraging the behavior of pathologists as a reference during the design process. In practice, pathologists usually zoom in to observe several important cancerous sites in the image and rely on image features to make a diagnosis for the patient [87], resulting in the design of a two-stage diagnostic algorithm that first detects important locations in the full image that affect the outcome and later classifies the locations. BenTaieb et al. [93] used a pyramid-shaped network to achieve the recognition of image features at different magnifications.

AI provides new knowledge for post-treatment analysis

The analysis of patient prognosis, recurrence or survival can guide the choice of treatment options [94–101]. In particular, the prognosis of the post-treatment effect in the early stages of cancer is key to optimizing clinical interventions. Generic models that directly use post-treatment metrics as labels are similar to diagnostic tasks, and researchers can corroborate the results of the models by analyzing the calculated characteristics in correspondence with other patient information [102, 103]. Therefore, the studies that can propose new criteria for clinical discriminations deserve more attention.

Pathological images reflect the morphological features and tissue architecture of tumors and contain a wealth of post-treatment information [104]. Traditional machine learning approaches apply predefined image features to resolve tissue patterns [105, 106]. On the contrary, deep learning can automatically identify cellular-level or structural-level patterns from multiple scales for fusion analysis [107]. For example, image features focus researchers’ attention on subclasses of a disease or subregions of an image, highlighting the relevance of class differences for post-treatment [108, 109]. Researchers measured the validity of the identified features using molecular expression information or correlation analysis as a bridge to analysis. Based on the idea that different molecules are expressed in different regions of the tumor, Zadeh Shirazi et al. [110] first segmented the pathological images into eight regions such as infiltrating tumor, and then correlated the mutation status with the survival rates. Yu et al. [111] first validated the ability of VGGnet as a backbone network to extract reasonable image features on simple cancer detection, grading and transcriptome subtype classification tasks. The network was then applied to predict the platinum-free interval (PFI), comparing proteomic and transcriptomic correlation analysis, combining biological mechanisms with morphological representation. Skrede et al. [102] developed a prognostic marker for colorectal cancer that can complete prognostic classification simply using H&E images. This method adopts a multi-scale MIL algorithm on a large-scale population.

The above paper shows that morphological information can give different forms of complementary evidence to post-treatment analysis, pushing AI models to ‘renovate’ existing empirical knowledge. Just as doctors know that different classes of conditions have different tissue compositions, in practice, this is still a difficult pattern distribution to determine. Neural networks can translate the distribution patterns of objectively complex and rich multicellular environments into quantitative clinical discriminators with a single value [98, 112]. Wang et al. [113] proposed the T/MLN for standardizing the N-staging criteria in gastric cancer. This score was calculated from the tumor classification area within the segmented lymph node region and was able to correct the cases diagnosed by doctors. In more complicated immune infiltration-related studies, it is a strong prognostic biomarker that relies on manual scoring [114–116]. With the aid of a computer, we can clearly detect immune cells in the tumor area and derive evaluation indicators of prognostic value from the number and distribution of cells (such as stromal TILs density statistics) [96, 117, 118]. This is the necessary research for clinical application of image features to drive the digital pathology assistance system to the ground.

Fusion approaches in multimodal digital pathology

During daily cancer diagnoses and treatments, even if the evaluation of tissue sections by trained pathologists is the gold standard, they still need the aid of some biomarker tests. The advent of high-throughput sequencing technology, the collection of electronic medical records and the improvement of hospital laboratory facilities have given us access to more information. The multiple information resources have been contributing to a new multimodal direction for research [119]. Multimodal studies refer to the processing of information from multiple modalities with different formation methods and internal structures and learning the correlations between separated modality datasets. This paper focuses on the integration of H&E images with nucleic acid data, additional images and clinical indicators. Nucleic acid data refer to the specific states detected during genome and transcriptome sequencing, and these often represent the inherent factors in the development of disease [120–122]. Additional images refer to the images generated when the same subject takes different imaging methods or stains. The paired images can help high-dimensional feature abstraction posed by H&E oversized images during deep learning feature extraction [123]. Clinical indicators are quantitative indicators contained in the patient’s electronic medical record, such as age, tumor stage, etc. All three types of data listed can be used to improve the effectiveness of deep learning algorithms from different perspectives.

The human brain is a high-level organ with the ability to process complex information, but the computer only completes computational procedures according to instructions. Two ways to solve the modal problem are shown in Figure 3. Independent inputs and outputs with unimodality can help us discover the correspondence between different modal information by running at high speed. Multiple modalities are used as the input of AI models, and each modality should be integrated with another. Research in both directions is addressing an important difficulty—aligning the features between different modalities [124].

Figure 3.

Description of multimodal generalized ways for AI application on digital pathology images. (A) H&E images are predicted by feature extractor for gene expression, other stained images, or clinical information to achieve alignment of different kinds of information content. (B) H&E images and multiple types of data are jointly predicted by feature extractor for the target, and the kinds of feature dimensions need to be aligned in the feature extraction process [169].

H&E images integration nucleic acid detection

The expression of nucleotides dominates the general patterns of organismal functioning. In unimodal studies, we commonly use RNA-seq data as validation metrics to discriminate whether the model learns the correct image features [98, 125]. The prediction result map of deep learning extracts human interpretable features that correlate with molecular phenotypes and can demonstrate the effectiveness of deep learning [126].

Unimodal input–output studies that extend the model on single pathological images to the fusion of pathological and molecular features guide the clinical system to make fine-grained decisions [127, 128]. It was Valieris et al. [129] who applied the model design of Campanella et al. [91] to DNA repair deficiency (DRD) detection. The morphological characteristics of the pathological pictures contain differential expression of gene methylation [130]. Tavolara et al. [131] used deep learning to predict gene expression, and then xgboost to classify gene expression as ‘supersusceptible’ or not.

Genetic mutations can cause a variety of cancers [132, 133]. Coudray et al. [134] used the same network to demonstrate that deep learning models can not only accomplish the classification of tumors, lung adenocarcinoma and lung squamous cells, but also predict specific gene mutations. The same problem was extended to pan-cancer by Noorbakhsh et al. [135] using the identical basic Inception v3 architecture. The pan-cancer study focuses on discovering similarities and differences between genomic and cellular alterations in different tumor types and finding universal patterns. The problem of less available data encountered in the above study, Bian et al. [136] proposed a semi-supervised approach to accomplish data labeling and screening, and the gene mutation prediction task designed based on this study achieved good results.

Simultaneous multimodal input of image and molecular information can predict survival. The fusion of multimodal data needs to address the validity of the information of each modal data. In the field of diffuse glioma survival research, Mobadersany et al. [137] first applied survival convolutional neural networks (SCNN) in regions of interest (ROIs), and the model combined CNN and traditional COX analysis to demonstrate a median c index of 0.754. After validation, the SCNN model was upgraded by adding isocitrate dehydrogenase (IDH) mutation status and 1p/19q codeletion as input to genomic survival convolutional neural network (GSCNN), the data of the two modalities together undergo fully connected network and COX analysis, and the index is upgraded to 0.801, which achieved the performance improvement on a single model. New techniques have been introduced to optimize feature alignment engineering across various modality data. The study by Chen et al. [138] used the co-attention transformer to achieve survival prediction. The co-attention visualization in the paper allows for complementary matching of sequence information with spatial information. The model’s performance surpasses the unimodal model as well as the late fusion algorithm.

Multiple digital pathology images combination

Magnetic resonance imaging (MRI) with WSI allows for simultaneous analysis of patient prognosis from both macroscopic and microscopic perspectives. Zhang et al. [139] extracted features from two scales separately to form a multiscale nomogram and verified that the features were associated by genetic variation. When it is necessary to understand the expression of biomarkers, other specific stains are needed to assist [44, 100, 115, 140–142]. Stain transformation techniques are mentioned in the ‘information preparation’ section, and this section discusses the role of different stains in accomplishing the task.

IHC combines the specificity of the immune response, the visibility of histochemistry, with qualitative, localization and quantitative determination of the corresponding antigen. Manual analysis of immune tissue environmental characteristics requires cytokeratin and IHC to identify each tissue and cell. Especially in complex tumor models, there is an insurmountable difficulty for humans to accurately calculate the number of each type of cells in the matrix [96]. Martino et al. [143] used IHC stained pictures as ground truth and extracted cell nuclear related features using QuPath to predict the expression of protein Ki67. This type of image is also commonly used to measure the model’s new biomarker [99, 117, 118] and how well the model scoring results matches the objective facts [144–146].

To investigate the performance differences of the different staining schemes on the same model structure and task [147–151], Jayapandian et al. [152] achieved segmentation of six kidney tissue structures on four stains: H&E, Periodic Acid Schiff (PAS), trichrome (TRI) and silver (SIL). Besides, one of the practical applications of the different staining methods is to achieve patient stratification and to apply multiple images to the diagnostic workup, reducing doctors’ workloads. Gehrung et al. [153] exploited a triage aid system to screen only H&E pictures with columnar epithelium into the second step of TFF3 staining assessment to select equivocal patients for pathologist to review.

H&E images incorporating multiple modal information

In clinical settings, data modalities are abundant. When some patients are admitted to the hospital, the first thing is to improve the patient demographics in the electronic medical record system. As a relatively common and easily obtained data modality, electronic records of clinical patients can be integrated with pathology images to enhance the reliability of image features and improve model performance [154, 155]. Brinker et al. [156] integrated inputs from three perspectives: clinical features, cellular features and image features, followed by features that went through a neural network feature extractor to jointly predict sentinel lymph node status, respectively. Although the results are not good enough, they provide new ideas for feature extraction. In [97], Cox regression analysis of the multinucleation index (MuNI) and clinical characteristics showed a significant improvement in hazard ratio (HR).

Vast amounts of omics data, bringing about detailed classification, early diagnosis, and prognosis of the disease, can be combined with digital pathology image data to extract useful patterns from these diverse data modalities. Multi-omics data can interpret image features [145]. Features extracted by deep learning models based only on pathology images with implicit information mining based on various types of histology data can help clinical translation for creating new clinical prognostic or grouping metrics [157]. Alternatively, sequencing data based on pathological image information and few important different multi-omics can support the performance of metrics for practical clinical applications. In [98], Cox analysis was carried out together with UICC stage, gender, and age, and it was verified that deep stroma score is an independent prognostic factor with high value. The application value of the index is reflected by the comparison with the gold standards, including manual annotation estimation and gene expression, thus confirming a universally available clinical application of H&E images. When copy-number alterations (CNA) correspond to transcriptome data with an unsupervised deep learning network (named CNx), the correlation of their middle layer activation values with the features can be found.

How to simultaneously input multiple histological data into a single network and tissue feature dimension alignment are the primary challenges in multimodal studies. Manual filtering of features is the general approach. Yu et al. [158] combined histopathological, genomic, transcriptomic, proteomic and clinical information. Firstly, quantitative features were extracted in the images. Then the images were graded based on clinical information, and the relationships between the filtered genomic, proteomic and transcriptomic features, and the pictures were explored. It was confirmed that there was a strong correlation between the enriched genes or proteins and tumor differentiation. When the multi-omics information was put into the COX analysis collectively, it showed excellent performance that was not available in separate analysis.

Multi-task learning with H&E images

Multi-task learning means that model features embedding and sharing for different tasks uncover more biological insights in different dimensions, such as, WSI, a domain of medical imaging that contains tissue and cell samples. Several researchers have applied deep learning on WSIs for various clinical tasks, including cell detection, segmentation and classification. Furthermore, while the similarly between the different tasks maybe difficult to define formally, some transfer learning methods between source tasks and target tasks have been shown to help convergence and reduce overfitting.

When extracting image features, we can perform cell segmentation and classification at the same time. One approach is to share feature extraction and then use different branches for different tasks. Graham et al. [159] completed the segmentation and classification of cell nuclei using Hover-Net, which is based on feature maps calculated from horizontal and vertical distances. The segmentation task is accomplished by parallel cell nucleus-to-background distinction branch, and individual cell nucleus delineation branch. Hover-Net demonstrates superb performance across diverse datasets. Wang et al. [160] focused on the pixel depiction of tissue edges and used two branches to complete the segmentation of tumor tissue and tumor against normal tissue, respectively; another auxiliary branch was designed to complete the classification task. Based on the deep learning results, the graph features are manually extracted to find new biological meanings. Wang et al. [161] completed the tasks of individual cells based on features extracted by Mask R-CNN, and the calculated quantitative depiction features set up a new subtyping criterion for hepatocellular carcinoma images. When having cell nucleus type labels, the classification task can be implemented simultaneously as an information supplement. Another approach is to process the segmentation and classification tasks sequentially. The segmentation task requires pixel-level annotation, while classification requires only image-level annotation since pixel-level annotation is often both time-consuming and error-prone. Because the segmentation mask contains the information on the classification to some extent, Ciga et al. [162] first trained the segmentation network using quantity-limited pixel-labeled images, then connected the classification layer after the network and borrowed the image level labels to adjust the segmentation network.

Clinical tasks with digital pathology are adaptive in different circumstances. To infer various properties of biological samples through multi-task learning or transfer learning is a big challenge. From the biological viewpoint, the human body can be viewed as a cell-tissue-organ level integration, which can be matched to the various resolution clinical tasks including image-based-only or image-omics-fusion tasks. In modern clinical medicine, especially with the diagnosis and treatment of complex diseases such as cancer, multiple various medical examinations are required, and they produce different data modalities. Using the different examination indicators as output, the genotype can be visualized spatially on the pathology images [163]. Schmauch et al. [144] implemented the HE2RNA model based on the idea of multilayer perceptron to study the accuracy of RNA-Seq prediction among 28 cancers. They found that, within the appropriate data selection, the model was consistent with the objective fact of IHC staining for gene expression prediction in terms of immunity, etc. They then transferred the transcriptomic representations learned by the model to MSI predictions, and the model performance reached a high improvement over the previous ones in a 3/4 versus 1/4 data division setting, demonstrating the validity of the representations. The multiclassification generic framework of [82] was used for cancer of unknown primary to predict whether it is metastatic and where the origin is. Convolutional features that concatenate gender features performed well on each point of origin, each cancer, its subclass, and individual. Top-k calculation results successfully narrowed down the IHC staining detection to assist doctors in diagnosis [164].

Opportunities and challenges in AI-based pathology

With the rise of big data analytics that enables the structuring and intelligent indexing of data, the current trend in the development of datasets is very favorable for research efforts, as the use of deep learning becomes more ubiquitous and practical in digital pathology [165–168]. Table 2 lists publicly available datasets of H&E images [169–192]. AI research motivates computers to simulate human learning habits, similar to adjusting the solution process based on the answers to exercises, which allows the model to capture the connection between data features and various labels and gain the ability to correctly discriminate known patterns after a series of adjustments and refinements. In the face of pathological images, features and labels are presented in different ways. Features can refer to whole image, tissue and cellular level image information extracted by the model, as well as clinical and genetic testing type information of the patient. The labels, on the other hand, are set according to the target task, with discrete numbers for classification tasks and pathologists’ annotations on the images for detection and segmentation tasks.

Table 2.

Digital pathology public datasets summary

Dataset name	Type	Task	Tissue	Staining method	ImageNum	Year	Refsa	Link
TCGA	Dataset	—	—	—	—	2008	[167]	https://gdc.cancer.gov/
ICPR 2012	Challenge	Automated detection of mitotic cells	Breast	H&E	5	2012	[170]	http://ludo17.free.fr/mitos_2012/
MITOS & ATYPIA 2014	Challenge	Detection of mitosis and evaluation of nuclear atypia score	Breast	H&E	32	2014	[171]	https://mitos-atypia-14.grand-challenge.org/
GlaS	Challenge	Segmentation of glands	Colon	H&E	165	2015	[172]	https://warwick.ac.uk/fac/cross_fac/tia/data/glascontest
CAMELYON16	Challenge	Detection of micro- and macro-metastases in lymph node digitized images	Breast	H&E	400	2016	[173]	https://camelyon16.grand-challenge.org/Home/
TUPAC	Challenge	Predicting tumor proliferation fraction	Breast	H&E	500 and auxiliary dataset	2016	[174]	https://tupac.grand-challenge.org/
CAMELYON17	Challenge	Detection and classification of breast cancer metastases in lymph nodes	Breast	H&E	1399	2017	[175]	https://camelyon17.grand-challenge.org/Home/
LSVM_CTXT	Dataset	Subtype classification of ovarian cancer	Ovarian	H&E	133	2017	[93]	http://www.sfu.ca/~abentaie/LSVM_CTXT/LSVM_CTXT.html
MCSU	Dataset	Quantification of tumor hypoxia	Eight tumors	H&E and (immuno-)fluorescence	178	2017	[151]	https://cs.adelaide.edu.au/~carneiro/humboldt.html
nucleisegmentation	Dataset	Segmentation of nuclei	Seven organs	H&E	30	2017	[41]	https://nucleisegmentationbenchmark.weebly.com/
BACH	Challenge	Classification of the four types of breast cancer	Breast	H&E	400	2018	[176]	https://iciar2018-challenge.grand-challenge.org/Home/
Pcam	Dataset	Differentiation of the presence for metastatic tissue	Lymph node	H&E	327 680	2018	[177]	https://github.com/basveeling/pcam
ANHIR	Challenge	Registration of stained images by different biomarkers	Lung, kidney, breast etc.	H&E, Cc10, proSPC, Ki67 etc.	355	2019	[178]	https://anhir.grand-challenge.org/Intro/
BCSS	Dataset	Semantic segmentation of breast cancer	Breast	H&E	151	2019	[179]	https://github.com/PathologyDataScience/BCSS
DigestPath2019	Challenge	Detection of signet ring cells & segmentation and classification of colonoscopy tissue	Gastric mucosa, intestine and colon	H&E	687 & 872	2019	[180]	https://digestpath2019.grand-challenge.org/Home/
Gleason 2019	Challenge	Automated Gleason grading	Prostate	H&E	333	2019	[81]	https://gleason2019.grand-challenge.org/Home/
MITOS_WSI_CCMCT	Dataset	Detection of mitosis	Skin(dog)	H&E (manually expert labeled dataset)	32	2019	[181]	https://github.com/DeepPathology/MITOS_WSI_CCMCT
NCT-CRC-HE-100 K	Dataset	Automated tissue decomposition	Colon	H&E	100 000	2019	[98]	http://dx.doi.org/10.5281/zenodo.1214456
PAIP 2019	Challenge	Detection of liver cancer	Liver	H&E	100	2019	[182]	https://paip2019.grand-challenge.org/Home/
HEROHE	Challenge	Classification of her2-positive and negative	Breast	H&E	360	2020	[183]	https://ecdp2020.grand-challenge.org/Home/
MITOS_WSI_CMC	Dataset	Detection of mitosis	Breast(dog)	H&E (manually expert labeled dataset)	21	2020	[184]	https://github.com/DeepPathology/MITOS_WSI_CMC/
MoNuSAC	Challenge	Segmentation of nuclei of multiple cell-types	Lung, prostate, kidney, and breast	H&E	71	2020	[185]	https://monusac-2020.grand-challenge.org/Home/
PANDA	Challenge	Automated Gleason grading	Prostate	H&E	11 000	2020	[186]	https://www.kaggle.com/c/prostate-cancer-grade-assessment/overview
BCNB	Dataset	Prediction of the metastatic status of ALN, histological grading and molecular subtype etc.	Breast	H&E and clinical characteristics	1058	2021	[187]	https://bcnb.grand-challenge.org/Home/
MIDOG 2021	Challenge	Identification of mitotic figure	Breast	H&E	150	2021	[188]	https://imig.science/midog/
NuCLS	Dataset	Classification, localization and segmentation of the cell nucleus	Breast	H&E	125	2021	[189]	https://nucls.grand-challenge.org/NuCLS/
PAIP 2021	Challenge	Detection of perineural invasion in multiple organ cancer	Colon, prostate and pancreas	H&E	240	2021	[190]	https://paip2021.grand-challenge.org/Home/
WSSS4LUAD	Challenge	Prediction of three tissue types	Lung	H&E	87	2021	[191]	https://wsss4luad.grand-challenge.org/WSSS4LUAD/
CoNIC	Challenge	Segmentation and classification of nuclear and prediction of cellular composition	Colon	H&E	4981	2022	[192]	https://conic-challenge.grand-challenge.org/Home/

^aThe reference numbers are consistent with the text.

More effective treatments in modern medicine are in need of accurate tests tailored for various patients instead of common one-size-fits-all tests. In conventional treatment, pathological sections are simply viewed under a microscope to make initial conclusions. Results are very dependent on the material quality and doctors’ skills, which tend to be lacking in areas with poor medical resources.

As we have discussed in the application of unimodal digital pathology, computers are attempting to replace manual operation. AI has different research priorities with different phases of medical tasks. The information preparation stage can simplify data processing steps, the diagnosis stage is able to replace doctors’ task of differential diagnosis, and the post-treatment stage proposes new criteria to refine treatment goals. AI helps doctors take digitized sections as inputs directly into the computer. The computer then automatically optimizes images’ quality and allows algorithms to assist the less experienced doctor in making a qualitative diagnosis. More accurate results help patients to gain early access to medical treatments, thus preventing further deterioration in unknown cases. Unimodal digital pathology makes traditional medical work faster and more accurate.

Unimodal pathology image processing and information extraction form the basis for multimodal applications. Multimodal pathology no longer requires the computer to simulate the behavior of physicians. In other words, it is more reliant on the complementary information in precision medicine. Genetic sequencing and special stained images are necessary to confirm the diagnosis of diseases such as cancer. In addition to resource-poor regions, the full set of tests is inaccessible to patients with limited physical and financial conditions. We discuss two modality fusion approaches in the context of multimodal digital pathology applications. One is to input pathology images and output another modality (sequences, images in other formats). This approach no longer simply summarizes the information from pathology images, but enables a transformation of the information. It shows that patients without medical conditions can also have equal treatment with other indicators. Another way in which multiple modalities are fed into the computer model at the same time achieves a correspondence between different data. For example, the mutation status of each lesion can be seen in images. This allows precision medicine to be more precise than just the individual but also the lesion. Both approaches try to uncover associations between changes in the body in different dimensions brought about by the disease. AI in multimodal or multi-task research solves the problem of the mismatching multiple information sources and inadequate mining of single information sources.

It is evident that advances in the computer vision field contributed to the new diagnostics methods that emerged with the fusion of pathology images and AI [193]. Researchers have been extending the applicability of models from several perspectives. Most approaches transfer pre-trained models on natural images to their specific research problems. But there are many differences between the two kinds of images. The objects of interest in natural images are usually large subjects such as people, animals and cars, while pathological images are concerned with tiny objects, such as cell nuclei and tissues. The generality of the abstract representation of both images in the hidden space is still problematic. Furthermore, it is not clear how many cells contained in a single image can show biological significance. Directly slicing pathology images to the commonly used 224*224 pixel size may result in information loss. Many studies have used class activation mapping to visualize model results to prove that valuable information had been learned; however, some heatmaps are difficult for pathologists to interpret. Thus, it is more reliable to do pre-training with public datasets, like TCGA, or cut patches like cellular connectivity maps and superpixels.

Large orders of magnitude of data from multiple sources also led to the issue of standards for data processing. Digital pathology section databases have achieved a wide collection of researchers in terms of data quantity, but not in terms of quality (class imbalance, artificial contamination, staining differences, etc.). To enhance the generalization of the model, validation experiments on data from multiple sources are necessary. Meanwhile data batch effects [194] place high demands on data preprocessing [195]. For addressing the data noise problem, standardized data preprocessing systems [196, 197] or uniform collection standards should be proposed accordingly. Otherwise, attention needs to be paid to the overfitting problem and the development of highly accurate AI algorithms on small data sets. Another source of noise is the inconsistency in the data annotation process. Secondary review among pathologists can reduce errors, but different countries and hospitals have different classification standards. There is also the challenge of focusing on multimodal research. Image data are ultra-high dimensional and sparse in AI models. But other clinical information can very easily become overwhelming, which makes it difficult to bring out all useful information [198].

There are also a few issues that may be met during the implementation of the algorithm. Focusing on the robustness of the model in testing and false-negative prediction results will help convince doctors the computer’s effectiveness, promoting its clinical application. In addition, not all hospitals have large GPU servers to compute the results [199] and scaling down the hardware requirements of the model is a necessary task.

In recent years, the research hotspot of digital pathology has shifted from unimodal bi-classification to multimodal multitasking, which has brought many benefits to doctors’ daily work. The automatic detection of identification at the cellular level and tissue level not only saves time in labeling molecular makers experiments, but also reduces the probability of misidentification by physicians in small regions. The direct application to the diagnosis process is semi- or fully automated by replacing the pathologist with a model to complete the process of identifying diagnostic results from images and reducing the stress of repetitive work for the doctors. Diagnosis-related tasks shorten the time to generate pathology reports, and prognosis-related tasks bring more opportunities to treat patients correctly and in a timely manner. The introduction of multimodal data taps into the association between external phenotypes and internal expressions, broadening the understanding of life. Many studies have demonstrated the improvements that AI can bring by comparing the accuracy between doctors and computers or the efficiency of doctors’ diagnosis with or without computer assistance [79, 200–203]. Building a universal model for all problems is not realistic. What AI can do is to reduce the burden on physicians and avoid the lack of treatment due to the lack of resources.

Conclusion

Digital pathology, combined with artificial intelligence, is one of the most promising fields for the delivery of precision medicine. The research problem is gradually switching from the understanding of unimodal data to the cooperation of multimodal data. The integration and evaluation criteria of digital pathology with electronic medical records, CT, or other clinical data for different diseases are urgently needed to be established, as they are important for the enhancement of the integrated use of medical information. Effective supporting systems can be produced based on each step of the complete process of doctors and researchers’ collaboration (scientific problem identification, clinical application criteria setting). Many people fear that this will lead to a regression in doctors’ skills, but, when computers replace basic repetitive tasks, doctors can have more time to focus on solving difficult problems. Currently, hospitals still lack the supporting landing facilities regarding automated pathology diagnosis and treatment systems. However, we cannot stop the research on digital pathology to promote the development of precision medicine. AI-based digital pathology delivers on detection, quantification, classification (such as tumor subtype), prognosis (in terms of combining clinical and genomics information) and prediction. With multiple modalities fusion analysis, artificial intelligence approaches will improve quality and efficiency and transforms pathology data into clinically actionable knowledge, and we can make all this information accessible to precision medicine.

Key Points

• The availability of digitizing whole-slide images in various diseases has led to the advent of artificial intelligence (AI) and machine learning tools in digital pathology.

• In this review, we critically summarized all kinds of Al-based computational approaches for digital pathology.

• The review showed the role that deep learning technology development has in medical assistance and explored the opportunities and challenges of AI.

Author Biographies

Yixuan Qiao is a PhD candidate of the Research Center for Ubiquitous Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences and the University of Chinese Academy of Sciences. Her research interests include multi-modal deep learning.

Lianhe Zhao is a post-doc at the Research Center for Ubiquitous Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, University of Chinese Academy of Sciences. Her research interests include bioinformatics and deep learning.

Chunlong Luo is a PhD candidate of the Institute of Computing Technology, Chinese Academy of Sciences and the University of Chinese Academy of Sciences. His research interests include medical image analysis.

Yufan Luo is a PhD candidate of the Institute of Computing Technology, Chinese Academy of Sciences and the University of Chinese Academy of Sciences. His research interests include deep learning.

Yang Wu is an assistant investigator at the Institute of Computing Technology, Chinese Academy of Sciences. Her research interests include bioinformatics.

Shengtong Li is an undergraduate student at Massachusetts Institute of Technology. Her research interest include bioinformatics and artificial intelligence.

Dechao Bu is a an associate researcher at the Institute of Computing Technology, Chinese Academy of Sciences. His research interests include bioinformatics and cancer immunology.

Yi Zhao is a professor at the Research Center for Ubiquitous Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences; University of Chinese Academy of Sciences and Shandong First Medical University & Shandong Academy of Medical Sciences. His research interests include bioinformatics and deep learning.

Authors’ contributions

Yixuan Qiao: Data Curation, Methodology, Writing - Original Draft, Writing - Review & Editing; Lianhe Zhao: Methodology, Writing - Original Draft, Writing - Review & Editing; Chunlong Luo: Investigation; Yufan Luo: Investigation; Yang Wu: Investigation; ShengTong Li: Writing - Review & Editing; Dechao Bu: Writing; Yi Zhao: Conceptualization, Methodology, Supervision, Project administration.

Funding

The National Key R&D Program of China (2021YFC2500203); The Strategic Priority Research Program of the Chinese Academy of Sciences (XDA16021400); Zhejiang Provincial Natural Science Foundation of China (LY20C060001); Innovation Fund of Institute of Computing and Technology, CAS (E161080); Zhejiang Provincial Research Center for Cancer Intelligent Diagnosis and Molecular Technology (JBZX-202003); Talent funding of Shandong First Medical University & Shandong Academy of Medical Sciences (922-001003130 185RC).

Competing interests

All the authors declare no competing financial or non-financial interests.

Data availability

The high-definition charts have been made publicly available here: http://bioinfo.org/mdp/. The data used to generate the figures and tables are publicly available to researchers through the National Library of Medicine. Additional inquiries are welcome to the corresponding author.