AI Applications in Renal Pathology

Abstract

The explosive growth of artificial intelligence (AI) technologies, especially deep learning methods, has been translated at revolutionary speed to efforts in AI-assisted healthcare. New applications of AI to renal pathology have recently become available, driven by the successful AI deployments in digital pathology. However, synergetic developments of renal pathology and AI require close interdisciplinary collaborations between computer scientists and renal pathologists. Computer scientists should understand that not every AI innovation is translatable to renal pathology, while renal pathologists should capture high-level principles of the relevant AI technologies. Herein, we provide an integrated review on current and possible future applications in AI-assisted renal pathology, by including perspectives from computer scientists and renal pathologists. First, the standard stages, from data collection to analysis, in full-stack AI-assisted renal pathology studies are reviewed. Second, representative renal pathology-optimized AI techniques are introduced. Last, we review current clinical AI applications, as well as promising future applications with the recent advances in AI.

1. Introduction

Artificial intelligence (AI) represents any technique that enables computers to mimic human intelligence^{1, 2}. The explosive development of its sub-discipline of Machine Learning³, especially Deep Learning⁴, has led to paradigm shifts in our daily routines ranging from computer vision⁵, natural language processing⁶, social media⁷, autonomous driving⁸ and healthcare⁹. AI, for years, has generated enthusiasm in healthcare for its potential to support clinical diagnosis, patient management, and therapeutic planning^{9, 10}. The increasing availability of enormous data (e.g., imaging, omics data, electronic medical record, clinical outcomes, etc.) and unprecedented computing power (e.g., big data management, high-performance computing, high-end networking, etc.) are driving large-scale and data-driven solutions to a variety of imaging-related fields, including, but not limited to, radiology¹¹, surgical intervention¹², oncology¹³ and digital pathology^{14, 15}.

Renal pathology is a subspecialty of general pathology that characterizes medical and transplant kidney diseases based on diagnostic assessment from light microscopy (LM), immunofluorescence (IF) microscopy, and electron microscopy (EM). Renal pathologists examine the renal biopsy to render a definitive clinicopathologic diagnosis, closely working with nephrologists. New applications of AI to renal pathology have recently become available, driven by the successful AI deployments in digital pathology¹⁶ and the more widespread use of digital diagnostic imaging, in particular, the recent advances in whole slide imaging (WSI) technology¹⁷.

Machine learning is typically regarded as a subset of AI, which focuses on empowering computer systems with the ability to “learn from data”³, rather than mimicking other aspects of human intelligence, such as body motion or human emotion. Since most AI techniques in renal pathology are developed to learn essential knowledge from data, AI and machine learning are often used as interchangeable terms in this context.

Deep learning is the latest breakthrough of AI/machine learning, which offers an unparalleled ability to efficiently manage patients, accelerate diagnosis, and guide treatments¹⁸. The performance of a deep learning model is typically constantly improved when more training examples are provided, while traditional AI can easily hit a learning plateau without further improvements¹⁹. Interestingly, the core computation algorithms of deep learning, such as artificial neural networks (ANN)^{20, 21} and convolutional neural networks (CNN), were proposed in the 1990s or earlier²². The exceptional learning capability of deep learning is largely driven by recent advances in computing software²³ and hardware²⁴, supporting deeper layers²⁵, more parameters²⁶ and large-scale training data²⁷. With unprecedented learning capability, AI can learn extensive medical knowledge from large-scale renal pathology data, to support clinical decisions and reduce diagnostic and therapeutic errors in clinical practice.

Renal pathologists play a critical role in the work-up of patients to render a correct biopsy diagnosis, with prognostic assessment, activity and chronicity grading and give accurate information to nephrologists to guide their choice of effective therapy. However, these tasks require both task-specific machine learning techniques for AI and deep clinical understanding. Therefore, the synergetic developments of renal pathology and AI require close interdisciplinary collaborations between computer scientists, nephrologists and renal pathologists¹⁶. To design an optimal AI algorithm for renal pathology, computer scientists should comprehend the current practice scenarios and barriers in renal pathology, while renal pathologists should understand high-level principles of the different AI technologies. Herein, we review the recent synergetic advances in AI-assisted renal pathology, with integrated perspectives from both computer scientists and renal pathologists. First, we review the standard stages in full-stack AI-assisted renal pathology studies, from data collection to analysis. Second, we introduce representative renal pathology-optimized AI techniques beyond using off-the-shelf algorithms. Last, we review the current clinical AI applications and promising future applications with recent advances in AI. The goal of this review is to provide an integrated vision of AI-assisted renal pathology by combining engineering and clinical perspectives, across research design, algorithm developments and clinical applications.

2. Standard Workflow of AI-assisted Renal Pathology

The standard workflow of performing an AI-assisted renal pathology study typically consists of four stages: (1) problem definition, (2) data collection and annotation, (3) AI development and training and (4) data fusion and analysis. An example of the workflow for a biopsy using AI integrated with clinical information and genotype is illustrated in Fig. 1. Briefly, AI can be used to count the percentage of glomeruli with global glomerulosclerosis (GS), with the fine-grained classification of these globally sclerotic glomeruli as obsolescent, solidified or disappearing. Obsolescent GS is associated with normal aging, while solidified/disappearing GS is characteristic of hypertension-associated disease²⁸. The upper panel in Fig. 1 shows a representative pipeline of AI-assisted GS characterization, using a WSI image acquired from anAfrica American man with a biopsy diagnosis of hypertension-associated nephropathy. The image features can be further aggregated with genetic and clinical features by another AI fusion algorithm, shown in this example as a computer-aided diagnosis system (Fig. 1). For this patient, the morphologic diagnosis is arterionephrosclerosis, with 4/19 glomeruli GS in a solidified pattern and one disappearing GS, the end stage of a solidified glomerulus as it becomes contiguous with the fibrotic interstitium, with additional characteristic vascular lesions. Here we have integrated these quantitative AI-derived assessments of morphologic lesions, with the patient’s ethnicity (Africa American), and genotype, with two ApoL1 high risk variants.

Figure 1.

A representative workflow of AI-assisted renal pathology. In the upper panel, AI is used to count the percentage of glomeruli with global glomerulosclerosis (GS), with the fine-grained classification of obsolescent, solidified or disappearing types. Then, the image features are further aggregated with genetic and clinical features by another AI fusion algorithm. The lower left panel shows the representative tasks of AI-assisted quantification, while the lower right panel shows the representation modalities for AI-based multi-modal data fusion.

2.1. Problem Definition

AI projects in renal pathology are commonly driven by specific problems from scientific research or clinical practice. There are two main motivations for employing AI in renal pathology studies. The first motivation is to help pathologists accomplish specific tasks with less manual efforts, higher efficiency and even better performance^29–32. A straightforward example is to compute the percentage of sclerotic glomeruli in a biopsy. The second motivation is to offer new capabilities to investigate problems that are unscalable or impractical for pathologists. For example, AI can “visualize” all voxels inside a 3D volume simultaneously, using advanced 3D microscopy imaging^33–35, while the human vision is limited by understanding serial 2D sections or projections.

2.2. Data Collection and Annotation

Once the problem is defined, the next step is to collect corresponding renal tissue samples, including standard tissue sections, digitizing metadata, digital imaging of stained tissue sections, and quality control^{36, 37}. The learning strategies can be classified as three types: supervised learning, semi-supervised learning, and unsupervised learning³⁸. The key differences are whether human annotations are required for training AI to understand the digitized images. The “supervision” can be simply interpreted as annotations in AI. Briefly, the supervised learning methods utilize well-annotated data to train AI systems, while the unsupervised learning methods encourage the AI systems to learn clinically relevant information from unlabeled data. The semi-supervised learning strategy utilizes both annotated and unlabeled data. Supervised learning is still the most prevalent design of AI-assisted solutions for renal pathology, due to its superior performance of distinguishing subtle morphological differences that are difficult to recognize by unsupervised self-learning. Moreover, even for the unsupervised learning algorithms, annotated data are still needed for validation purposes.

Annotations in renal pathology are typically achieved from manual efforts from pathologists^{39, 40}. The annotations train the AI algorithms to understand different levels of image concepts⁴¹. For example, the annotations can be offered at image-level (for image classification), region-level (for object detection) or pixel-level (for image segmentation). However, inadequate annotations are one major bottleneck in training deep learning algorithms in renal pathology⁴², as precise annotations require extensive clinical knowledges from trained practitioners, rather than using crowdsourcing annotations from lay person annotators. For renal pathologists, it is not only tedious but unscalable to annotate large-scale data for training deep learning algorithms, with the additional challenges of inter-rater variabilities and protocol consistency. Fortunately, more feasible AI-assisted annotation algorithms have been proposed to reduce the manual workload for human annotators, such as human-in-the-loop annotation or AI-based interactive annotation^{37, 38}. Meanwhile, a collaborative data curation pipeline is appealing to distribute the workload between clinical experts and lay annotators.

2.3. AI Development and Training

In healthcare, AI refers to computer techniques that mimic human physicians’ decisions and actions. Most AI algorithms in renal pathology have thus been invented to understand biopsies by simulating the behaviors of renal pathologists, including object detection (e.g., identify glomeruli, tubules, arteries), classification (e.g., phenotyping, diagnosis, therapeutic planning), segmentation (e.g., trace contours of objects), synthesis (e.g., staining), and rendering diagnosis (e.g., fuse information from multiple modalities or resources)^{29–31, 43–53}. Historically, such learning algorithms were built upon traditional machine learning approaches, by capturing human-designed imaging features among digital microscope images (e.g., color, edge, shape). However, such methods were usually limited by the lack of generalizability and completeness of the hand-crafted features. Deep learning-based AI algorithms are now increasingly used to learn discriminative features from large-scale training images^{16, 32, 54}. In the following sections, we introduce four standard techniques in AI-assisted renal pathology, which utilize data-driven features in different ways.

2.3.1. Classification

Image classification is a fundamental task in computer vision to provide a specific label for each image⁴. Specific to renal pathology, the label can be a phenotype, a diagnosis, or a therapeutic plan. Recently, deep learning-based AI has been widely used for different classification tasks in renal pathology, which typically consist of two parts: feature extraction and diagnosis^{43, 44, 55–58}. In contrast to previous AI algorithms, deep learning algorithms can extract multi-scale features (from local to global), which benefit from the deeper (more layers) network design. In a canonical CNN design⁵⁹, the initial layers focus on extracting local image features (e.g., color, shape, and morphology of a nucleus), while the deeper layers extract global image features (e.g., staining, shape, and morphology of a glomerulus). Then, a final diagnosis is typically made by aggregating the multi-scale features using a multi-layered perceptron (MLP) network⁶⁰. The process is similar to rendering a biopsy interpretation by a renal pathologist, where the pathologist might first screen the local and global image features, and then aggregate all evidences to achieve a final diagnosis.

2.3.2. Detection

Detection is regarded as a higher-level visual perception compared with classification, since it offers a region-level classification of each detected object (e.g., a glomerulus within a rectangular bounding box), instead of image-level characterization in classification^{45–48, 61, 62}. In AI implementation, a feature extraction step is first performed to obtain local and global features, which is similar to the feature extractor used in classification. Next, key landmarks of each object (e.g., corners of bounding boxes) are achieved by aggregating extracted features from multi-scale data. Last, a classification sub-network is employed to provide a label for each object, within each bounding box. Using this design, many deep learning-based detection algorithms (e.g., Faster-RCNN⁶³) have outperformed traditional model-based approaches in terms of accuracy and efficiency^{64, 65}. Some AI approaches have even provided real-time capability of object detections, which have led to a new line of products, called the smart microscope⁶⁶. The smart microscope might change the current paradigm of inspecting biopsies, as it offers real-time support to pathologists analyzing samples, detecting objects, rendering diagnosis and generating reports⁶⁶. Using the smart microscope, AI quantifications are conducted when pathologists perform routine pathology workflow, such as visual inspection with a microscope, without extra effort.

2.3.3. Segmentation

Image segmentation is the most detailed quantification of an image since each pixel is assigned with a label, as opposed to image-level or region-level characterization^{31, 43, 49–53, 67–74}. Technically, the segmentation task can be regarded as an extreme case of classification or detection, where the classification is performed on each pixel. Pixel-level quantification provides us the precise spatial and quantitative measurements of objects at different scales. However, extensive pixel-level classification is time consuming. To speed up such processes, fully convolutional network (FCN)-based⁷⁵ approaches were proposed to perform classification on all pixels simultaneously. Recently, to achieve instance object level segmentation, FCN networks have been incorporated with detection networks to form holistic instance segmentation approaches (e.g., Mask-RCNN⁷⁶). Although segmentation is not typically involved in routine pathology workflow, this approach provides a promising direction to replace the current semi-quantitative phenotyping methods with fully-quantitative solutions.

2.3.4. Synthesis

Image synthesis is one class of AI algorithms to synthesize the image from one appearance to another, which is typically referred to as generative adversarial networks (GAN)^77–79. Instead of providing labels for image, region, or pixel, the aim of image synthesis is to generate a new “real look” image from another existing image, or even from a random noise or a prior shape. In renal pathology, image synthesis has been used for virtual staining, which transfers the appearance of a digitized tissue from one stain to another^{78, 80–84}. As the synthetic images are generated digitally rather than physically, there is almost no extra imaging cost to generate large-scale new data as sample size augmentation to train a deep neural network. For instance, GAN technologies were utilized to develop a generalizable glomerular segmentation model⁸³, without the need of large databases with different staining patterns⁸⁵. Synthetic images have been used in next-generation renal histomorphometry³², to perform label-free object classification^{79, 86}, detection⁸⁷ and segmentation^88–90 from renal pathological images. The summarization of AI methods in renal pathology is provided in Table 1.

Table 1.

AI methods in renal pathology

Methodology	Authors	Journal	Year	Tools	Applied task	Number of WSIs or cases
Classification	Ginley et al. [43]	J. Am. Soc. Nephrol.	2019	RNN, CNN	Glomerulus classification, Multiclass segmentation of renal morphology	54 WSIs, human 25 WSIs, mouse
	Uchino et al. [44]	International Journal of Medical Informatics	2020	CNN	Glomerulus classification	283 cases, human
	Ginley et al. [55]	Proc SPIE Int Soc Opt Eng	2020	RNN, CNN	Glomerulus segmentation, Recurrent biopsy classification, Glomerular component analysis	82 WSIs (65 cases), human
	Zee et al. [56]	Arch Pathol Lab Med	2018	Scoring System	Glomerulus morphology assessment	236 WSIs (glomeruli), human
	Chagas et al. [57]	Artif Intell Med.	2020	CNN, SVM	Glomerulus hypercellularity	811 WSIs (glomeruli), human
	Ledbetter et al. [58]	arXiv.	2017	CNN	Prediction of kidney function	80 cases, human
Detection	Temerinac-Ott et al. [45]	Proc Int Symp Image Signal Process Anal	2017	CNN, HOG	Glomerulus detection	6 cases, human
	Bukowy et al. [46]	J Am Soc Nephrol	2018	CNN	Glomerulus detection	87 WSIs, human
	Simon et al. [47]	Sci Rep	2018	SVM, CNN	Glomerulus detection	15 WSIs, mouse 25 WSIs, human
	Marée et al. [48]	Proc IEEE Int Symp Biomed Imaging	2016	Commercial software	Glomerulus detection	200 WSIs, human
	Kato et al. [61]	Bmc Bioinformatics	2015	SVM, HOG	Glomerulus detection	20 WSIs, mouse
	Gallego et al. [62]	J Imaging	2018	CNN	Glomerulus detection, Glomerulus classification	108 WSIs, human
	Yang et al. [91]	Med Image Comput Comput Assist Interv	2020	CNN	Glomerulus detection	42 WSIs, human
Segmentation	Hermsen et al. [31]	J Am Soc Nephrol	2019	CNN	Multiclass segmentation of nephrectomy and transplant biopsies	50 WSIs, human
	Ginley et al. [43]	J Am Soc Nephrol	2019	RNN, CNN	Glomerulus segmentation, Multiclass segmentation of renal morphology	54 WSIs, human 25 WSIs, mouse
	Bueno et al. [49]	Comput Methods Programs Biomed	2020	CNN	Glomerulus segmentation, Glomerulus classification	47 WSIs, human
	Kannan et al. [50]	Kidney Int Rep	2019	CNN	Glomerulus segmentation	275 WSIs (171 cases), human
	Gadermayr et al. [51]	arXiv preprint	2017	CNN	Glomerulus segmentation	24 WSIs, mouse
	Gadermayr et al. [52]	Comput Biol Med	2017	SVM	Glomerulus detection, Glomerulus segmentation	8 WSIs from mouse kidneys
	Ginley et al. [53]	arXiv preprint	2020	CNN	Multiclass segmentation of renal morphology	65 cases, human
	Santo et al. [67]	Proc SPIE Int Soc Opt Eng	2020	Feature engineering	LN biopsies segmentation	21 WSIs, human
	Gupta et al. [68]	Proc Machine Learning Res	2019	CNN	Glomerulus segmentation	22 WSIs, mouse
	Ginley et al. [69]	J Med Imaging	2017	Gabor Filterbank, Statistical Testing	Glomerulus segmentation	1000 images, mouse
	Sarder et al. [70]	Proc SPIE Int Soc Opt Eng	2020	Gabor Filterbank, Statistical Testing	Glomerulus segmentation	15 WSIs, mouse
	Tey et al. [71]	Comput Methods Programs Biomed	2018	Alternating Decision Trees, SVM	Glomerulus segmentation	286 WSIs (70 cases), human
	Altini et al. [72]	Electronics	2020	CNN	Glomerulus segmentation, Glomerulus classification	26 WSIs (19 cases), human
	Bouteldja et al. [73]	J Am Soc Nephrol	2020	CNN	Kidney tissue segmentation	168 WSIs, mouse
	Jha et al. [92]	J Med Imaging	2020	CNN	Glomerulus segmentation	1454 images, human
	Jayapandian et al. [74]	Kidney Int	2020	CNN	Kidney tissue segmentation	459 WSIs (125 cases), human
Synthesis	Murali et al. [78]	Proc SPIE Int Soc Opt Eng	2020	GAN	Renal histopathology images synthesis	20K images, human
	Lutnick et al. [79]	Proc SPIE Int Soc Opt Eng.	2020	VAE-GAN	Glomerulus images synthetic, Glomerulus classification	59930 images, human 27508 images, mouse
	Zhang et al. [81]	Light: Sci & Appl	2020	GAN	Digital stain	12 WSIs, human
	Gadermayr et al. [82]	Med Image Comput Comput Assist Interv	2018	GAN, CNN	Digital stain, Glomerulus segmentation	59 WSIs, mouse
	Gadermayr et al. [83]	IEEE Trans Med Imaging	2019	GAN, CNN	Digital stain, Glomerulus segmentation	41 WSIs, mouse
	de Bel et al. [84]	Med Imaging Deep Learning	2020	GAN	Digital stain	64 WSIs, human
	Wu et al. [86]	Proc Conf AAAI Artif Intell	2019	GAN, CNN	Digital stain, Glomerulus classification	209 cases, human
	Gadermayr et al. [88]	Med Imaging Deep Learning	2019	GAN	Glomerulus synthetic segmentation	23 WSIs, mouse
	Gupta et al. [89]	Med Image Comput Comput Assist Interv	2019	GAN, CNN	Digital stain, Glomerulus segmentation	59 WSIs, mouse
	Mei et al. [90]	IEEE ICASSP	2019	GAN, CNN	Digital stain, Glomerulus segmentation	819 images, human

^*CNN = convolutional neural network, GAN = generative adversarial network, VAE = variational autoencoder, SVM = support vector machine, RNN = recurrent neural network, HOG = histogram of oriented gradients.

^*WSI = whole slide imaging, LN = lupus nephritis.

2.4. AI-based Data Fusion and Analysis

The same lesion by light microscopy may be due to varied underlying diseases, e.g. nodular glomerulosclerosis in diabetic nephropathy vs light chain deposition disease or other entities. Renal pathologists thus integrate various data to make diagnoses, including light microscopy, immunofluorescence, electron microscopy and clinical setting, and even specific genetic testing^{93, 94}. AI algorithms have led to successful clinical applications by aggregating heterogeneous healthcare data, advanced by fusing data with different dimensions and representations in a fully quantitative and data-driven manner⁹⁵. AI-assisted multimodal data fusion and analysis algorithms can also play important roles in renal pathology. One AI-based multi-modal data fusion strategy is “late fusion”⁹⁶, which is similar to human decision making. Briefly, different AI networks are typically employed to extract the useful knowledge (features) from different modalities⁹⁵. Then, such features are concatenated and aggregated (e.g., using MLP⁶⁰) into final outcomes, analogous to decision-making from comprehensive clinical assessments integrated with various tests by physicians. AI-based approaches can also perform “early fusion”⁹⁶, such as conducting pixel-wise alignment and integration between different imaging modalities⁹⁷, or mapping single-cell RNA-seq with pathological images⁹⁸.

3. Renal Pathology Optimized Deep Learning

Currently, a majority of AI-assisted renal pathology studies utilize off-the-shelf deep learning algorithms that have been developed in the computer vision community, such as Faster-RCNN⁶³, Mask-RCNN⁷⁶, U-Net⁹⁹ etc. The rationale is that both renal pathology and computer vision fields deal with similar targets -- images. Therefore, it is a natural and reasonable choice to apply prevalent algorithms in computer vision to renal pathology. However, computer vision-oriented AI algorithms are not necessarily optimized for biomedical objects. Therefore, deep learning optimized for renal pathology is appealing to further boost the performance of AI in renal pathology. In this section, we review cases of such optimized deep learning algorithms, including optimized data representation, optimized algorithm and optimized dimensionality (Fig. 2).

Figure 2.

Illustration of AI algorithms optimized for renal pathology. This figure compares off-the-shelf AI algorithms in computer vision and AI algorithms optimized for renal pathology. The optimized AI algorithms have been developed to address unique challenges in renal pathology, including optimized data representation, optimized algorithm and optimized dimensionality.

3.1. Optimized Representation

Representation means the format in which data are stored, processed, and transmitted to efficiently convey quantitative information. In AI-assisted renal pathology, the “computer vision” oriented representation is not necessarily optimized for biomedical objects. For instance, the widely used rectangular encoder of representing glomerular objects yielded many false positive due to the extensive variations in e.g. number of diverse pathogenetic mechanisms¹⁰⁰. Therefore, Kato et al. proposed a new renal pathology optimized descriptor to perform comprehensive glomerular detection⁶¹. Similarly, Simon et al. introduced a circular neighborhood image features extractor, which achieved superior detection performance of spherical shape glomeruli⁴⁷. In the deep learning era, the computer vision oriented bounding boxes representation is employed in most glomerular detection studies. However, the bounding box might not be the most effective representation for the ball-shaped glomeruli, which yields less reproducibility with image rotation, even for the same tissue. Recently, we proposed a new circle representation⁹¹ as an optimized data representation for glomerular detection, which notably improved the detection performance.

3.2. Optimized Algorithm

An obvious obstacle that limits the performance of off-the-shelf deep learning algorithms in renal pathology applications is the fundamental difference in image resolution. In computer vision, even a high-resolution image (e.g., “4K” resolution, 3840×2160 pixels) is relatively small compared with “Gigapixel” WSI (the resolution can be higher than 32,768×32,768 pixels). Therefore, the scale of AI algorithms in computer vision might not be optimized for renal pathology. To bridge the gap of resolution, use of tiling and down-sampling are typically inevitable when deploying standard computer vision AI approaches to renal pathology. For example, a single glomerulus in a 40× WSI can have a resolution of more than 1000×1000 pixels, yet its feature maps are compressed to 28×28 pixels in the standard Mask-RCNN segmentation approach⁷⁶. Therefore, AI algorithms optimized for renal pathology are required to address the unique challenges in high resolution diagnostic imaging. In this regard, Gadermayr et al. proposed a cascade segmentation method to first segment the glomeruli on the low resolution image using a shallower CNN, and then segment the high resolution glomerular regions with a deeper CNN⁵¹. Recently, we proposed a “detect-then-segment” instance segmentation approach⁹² to balance the computation efficiency without significantly compromising image resolution for analysis of renal pathology WSI.

3.3. Optimized Dimensionality

Dimensionality is a critical property when designing a proper strategy of understanding any types of imaging data. In computer vision, the input images for AI are typically 2D natural images, acquired from digital cameras. In radiology, the input images are typically 3D volumes, acquired from computed tomography (CT) or magnetic resonance imaging (MRI). By contrast, in renal pathology, a set of serial sections from the same tissue can be acquired as a stack of images from WSI. However, the stacks of 2D images are not aligned perfectly as they are for radiological images, due to large tissue deformation, missing areas of tissues, and artifacts from WSI. Therefore, such images are typically processed as independent 2D images when AI algorithms are deployed. However, if the 3D context of glomeruli can be quantified holistically from all 2D WSI serial sections, the robustness and reproducibility of glomerular phenotyping would be improved significantly. The advantages of 3D quantification have been elucidated using advanced microscopy imaging techniques, such as confocal microcopy¹⁰¹, electron microscopy^{35, 102, 103} and automated tape collecting microtome¹⁰⁴. To further enable the 3D quantification of glomeruli for serial WSI images, we proposed an AI method¹⁰⁵ to perform 3D identification and association of glomeruli. Using this method, the time needed to quantify atubular glomeruli, (i.e. those without patent connection through the proximal tubule outlet) in a 3D stack of whole mouse kidney sections form one kidney was reduced from 30 hours (done maually by humans) to 30 mins.

4. Clinical AI Applications in Renal Pathology

AI-assisted pathologists demonstrated higher accuracy than either AI alone or the pathologist alone in detecting cancer micrometastases¹⁰⁶. New molecular technologies and the big data revolution are paving the way to a new era in renal pathology. In this section, the applications of AI in renal pathology are reviewed, from integrated pathology to precision medicine (Fig. 3).

Figure 3.

AI applications in renal pathology and nephrology. An example of applications of AI in renal pathology is presented, from integrated pathology to precision medicine.

4.1. From Morphologic to Integrated Pathology

Current histopathological categories used in renal pathology are irrespective of different molecular mechanisms. A diagnosis is made based on the following flow: recognition of injury patterns, grouping/clustering of patterns and integration with clinical data. AI could assist in pattern recognition and clustering. By using CNN technology, 10 different types of renal structures could be segmented, including e.g. glomeruli, Bowman’s capsule, proximal tubules, distal tubules, arteries and interstitium³¹. Recurrent neural network (RNN) technology was successfully used to classify diabetic nephropathy⁴³. Adversarial learning could play a potentially important role in robust AI, especially considering disease variability and the challenges of rare disease. For instance, if the AI system has self-awareness of uncertainty using adversarial learning as a posterior probability, the computer would request help from human experts to reduce inaccurate diagnoses and reduce the workload of pathologists by letting AI figure out the “easy” cases. In addition, AI could distill more useful information from gigapixel images than the human eye, which means more sub-visual morphology could be discovered by AI. An algorithm based on the auto-encoder network, which is an unsupervised deep learning technique that learns how to efficiently extract (encode) intrinsic features from high-dimensional images, will be more stable and effective for such efforts¹⁰⁷. Therefore, morphology profiles, named morphomics, can be expanded by AI. The identification of previously underrecognized patterns, and their potential correlations with outcomes, treatment responses or even specific injury pathways, will change paradigms for renal pathology categorization. Moreover, the deep Cox model¹⁰⁸ and multimodal learning could be used for prediction of prognosis.

AI also could contribute to a move from morphologic classification to mechanism-based disease definition. One major concern of morphology-based diagnosis is that one disease can present different patterns and one injury pattern can be induced by different diseases. IgA nephropathy shows a broad range of glomerular changes by light microscopy, from normal or just mesangial cell proliferation to sclerosis or even crescents, while cellular crescents could be related to e.g. anti-GBM antibody-mediated disease or ANCA vasculitis. Morphology-based disease definitions/classifications have as a goal to show some clinical relevance. The Columbia classification of FSGS has distinctive correlation to clinical features and outcomes, although the etiologies may be quite varied, with potential added insights providing better personalized treatment approaches¹⁰⁹. On the other hand, the old classification of membranoproliferative glomerulonephritis (MPGN) based on morphology in EM did not allow recognition of specific etiologies underlying the lesions. The current MPGN classification adding immunofluorescence findings allows recognition and distinction of MPGN lesions driven by complement dysregulation vs. immune complexes or monoclonal immunoglobulin deposition, which significantly fits in implication for therapy. Diseases have molecular signatures, which can potentially be detected by genomic, epigenomic, proteomic and/or metabolomic profiles. Morphologic changes, assessed and aided by sub-visual signals detectable by AI, could be linked to these molecular features, to achieve more accurate, comprehensive, and clinically relevant pathology diagnosis/classification.

This integrated pathology approach will require matching and clustering multi-dimensional data (morphomics, molecular omics, and clinical data) (Fig. 3). Linear or non-linear data transformations are widely used to cluster. If data have a complex structure, these techniques would be unsatisfying for clustering.

4.2. From Descriptive to Quantitative Pathology

Although most pathology reports are descriptive, some morphologic scores have been incorporated into clinical practice, such as activity/chronicity indices of lupus nephritis (LN) and MEST-C score of IgA nephropathy. These scores are useful in predicting renal outcomes^{110, 111}. Other parameters, such as fractional interstitial area, average glomerular tuft volume, and cortical density of glomeruli, which only could be measured by morphometry on scanned slides and cannot be precisely achieved with the human eye, correlate with various aspects of renal function¹¹². Quantifying lesions may be time-consuming and have variable reproducibility. For instance, the chronicity index for LN showed a moderate interobserver agreement (intraclass correlation coefficient 0.494) when assessed by five specialized renal pathologists¹¹³. The benefits of transforming from descriptive pathology to quantified lesions are obvious. Image quantification is more objective than description and can overcome the observer’s potential bias. Compared to the current semi-quantitation system often used, e.g. 0–4+, the use of continuous variable quantitation is more sensitive and may be more useful for prediction of outcome. With the common use of WSI, enhancement of computer hardware and server capacity, AI-assisted quantitative analysis of pathological images can be realized. A recent study showed that AI can quantify interstitial fibrosis in renal biopsy specimens better than pathologists¹¹⁴. Thus, AI-assisted pathological quantitation reduces human effort, and can ensure consistency, reproducibility and standardization.

4.3. Towards Precision Medicine

Precision medicine aims to classify patients into mechanistically defined subgroups with different disease susceptibility, prognosis and therapeutic responsiveness. Review of failed clinical trials sometimes shows positive results in subgroups of patients although results were negative in the overall group¹¹⁵. In oncology, disease is now not categorized by tissue of origin, e.g. cancer of the lung, but rather by mutation that can be targeted by a specific drug, e.g. ALK mutation positive. The feasible strategies to tailor an individualized treatment are as follows: clarify the molecular pathology spectrum of a certain disease; develop appropriate drugs for single or multiple spectra; evaluate the spectrum in the individual patient by integrating morphomics, molecular omics, and clinical data; and finally choose treatment drugs to achieve “the right treatment for the right patient”. The combination and analysis of these large-scaled, disparate dimensional datasets are challenging. AI and use of tailored different algorithms could be used for this strategy (Fig. 3).

Deep reinforcement learning is a promising method in precision medicine, since this approach has unique advantages to explore the problem set by itself in an unsupervised manner, to for instance determine optimal chemotherapy drug dosage and optimal scheduling of radiation therapy for cancer treatment^{116, 117}. Deep reinforcement has been used to select optimal image patches for automated immunohistochemical scoring¹¹⁸. One machine learning algorithm LOBICO, a logic optimization for binary input to continuous input, has been successfully applied to merge omics data and identify biomarkers predicting drug response in cancer cells¹¹⁹.

Currently, most AI-based computer-aided diagnosis systems function as black boxes, with limited capability of explanation and reasoning in terms of why the computer system made specific decisions. Recently, a new direction of algorithms, named explainable AI, has emerged to provide diagnostic evidence that can be verified by humans, accelerating deployment of AI in practice¹²⁰. For instance, the AI attentions from a classification task can be visualized using CAM¹²¹, GradCAM¹²² or Lime¹²³ tools, which highlights the regions that the AI algorithm is “looking at” when making decision. Ultimately, with stronger interpretation capability, explainable AI techniques would contribute to discover sub-visual biomarkers, which will play a vital role in the development of therapies, personalized diagnosis and treatment of diseases.

Recent AI-based clustering technologies also provide powerful means to group histopathological images with similar morphological and pathological features across large-scale patient cohorts. One example would be large-scale histopathological image retrieval systems. Using deep texture representation algorithms, such web-based systems could help clinicians to retrieves similar histopathological images from a large database, even with the patients’ diagnoses, treatments, and prognoses as references. The challenges and proposed solutions of AI-based techniques are summarized in Table 2.

Table 2.

Some of the limitations of AI methods and proposed solutions

Methodology	Limitation	Explanation	Proposed Solution
Classification	Class imbalance	The anatomy of interest or pathological tissues only occupy a very small portion of WSI images.	Weight rebalance techniques can be used to assign higher penalties when images in minor classes are misclassified.
Detection	Balance recall and precision	To balance false positive and false negative is a dilemma since only positive samples are typically annotated in a standard object detection task.	To actively mine hard negative samples is an effective solution to enlarge decision margins in detection tasks, to reduce ambiguities.
Segmentation	Annotated data	Pixel-wise annotations for training a segmentation AI are resource intensive for digital renal pathology.	Semi-supervised learning algorithms can utilize a smaller cohort of annotated data and a larger cohort of unannotated data, to enhance segmentation performance.
Synthesis	Hallucination	Hallucination is a major limitation of applying image synthesis. It is challenging to validate whether the synthetic images are biologically plausible.	Instead of directly using synthetic images in diagnosis, synthetic images can be used as an intermediate modality for downstream tasks with transfer learning.
Shared limitations across methodologies	Generalizability	The AI model, trained from specific cohorts or sites, might not generalize well on other cohorts or sites.	Federated learning introduces a new computing solution to train a generalizable deep learning model using heterogenous multi-site datasets in an efficient manner without moving patients’ data outside each site.
	Dimensional issue	Images can be Gigapixel size, leading to fundamental computing challenges when training and deploying models at scale.	Patch-wise training can reduce the input size for an AI model. Parallel computing algorithms with more processor cores can accelerate the processing speed on large images.
	Explainability	Most AI algorithms currently function as “black boxes”, with limited capability of explaining why certain decisions are made.	Attention-based mechanisms and recent explainable AI techniques are promising solutions to provide stronger interpretability of AI models.
	Data fusion	Effective algorithms are needed to aggregate different data including light, immunofluorescence and electron microscopy, with clinical setting, and even specific genetic testing.	Multi-task learning and multi-modality fusion algorithms can normalize and aggregate different feature representations to support holistic clinical decisions with multi-modality data.

5. Summary

AI technologies are playing increasingly important roles in renal pathology. These approaches have a strong potential to shift the current clinical paradigms in terms of accelerating workflow, supporting clinical decision-making and determining personalized treatment plans. However, there are still essential challenges that hinder the broader applications of AI in renal pathology, such as limited annotated data, complicated multi-modal data, and lack of clinical validation. Therefore, more seamless collaboration between pathologists and engineers will leverage further advances in developing next generation AI algorithms in renal pathology, with potential for major impacts in diagnosis and understanding of kidney diseases.