Artificial intelligence driven next-generation renal histomorphometry

Abstract

Purpose of review

Successful integration of artificial intelligence into extant clinical workflows is contingent upon a number of factors including clinician comprehension and interpretation of computer vision. This article discusses how image analysis and machine learning have enabled comprehensive characterization of kidney morphology for development of automated diagnostic and prognostic renal pathology applications.

Recent findings

The primordial digital pathology informatics work employed classical image analysis and machine learning to prognosticate renal disease. Although this classical approach demonstrated tremendous potential, subsequent advancements in hardware technology rendered artificial neural networks ‘(ANNs) the method of choice for machine vision in computational pathology’. Offering rapid and reproducible detection, characterization and classification of kidney morphology, ANNs have facilitated the development of diagnostic and prognostic applications. In addition, modern machine learning with ANNs has revealed novel biomarkers in kidney disease, demonstrating the potential for machine vision to elucidate novel pathologic mechanisms beyond extant clinical knowledge.

Summary

Despite the revolutionary developments potentiated by modern machine learning, several challenges remain, including data quality control and curation, image annotation and ontology, integration of multimodal data and interpretation of machine vision or ‘opening the black box’. Resolution of these challenges will not only revolutionize diagnostic pathology but also pave the way for precision medicine and integration of artificial intelligence in the process of care.

INTRODUCTION

In the age of modern medicine and artificial intelligence integration, computational image analysis and machine learning are revolutionizing diagnostic pathology [1^▪]. The advent of algorithms for automated classification of kidney morphology in digital histopathology has potential to enable computer-aided detection, diagnosis and prognosis of renal disease [2^▪▪]. Integration of such machine learning applications with extant clinical practices will offer rapid and precise patient assessment on the part of the pathologist, with significant implications for patient outcome. Further expansion of computational and machine learning analysis to clinical biometrics and omics data will potentiate patient prognostication and personalized medicine. Beyond research and development of artificial intelligence tools, successful adoption of artificial intelligence in patient care requires physician comprehension and interpretation of computer vision’s methods and results, respectively. The goal of this review is to guide readers through the computational image analysis and machine learning pipelines applied to digital renal histopathology (Fig. 1), while also summarizing published, kidney-centric artificial intelligence tools (Table 1).

FIGURE 1.

Clinical and computational workflow for artificial intelligence driven assessment of digital renal pathology. Following data acquisition and preparation, classical and modern artificial intelligence approaches provide novel insights for patient assessment and medical image analysis.

Table 1.

Kidney-centric artificial intelligence tools

Publication	Journal	Authors	Year	Tools	Applied Task
Computational pathology analysis of tissue microarrays predicts survival of renal clear cell carcinoma patients	Lect. Notes Comput. Sc., MICCAI	Fuchs et al. [3]	2008	Local Binary Patterns (LBP) Random Forest Survival Analysis	Tissue Microarray (TMA) Analysis Nuclei Detection Prognostication
Weakly supervised cell nuclei detection and segmentation on tissue microarrays of renal clear cell carcinoma	Lect. Notes Comput. Sc., DAGM	Fuchs et al. [3]	2008	Edge Detection Morphological Image Processing Support Vector Machine (SVM)	TMA Analysis Nuclei Detection
Computational TMA analysis and cell nucleus classification of renal cell carcinoma	Lect. Notes Comput. Sc., DAGM	Schuffler et al. [4]	2010	Graph-Cut Optimization Feature Engineering SVM	TMA Analysis Nuclei Segmentation Nuclei Classification
Segmental HOG: new descriptor for glomerulus detection in kidney microscopy image	BMC Bioinformatics	Kato et al. [5]	2015	Segmental Histogram of Oriented Gradients SVM	Glomerulus Detection
Unsupervised labelling of glomerular boundaries using Gabor filters and statistical testing in renal histology	J. Med. Imaging	Ginley et al. [6]	2017	Gabor Filterbank Statistical Testing	Glomerulus Segmentation
Multiradial LBP features as a tool for rapid glomerular detection and assessment in whole slide histopathology images	Sci. Rep.	Simon et al. [7]	2018	Multiradial Colour LBP SVM Convolutional Neural Network (CNN)	Glomerulus detection Glomerulus subclassification
Region-based convolutional neural nets for localization of glomeruli in trichrome-stained whole kidney sections.	J. Am. Soc. Nephrol.	Bukowy et al. [8]	2018	Region-based CNN	Glomerulus detection
Association of pathological fibrosis with renal survival using deep neural networks	Kidney Int. Reports	Kolachalama et al. [9▪▪]	2018	CNN	Prediction of clinical phenotype from trichrome image features
Glomerulus classification and detection based on convolutional neural networks	J. Imaging	Gallego et al. [10]	2018	CNN	Glomerulus detection
Segmentation of glomeruli within trichrome images using deep learning	Kidney Int. Reports	Kannan et al. [11▪▪]	2019	CNN	Glomerulus detection Glomerulus classification
An integrated iterative annotation technique for easing neural network training in medical image analysis	Nat. Mach. Intell.	Lutnick et al. [12▪]	2019	CNN - Semantic segmentation	Glomerulus segmentation Multiclass segmentation of renal morphology
Computational segmentation and classification of diabetic glomerulosclerosis	J. Am. Soc. Nephrol.	Ginley et al. [13▪]	2019	Feature engineering Recurrent Neural Network (RNN)	Glomerulus Classification Diabetic nephropathy classification
Deep learning-based histopathologic assessment of kidney tissue	J. Am. Soc. Nephrol.	Hermsen et al. [2▪▪]	2019	CNN - Semantic segmentation	Multiclass segmentation of renal morphology

In recent years, the field of renal pathology has seen a surge in the volume of works applying computational image analysis and machine learning for analysis of benign renal histomorphometry.

THE DATA: DIGITAL RENAL HISTOPATHOLOGY

Terminology

Computational image analysis and machine learning utilize a wealth of technical jargon. To achieve the aims of this review, several terms are bolded and defined throughout. For additional terminology, we recommend Roeder et al.’s Computational Image Analysis Glossary for Biologists [14] and Deo’s Machine Learning in Medicine [15].

The advent of digital pathology

The implementation of telepathology in the 1960s demonstrated the power of data sharing and communication for diagnostic assessment [16]. Later development of whole-slide imaging technologies enabled digitization of pathology slides, establishment of digital pathology platforms and curation of whole-slide image (WSI) repositories. Modern digital pathology capitalizes upon these developments, providing a WSI-based, data-driven environment for pathology research, education and clinical practice [17,18]. Primordial digital pathology informatics efforts employed computational image analysis, feature engineering and classical machine learning to prognosticate survival in patients with renal cellcarcinoma (RCC) [3]. More recent works focus on the extension of these methods to benign processes of the kidney.

Data sources and challenges

Benign renal histopathology data are primarily derived from human kidney biopsies performed in patients with primary renal or allograft dysfunction. To date, exemplary renal pathology repositories featuring comprehensive human data include the Nephrotic Syndrome Network (NEPTUNE) [19] and CureGN [20]. Enlistment of these vast data for computational purposes is ideal. However, these datasets lack ground-truth annotations. As a direct result, all projects investigating kidney morphology for classification or predictive purposes must invest significant time and manpower to generate annotation sets. Depending on the kidney structure of interest, machine learning algorithms require anywhere from hundreds to thousands of examples [21] to achieve optimal performance. This annotation burden [12^▪] is but one of many data-related challenges facing digital pathology research.

Large-scale analysis of digital pathology is further complicated by data heterogeneity. The lack of standardized protocols for sample preparation and digitization results in heterogeneous, multiinstitutional datasets, which feature significant variation in stain and image quality. To mitigate this issue, de Bel et al. [22^▪] recently proposed an innovative solution to the shortage and heterogeneity of kidney data: cycle generative adversarial networks (cycleGANs) for stain transformation and structural segmentation of kidney histopathology. GANs are deep neural network architectures [23], which learn to generate synthetic images resembling authentic input images. In the aforementioned work, the cycleGAN is trained to take input histology images of disparate stain quality and generate images featuring an identical colourspace. The ability to generate synthetic images representative of renal histopathology navigates disparities in tissue preparation. An additional computational tool for quality control of digital pathology data is HistoQC [24^▪▪]. Designed to automatically detect image artefacts and normalize sample stain variation, HistoQC offers a rapid and reproducible alternative to the manual review of WSIs.

DETECTION AND CHARACTERIZATION OF KIDNEY STRUCTURES IN WHOLE-SLIDE IMAGES

Traditionally, diagnostic renal pathology involves examination of kidney biopsies for known structural abnormalities indicative of disease [13^▪]. On the basis of the type and severity of abnormality, pathologists classify biopsies and provide a patient diagnosis. Automation of this process requires the development of artificial intelligence tools, which automatically detect kidney structures in biopsy WSIs. To achieve this end, engineers employ computer vision: the scientific field that explores how computers can achieve high-level understanding from digital images [25]. When provided and trained with digital pathology data, computers have the potential to detect, analyse and classify kidney structural morphology, effectively emulating pathologist visual assessment. Computers accomplish these tasks by conducting pattern recognition via image analysis [18].

Image analysis is the extraction of meaningful information from digitized images via image processing techniques [25]. Given that images are composed of pixels, or a matrix of numbers, these processing techniques involve mathematical calculations to detect and characterize specific patterns in image pixels [25]. To selectively analyse image regions of interest, we apply segmentation: the partitioning of an image into multiple, distinct segments [25] of select pixels. For example, when analysing kidney morphology, WSIs must be segmented or partitioned into regions featuring anatomical structures of interest such as glomeruli, tubules, vessels or inflammatory cells. Once segmented, kidney structures may be quantified – enumerated and measured – to provide a set of descriptive features for subsequent classification by type and morphological variation [26,27].

Classical feature extraction and classification from renal biopsy whole-slide images

A myriad of classical image analysis algorithms has been developed for exploration of kidney morphology in WSIs [27]. These algorithms capitalize upon textural and morphological image processing techniques [28] to detect regions wherein disease may manifest. In image processing, morphology describes the shapes of image regions, while texture describes the arrangement and heterogeneity of colour or pixel intensities. Object edges in images are described as discrete regions wherein an abrupt change in intensity is observed for a stretch of contiguous pixels. Textural operations may be employed to detect object edges or distinct textures in images. For example, the texture of glomeruli is very different from the surrounding tubulo-interstitium. This observation, among many others, facilitated the application of textural analysis techniques for glomerulus detection and feature engineering, including Gabor filtering [6], local binary pattern (LBP) [7] and histogram of oriented gradients [5].

Morphological processing techniques have been used for detection of kidney microstructures that have characteristic morphologies. For example, the glomerular unit is typically elliptical in histology images, and therefore may be detected via minimization of the variance between the glomerulus boundary and a predefined shape (e.g. ellipse) [29]. Unfortunately, despite reproducible detection of glomerulus boundaries, these shape fitting methods have a substantial false-positive rate. More specifically, the similarity of the glomerular boundary to large, transversely cut tubules and arterioles renders each candidate for detection [29]. Algorithms based upon textural analysis also have drawbacks. In pathologic states, such as renal parenchymal scarring, there is a decrease in textural contrast between globally sclerosed glomeruli and surrounding cortical fibrosis that may hinder texture-based segmentation [30]. Therefore, algorithms may perform poorly when analysing disease tissue. To date, some of the most successful algorithms use morphological and textural analysis in tandem to provide for robust detection, segmentation and characterization of glomeruli, intraglomerular compartments and tubules [13^▪,31].

Once segmented, image regions are further processed to derive discriminative features for subsequent classification tasks. Features descriptive of image texture can be as simple as the standard deviation of pixel intensities in a given region [7], or as complex as the amplitude response of a Gabor filter [6]. Similarly, morphological features may be more apparent, such as image object area defined as the sum of resident pixels, or the computation of nuclear shape signatures that compare object radii at incremental polar angles [32]. Given the ability to selectively apply techniques for image analysis, derived features are often referred to as handcrafted or engineered. Similar to segmentation tasks, handcrafted feature sets derived via a combination of textural and morphological processes provide for optimal image classification. Although classical classification algorithms in machine learning are numerous, methods commonly used for classification of kidney microstructures using handcrafted image features include support vector machines (SVMs) [7,31,33] and decision trees [29]. Strong classifier performance has also been reported when features are extracted using artificial neural networks (ANNs), and subsequently classified using a SVM [33].

Neural networks for feature extraction and classification from renal biopsy whole-slide images

In recent years, ANNs have taken the field of image analysis by storm, exploiting their tremendous depth of vision and comprehension for a myriad of image-based applications. With respect to computational renal pathology, ANNs have emerged as the ‘panacea’ for existing challenges, enabling not only detection and classification of kidney morphology [2^▪▪,8,11^▪▪,12^▪,26,33,34,35] but also prediction of disease status and outcome [13^▪,9^▪▪,35,36,37,38^▪▪,39]. ANNs achieve high-level understanding through their biologically inspired architecture, wherein interconnected artificial neurons simulate the neuronal pathways of the human brain [40]. Input data (observations) are passed through a series of layers, or mathematical processes, which transform the input into a desired output. As data are passed from layer to layer, the network gains increasingly abstract insight and learns to recognize trends in the input data. Similar to the brain, ANNs also feature weighted connections between their artificial neurons [41]. These connections enable neurons to selectively influence each other, with higher number weights corresponding to greater influence. This fully connected, weighted design facilitates ANN learning or training.

During training, ANNs are provided with vast input data and ground truth labels defining each input’s expected output. Comparison of transformed output with the ground-truth defines the error of fitting, which the machine optimizes using an operation called back propagation [41]. Back propagation involves adjustment of learning parameters, such as the weights of neuronal connections, to produce updated transformed outputs. When the actual and expected outputs converge, error is minimized, and the network achieves optimal performance.

Initially, the computationally intensive nature of network parameter (weights) estimation and optimization deterred use of ANNs for image-based applications. Later discovery of convolutional neural networks (CNNs) – a more efficient network design – rendered ANNs ideal for image analysis tasks. Digital pathology has since seen an explosion in the application of CNNs for morphological and textural analysis of tissue structure in disease [1^▪,39,42,43]. CNNs pass input images through a series of convolutional filters (layers) and extract image features via convolution, a signal processing operation. The progressive complexity of convolutional filters at increasing network depth results in the extraction of a hierarchy of image features, branching from simple (edges and textures) to abstract (objects). These hierarchical features, along with the input image’s ground truth label, provide for subsequent image segmentation and classification tasks.

To date, CNNs have been applied for the detection and classification of kidney morphological structures, including glomeruli, tubules, interstitium, arteries and even empty Bowman’s spaces [2^▪▪,8,11^▪▪,12^▪,26,33,34]. In addition, CNNs have enabled discrimination of healthy and pathologic kidney morphology [2^▪▪,13^▪,34]. Many of these methods capitalize upon select histological stains for multiclass structural segmentation. For example, WSIs stained with Masson’s trichrome enabled detection and quantification of glomeruli and renal scarring. In addition, PAS-stained WSIs provided for comprehensive compartmentalization of both healthy and pathologic kidney morphology [2^▪▪]. These methods brought to light challenges facing multiclass segmentation of renal tissue, including reproducible detection and subclassification of tubules and sclerosed glomeruli, and overestimation of renal pathologies in distinct stains. Despite significant training data, composed of both pathologist manual segmentations and ground truth labels, networks struggle to reproducibly detect tubules (esp. atrophied) and sclerosed glomeruli. In addition, studies have shown that quantification of trichrome stain tends towards overestimation of renal fibrosis [44]. Taken together, these two challenges limit network performance for detection and classification of pathologic kidney morphologies essential to disease assessment.

Lutnick et al. [12^▪] recently developed Human-Artificial Intelligence-Loop (H-AI-L), a unique, iterative technique for multiclass semantic segmentation of kidney structures from WSIs. Rather than providing a single image label, semantic segmentation involves linking a class label to each pixel in an image. Pixel class labels are assigned based on ground truth annotations defining distinct image regions of interest. H-AI-L allows humans to interact with the machine through manual correction of structural boundary predictions for additional rounds of network training [12^▪]. The ability to iteratively train the network and guide the machine to identify kidney structures, as pathologists do, offers a solution to the aforementioned challenges in multiclass image segmentation. In addition, H-AI-L’s ability to automatically generate structural boundary predictions will ease the annotation burden in medical image analysis.

Recent works have demonstrated the successful application of an alternative ANN architecture – Recurrent Neural Networks (RNNs) [45] – for classification of digital pathology [13^▪,46]. Traditional ANNs process inputs independently, providing an output for each input. In comparison, RNNs inter-relate all inputs, using the output of a previous input to provide context for the next. As a result, RNNs have the ability to recognize the sequential characteristics of a dataset, and use learned patterns in data to make informed predictions and effectively exhibit memory. Ginley et al. [13^▪] capitalized upon this network ‘memory’ to predict the stage of diabetic nephropathy from WSIs of kidney biopsies. Given that current diagnostic schemes involve holistic evaluation of a biopsy’s glomerular pathologies, machine learning applications designed for automated diagnosis must analyse and classify all glomeruli in a WSI. To achieve this, Ginley et al. [13^▪] strategically trained a RNN with a sequence of labelled feature sets for each glomerulus in a biopsy, and trained the network to predict an ‘average’ stage for the entire input dataset (kidney biopsy) based on learned data trends. For the input data, glomeruli were staged by an expert pathologist, and glomerular features were engineered via computational image analysis. This work’s successful prediction of disease stage through a combination of computational image analysis, feature engineering and modern machine learning demonstrates the tremendous potential for artificial intelligence in diagnostic pathology.

Correlation of image features with clinical biometrics and progression data

ANNs are not limited to image data. The ability of ANNs to extract features and classify image, numerical and categorical data, renders them ideal for correlation of histopathology with clinical phenotypes. For example, multiple CNN models have been trained to correlate and predict CKD stage, proteinuria and other measures of kidney function at time of biopsy based on input histopathology images derived from renal biopsies [9^▪▪,35]. Additional CNN models predict 1, 3 and 5-year renal survival based on input biopsy images [9^▪▪]. At this time, few kidney-centric works feature such predictive capacity. However, a variety of ANN models have been developed for other types of histopathology data. Landmark discoveries in the field of cancer pathology have been made as a product of artificial intelligence methods for evaluation of histopathology data [38^▪▪,39,47,48]. The predictive capacity of these models for patient biometrics and survival rates demonstrates artificial intelligence’s potential in clinical diagnostics, prognostics and theragnostics.

CONCLUSION

Despite the successful application of ANNs in digital pathology, our interpretation of machine vision and the metaphorical ‘black box’ remains unclear [49]. Elucidation of the mechanisms governing machine decision making is essential to exploiting the maximal potential of these algorithms and gaining insight beyond extant clinical knowledge. Such insights will facilitate streamlined ANN training for recognition of rare, yet highly informative image features (biomarkers) indicative of distinct clinical phenotypes. Reproducible detection of clinical phenotypes from digital pathology will further the development of diagnostic and prognostic applications requiring analysis of multimodal data. In truth, the greatest obstacle facing the precision medicine initiative is integration of omic – genomic, proteomic – and image data [50]. Although select biomarkers have been identified in digital pathology, a method for comprehensive correlation of image biomarkers with patient omic data has yet to be developed. Additional challenges facing digital pathology include expansion of computational pipelines from two-dimensional tissue sections to three-dimensional reconstructions, and establishment of image ontology protocols [51–53] for curation of optimal machine learning datasets. Resolution of these challenges will revolutionize our understanding of renal pathology and will actualize future integration of artificial intelligence in the process of care.

KEY POINTS.

• The ability of ANNs to not only extract image features but also classify image, numerical and categorical data potentiates the integration of omic and image data in pursuit of precision medicine.

• Given that machine learning algorithms require a high volume of annotated examples to learn to reproducibly detect and classify distinct kidney structures, computational tools (e.g. H-AI-L) offer a unique solution to ease pathologists’ annotation burden and augment machine learning data sets.

• Due to the lack of standardized sample preparation and imaging protocols, robust algorithms must be developed to perform data quality control and navigate disparities in digital pathology data.

• Although modern machine learning with ANNs provides rapid, reproducible results, the handcrafted or engineered features of classical machine learning are more biologically and clinically interpretable.

• RNNs have enabled the evaluation of whole kidney biopsies, taking into account all pathologically relevant regions of interest, and strong approximation of pathologist visual assessment.