Abstract
The emergence of digital pathology — an image-based environment for the acquisition, management and interpretation of pathology information supported by computational techniques for data extraction and analysis — is changing the pathology ecosystem. In particular, by virtue of our new-found ability to generate and curate digital libraries, the field of machine vision can now be effectively applied to histopathological subject matter by individuals who do not have deep expertise in machine vision techniques. Although these novel approaches have already advanced the detection, classification, and prognostication of diseases in the fields of radiology and oncology, renal pathology is just entering the digital era, with the establishment of consortia and digital pathology repositories for the collection, analysis and integration of pathology data with other domains. The development of machine-learning approaches for the extraction of information from image data, allows for tissue interrogation in a way that was not previously possible. The application of these novel tools are placing pathology centre stage in the process of defining new, integrated, biologically and clinically homogeneous disease categories, to identify patients at risk of progression, and shifting current paradigms for the treatment and prevention of kidney diseases.
Subject terms: Kidney diseases, Systems biology, Kidney diseases, Systems biology
Developments in digital pathology and computational image analysis have the potential to identify new disease mechanisms, improve disease classification and prognostication, and ultimately aid the identification of targeted therapies. In this Review, the authors provide an outline of the digital ecosystem in nephropathology and describe potential applications and challenges associated with the emerging armamentarium of technologies for image analysis.
Key points
The introduction of digital pathology in clinical research, trials and practice has catalysed the development of novel machine-learning models for tissue interrogation with the potential to improve our ability to discover disease mechanisms, identify comprehensive, patient-specific phenotypes, classify kidney patients into clinically relevant categories, predict disease outcome and, ultimately, identify more targeted therapies.
The development of computational image analysis tools for tissue interrogation has brought pathology to the forefront in this process of re-defining kidney diseases.
The new nephropathology ecosystem offers several advantages over conventional pathology but also brings some challenges that need to be addressed collectively by all the stake holders, including pathologists, nephrologists, computer scientists, regulatory agencies and patient’s representatives; overcoming these challenges is a pre-requisite for these new machine-learning and computational pathology models to be fully deployed for patient care.
The development of novel powerful computational tools for image analysis and data integration in kidney diseases has exposed the need to revise the curriculum for medical professionals to prepare the next generation to fully operate within the new digital pathology ecosystem.
Ultimately, our ability to treat kidney diseases (actionable intelligence) will be largely based on the application of artificial (augmenting) intelligence tools and the establishment of synergistic human–machine protocols that integrate digital pathology data with clinical and molecular data for personalized nephrology.
Introduction
The advent of whole slide digital imaging systems has catalysed new interactive models for pathology education, expedited clinical workflows and provided opportunities for the development of new medically actionable image analysis tools and investigative techniques. Although still evolving, this technology has provided the clinical and research community with new solutions that will need to be integrated with established models for contemporary care. Ultimately, we anticipate that a new system of care will emerge, whereby these innovative technologies are integrated into routine practice in a rational and fully validated manner.
In this Review, we discuss how developments in digital pathology and computational image analysis are shaping a new digital era for nephropathology and how these technologies might be most expeditiously considered for clinical deployment. We provide an overview of the current status of the digital ecosystem in nephropathology, which encompasses human expertise in renal pathology, artificial intelligence (AI), omics science and powerful computational tools, as well as potential applications and challenges associated with the emerging armamentarium of technologies for image analysis. Finally, we present what might be a prescient vision of the imminent future of clinical research and practice made possible by these technologies.
Nephropathology in the digital era
Only a decade ago, virtual microscopy was considered to be a fairly novel technology and of insufficient maturity for primary diagnostic purposes. Although the technology was rapidly embraced for educational uses, its role in clinical practice was less clear given the preference of conventionally trained pathologists to work with conventional light microscopy. This status quo has been shattered by the emergence of rapidly evolving technologies such as telepathology and digital pathology, in combination with developments in computational image analysis. The collective subspecialties of pathology are now witnessing a far-ranging and transformative set of new practice models, with implications for both intramural and remote primary and consultative diagnosis. Similarly, delivery models for education and companion diagnostics are positively affected by the emergence of these technologies. In particular, content-based image retrieval1 — a machine vision technique that searches images by analysing their content (according to their colour, shape and/or texture) rather than their associated metadata — is now a reality that opens fascinating new possibilities to leverage the vast and under-utilized slide and image archives long held by pathology departments. The main uses of digital imaging in nephropathology can be divided into three major operational models: telepathology, digital pathology and computational image analysis, as outlined below.
Telepathology
Telepathology involves the transmission of one or more microscopic images from an originating location to an imaging workstation at a distant location. Its use has been extensively validated and it is now a common tool for real-time assessment of tissue adequacy and diagnosis2,3. In nephropathology, dynamic telepathology has been deployed for the remote assessment of donor implant biopsy samples by trained renal pathologists. This approach overcomes some of the limitations of current practice, whereby cadaveric donor kidneys are often harvested and evaluated after-hours, in non-academic hospitals and by general surgical pathologists4,5. Telepathology has also enabled renal pathologists to assess the quality of fresh tissue biopsy at the time of the biopsy procedure without needing to be physically present in the interventional radiology suite. Its use gives the subject matter expert (rather than the available non-renal pathologist) the ability to triage the kidney biopsy samples for subsequent histology, immunofluorescence and electron microscopy analysis, without significant disruption of their workflow1.
Digital pathology
Digital pathology is a general term that refers to the assemblage of digital workflow and imaging solutions that are geared towards creating a digital image-based practice environment in which a whole slide image (WSI) or other digital image is acquired, managed, interpreted and searched for specific content. This new approach is being increasingly adopted in clinical trials6, for education purposes7–9 and for clinical research1,10–16. Since 2017, the FDA has approved the use of commercial WSI platforms, facilitating the use of digital pathology as a tool for primary diagnosis17–22. In response, many pathology practices and laboratories, particularly in Europe and Canada, are now in the early or mid-phases of implementing digital pathology solutions for purposes other than image archiving or for second-opinion consultation23,24. WSIs, which can now be scanned in less than a minute, can serve as an effective surrogate for traditional microscopy-based pathology20,25. However, WSI workflow models have known limitations including the need for substantial network bandwidth to handle large file sizes, the lack of Z-dimensional focus (except in specialized cases, where a ‘Z-Stack’ image might be generated for a region of interest (ROI) from the larger total slide area), which limits their utility for cytopathological and renal biopsy evaluation, and imperfect control software in the scanning appliances themselves, which can contribute to various imaging artefacts (for example, out-of-focal plane dust on the glass slide triggering acquisition of the wrong focal plane)26.
These limitations support the subjective perception held by many pathologists that digital pathology can be less accurate and more time consuming than glass slide microscopy for diagnosis, although direct quantitative comparisons show the opposite. Processes might indeed be slower when digital pathology workflows are first adopted; however, following an initial training and acclimatization period, overall pathologist performance greatly increases for measurements of both speed and accuracy. As a result, perceptions are changing, with digital pathology becoming increasingly accepted, helping the pathology specialty move towards fully digital workflows and associated platforms for clinical operations26–29.
Computational image analysis
Advances in scanning technology paired with the increasing availability of large datasets of digital images have enabled pathologists to collaborate with technology experts, including data scientists, computational engineers and imaging physicists, to explore the potential of a newly established branch in the field of pathology: machine vision, a component of computational image analysis. The primary objective of this field is to extract and/or generate quantitative data from digital image subject matter, either in isolation or in tandem with other classes of spatially or non-spatially based biological or omics data. Recognizing the high-dimensionality of the data generated by computational image analysis, AI and machine-learning (ML)30–33 techniques, which are ideally positioned to interrogate such complex datasets in an exhaustive manner, have been deployed to extract features, patterns and information from histopathological subject matter that cannot otherwise be analysed by human-based image interrogation alone. Deep learning (DL) — also known as deep structured learning — is a particularly important branch of ML that provides opportunities to interrogate images at greater depths than previously possible30,31,34,35. In pathology, DL has been used for the detection, annotation, segmentation, registration, processing and classification of WSIs. The learning process uses neural networks mimicking the human intelligence and contains multiple layers to progressively extract deep features from the input image, first transcribing imaging information into structured data, and then translating them into knowledge.
AI models, including ML and DL, can be used for various purposes, including the automatic detection, segmentation and quantification of histological parameters and structural changes, and for disease diagnosis. These tasks are considered to be ‘low-level’ tasks. This is in contrast to ‘high-level tasks’, in which AI models are used to concurrently interrogate and integrate multiple classes of primary data — for example, histopathological image data spatially coupled to transcriptomic data — providing opportunities for the prediction of processes such as disease aggression (for example, the biological potential of a malignancy), patient outcome, organ engraftment survival and therapeutic response. Thus, use of AI has the potential to elevate digital images from their basic use as a tool for visual assessment of disease status to a much more complex and comprehensive role as a tool to facilitate prediction of disease trajectory. These new capabilities enable the extraction of useful information in a way that was not previously possible with the use of conventional microscopy in the context of direct human cognitive and visual assessment.
The increased demand for predictive assays for patient profiling is driven not only by the need to better stratify patients according to their clinical characteristics and outcomes but also by the evolving discovery and development of drugs that target specific molecular pathways36. Novel approaches to aid in the understanding of the relationships between structural changes in tissue and clinical and/or molecular phenotypes would be particularly beneficial to the nephrology community, as current classification systems are inadequate in their ability to capture the heterogeneity of kidney diseases, or predict disease outcomes. New, robust algorithms derived from the application of AI approaches and integration of pathology data with other relevant datasets will have utility for both pathologists and nephrologists, with the potential for a meaningful impact on patient care and precision medicine.
Many parallels can be drawn between machine vision in pathology and developments in radiography, which underwent its own digital revolution in the late twentieth century with the development of native digital radiographic modalities such as MRI and CT, which allowed interrogation of digital anatomically coupled subject matter for the first time. Additional innovations in radiology, such as cassette-based storage phosphor image plates (which allowed X-ray images to be recorded digitally37) and the implementation of charge-coupled devices, facilitated the rapid development of digital detectors38–41 and further accelerated the adoption of digital technologies. Similar to the current interplay between digital and computational image analysis, the mainstream availability of digital radiographic images has enabled quantitative imaging analysis techniques to be tested and introduced for use in clinical trials and for patient monitoring42,43.
Digital pathology repositories
The nephrology community has witnessed the formation of several international consortia that collect biospecimens from patients with a variety of kidney diseases (Table 1). The common goals of these consortia10,11,13,44–52 are to better understand the pathogenesis of kidney diseases and to improve their classification and treatment through comprehensive analysis of clinical, morphological and molecular data — a process that has required the establishment of digital pathology repositories (DPRs) for the banking and organization of digital images from renal biopsies and their associated metadata.
Table 1.
Multicentre digital pathology repositories for kidney disease
| Name | Location | Data collection centre/central hub | Target diseases | DPR (year established) | Consortium website |
|---|---|---|---|---|---|
| NEPhrotic syndrome sTUdy NEtwork (NEPTUNE) | North America | University of Michigan, MI, USA | MCD; FSGS; MN | 2010 | https://neptune-study.org/ |
| NEPhrotic syndrome sTUdy NEtwork – CHINA (NEPTUNE-CHINA) | China | National Clinical Research Center of Kidney Diseases; Nanjing University, China | MCD; FSGS; MN | 2013 | NA |
| Cure GlomeruloNephropathies (CureGN) | North America; Europe (Poland, Italy) | Arbor Research, Michigan, MI, USA | MCD; FSGS; MN, IgAN | 2015 | https://curegn.org/ |
| EURenOmics | Europe, North America | University of Heidelberg, Germany | Paediatric kidney diseases | 2015 | https://www.eurenomics.eu/ |
| Transformative Research In DiabEtic NephropaThy (TRIDENT) | North America | University of Pennsylvania, PA, USA | Diabetic kidney disease | 2016 | https://www.med.upenn.edu/trident/ |
| Biomarker Enterprise to Attack Diabetic Kidney Disease (BEAt-DKD) | Europe | Lund University, Sweden | Diabetic kidney disease | 2018 | https://www.beat-dkd.eu/ |
| European Rare Kidney Disease Network (ERK-Net) | Europe | University of Heidelberg, Germany | Glomerulopathies; thrombotic microangiopathies; renal and urinary tract malformations; tubulopathies; metabolic nephropathies; familial cystic diseases; any rare kidney disease; paediatric transplantation | 2018 | https://www.erknet.org/ |
| German Focal Segmental Glomerulosclerosis and Minimal Change Disease Registry (FOrME) | Europe | University of Cologne, Germany | MCD; FSGS | 2019 | https://clinicaltrials.gov/ct2/show/NCT03949972 |
| Kidney Precision Medicine Project (KPMP) | North America | University of Washington, WA, USA; University of Michigan, MI, USA; Mount Sinai School of Medicine, NY, USA | Chronic kidney disease; acute kidney injury | 2019 | https://kpmp.org/ |
| Japan Renal Biopsy Registry (J-RBR) | Japan | Niigata University | Glomerulopathies; tubulopathies; transplantation | 2019 | https://pubmed.ncbi.nlm.nih.gov/21437579/ |
| Human Heredity and Health in Africa (H3 Africa) | Africa | University of Michigan (for the DPR), MI, USA | Non-communicable disorders (e.g. heart and kidney disease), as well as communicable diseases (e.g. tuberculosis) | 2020 | https://h3africa.org/ |
| Acceleration of Medicine Partnership in RA/Systemic Lupus Erythematosus (AMP RA/SLE) | North America | University of Michigan (for the DPR), MI, USA | Lupus nephritis | 2020 | https://amp-ralupus.stanford.edu/ |
| APLO1 Long-term Kidney Transplantation Outcomes Network (APOLLO) | North America | Wake Forest, NC, USA | Kidney transplantation in the African American population | Under consideration | https://clinicaltrials.gov/ct2/show/NCT03615235 |
This table lists major, multicentre studies that are known to the authors, but it is not exhaustive. DPR, digital pathology repository; FSGS, focal segmental glomerulosclerosis; IgAN, IgA nephropathy; MCD, minimal change disease; MN, membranous nephropathy; NA, not available.
The establishment of DPRs has driven the concept of a digital biobank and a convergence of research on digital rather than physical assets. Conceptually, the formation of a digital biobank is the same as forming other types of biobank, starting with rigorous training of personnel involved in the collection, de-identification and transfer of the pathology material. This part of the process requires adherence to quality control (QC) steps to protect the privacy of individuals and their health information in accordance with the Health Insurance Portability and Accountability Act, to ensure study protocol compliance and that the renal biopsy pathology materials are adequate, complete, organized in the appropriate patient and disease category-designated space, and properly identified (labelled) for searchability11,14. DPRs are created to facilitate shareable data access; thus, considerations are needed at the consortium and regulatory agency level to facilitate secure, seamless and timely access for the end user.
The digital nephropathology ecosystem
Digital pathology applications are progressively modifying the ecosystem in which pathologists operate, offering considerable advantages over traditional pathology approaches, as discussed below. However, these applications also present challenges that require consideration of factors such as the redistribution of resources, standardization and regulation of processes, approaches to standardizing and integrating data, the consequences of these new approaches for diagnostic paradigms and effects on the medical curriculum.
Investment in digital pathology
DPRs are a cost-efficient way of systematically organizing pathology resources. The investment, although initially burdensome, is long term and allows for the establishment of a permanent library of sharable images.
In translational research, the return of such investment resides in the nature of the digital library itself, which can be accessed by multiple investigators, thereby supporting and facilitating a variety of pathology-based applications and related new discoveries within and across populations15,16,53–56. Additionally, DPRs allow barriers that were previously not addressable to be overcome: they provide transparency for regulatory agencies and enable collaboration both within and between consortia. The reproducibility of observations is also improved, facilitated by digital annotation11,12,15,16 and computer-aided quantitative assessment, which offers a numerical value to the observations and measurements of parameters (i.e. density, spatial distribution and relationships of histological primitives) that cannot be achieved by conventional analysis1. Thus, with the increasing role of big data and the ability to integrate separate knowledge domains (pathology, clinical and molecular domains) to generate new diagnostic categories and predictors of outcome, the launching of digital platforms for the collection and analysis of tissue structural changes has become a necessary investment for clinical research.
Improvements in image resolution and digital scanner capacity, as well as the development and regulatory approval of whole slide scanners for primary diagnosis and tools for the integration of image management software with laboratory information systems, have positioned digital pathology to be deployed for clinical applications17,19,28. Studies have shown that the implementation of digital pathology in clinical settings decreases overall operational costs while increasing clinical and operational efficiency28. Most recently, the COVID-19 pandemic has forced the practising pathology community to explore approaches to enabling remote histopathological analysis on a large scale. In the USA, a critical step in facilitating this move was an amendment to the 1988 Clinical Laboratory Improvement Act (CLIA ’88), which relaxed the location-specific licensing requirements that regulate where a pathologist is allowed to diagnose cases. This change enabled a number of pathology practices and health networks to rapidly employ remote diagnosis workflow models, allowed the field of pathology to gain greater proficiency in rendering primary diagnoses by digital means and facilitated the accelerated adoption of modality-specific workflow models and digital consultative review. These developments all bode well for the prospect of nephropathology benefiting from the expanded use of WSI solutions for expedited primary diagnosis.
However, the potential benefit of a fully digital pathology operation on clinical applications in nephropathology extends beyond operational aspects alone. Integration of knowledge generated by these new computational tools can lead to new discoveries with the potential to introduce new paradigms in clinical practice.
Standardization of analytics
The deployment of digital pathology into research, clinical trials, and ultimately clinical practice depends on a variety of elements, from the standardization of analytics to allow the sharing of images to the application of image analysis tools (Table 2). The process of transforming glass slides into WSIs involves a series of steps, each contributing to the final quality of the digital image that is visualized on a computer monitor. Protocols for the processing of tissue into histology sections and the scanning and transmission of image-associated information need to be standardized so that the digital pathology applications can augment the work of the pathologist on a large scale. Such processes demand a revision of traditional criteria for QC and standardization of analytics beyond current recommendations and regulations that apply to conventional histology approaches, as provided by the College of American Pathologists, CLIA ‘88, the Digital Pathology Association57 and the Renal Pathology Society (RPS). In 2018, The Royal College of Pathologists in the UK published best-practice recommendations for implementing digital pathology, highlighting the diagnostic challenges of WSIs and providing practical solutions for reducing the risk of discordant diagnoses58,59. In the past few years, the College of American Pathologists has also provided guidelines for validating digital pathology approaches60, and the National Society for Histotechnology has initiated a programme to improve the quality of WSIs60,61. However, we and others recommend that all stake-holders participate in regulating and monitoring the use and performance of digital pathology-based protocols62.
Table 2.
Current limitations and proposed solutions for computer-based image analysis
| Issue | Problem | Proposed solution |
|---|---|---|
| Standardization of tissue analytics | Variability of harvesting and tissue processing |
Revision of histology protocols across centres to optimize downstream quantitative analysis Involvement of subspecialty societies |
| Standardization of imaging analytics | Scanner variability and image inconsistencies |
Implementation of standard quality assurance and calibration procedures (e.g. image linearity, uniformity, reproducibility) Implementation of DICOM implementation in pathology Evaluation of scanner variability across manufacturers Involvement of subspecialty societies |
| Selection of the best approach for each task or problem | Quantitative techniques are specific to the application, region of interest, or the clinical question |
Parallel training, validation and testing of models and techniques Prospective evaluation (trials) to assess the clinical relevance of the application |
| Studying rare kidney diseases | Inherently small datasets |
Knowledge-based data science Data augmentation Inter-institutional consortia for data sharing |
| Data integration | Data extraction, representation, and fusion across different length scales |
Standardization of data collection and storage Ontology development Development of new data fusion algorithms |
| Knowledge integration | Lack of didactic training across disciplines |
Revision of curricula for health-care providers (e.g. medical students, residents, fellows) and health data scientists Interdisciplinary nephrology boards |
| Deployment and efficacy in clinical practice | Lack of comprehensive prospective evaluation |
Comprehensive prospective evaluation in clinical trials Development of new regulatory guidelines |
| Data-sharing ethics | Lack of regulatory guidelines |
Revision of IRB oversight Revision of Data and Material Transfer and/or Access Agreements Involvement of subspecialty societies and the medical ethics community |
DICOM, Digital Imaging and Communications in Medicine; IRB, Institutional Review Board.
As outlined below, standardization of pathology analytics requires consideration of three major components (Fig. 1): a pre-analytical phase, which includes the steps from tissue procurement to fixation, processing, cutting and drying time; an analytical phase, which includes a histology phase (involving stain selection, optimization and validation) and a digital phase (scanning, consideration of image and monitor resolution, number of colours, colour distribution, compression ratio and image format63); and a post-analytical phase, which involves the analysis and interpretation of results, the recording and reporting of data, and pathologist and ML performance metrics1,10,12,16,64–66 (Fig. 1).
Fig. 1. The analytical phases of digital pathology.
The pre-analytical phase includes steps involved in tissue procurement, processing and fixation. The analytical phase involves a histology phase (which includes selection of the stain to be used, optimization and validation of the staining procedure) and a digital phase (which includes scanning the slides as whole slide images (WSIs) and curation of the digital library). The post-analytical phase involves data extraction, analysis and interpretation of results. Data extraction is accomplished using human visual analysis and/or machine vision, using human–machine synergistic protocols. Data extracted from the images using human and machine vision are integrated and reported. Data integration can be achieved using computational tools as well as human intuition and domain expertise.
Standardization of the pre-analytical and analytical phases
The current criteria for ensuring QC of the histology preparations allow for broad variation of pre-analytical and analytical artefacts across and within laboratories. The spectrum of artefacts varies in severity and quality. Some can affect the glass slide (for example, pen marks, dirt and bubbles), whereas others affect the tissue (for example, the presence of tissue folds, or differences in thickness or stain intensity) or the scanning process (for example, differences in focus or the gridding effect of the WSI). While pathologists train themselves to make an interpretation using the stain protocol they are accustomed to and read through artefacts on the glass slide, tissue section or digital image, computers must be trained to adapt to such heterogeneity in image presentation. For computer-aided algorithms to leverage digital pathology and enable quantification, classification and prognostication in a scalable manner, the algorithms must be generalizable across datasets. Although variability in WSI training sets may actually represent a potential advantage for the generalizability of the resulting algorithms, the performance of algorithms is intimately linked to the control of analytics and homogeneity of the datasets63.
High-quality datasets can be achieved by proactively gaining tighter control over analytical practices within and across laboratories, and by re-modelling current criteria for the minimum QC threshold based on the needs of current technologies and their applications61. As the use of image-based data by AI technologies requires the conversion of analogue data into numerical values (that is, digitization), understanding of the intimate association between stain quality and image quality becomes essential. Similar to the field of radiology, where imaging physicists are employed to ensure QC of clinical images and imaging systems, standardization and quality assurance at both the tissue staining phase and the image acquisition phase will likely require on-site technical expertise. This support will ensure patient safety and reliable diagnostics. We expect that this area of optics and light microscopy will be an important focus of discovery and standardization in the coming years.
One relevant issue that requires better understanding before computational tools are deployed for image analysis in the clinical setting is how variation in image quality caused by the physical abstraction layer of the WSI scanning platform itself affects the performance of AI approaches. Different commercial WSI solutions digitally reproduce colours based on a reference colour source slide with uneven variability. This variability can extend to inter-device variability, even within a single model of a WSI scanner. Although colour correction software can be used to standardize WSIs across different scanning platforms, this approach is still in its infancy. Recognizing that AI algorithms can be (and often are) dependent on consistent colour representation, standardization of colour becomes essential to establish robust and reproducible quantitative measurements and analysis63,67,68. This issue is of critical importance in the context of multicentre DPRs and will force investigators to weigh the pros and cons of central versus local scanning.
QC of image datasets can also be achieved retrospectively (that is, after WSI acquisition) using computational post-processing techniques. QC tools have now been developed to automatically assess WSI quality, capturing variations in image presentation such as stain distribution, brightness, or glass, tissue and scanning artefacts. One example of such a QC algorithm is HistoQC, an open source QC tool that quantitatively assesses the heterogeneity of WSI image datasets and, similarly, identifies artefacts present on the source glass slides or on the scanned image69. In the context of clinical research, HistoQC is currently being evaluated in NEPTUNE, CureGN and Kidney Precision Medicine Project (KPMP) for curation of the image dataset prior to experimental analysis (Table 1). In clinical practice, the ability to identify compromised WSIs before they are assessed by the pathologist or computer-aided tools reduces downstream delays in clinical diagnosis.
Standardization of the post-analytical phase
The digital ecosystem has the potential to facilitate the harmonization of data collection both within and between consortia via the use of shared protocols10, and to improve accuracy15 and concordance among pathologists12,16. However, agreement among nephropathologists in diagnostic and tissue assessment remains the limiting factor and a pre-requisite for the training of AI algorithms5,12,16,70–73. Awareness that the standardization of pathology definitions and use of a controlled vocabulary are central elements that drive the reproducibility and accuracy of observations and/or of data generated for both human and computer-aided analysis is reflected by several recent initiatives. For example, NEPTUNE and the international digital nephropathology network, INTEGRATE, have generated a list of individual observational lesions (descriptors) and their associated definitions, obtained by international consensus, for the assessment of WSIs10,12. Similarly, the Renal Pathology Society has undertaken an initiative to develop a universal lexicon by compiling detailed and unambiguous definitions of pathological lesions and structures with the goal of harmonizing classifications, whereas the KPMP investigators have applied a controlled vocabulary to assemble libraries of conventional diagnoses, disease categories and descriptors. In KPMP, the descriptor library will be relevant to informing and guiding the assembly of the KPMP feature vector library, a ML approach to the automatic detection of normal and abnormal histological primitives, and to generate a functional ontology and atlas of kidney diseases.
Shifting paradigms in renal pathology
The transformative approach to the study of kidney diseases introduced by systems biology science is calling for a change of paradigms in renal pathology for the interrogation of WSIs. The current ‘one-size-fits-all’ approach, whereby pathologists generate classifications using a few preselected parameters and test them against disease outcome or disease biomarkers, provides neither sufficient granularity nor a quantitative metric that adequately represents the complexity of structural and cellular changes that can exist in the kidney. The new digital ecosystem and the development of novel technologies offers an opportunity to establish new protocols and methodologies — either visual or computer-aided — to retrieve and represent, as interpretable data, the information contained in kidney biopsy samples.
The introduction of digital pathology and use of descriptors to identify discrete elements of structural changes in tissue has enabled a more comprehensive visual assessment of the kidney on annotated WSIs compared with conventional methodologies10,11. As discussed above, the standardization of pathology vocabulary is a prerequisite for the interoperability of image data collection and reporting. Use of standardized vocabulary and DPR protocols has facilitated the harmonization of data collected across multiple consortia, providing insights into the structural manifestations of kidney diseases within and across populations10,53. For example, one study that used visual assessment of samples from three cohorts in North America and China showed that the age-adjusted percentage of global glomerulosclerosis is a useful predictor of kidney disease progression across diseases and populations53. Similarly, the percentage of interstitial fibrosis was associated with the expression of specific genes, transcriptional regulators and urinary biomarkers across cohorts from North America and Europe54,55. However, visual (that is, manual) assessment of WSIs is time consuming and has limited reproducibility71. Computational imaging algorithms are likely to provide more efficient and reproducible automated quantitative assessment across diseases and populations, increasing the robustness of the observations and scalability of such studies. The integration of computational image analysis and AI into the digital pathology ecosystem will induce further shifts in renal pathology paradigms towards the integration of structural changes, with multiaxial datasets to redefine categories and subcategories of patients with similar biological mechanisms, morphological profiles and clinical trajectories.
Integration and interoperability
Multi-domain datasets generated by systems biology approaches are represented by different length scales; for example, serum creatinine is measured quantitatively at different time points during the disease course, morphology data are collected using different methodologies, human or machine vision, and metrics, at the time of the biopsy, genetic data do not change with time, whereas the expression of genes and associated regulators are dependent on disease status. The extraction, alignment, integration, interpretation and reporting of such multi-scale information is an emerging challenge in nephrology (Table 2). In the new data integration ecosystem, foundational, structural and semantic interoperability across and within datasets and studies is intimately linked to the standardization of analytics, data sharing and data integration approaches. The interoperability of image analysis alone carries challenges, from annotation to algorithms and image features. For example, image annotations created by vendor-supplied software programs in one community are often incompatible with those created by other communities who might make use of a different program. A valuable approach to facilitating the interoperability of heterogeneous data within and across domains is to use ontologies, which encompass and represent properties and relations between data, categories, concepts and entities, and to organize the data into information and knowledge74,75. The Quantitative Histopathology Image Ontology is one example of an effort to establish standardized approaches to ontological pathology image data for their integration with data from other domains — such as clinical or demographic data. The Quantitative Histopathology Image Ontology is aimed at standardizing terminology at multiple levels, including data input and output, parameters for image acquisition and analysis, and the execution phase whereby data are recorded and reported76. Integration and interoperability are also centre-stage concepts in KPMP, where the development of a kidney cell atlas will be based on a comprehensive ontology harmonized with pre-existing ontologies. Much can be learned by studying other disciplines, such as oncological pathology42,77–85 and radiology, where considerable progress in integrating information has already been achieved42, and where ML approaches have been used to combine pathology and molecular data for improved prediction of cancer recurrence81,82,86.
The data ecosystem
The confluence of large-scale studies, often involving clinical data from electronic health records or data repositories, and the capacity to represent entire research studies in electronic form, has altered the way in which researchers collect, store, use and integrate data. The oncology field has integrated data from different domains for quite some time, with the advent of secure data environments paired with flexible database design and cloud computing technology aiding the evolution of the data ecosystem. Rather than attempt to place data in a single database in which a single framework unifies disparate data types, the trend is to now store data in individually optimized databases that are then connected by data aggregators. Functionally, such a set-up enables end-users to perform a search from one database as simultaneous queries against multiple databases. Moreover, this process enables image data to be linked with both clinical and molecular data.
Data sharing
Data sharing encompasses several diverse concepts, from transparency and regulation, to interoperability (discussed above). A key aspirational goal for data sharing has been, and remains the attainment of interoperability, under the umbrella of the now-recognized FAIR principle (Findable, Accessible, Interoperable and Reusable)87. The past decade has seen the emergence of several policies relating to data sharing and research transparency by the Office of Research Integrity, National Institute of Health or other organizations88. Providing access to bio-specimen repositories and research data increases their potential utility for health discovery and validation, and optimizes their long-term value89–95. This benefit is particularly relevant in the context of rare diseases, where data sharing is often needed to obtain sufficient sample sizes. Historically, renal pathologists have used local datasets for research purposes, with limited opportunities for external QC or data exchange. The availability of digital WSIs fosters opportunities for collaboration in a transparent environment. Transparency is not only relevant to QC for internal research purposes, for example, when clinical trials are conducted using a digital pathology platform for the pathology analysis, the relevant pathology data elements can be easily audited by regulatory agencies6.
However, the concept of data sharing raises several legal and ethical issues. Some of these, including issues related to inclusiveness, ethically responsible research, privacy and return of results, are not new and apply to many forms of research. Consideration must also be given to the sharing of metadata, including validated feature detector ensembles and associated image sets that would be submitted in the curation process in parallel with the bio-specimens themselves. In addition, although best practices relating to data sharing and material transfer agreements have been articulated in many documents, particularly in the context of rare disease research89–95, guidance specific to the sharing of pathology images and use of DPRs does not yet exist and is urgently needed (Table 2).
An additional consideration is the use of WSI data for retrospective analysis, beyond the original stated research intent. The digital permanent image libraries that will originate at the institutional level, or under individual consortium oversight, will naturally outlive the scope of the originally stated informed consent, the specific aims of the original study, and technology that was available at the time of the image data collection. Given that new technologies empower investigators to generate hypotheses and ask scientific questions that were not conceivable at the time of the collection, it will be essential to establish standard processes to enable permission lists of original data use to be extended. These extensions will require a standardized adjudicative process that can operate without the need to re-consent participants, as their location and identity may no longer be known. Such broad consent documents, which essentially ‘future-proof’ possible unforeseen uses of curated tissues and images are already used by researchers in various omics fields. One controversy from the oncology field96, involving the alleged exclusive sharing of pathology datasets originally intended for general, not-for-profit use with a commercial AI developer, underscores the need for the pathology field to rethink not only how informed consents should be formulated but also who should benefit from the sharing of digital datasets and associated health information.
It is imperative that guidelines for the responsible sharing of data in the digital pathology ecosystem are formally established. Such guidelines could be modelled on previously published documents, such as the international code of conduct for genomic and health-related data sharing, the founding principles of which are based on the goals of promoting health and advancing research, and associated ethical elements (for example, respect of the parties involved and their privacy, distribution of benefits, trust, integrity, accountability, reciprocity, and data security and quality)97.
Image analysis
Two main approaches are used for the interrogation of WSIs: human visual assessment and machine vision.
Human visual assessment
A variety of methods have been used for the visual assessment of WSIs, including conventional diagnostic5 and descriptor-based approaches10, quantification of structural and cellular elements6,12, and morphometric analysis56. The collection and quantification of visual data can be further facilitated by digital or manual annotation tools of the regions or objects of interest on the WSIs, resulting in increased accuracy and reproducibility of the observations15. Visual assessment of WSIs has been used to identify features that correlate with outcomes or the presence of biomarkers, and for the discovery of clinically relevant disease categories. As mentioned earlier, features such as extent of age-adjusted global glomerulosclerosis or interstitial fibrosis can be reliably assessed visually and are associated with outcomes including disease progression53,54. Other quantitative information that can be obtained from WSIs of renal biopsy samples, such as the interstitial capillary and podocyte density, might enhance our ability to predict predisposition to chronic kidney disease progression, hypertension98 and the development of global glomerulosclerosis99–105 but cannot be easily visually assessed, at least not without a significant amount of labour.
Machine vision
Machine vision refers to the computational techniques that are used to convert digital pathology images into mineable data, using AI techniques such as ML, DL and pathomics. This approach is driving a fundamental paradigm shift in digital pathology in which digital renal biopsy samples are interpreted as quantitative datasets. The performance of the AI techniques depends on the quantity and quality of the primary data, where ‘quality’ refers to the cleanliness, provenance, accuracy, signal-to-noise ratio and comprehensiveness to achieve the maximum predictive performance. Obtaining primary data of sufficient quality or quantity can be challenging for datasets associated with rare kidney diseases.
The two main categories of machine vision that are applicable to WSIs of renal biopsy samples are supervised learning and unsupervised learning. In unsupervised machine vision, features are identified based on their similarity to each other and group or category assignment is based on their relative proximity. By contrast, supervised learning uses an annotated training dataset to learn the relationship between individual features and categorical labels. Supervised and unsupervised learning can be used to convert digital pathology images into minable data, using two general techniques: ‘hand-crafted pathomics’ (also known as conventional pathomics) and ‘discovery pathomics with DL’.
Hand-crafted pathomics
This approach involves the segmentation (by manual or automated means) of an ROI on an image (for example, glomeruli, tubules, arteries or interstitial capillaries), from which high-throughput quantitative features known as hand-crafted features are extracted to derive a pathomic feature space, or digital image signature. This approach is essentially a transcription process, whereby unstructured imaging data (that is, pixels within the ROI) are converted into structured imaging data (that is, the pathomic feature space) so that it can be recognized by learning algorithms.
Pathomic signatures, which are digital imaging signatures obtained through hand-crafted pathomic feature extraction, are typically engineered to capture a number of features including morphology, intensity, texture, higher-order and spatial features. Morphology features quantify the size and shape of the ROI, such as its compactness, border irregularity or maximum diameter. Intensity features capture the distribution of pixel values within the ROI, based on the image’s histogram, which is a probability distribution that encodes the frequency of occurrence of all pixel intensities. Of note, intensity features do not capture spatially encoded information. For instance, the entropy of an image can explain the degree of randomness in its pixel intensities but not where the randomness occurs. Texture features quantify the spatial distribution of pixel information in the image, based on joint-probability distribution functions. These features quantify aspects such as overall image brightness, local heterogeneities and non-linear relationships between pixels. Fine- and coarse-texture features capture heterogeneities at different orders of magnitude. Higher-order features capture information specific to certain frequency bandwidths such as wavelet decomposition and multi-scale invariance such as fractal geometry. Spatial features capture the spatial organization or architecture of specific histological primitives (for example, glomeruli or tubules) in the tissue. These typically involve graph network-based algorithms such as the Voronoi tessellation or minimum spanning tree.
Discovery pathomics with deep learning
In contrast to hand-crafted pathomics, discovery pathomics uses DL to extract features directly from images, without explicit mathematical definitions. DL algorithms are often able to uncover novel and important patterns from sets of input images with case-level annotations and without human guidance, given their ability to excel in unsupervised feature extraction. This approach can be particularly powerful for applications that remain as unstructured imaging data (for example, generating segmentations of a histological primitive), or when a problem is too complex for conventional feature engineering. Although DL can be interpreted as a complete classifier, it can also be used as a type of image representation analogous to the hand-crafted pathomic approach. In fact, DL approaches commonly encode an input signal into structured data form (that is, a transcription process similar to that used for feature engineering), followed by a decoding of the data to generate an output signal (that is, a translation process akin to that used by conventional ML algorithms). The most common type of DL used with images is based on convolutional neural networks (CNNs), whereby a series of shift-invariant filters are used to generate abstract feature maps to learn hierarchical patterns embedded within the input image. CNN architectures have been used to automatically detect and segment normal and sclerotic glomeruli, nodular glomerulosclerosis, tubules, interstitial space and fibrosis, peritubular capillaries and arteries72,73,106 (Fig. 2). Other popular DL architectures (that is, artificial neural networks) include fully convolutional networks, recurrent neural networks and generative adversarial networks42 (Table 3).
Fig. 2. Machine vision technology as a support tool in nephropathology.
Machine vision tools — including discovery pathomics with deep learning and hand-crafted pathomics — can be used to convert digital pathology images into minable data and provide support to pathologists. a | Machine vision can be used to automatically detect histological primitives. For example, a convolutional neural network can be used to detect glomeruli in frozen kidney sections stained with haematoxylin and eosin. b | Machine vision has also been used to establish a classifier for glomerular disease stage in patients with diabetic glomerulosclerosis. The left panel shows a paraffin-embedded section of a normal glomerulus (top) and a glomerulus with nodular glomerulosclerosis (bottom) stained with periodic acid Schiff. The middle panel shows detection of the cellular/nuclei (blue) and matrix (red) component of the normal (top) and diseased (bottom) glomerulus. The right panel shows the measurements of the glomerular characteristics for both the normal and the diseased glomerulus. c | Application of artificial intelligence-guided morphometry can provide quantitative assessment of the interstitial fractional space. On the left is a paraffin-embedded section stained with trichrome. The right panel shows the automatic detection of interstitial fractional space. The superimposition of a digital grid on the image enables digital morphometry. d | Machine vision can also be used to build models to aid prognostication. For example, qualitative and quantitative automatic detection of features of acute tubular injury may predict the course of the disease and response to therapy: the presence of only a few areas of vacuolization with specific qualitative characteristics could predict rapid recovery from an episode of acute renal failure, with normalization of serum creatinine levels, compared with a renal biopsy containing much greater levels of vacuolization. e | Computational imaging tools can be combined with other methodologies for parallel discovery. For example, computational image analysis tools can be applied to guide laser capture microdissection by identifying structures with similar pathomic signatures within the same biopsy sample and across biopsy samples. The structures with similar pathomic signatures can be captured and analysed separately, allowing for spatial mapping of pathogenomic signatures. Panel b reprinted courtesy of P. Sander and B. Ginley, University at Buffalo, NY, USA. Panel c reprinted courtesy of J. Hodgin, University of Michigan, MI, USA.
Table 3.
Use of methods for image analytics in digital nephrology studies
| Methodology | Stains | Histological primitive | Number of WSIs or cases | Task | Refs |
|---|---|---|---|---|---|
| CNN | PAS (paraffin sections) | Interstitial fibrosis, tubular atrophy, global glomerulosclerosis | 65 WSIs from transplant kidney biopsies | Segmentation of multi-classes of histological primitives | 119 |
| PAS (paraffin sections) | Glomeruli, empty Bowman capsule, globally sclerotic glomeruli, proximal tubules, distal tubules, atrophic tubules, not otherwise identified tubules, arteries | 142 WSIs from transplant kidney biopsies; 15 WSI from nephrectomies | Segmentation of multi-classes of histological primitives | 73 | |
| PAS (paraffin sections) | Non-sclerotic glomeruli, globally sclerotic glomeruli, podocyte nuclei, other nuclei, interstitial fibrosis/tubular atrophy | WSIs from mouse kidneys; WSIs from human biopsies (number of WSIs not provided) | Segmentation of multi-classes of histological primitives | 109 | |
| TRI (paraffin sections) | Glomeruli | 275 WSIs from 171 renal biopsies | Glomerular segmentation and classification | 117 | |
| H&E (frozen sections) | Non-sclerotic glomeruli; globally sclerotic glomeruli | 40 WSIs from donor kidney biopsies | Glomerular segmentation | 120 | |
| TRI (paraffin sections) | Interstitial fibrosis | 171 WSIs from native kidney biopsies | Prediction of clinical phenotype | 106 | |
| Feature-engineering RNN | PAS (paraffin sections) | Nuclei; glomerular capillary lumina; glomerular matrix | 54 WSIs from human renal biopsies and nephrectomies; 25 WSIs from mouse kidneys | Glomerular segmentation; glomerular nucleus; glomerular component detection; glomerular feature extraction; diabetic nephropathy classification/prediction | 72 |
| FCN (two-stage semi-supervised approach) | PAS (paraffin sections) | Glomeruli | 22 WSIs from mouse kidneys | Glomerular segmentation | 110 |
| CNN, SW-CNN and FCN | PAS (paraffin sections) | Glomeruli | 24 WSIs from mouse kidneys | Glomerular detection and segmentation | 107 |
| CNN and GAN | PAS, COL3, CD31, AFOG (paraffin sections) | Glomeruli, interstitial space, interstitial capillaries | 20 WSIs from mouse kidneys | Stain-dependent supervised segmentation; stain-independent unsupervised segmentation | 108 |
| Region-based CNN | TRI (paraffin sections) | Glomeruli | 87 WSIs from rat kidneys; 6 WSIs from human kidney biopsies | Glomerular localization/detection | 111 |
| Local binary patterns image feature vector and CNN | H&E, PAS, TRI, SIL, CR (paraffin sections) | Glomeruli | 17 WSIs from mouse kidney; 10 WSIs from rat kidney; 9 WSIs from native kidney biopsies | Glomerular detection; glomerular sub-classification | 110 |
| CNN, SW-CNN and FCN | PAS (paraffin sections) | Glomeruli | 24 WSIs from mouse kidneys | Glomerular segmentation | 114 |
| Segmental histogram of oriented gradients | Desmin (paraffin sections) | Glomeruli | 20 WSIs from rat kidneys | Glomerular detection | 113 |
| Colour segmentation and perceptual organization | H&E (paraffin sections) | Glomerular urinary space | WSIs from mouse kidneys | Glomerular detection | 135 |
AFOG, Acid Fuchsin Orange G; CNN, convolutional neural network; COL3, collagen type III; CR, Congo red; FCN, fully convolutional network; GAN, general adversarial network; H&E, haematoxylin and eosin; PAS, periodic acid Schiff; RNN, recurrent neural network; SIL, silver; SW-CNN, sliding window convolutional neural network; TRI, trichrome; WSIs, whole slide images.
Trade-off between hand-crafted and discovery pathomics
Hand-crafted pathomics and discovery pathomics are complementary approaches to quantitative WSI representation. Each technique has advantages and disadvantages, and their proper implementation therefore depends on the specific clinical case and intended use. Hand-crafted pathomic features are mathematically well defined, interpretable and can be easily combined with other structured health data, such as radiomics, transcriptomic or electronic health record data, to develop multi-modal digital biomarkers. However, hand-crafted features are limited to human intuition (that is, human involvement is required to define the features based on relevant domain knowledge) and can often lead to high-dimensional feature spaces that are non-trivial and computationally expensive to analyse. On the other hand, unsupervised feature generation does not require classical feature engineering (that is, features are instead generated automatically based on intrinsic patterns in the data that do not require direct human input). Furthermore, DL-based approaches are particularly useful when the input (for example, manual segmentation of a histological primitive) and output (for example, automatic segmentation) of a problem are both images or when conventional approaches cannot perform the task at hand. However, the trade-off of these benefits is loss of mathematical interpretation and the need for relatively larger datasets.
Applications and challenges
Pathomics and DL are complementary approaches to computational image analysis with both operationally and research-driven applications. These approaches facilitate the discovery of new scientific knowledge and push the boundaries of clinical intuition. By contrast, operationally driven applications provide longer term, stable solutions to existing clinical use-cases, facilitate more efficient clinical workflows and enable the use of existing knowledge in new ways72. Research- and operationally driven applications are far from independent of one another, as an inherent feedback loop exists between them. Operationally driven applications designed on the basis of existing clinical needs often lead to unanticipated tertiary findings, which may motivate new research-driven applications, resulting in updated scientific knowledge to be eventually deployed in the form of new operationally driven applications. In the next decade, nephropathology will be indisputably altered by this feedback loop between AI-driven research discoveries and their application in clinical practice.
Research-driven applications
In contrast to oncology42,77–85, only a few examples exist of research-driven applications of computational image analysis in the nephrology literature72,106–114 (Table 3). The complexity of the renal parenchyma, the heterogeneity of the pathological features and the conventional use of multiple stains for renal histology preparations, make computer-aided image analysis of renal biopsy samples challenging (Table 4). In order to produce a complete map or atlas of the entire renal parenchyma and its pathological variations, subject matter experts must manually annotate a large number of normal and abnormal histological primitives (covering all normal structures and cells, and all possible pathological variations) to train the DL algorithms. Definition of the object boundaries (for example, glomerular tuft versus glomerular unit) is an obvious important pre-requisite. Although oncological pathologists can focus their manual annotation on a single monomorphic group of cells that form a specific tumour and then apply pathomic tools to identify subvisual variations, each form of kidney and glomerular disease is represented by a very heterogeneous aggregation and variable combination of normal and abnormal cells, even within the same disease category.
Table 4.
Machine vision interrogation of nephropathology samples versus surgical pathology specimens
| Consideration | Nephropathology (kidney biopsy) sample | Surgical pathology |
|---|---|---|
| Tissue size | Generally small specimens (needle biopsies), which contain a variable amount of cortex and with limited sample of the objects of interest (e.g. focal lesions) | Resection specimens containing large fragments of tumours ± surrounding non-tumoural tissue |
| Tissue staining | A diversity of stains are routinely used. Several sections (levels) are stained for each biopsy sample, and each may contain a different representation of the objects of interest. The object of interest may not be represented in all the sections and stains | Routinely only haematoxylin and eosin |
| Tissue complexity | The kidney parenchyma contains a variety of histomorphological structures (glomerular unit, proximal and distal tubular segments, collecting ducts, interstitial space, glomerular and interstitial microvasculature, arteries, veins and lymphatics) and cell types (glomerular and tubular epithelial cells, glomerular/interstitial capillaries, venular and arterial endothelial cells) | Generally homogeneous collection of tumoural cells ± surrounding non-tumoural tissue |
| Histological complexity of disease manifestation | Quantitative and qualitative heterogeneity of disease manifestation at the structural and cellular level, often involving more than one structure or cell type at the same time. The association of diverse structural changes is not necessarily predictable | Limited histological heterogeneity. The histological manifestation of the disease can generally be graded based on a few pre-selected parameters. Although assessment of aggressiveness is based on the most aggressive cell type, there may be some variability of the tumour grade within each case |
| Clinical data complexity | A variety of clinical parameters are often present; the natural history of kidney diseases is heterogeneous across and within the same disease; genetic heterogeneity exists, and there is lack of standardization of therapies during the course of many diseases | Outcome is evaluated by time to response, progression or death |
| Data analysis | Most kidney diseases are rare; thus, the collection of samples from multiple institutions or laboratories is necessary to obtain a sufficient number of samples for meaningful studies | Many oncological diseases are fairly common. It is therefore easier to collect a large number of cases across only a few institutions or laboratories for analysis |
The choice of methodology for each purpose is an additional challenge. For example, several methodologies have been proposed for the segmentation of glomeruli: supervised classification with a separate detection and segmentation stage, which is highly dependent on the precise annotation of the glomerulus; segmentation of the white sickle space surrounding the Bowman capsule, which is highly dependent on the glomerulus relative cutting plane; and domain adaptation methods, which are dependent on a number of annotated target and source domain training data (Table 3). Generalization of the selected approach is virtually impossible, as the number of manual annotations required for each histological primitive can also vary and is dependent on the intrinsic structural heterogeneity of the histological primitive. Furthermore, as each stain might provide unique information, the algorithms will need to be generated and tested on all stains before current practice can be changed to limit AI applications to a single stain. Finally, as the spatial distribution of different pathological structures is uniquely important to renal diagnostics, future applications in 3D WSI reconstruction115,116 will facilitate the accurate assessment of histological primitives (for example, glomerular number in a biopsy sample) and the spatial interplay between pathological features (for example, interstitial fibrosis and inflammation) and associated molecular signatures.
To date, the majority of studies using DL technology with kidney tissue have targeted low-level tasks, such as the detection and segmentation of normal histological primitives (for example, glomeruli, tubules, interstitial capillaries and arteries) and universally recognized pathological primitives (for example, segmental and global glomerulosclerosis, nodular glomerulosclerosis, interstitial fibrosis and tubular atrophy) on WSIs from paraffin72,73,111,117–119 and frozen sections120 (Table 3; Fig. 2). More complex tasks have been executed by using a DL architecture to stage diabetic glomerulosclerosis72 or to determine the association of interstitial fibrosis with clinical phenotypes such as chronic kidney disease stage, serum creatinine level, nephrotic range proteinuria at the time of the biopsy and renal survival at 1, 3 and 5 years (Fig. 2). Interestingly, the trained CNN models outperformed the pathologist-estimated fibrosis score in predicting the output classes106.
Many pathomic tools for the prediction of outcomes, genotype and disease mechanisms have not yet been applied to renal pathology. The current lack of training and knowledge of the principles of computer-aided image analysis and AI among pathologists for manual annotation and of data scientists for algorithm development are just two factors that have prevented this field from moving forwards faster.
The development of computational imaging tools for nephropathology and pathology can exemplify the concept of parallel discoveries (Fig. 2). For example, high-throughput gene expression analysis and comparison of multiple samples at the same time can be achieved by applying tissue microarrays to glass slides containing fragments of tissue from numerous patients121. This technique provides an opportunity to apply computational imaging techniques to capture the desired histological primitive from an image, match histological and cytological features across a range of length scales, and combine the information to create a single construct from multiple images in an array-based format (an image microarray), on multiple patients at the same time. The combination of tissue and image microarrays enables the assemblage of many fields, reflecting highly specific morpho-omic signatures122.
Similarly, the combination of laser capture microdissection techniques to isolate structures and cells can be enhanced via computational imaging techniques122. The pairing of computational image analysis with laser capture microdissection can be purposed to automatically detect the ROI as predefined by the pathologist, thus substituting the pathologist for the task, and/or to augment the process with information that would otherwise not be captured by the pathologist, but that can offer unique opportunities for omics discovery. For example, automatic detection of histological primitives followed by cluster analysis of those primitives may identify structures that share subvisual features independently of their morphological similarity at the human eye. Visualization tools to map these discoveries back to changes at the tissue level might improve our understanding of disease mechanisms123.
Arguably the most complex task currently underway in the field of nephropathology is the development of a functional kidney cell atlas by KPMP investigators. The development of a histology-based atlas of normal and pathological renal biopsy samples, where an array of validated morpho-genomic and morpho-transcriptomic data extractions will be spatially coupled to the WSI data, will provide an interactive computational framework in which interrogation can take place across all data domains, within an anatomic framework. This interactive framework is expected to provide deeper insights into cell and disease classification and lead to the identification of actionable targets at the molecular level. Unlike most prior efforts, the KPMP effort makes digital histology data a centre point for possible initial discovery modes.
Operationally driven applications
As research-driven applications become operationalized, AI tools for image analysis are likely to take a role in decision support through computer-assisted diagnosis (which, importantly, differs from computer-generated diagnosis). These decision support tools may be generated by processes such as DL algorithms, pathomics with feature engineering and computational integration of image data with data from other domains (Fig. 3). For example, automated detection of histological primitives has potential operational applications owing to its ability to increase the reproducibility of estimates of the distribution, severity and quantity of structures and lesions that are known predictors of outcome or molecular signatures but which have limited reproducibility when assessed manually (for example, globally sclerotic glomeruli, interstitial fibrosis, diabetic glomerulosclerosis stage)4,5,12,16,71; and to provide a quantitative estimate of histological primitives that can be visualized but cannot be reliably counted manually (for example, podocytes, interstitial capillaries)98,100–104. Thus, DL quantification of these histological primitives can augment the ability of the pathologist to interpret and diagnose kidney diseases.
Fig. 3. The nephropathology digital ecosystem.
The digital ecosystem covers three phases: the digital pathology phase (analogue to digital conversion), the knowledge extraction phase, which relies on human and/or artificial intelligence (AI), and the actionable intelligence phase, in which integrated knowledge is applied to patient care. Each phase begins with input data and ends by generating output data that represent the input for the successive phase. In the digital pathology phase, glass slides from the renal biopsy sample (that is, the analogue input data), are converted into whole slide images (WSIs) (that is, digital output data). The WSIs represent the input data for the knowledge extraction phase, from which useful information is generated (output data). Knowledge extraction can be broken down into two types: human cognition and AI. AI techniques can be implemented as companion diagnostic tools alongside human cognition. Human cognition is employed when WSIs (input image data) are visually assessed or scored by a trained pathologist to generate diagnoses or morphological profiles as digital pathology-derived knowledge (output data). AI-based machine vision comprises both hand-crafted pathomics and discovery pathomics. For hand-crafted pathomics, image data (input) is transcribed into pathomic signatures (the pathomic feature space) and then translated into digital pathology-derived knowledge (the output data) using machine-learning models. For discovery pathomics, the image data (the input data) are transcribed or encoded into structured data and then decoded into an output signal (the output image data) using deep learning. Finally, in the actionable intelligence phase, the knowledge obtained from the digital images (input data) is integrated with other data types, for example, omics and clinical data, and used to diagnose, prognosticate and select targeted treatments (output data) for the patient.
The information contained in WSIs that the human cognitive system cannot directly perceive, known as subvisual features, provide an opportunity to pose new scientific questions and novel insights that have not yet been conceived or imagined based on conventional pathology approaches. Although the implementation of these approaches and integration of new knowledge arising from these findings into clinical practice will be complex and require extensive validation, future ML pipelines will be an integral component of the pathologist’s workflow as decision support tools. Indeed, some of these algorithms are already making their way into routine practice in the field of radiation oncology42,124,125. The ultimate challenge will be the integration and implementation of computational image analysis-generated data with other knowledge domains to establish fused datasets and associated morpho-omic classifications, prognostications and predictions for kidney diseases at the individual patient level (that is, fused morpho-omic signatures). These applications will also have a role in drug discovery36, in the identification of participants for enrolment in clinical trials and in rationally determining the dosage of therapeutic agents.
Despite the excitement about these new approaches, pathologists and data scientists are advocating for cautious scepticism in the interpretation of quantitative results derived from the application of image analysis techniques115–119. Similar caution is warranted when considering the potential clinical implementation of such approaches, recognizing that vigorous validation should always be an integral step of the deployment process126–130. Our understanding of the key drivers behind AI-based determinations is still incomplete, and we still do not possess sufficiently developed tools to assess performance metrics or enforce QC measures in the clinical setting. As algorithms become available for clinical deployment, regulatory bodies will need to work closely with the other stake-holders (such as pathologists, computer scientists, nephrologists, pathology laboratories and institutions) to establish a framework for their approval and implementation.
Limitations of sample size
As digital pathology becomes more widely accessible, the analytical tools of AI and ML will reveal new opportunities with which to develop large-scale, kidney-centric data ecosystems, to combine datatypes and datasets for discovery and validation, and for the practice of precision nephrology (Table 2). However, ML models are based on learning patterns in data and require sufficient numbers of training examples, with the added observation that most AI-driven solutions require large amounts of data during their training process to be both successful and generalizable. This requirement could prove to be a challenge in their application to rare kidney diseases, where sample sizes are inherently small. Fortunately, novel AI and ML approaches that can be applied to small training sets are now becoming available. These approaches use methodologies such as data augmentation, whereby computational perturbations (usually of a spatial or colorimetric nature) are used to increase the variation of a training set to improve model generalization; new approaches to high-dimensional data clustering131, whereby samples are grouped into similar categories on the basis of intrinsic data properties without the need for labelled training examples; or unsupervised machine-learning processes, which provide reasonably robust image segmentation performance when used to identify normal renal histological primitives100,118. All these capabilities, taken together, represent a promising future for these new computational tools, and we expect that they will augment the clinical practice of both nephrology and pathology in the near future.
The limitations of ML approaches for use on small datasets will also be addressed by the development of robust learning health-care systems. Aided by AI technology, learning health-care systems rely on the data-driven generation of knowledge to enable the practice of evidence-based medicine132–136. Implementation of this technology-driven health-care model will facilitate the sharing of data and ML hyper-parameters (that is, parameters that refer to the model selection task or to the quality and speed of the learning-process algorithm) between institutions, leading to aggregate models with better potential for generalizability. Furthermore, the continued establishment and management of large international nephrology consortia will also have a key role in the success of AI applications in rare kidney diseases. Given the nature of ongoing consortium-driven kidney research, which is most typically based on cross-institutional and multi-omics data13,137, it is highly likely that these consortia will serve as the blueprint for future collaborative AI endeavours in the renal space that operate at scale.
Training of medical professionals
Renal pathologists will have a critical role in establishing the digital nephropathology ecosystem, developing clinically relevant ML and AI algorithms, and deploying such algorithms in routine clinical practice. Renal pathologists are the subject matter experts in renal histology and, as such, they will need to work closely with data and machine vision scientists to design studies and protocols, ensure that relevant WSI datasets are made available, annotate WSI libraries, and finally curate, validate and interpret the data that arise from these efforts. It will be critical for pathologists to be well-versed in AI and ML applications and methods, including their limitations, so that these powerful tools can be applied to clinical practice quickly, effectively and reliably. Owing to the interdisciplinary nature of integrated computational pathology, the need for nephrologists to acquire new knowledge and skills is equally germane, as these individuals represent the penultimate consumers of this new class of information as generated by these approaches. Acknowledgment that AI will be integral part of medicine is the first step towards the re-design of curricula for medical schools and residency or fellowship training programmes to address the current inadequacy in preparing medical professionals for the imminent future138 (Table 2).
Conclusions
We can borrow concepts and analogies from biology to summarize the transformative process that is beginning to affect the nephropathology community. This transformative process is based on a variety of events that represent the first decisive steps in the long anticipated pathway towards a new digitally enhanced ecosystem139. Humans are learning to operate within the confines of this new paradigm, as represented by digital avatars of living components, the operability of which is closely linked to the non-living components of their environment. These biotic and abiotic constituents are linked together by new, interdependent cycles, dynamisms and flows. The next-generation nephropathology is expected to operate within this new digital ecosystem and apply artificial (augmenting) intelligence tools to implement human–machine synergistic protocols in clinical practice to better categorize, predict and prognosticate kidney diseases, and to identify candidates for conventional or novel therapeutic approaches. The critical steps needed for the success of this transformational process ultimately depend on practitioners in the field being open to acquiring new skills, and the adaptation of practice patterns to support the new discovery environments made possible by these technologies. Most importantly, it will be essential for renal pathologists and nephrologists to understand both the strengths and inherent limitations of digital pathology technologies and, at the same time, be willing to invest the time and effort needed to shoulder their adoption.
Acknowledgements
L.B. has received research support grants from the National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health (U54 DK083912, UM1 DK100845, 1R01-DK118431-01, R01 DK108805, M1701051) and the Nephcure Foundation. A.M. has received research support grants from the National Cancer Institute of the National Institutes of Health (1U24CA199374-01, R01CA202752-01A1, R01CA208236-01A1, R01 CA216579-01A1, R01 CA220581-01A1, 1U01 CA239055-01); the National Institute for Biomedical Imaging and Bioengineering (1R43EB028736-01); the National Center for Research Resources under award number 1 (C06 RR12463-01); the VA Merit Review Award IBX004121A from the United States Department of Veterans Affairs Biomedical Laboratory Research and Development Service; the DOD Breast Cancer Research Program Breakthrough Level 1 Award (W81XWH-19-1-0668); the DOD Prostate Cancer Idea Development Award W81XWH-15-1-0558; the DOD Lung Cancer Investigator-Initiated Translational Research Award (W81XWH-18-1-0440); the DOD Peer Reviewed Cancer Research Program (W81XWH-16-1-0329); the Ohio Third Frontier Technology Validation Fund; the Wallace H. Coulter Foundation Program in the Department of Biomedical Engineering and The Clinical and Translational Science Award Program (CTSA) at Case Western Reserve University. U.G.J.B. has received research support grants from the National Cancer Institute of the National Institutes of Health (U24CA199374, U2CDK114886, U54(MMDHHS), NIH1RC2DK01). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health, the U.S. Department of Veterans Affairs, the Department of Defense, or the United States Government.
Glossary
- Computational image analysis
The application of artificial intelligence for the extraction of meaningful information (output) from the digital images (the input).
- Artificial intelligence
(AI). A branch of computer science that deals with the ability of computers to mimic human intelligence or cognitive functions.
- Virtual microscopy
The use of microscopy and digital technology to capture an image from a stained tissue section and transmit it in real time as a static image (digital pathology) or live image (telepathology) over computer networks, so that the viewer can access the image on a remote computer.
- Companion diagnostics
Generally, indicates a test developed based on a companion biomarker and used as a companion to a medical intervention. Companion diagnostics can identify patients who are most likely to benefit from a therapeutic approach.
- Content-based image retrieval
Tissue interrogation using machine vision technology to search an image dataset by analysing information contained and derived from the images of a dataset. In the context of digital pathology, the content refers to the colours, shape and texture of the image.
- Machine vision
Settings in which image-based computational algorithms and processing pipelines are applied to provide automation to tasks that would otherwise represent substantial cognitive tasks for human observers alone.
- Dynamic telepathology
A process whereby a live video image is viewed with the assistance of an operator at the site from which the image is transmitted, or via a robotic mode, where the viewer remotely controls the transmission of the image at the originating site.
- Whole slide image
(WSI). High-resolution replica of a glass slide created by a slide scanner that can be viewed on a computer screen (a virtual slide).
- Z-dimensional focus
The coordinate axis representing the depth of an image.
- Dimensionality
The number of data attributes that make up a dataset; the number of degrees-of-freedom (i.e. features) in a machine-learning model.
- Machine-learning
(ML). A branch of artificial intelligence that builds mathematical models based on training data in order to make predictions or decisions without being explicitly programmed to perform the task.
- Image interrogation
The application of a variety of machine vision technologies to extract quantitative information from whole slide images.
- Deep learning
A class of machine-learning algorithms that has networks capable of unsupervised learning from data that are unstructured or unlabelled. It uses multiple layers to progressively extract higher level features from the input (in this case, the image).
- Segmentation
One of the most important tasks of machine vision. Manual segmentation (annotation) is performed by subject matter experts, who use digital annotation tools to delineate the boundaries of the object (region of interest) or histological primitive. Automatic segmentation is performed by the deep learning model trained to detect the boundaries of the object.
- Prediction
The result of a machine-learning model when operating on a new data point based on historical data samples.
- Health Insurance Portability and Accountability Act
An Act passed by US Congress in 1996 to reduce health-care fraud and abuse, enable transfer of insurance coverage when changing jobs, standardize health-care information on electronic billing and other formats, and protect confidentiality of health information.
- Computer-aided quantitative assessment
Use of a computer output to guide clinical decision-making and interpretation of clinical data.
- Histological primitives
Discrete normal or abnormal structures (e.g. glomerular tufts, globally sclerotic glomeruli, tubules, atrophic tubules and vessels) or cells (e.g. lymphocytes, podocytes and proximal tubular cells).
- Digital scanner
A device that optically scans images or glass slides and converts them into digital images or whole slide images.
- Image management software
Software that allows the acquisition, organization and viewing of pathology image data and its associated metadata.
- Generalizable
The ability of a machine-learning algorithm to remain effective across a wide range of data examples and applications.
- Analogue data
Data that are represented and analysed in a physical way (e.g. a glass slide) as opposed to a digital way (e.g. a whole slide image). Analogue data are stored in physical structures.
- Physical abstraction layer
The physical computational machines, which represent a hard stop with respect to the computational scale and storage capacity of a computer and typically cannot be overcome by the addition of more software.
- Feature vector
Typically, one or more spatial constructs selected by a subject matter expert that serves as a training template for any class of machine-learning algorithm, including convolutional neural networks.
- Ontology
Represented sharable knowledge, a specification of a conceptualization, in the form of concepts and categories for a specific domain, associated properties and relationships between them. Ontologies function using a controlled vocabulary to connect terms and use links, logical definitions, concepts and networks of well-defined relationships.
- Data aggregators
Aggregation or compiling of information from databases with the goal of preparing such information for data processing.
- Pathomics
The high-throughput extraction and analysis of features derived from digital pathology images, typically achieved using techniques from the machine vision field, image processing and machine learning.
- Hand-crafted pathomics
Pathomics with feature engineering — a process that uses domain knowledge to extract features (attributes or properties) from raw data.
- Discovery pathomics
An approach that uses deep-learning-based feature extraction to learn complex patterns in the data or image. It commonly uses artificial neural networks.
- Hand-crafted features
Mathematically well-defined features derived from data using feature engineering techniques.
- Digital image signature
A mathematical vector, the components of which denote various quantitative imaging features that collectively represent an image.
- Wavelet decomposition
A signal processing technique whereby an input signal (e.g. an image) is passed through a cascade of digital filters (i.e. high-pass filters or low-pass filters) to generate a set of bandwidth-limited signal components (e.g. filtered images), which collectively contain the full information of the original signal.
- Fractal geometry
A non-regular geometric shape, where the degree of non-regularity is invariant across different spatial scales.
- Voronoi tessellation
A mathematical approach where a plane (e.g. a whole slide image) is partitioned into sub-regions based on their relative proximity to a given set of objects (e.g. image landmarks, histological primitives, etc.).
- Minimum spanning tree
The subset of the edges of a weighted graph that connects all vertices together by minimizing total edge weight. For example, if the physical distances (edges) between a set of histological primitives (vertices) on a whole slide image is represented as a graph, then its minimum spanning tree is the combination of edges that minimize total physical distance, while still connecting all histological primitives.
- Feature engineering
The process of computationally deriving features from databases on domain knowledge. These features act as mathematical inputs to machine-learning algorithms.
- Classifier
A mathematical function, typically formulated via machine learning, that maps a set of input variables to discrete output variables, known as classes.
- Convolutional neural networks
(CNNs). Types of deep, feedforward networks, comprising multiple layers to infer an output from an input (the image). CNNs use building blocks to learn and extract feature maps from an image, and filters between the input and the output that are connected only to a fixed region from the previous layer (and not the entire layer). It also comprises pooling layers to reduce the dimensionality of the features.
- Artificial neural networks
A set of connected input (image) and output (a defined class or number) units where each connection has a weight associated with it. During the learning phase, the network learns by adjusting the weights to predict the correct class label (for categorical data) or regression value (for continuous data) of the input image. There are different types of artificial neural networks (e.g. convolutional, recurrent, generative adversarial).
- Fully convolutional networks
Convolutional networks that lack fully connected layers and comprise only a hierarchy of convolutional layers. They can be used to learn representations from every pixel and make pixel-level predictions.
- Recurrent neural networks
Acquire inputs over different time points, taking into account the status of the input at the different time points and learning from the discrete earlier inputs, displaying dynamic behaviours.
- Generative adversarial networks
Learn from implementing two simultaneous neural networks, one producing the data and the other evaluating the agreement between the generated data and the original input.
- Radiomics
The high-throughput extraction and analysis of features derived from radiological medical images, typically using techniques in machine vision, image processing and machine learning.
- High-dimensional feature spaces
A set of descriptive data attributes (i.e. features) measured across a cohort of samples, where the number of features (known as the dimension of the data) is significantly larger than the number of samples. Feature spaces are typically represented as matrices, where one dimension (i.e. matrix columns) represents different features and the other dimension (i.e. matrix rows) represents different samples.
- Use-cases
Descriptions of how to perform a task using examples, outlining the system’s behaviour in response to the request.
- Image microarray
The assembly of uniformly sized and resolution-matched images, representing key morphological features and fields of view, and aggregated into a single monolithic digital image file in an array format.
- Cluster analysis
Statistical data analysis technique that is aimed at grouping objects in homogeneous groups (clusters).
- Subvisual features
Features contained in the digital images that cannot be captured by the human eye, and require the application of machine vision technology. Synonymous with pathomic feature extraction.
- Fused morpho-omic signatures
The integration of patient-specific data derived from multiple omic domains (e.g. pathomics, genomics, transcriptomics and metabolomics) to define patient-specific phenotypes.
- Learning health-care systems
Systems built around evidence-based medicine and on knowledge generation processes embedded in daily clinical practice. They rely on interdisciplinary approaches to generating large electronic health datasets, which are used to generate new knowledge by research, resulting in quality improvement of patient care.
- Computational pathology
The science that includes big data generation and analysis, image processing, data mining and data fusion of digital pathology data.
Author contributions
All authors contributed to the general design of this Review with specific areas of emphasis or contribution enumerated as follows: K.J.L. contributed to the sections on machine vision and provided technical insight for the reference tables and figures. S.M.H. and A.M. provided historical context and general fact-checking throughout the manuscript. L.B. and U.G.J.B. carried out a thematic review of the current state of digital pathology, providing substantial narrative content and final editorial review.
Competing interests
L.B. is a consultant for Vertex, Sangamo and Protalix. She is also on the scientific advisory board for Vertex. Additionally, she is the chair of the pathology committee for NEPTUNE, director of the Digital Pathology Repository for CureGN and curator for the COVID-19 digital pathology repository hosted at the NIH/NCI. She is involved as a co-investigator in KPMP, and as co-PI in an R01 that uses digital pathology material. A.M. is an equity holder in Elucid Bioimaging and in Inspirata Inc. In addition, he has served as a scientific advisory board member for Inspirata Inc, AstraZeneca, Bristol Meyers-Squibb and Merck. Currently, he serves on the advisory board of Aiforia Inc. He also has sponsored research agreements with Philips and Bristol Meyers-Squibb. His technology has been licensed to Elucid Bioimaging. He is also involved in a NIH U24 grant with PathCore Inc, and three different R01 grants with Inspirata Inc. U.G.J.B. is a member of the advisory Board for Inspirata Inc, a company that provides digital pathology workflow solutions. The other authors declare no competing interests.
Footnotes
Peer review information
Nature Reviews Nephrology thanks M. Hermsen, P. Sarder and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Barisoni L, Hodgin JB. Digital pathology in nephrology clinical trials, research, and pathology practice. Curr. Opin. Nephrol. Hypertens. 2017;26:450–459. doi: 10.1097/MNH.0000000000000360. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Pantanowitz L, et al. Experience with multimodality telepathology at the University of Pittsburgh Medical Center. J. Pathol. Inf. 2012;3:45. doi: 10.4103/2153-3539.104907. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Dietz RL, Hartman DJ, Pantanowitz L. Systematic review of the use of telepathology during intraoperative consultation. Am. J. Clin. Pathol. 2019;153:198–209. doi: 10.1093/ajcp/aqz155. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Azancot MA, et al. The reproducibility and predictive value on outcome of renal biopsies from expanded criteria donors. Kidney Int. 2014;85:1161–1168. doi: 10.1038/ki.2013.461. [DOI] [PubMed] [Google Scholar]
- 5.Liapis H, et al. Banff histopathological consensus criteria for preimplantation kidney biopsies. Am. J. Transpl. 2017;17:140–150. doi: 10.1111/ajt.13929. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Barisoni L, et al. Novel quantitative method to evaluate globotriaosylceramide inclusions in renal peritubular capillaries by virtual microscopy in patients with Fabry disease. Arch. Pathol. Lab. Med. 2012;136:816–824. doi: 10.5858/arpa.2011-0350-OA. [DOI] [PubMed] [Google Scholar]
- 7.Pantanowitz L, Szymas J, Yagi Y, Wilbur D. Whole slide imaging for educational purposes. J. Pathol. Inf. 2012;3:46. doi: 10.4103/2153-3539.104908. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Saco A, Bombi JA, Garcia A, Ramirez J, Ordi J. Current status of whole-slide imaging in education. Pathobiology. 2016;83:79–88. doi: 10.1159/000442391. [DOI] [PubMed] [Google Scholar]
- 9.Kumar, N., Gupta, R. & Gupta, S. Whole slide imaging (WSI) in pathology: current perspectives and future directions. J. Digit. Imaging10.1007/s10278-020-00351-z (2020). [DOI] [PMC free article] [PubMed]
- 10.Barisoni L, et al. Digital pathology imaging as a novel platform for standardization and globalization of quantitative nephropathology. Clin. Kidney J. 2017;10:176–187. doi: 10.1093/ckj/sfw129. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Barisoni L, et al. Digital pathology evaluation in the multicenter nephrotic syndrome study network (NEPTUNE) Clin. J. Am. Soc. Nephrol. 2013;8:1449–1459. doi: 10.2215/CJN.08370812. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Barisoni L, et al. Reproducibility of the NEPTUNE descriptor-based scoring system on whole-slide images and histologic and ultrastructural digital images. Mod. Pathol. 2016;29:671–684. doi: 10.1038/modpathol.2016.58. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Mariani LH, et al. CureGN study rationale, design, and methods: establishing a large prospective observational study of glomerular disease. Am. J. Kidney Dis. 2019;73:218–229. doi: 10.1053/j.ajkd.2018.07.020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Nast CC, et al. Morphology in the digital age: integrating high-resolution description of structural alterations with phenotypes and genotypes. Semin. Nephrol. 2015;35:266–278. doi: 10.1016/j.semnephrol.2015.04.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Rosenberg AZ, et al. The application of digital pathology to improve accuracy in glomerular enumeration in renal biopsies. PLoS One. 2016;11:e0156441. doi: 10.1371/journal.pone.0156441. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Zee J, et al. Reproducibility and feasibility of strategies for morphologic assessment of renal biopsies using the nephrotic syndrome study network digital pathology scoring system. Arch. Pathol. Lab. Med. 2018;142:613–625. doi: 10.5858/arpa.2017-0181-OA. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.US Food and Drug Administration. FDA allows marketing of first whole slide imaging system for digital pathology. FDAhttps://www.fda.gov/news-events/press-announcements/fda-allows-marketing-first-whole-slide-imaging-system-digital-pathology (2017).
- 18.Amin M, et al. Integration of digital gross pathology images for enterprise-wide access. J. Pathol. Inf. 2012;3:10. doi: 10.4103/2153-3539.93892. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Leica Biosystems. Leica biosystems receives FDA 510(k) clearance to market a digital pathology system for primary diagnosis. PR Newswirehttps://www.prnewswire.com/news-releases/leica-biosystems-receives-fda-510k-clearance-to-market-a-digital-pathology-system-for-primary-diagnosis-300857825.html (2019).
- 20.Evans AJ, et al. US food and drug administration approval of whole slide imaging for primary diagnosis: a key milestone is reached and new questions are raised. Arch. Pathol. Lab. Med. 2018;142:1383–1387. doi: 10.5858/arpa.2017-0496-CP. [DOI] [PubMed] [Google Scholar]
- 21.Imaging Technology News. FDA grants breakthrough designation to Paige.AI. Imaging Technology Newshttps://www.itnonline.com/content/fda-grants-breakthrough-designation-paigeai (2019).
- 22.Menter T, Nicolet S, Baumhoer D, Tolnay M, Tzankov A. Intraoperative frozen section consultation by remote whole-slide imaging analysis — validation and comparison to robotic remote microscopy. J. Clin. Pathol. 2019;73:350–352. doi: 10.1136/jclinpath-2019-206261. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Brachtel E, Yagi Y. Digital imaging in pathology — current applications and challenges. J. Biophotonics. 2012;5:327–335. doi: 10.1002/jbio.201100103. [DOI] [PubMed] [Google Scholar]
- 24.van Diest PJ, et al. Pathology image exchange: the Dutch digital pathology platform for exchange of whole-slide images for efficient teleconsultation, telerevision, and virtual expert panels. JCO Clin. Cancer Inf. 2019;3:1–7. doi: 10.1200/CCI.18.00146. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Madabhushi A, et al. Integrated diagnostics: a conceptual framework with examples. Clin. Chem. Lab. Med. 2010;48:989–998. doi: 10.1515/CCLM.2010.193. [DOI] [PubMed] [Google Scholar]
- 26.Baidoshvili A, et al. Validation of a whole-slide image-based teleconsultation network. Histopathology. 2018;73:777–783. doi: 10.1111/his.13673. [DOI] [PubMed] [Google Scholar]
- 27.Baidoshvili A, et al. Evaluating the benefits of digital pathology implementation: time savings in laboratory logistics. Histopathology. 2018;73:784–794. doi: 10.1111/his.13691. [DOI] [PubMed] [Google Scholar]
- 28.Hanna MG, et al. Implementation of digital pathology offers clinical and operational increase in efficiency and cost savings. Arch. Pathol. Lab. Med. 2019;143:1545–1555. doi: 10.5858/arpa.2018-0514-OA. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Retamero JA, Aneiros-Fernandez J, Del Moral RG. Complete digital pathology for routine histopathology diagnosis in a multicenter hospital network. Arch. Pathol. Lab. Med. 2019;144:221–228. doi: 10.5858/arpa.2018-0541-OA. [DOI] [PubMed] [Google Scholar]
- 30.Yao X. Evolving artificial neural networks. Proc. IEEE. 1999;87:1423–1447. [Google Scholar]
- 31.Haykin, S. Neural Networks (Prentice Hall, 1994).
- 32.McCarthy J, Minsky ML, Rochester N, Shannon CE. A proposal for the Dartmouth summer research project on artificial intelligence, August 31, 1955. AI Mag. 2006;27:12–14. [Google Scholar]
- 33.McCarthy, J. J., Minsky, M. L. & Rochester, N. Artificial intelligence (Research Laboratory of Electronics (RLE) at the Massachusetts Institute of Technology (MIT), 1959).
- 34.Deng L. Deep learning: methods and applications. Trends Signal. Process. 2014;7:197–387. [Google Scholar]
- 35.LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521:436–444. doi: 10.1038/nature14539. [DOI] [PubMed] [Google Scholar]
- 36.Vamathevan J, et al. Applications of machine learning in drug discovery and development. Nat. Rev. Drug. Discov. 2019;18:463–477. doi: 10.1038/s41573-019-0024-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Moore R. Computed radiography. Med. Electron. 1980;11:78–79. [PubMed] [Google Scholar]
- 38.Zhao W, Rowlands JA. X-ray imaging using amorphous selenium: feasibility of a flat panel self-scanned detector for digital radiology. Med. Phys. 1995;22:1595–1604. doi: 10.1118/1.597628. [DOI] [PubMed] [Google Scholar]
- 39.Neitzel U, Maack I, Gunther-Kohfahl S. Image quality of a digital chest radiography system based on a selenium detector. Med. Phys. 1994;21:509–516. doi: 10.1118/1.597389. [DOI] [PubMed] [Google Scholar]
- 40.Kandarakis I, Cavouras D, Panayiotakis GS, Nomicos CD. Evaluating X-ray detectors for radiographic applications: a comparison of ZnSCdS:Ag with Gd2O2S:Tb and Y2O2S:Tb screens. Phys. Med. Biol. 1997;42:1351–1373. doi: 10.1088/0031-9155/42/7/009. [DOI] [PubMed] [Google Scholar]
- 41.Antonuk LE, et al. A real-time, flat-panel, amorphous silicon, digital x-ray imager. Radiographics. 1995;15:993–1000. doi: 10.1148/radiographics.15.4.7569143. [DOI] [PubMed] [Google Scholar]
- 42.Bera K, Schalper KA, Rimm DL, Velcheti V, Madabhushi A. Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 2019;16:703–715. doi: 10.1038/s41571-019-0252-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Khorrami M, et al. Changes in CT radiomic features associated with lymphocyte distribution predict overall survival and response to immunotherapy in non-small cell lung cancer. Cancer Immunol. Res. 2020;8:108–119. doi: 10.1158/2326-6066.CIR-19-0476. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Gadegbeku CA, et al. Design of the nephrotic syndrome study network (NEPTUNE) to evaluate primary glomerular nephropathy by a multidisciplinary approach. Kidney Int. 2013;83:749–756. doi: 10.1038/ki.2012.428. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Lochmuller H, et al. RD-connect, NeurOmics and EURenOmics: collaborative European initiative for rare diseases. Eur. J. Hum. Genet. 2018;26:778–785. doi: 10.1038/s41431-018-0115-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Consortium HA, et al. Research capacity. Enabling the genomic revolution in Africa. Science. 2014;344:1346–1348. doi: 10.1126/science.1251546. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Alyodawi K, et al. Compression of morbidity in a progeroid mouse model through the attenuation of myostatin/activin signalling. J. Cachexia Sarcopenia Muscle. 2019;10:662–686. doi: 10.1002/jcsm.12404. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Heinzel A, et al. Validation of plasma biomarker candidates for the prediction of eGFR decline in patients with type 2 diabetes. Diabetes Care. 2018;41:1947–1954. doi: 10.2337/dc18-0532. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Kammer M, et al. Integrative analysis of prognostic biomarkers derived from multiomics panels helps discrimination of chronic kidney disease trajectories in people with type 2 diabetes. Kidney Int. 2019;96:1381–1388. doi: 10.1016/j.kint.2019.07.025. [DOI] [PubMed] [Google Scholar]
- 50.Perco P, Pena M, Heerspink HJL, Mayer G, Consortium BE-D. Multimarker panels in diabetic kidney disease: the way to improved clinical trial design and clinical practice? Kidney Int. Rep. 2019;4:212–221. doi: 10.1016/j.ekir.2018.12.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Freedman BI, Julian BA. Evaluation of potential living kidney donors in the APOL1 era. J. Am. Soc. Nephrol. 2018;29:1079–1081. doi: 10.1681/ASN.2018020137. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Freedman BI, Moxey-Mims M. The APOL1 long-term kidney transplantation outcomes network — APOLLO. Clin. J. Am. Soc. Nephrol. 2018;13:940–942. doi: 10.2215/CJN.01510218. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.Hommos MS, et al. Global glomerulosclerosis with nephrotic syndrome; the clinical importance of age adjustment. Kidney Int. 2018;93:1175–1182. doi: 10.1016/j.kint.2017.09.028. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Mariani LH, et al. Interstitial fibrosis scored on whole-slide digital imaging of kidney biopsies is a predictor of outcome in proteinuric glomerulopathies. Nephrol. Dial. Transpl. 2018;33:310–318. doi: 10.1093/ndt/gfw443. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Ju W, et al. Tissue transcriptome-driven identification of epidermal growth factor as a chronic kidney disease biomarker. Sci. Transl. Med. 2015;7:316ra193. doi: 10.1126/scitranslmed.aac7071. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 56.Lemley KV, et al. Morphometry predicts early GFR change in primary proteinuric glomerulopathies: a longitudinal cohort study using generalized estimating equations. PLoS One. 2016;11:e0157148. doi: 10.1371/journal.pone.0157148. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.Abels E, et al. Computational pathology definitions, best practices, and recommendations for regulatory guidance: a white paper from the Digital Pathology Association. J. Pathol. 2019;249:286–294. doi: 10.1002/path.5331. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58.Cross S., et al. Best practice recommendations for implementing digital pathology (The Royal College of Pathologists, 2018).
- 59.Williams BJ, Treanor D. Practical guide to training and validation for primary diagnosis with digital pathology. J. Clin. Pathol. 2019;73:418–422. doi: 10.1136/jclinpath-2019-206319. [DOI] [PubMed] [Google Scholar]
- 60.Pantanowitz L, et al. Validating whole slide imaging for diagnostic purposes in pathology: guideline from the college of American Pathologists Pathology and Laboratory Quality Center. Arch. Pathol. Lab. Med. 2013;137:1710–1722. doi: 10.5858/arpa.2013-0093-CP. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 61.Newitt, V. N. Program zeroes in on histology digital scan connection. CAP Todayhttps://www.captodayonline.com/program-zeroes-in-on-histology-digital-scan-connection/ (2018).
- 62.Yagi Y, Gilbertson JR. Digital imaging in pathology: the case for standardization. J. Telemed. Telecare. 2005;11:109–116. doi: 10.1258/1357633053688705. [DOI] [PubMed] [Google Scholar]
- 63.Yagi Y. Color standardization and optimization in whole slide imaging. Diagn. Pathol. 2011;6(Suppl 1):15. doi: 10.1186/1746-1596-6-S1-S15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 64.Bellur SS, et al. Reproducibility of the Oxford classification of immunoglobulin A nephropathy, impact of biopsy scoring on treatment allocation and clinical relevance of disagreements: evidence from the VALidation of IGA study cohort. Nephrol. Dial. Transpl. 2019;34:1681–1690. doi: 10.1093/ndt/gfy337. [DOI] [PubMed] [Google Scholar]
- 65.Working Group of the International IgA Nephropathy Network et al. The Oxford classification of IgA nephropathy: rationale, clinicopathological correlations, and classification. Kidney Int. 2009;76:534–545. doi: 10.1038/ki.2009.243. [DOI] [PubMed] [Google Scholar]
- 66.Working Group of the International IgA Nephropathy Network et al. The Oxford classification of IgA nephropathy: pathology definitions, correlations, and reproducibility. Kidney Int. 2009;76:546–556. doi: 10.1038/ki.2009.168. [DOI] [PubMed] [Google Scholar]
- 67.Leo P, et al. Evaluating stability of histomorphometric features across scanner and staining variations: prostate cancer diagnosis from whole slide images. J. Med. Imaging. 2016;3:047502. doi: 10.1117/1.JMI.3.4.047502. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 68.Yoshida H, et al. Meeting report: the international workshop on harmonization and standardization of digital pathology image, held on April 4, 2019 in Tokyo. Pathobiology. 2019;86:322–324. doi: 10.1159/000502718. [DOI] [PubMed] [Google Scholar]
- 69.Janowczyk A, Zuo R, Gilmore H, Feldman M, Madabhushi A. HistoQC: an open-source quality control tool for digital pathology slides. JCO Clin. Cancer Inf. 2019;3:1–7. doi: 10.1200/CCI.18.00157. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 70.Bajema IM, et al. Revision of the International Society of Nephrology/Renal Pathology Society classification for lupus nephritis: clarification of definitions, and modified National Institutes of Health activity and chronicity indices. Kidney Int. 2018;93:789–796. doi: 10.1016/j.kint.2017.11.023. [DOI] [PubMed] [Google Scholar]
- 71.Dasari S, Chakraborty A, Truong L, Mohan C. A systematic review of interpathologist agreement in histologic classification of lupus nephritis. Kidney Int. Rep. 2019;4:1420–1425. doi: 10.1016/j.ekir.2019.06.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 72.Ginley B, et al. Computational segmentation and classification of diabetic glomerulosclerosis. J. Am. Soc. Nephrol. 2019;30:1953–1967. doi: 10.1681/ASN.2018121259. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 73.Hermsen M, et al. Deep learning-based histopathologic assessment of kidney tissue. J. Am. Soc. Nephrol. 2019;30:1968–1979. doi: 10.1681/ASN.2019020144. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 74.Smith B, et al. Biomedical imaging ontologies: a survey and proposal for future work. J. Pathol. Inf. 2015;6:37. doi: 10.4103/2153-3539.159214. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 75.Erdal S, et al. A knowledge-anchored integrative image search and retrieval system. J. Digit. Imaging. 2009;22:166–182. doi: 10.1007/s10278-007-9086-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 76.Gurcan MN, et al. Developing the quantitative histopathology image ontology (QHIO): a case study using the hot spot detection problem. J. Biomed. Inf. 2017;66:129–135. doi: 10.1016/j.jbi.2016.12.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 77.Ali S, Lewis J, Madabhushi A. Spatially aware cell cluster(spACC1) graphs: predicting outcome in oropharyngeal pl6+ tumors. Med. Image Comput. Comput. Assist. Interv. 2013;16:412–419. doi: 10.1007/978-3-642-40811-3_52. [DOI] [PubMed] [Google Scholar]
- 78.Ali S, Veltri R, Epstein JA, Christudass C, Madabhushi A. Cell cluster graph for prediction of biochemical recurrence in prostate cancer patients from tissue microarrays. SPIE Med. Imaging. 2013 doi: 10.1117/12.2008695. [DOI] [Google Scholar]
- 79.Ertosun MG, Rubin DL. Automated grading of gliomas using deep learning in digital pathology images: a modular approach with ensemble of convolutional neural networks. AMIA Annu. Symp. Proc. 2015;2015:1899–1908. [PMC free article] [PubMed] [Google Scholar]
- 80.Heindl A, et al. Relevance of spatial heterogeneity of immune infiltration for predicting risk of recurrence after endocrine therapy of ER+ breast cancer. J. Natl Cancer Inst. 2018 doi: 10.1093/jnci/djx137. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 81.Lee G, et al. Supervised multi-view canonical correlation analysis (sMVCCA): integrating histologic and proteomic features for predicting recurrent prostate cancer. IEEE Trans. Med. Imaging. 2015;34:284–297. doi: 10.1109/TMI.2014.2355175. [DOI] [PubMed] [Google Scholar]
- 82.Lee G, et al. Co-occurring gland angularity in localized subgraphs: predicting biochemical recurrence in intermediate-risk prostate cancer patients. PLoS One. 2014;9:e97954. doi: 10.1371/journal.pone.0097954. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 83.Lucas M, et al. Deep learning for automatic Gleason pattern classification for grade group determination of prostate biopsies. Virchows Arch. 2019;475:77–83. doi: 10.1007/s00428-019-02577-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 84.Osareh, A., Shadgar, B. 2010 5th IEEE International Symposium on Health Informatics and Bioinformatics, 114–120 (IEEE, 2010).
- 85.Saltz J, et al. Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images. Cell Rep. 2018;23:181–193.e7. doi: 10.1016/j.celrep.2018.03.086. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 86.Ginsburg SB, Lee G, Ali S, Madabhushi A. Feature importance in nonlinear embeddings (FINE): applications in digital pathology. IEEE Trans. Med. Imaging. 2016;35:76–88. doi: 10.1109/TMI.2015.2456188. [DOI] [PubMed] [Google Scholar]
- 87.Kush RD, et al. FAIR data sharing: the roles of common data elements and harmonization. J. Biomed. Inform. 2020;107:103421. doi: 10.1016/j.jbi.2020.103421. [DOI] [PubMed] [Google Scholar]
- 88.Medicine, I. O. Sharing Clinical Research Data: Workshop Summary (The National Academies Press, 2013). [PubMed]
- 89.Mascalzoni D, et al. International charter of principles for sharing bio-specimens and data. Eur. J. Hum. Genet. 2015;23:721–728. doi: 10.1038/ejhg.2014.197. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 90.Sampson MG, Kang H. M. Using and producing publicly available genomic data to accelerate discovery in nephrology. Nat. Rev. Nephrol. 2019;15:523–524. doi: 10.1038/s41581-019-0166-z. [DOI] [PubMed] [Google Scholar]
- 91.Knoppers BM, et al. A P3G generic access agreement for population genomic studies. Nat. Biotechnol. 2013;31:384–385. doi: 10.1038/nbt.2567. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 92.Ness RB, American College of Epidemiology Policy Committee. Biospecimen “ownership”: point. Cancer Epidemiol. Biomarkers Prev. 2007;16:188–189. doi: 10.1158/1055-9965.EPI-06-1011. [DOI] [PubMed] [Google Scholar]
- 93.Official Journal of the European Union. Council recommendation of 8 June 2009 on an action in the field of rare diseases. Official Journal of the European Unionhttps://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:C:2009:151:0007:0010:EN:PDF (2009).
- 94.OECD. https://www.oecd.org/sti/inno/38500813.pdf (2007).
- 95.Toronto International Data Release Workshop Authors Prepublication data sharing. Nature. 2009;461:168–170. doi: 10.1038/461168a. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 96.Thomas, K. & Ornstein, C. Sloan Kettering’s cozy deal with start-up ignites a new uproar. The New York Timeshttps://www.nytimes.com/2018/09/20/health/memorial-sloan-kettering-cancer-paige-ai.html (2018).
- 97.Sugano S. International code of conduct for genomic and health-related data sharing. HUGO J. 2014;8:1. doi: 10.1186/1877-6566-8-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 98.Afsar B, et al. Capillary rarefaction from the kidney point of view. Clin. Kidney J. 2018;11:295–301. doi: 10.1093/ckj/sfx133. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 99.Venkatareddy M, et al. Estimating podocyte number and density using a single histologic section. J. Am. Soc. Nephrol. 2014;25:1118–1129. doi: 10.1681/ASN.2013080859. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 100.Hodgin JB, et al. Glomerular aging and focal global glomerulosclerosis: a podometric perspective. J. Am. Soc. Nephrol. 2015;26:3162–3178. doi: 10.1681/ASN.2014080752. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 101.Naik AS, et al. Quantitative podocyte parameters predict human native kidney and allograft half-lives. JCI Insight. 2016;1:e86943. doi: 10.1172/jci.insight.86943. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 102.Ding F, et al. Accelerated podocyte detachment and progressive podocyte loss from glomeruli with age in Alport syndrome. Kidney Int. 2017;92:1515–1525. doi: 10.1016/j.kint.2017.05.017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 103.Kikuchi M, Wickman L, Hodgin JB, Wiggins RC. Podometrics as a potential clinical tool for glomerular disease management. Semin. Nephrol. 2015;35:245–255. doi: 10.1016/j.semnephrol.2015.04.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 104.Nishizono R, et al. FSGS as an adaptive response to growth-induced podocyte stress. J. Am. Soc. Nephrol. 2017;28:2931–2945. doi: 10.1681/ASN.2017020174. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 105.Kather JN, et al. Predicting survival from colorectal cancer histology slides using deep learning: a retrospective multicenter study. PLoS Med. 2019;16:e1002730. doi: 10.1371/journal.pmed.1002730. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 106.Kolachalama VB, et al. Association of pathological fibrosis with renal survival using deep neural networks. Kidney Int. Rep. 2018;3:464–475. doi: 10.1016/j.ekir.2017.11.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 107.Gadermayr M, et al. Segmenting renal whole slide images virtually without training data. Comput. Biol. Med. 2017;90:88–97. doi: 10.1016/j.compbiomed.2017.09.014. [DOI] [PubMed] [Google Scholar]
- 108.Gadermayr M, et al. Generative adversarial networks for facilitating stain-independent supervised and unsupervised segmentation: a study on kidney histology. IEEE Trans. Med. Imaging. 2019;38:2293–2302. doi: 10.1109/TMI.2019.2899364. [DOI] [PubMed] [Google Scholar]
- 109.Lutnick B, et al. An integrated iterative annotation technique for easing neural network training in medical image analysis. Nat. Mach. Intell. 2019;1:112–119. doi: 10.1038/s42256-019-0018-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 110.Gupta L, Klinkhammer BM, Boor P, Merhof D, Gademayr M. Iterative learning to make the most of unlabeled and quickly obtained labeled data in histology. Proc. Mach. Learn. Res. 2019;102:215–224. [Google Scholar]
- 111.Bukowy JD, et al. Region-based convolutional neural nets for localization of glomeruli in trichrome-stained whole kidney sections. J. Am. Soc. Nephrol. 2018;29:2081–2088. doi: 10.1681/ASN.2017111210. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 112.Simon O, Yacoub R, Jain S, Tomaszewski JE, Sarder P. Multi-radial LBP features as a tool for rapid glomerular detection and assessment in whole slide histopathology images. Sci. Rep. 2018;8:2032. doi: 10.1038/s41598-018-20453-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 113.Kato T, et al. Segmental HOG: new descriptor for glomerulus detection in kidney microscopy image. BMC Bioinforma. 2015;16:316. doi: 10.1186/s12859-015-0739-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 114.Gadermayr M, Dombrowski A.-K., Klinkhammer BM, Boor P, Merhof D. CNN cascades for segmenting sparse objects in gigapixel whole slide images. Comput. Med. Imaging Graph. 2019;71:40–48. doi: 10.1016/j.compmedimag.2018.11.002. [DOI] [PubMed] [Google Scholar]
- 115.Jansen I, et al. Three-dimensional histopathological reconstruction of bladder tumours. Diagn. Pathol. 2019;14:25. doi: 10.1186/s13000-019-0803-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 116.Jansen I, et al. Histopathology: ditch the slides, because digital and 3D are on show. World J. Urol. 2018;36:549–555. doi: 10.1007/s00345-018-2202-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 117.Kannan S, et al. Segmentation of glomeruli within trichrome images using deep learning. Kidney Int. Rep. 2019;4:955–962. doi: 10.1016/j.ekir.2019.04.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 118.Gallego J, et al. Glomerulus classification and detection based on convolutional neural networks. J. Imaging. 2018;4:20. [Google Scholar]
- 119.Ginley B. et al. Neural network segmentation of interstitial fibrosis, tubular atrophy, and glomerulosclerosis in renal biopsies. Preprint at arXivhttps://arxiv.org/abs/2002.12868 (2020).
- 120.Marsh JN, et al. Deep learning global glomerulosclerosis in transplant kidney frozen sections. IEEE Trans. Med. Imaging. 2018;37:2718–2728. doi: 10.1109/TMI.2018.2851150. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 121.Avninder S, Ylaya K, Hewitt SM. Tissue microarray: a simple technology that has revolutionized research in pathology. J. Postgrad. Med. 2008;54:158–162. doi: 10.4103/0022-3859.40790. [DOI] [PubMed] [Google Scholar]
- 122.Hipp JD, et al. Computer-aided laser dissection: a microdissection workflow leveraging image analysis tools. J. Pathol. Inf. 2018;9:45. doi: 10.4103/jpi.jpi_60_18. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 123.Corredor G, Whitney J, Arias V, Madabhushi A, Romero E. Training a cell-level classifier for detecting basal-cell carcinoma by combining human visual attention maps with low-level handcrafted features. J. Med. Imaging. 2017;4:021105. doi: 10.1117/1.JMI.4.2.021105. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 124.Lafata KJ, et al. Association of pre-treatment radiomic features with lung cancer recurrence following stereotactic body radiation therapy. Phys. Med. Biol. 2019;64:025007. doi: 10.1088/1361-6560/aaf5a5. [DOI] [PubMed] [Google Scholar]
- 125.Lafata KJ, et al. An exploratory radiomics approach to quantifying pulmonary function in CT images. Sci. Rep. 2019;9:11509. doi: 10.1038/s41598-019-48023-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 126.Glassy EF. Digital pathology: quo vadis? Pathology. 2018;50:375–376. doi: 10.1016/j.pathol.2018.03.001. [DOI] [PubMed] [Google Scholar]
- 127.Van Es SL. Digital pathology: semper ad meliora. Pathology. 2019;51:1–10. doi: 10.1016/j.pathol.2018.10.011. [DOI] [PubMed] [Google Scholar]
- 128.Van Es SL, Madabhushi A. The revolving door for AI and pathologists — docendo discimus? J. Med. Artif. Intell. 2019;2:12. doi: 10.21037/jmai.2019.05.02. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 129.Dietz RL PL. The future of anatomic pathology: deus ex machina? J. Med. Artif. Intell. 2019;2:4. [Google Scholar]
- 130.Flotte TJ, Bell DA. Anatomical pathology is at a crossroads. Pathology. 2018;50:373–374. doi: 10.1016/j.pathol.2018.01.003. [DOI] [PubMed] [Google Scholar]
- 131.Lafata KJ, et al. Data clustering based on Langevin annealing with a self-consistent potential. Q. Appl. Mathematics. 2019;77:591–613. [Google Scholar]
- 132.McLachlan S, et al. The heimdall framework for supporting characterisation of learning health systems. J. Innov. Health Inf. 2018;25:77–87. doi: 10.14236/jhi.v25i2.996. [DOI] [PubMed] [Google Scholar]
- 133.National Academies Press. The Learning Healthcare System: Workshop Summary (National Academies Press (US), 2007). [PubMed]
- 134.Institute of Medicine. Digital Infrastructure for the Learning Health System: The Foundation for Continuous Improvement in Health and Health Care: Workshop Series Summary (Institute of Medicine, 2011). [PubMed]
- 135.Samsi, S., Jarjour, W. & Krishnamurthy, A. Glomeruli segmentation in H&E stained tissue using perceptual organization. Proceedings of the IEEE Signal Processingin Medicine and Biology Symposium, 1–5 (IEEE, 2012).
- 136.McLachlan S, et al. Learning health systems: the research community awareness challenge. J. Innov. Health Inf. 2018;25:981. doi: 10.14236/jhi.v25i1.981. [DOI] [PubMed] [Google Scholar]
- 137.Sampson MG, et al. Integrative genomics identifies novel associations with APOL1 risk genotypes in black NEPTUNE subjects. J. Am. Soc. Nephrol. 2016;27:814–823. doi: 10.1681/ASN.2014111131. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 138.Kolachalama VB, Garg PS. Machine learning and medical education. NPJ Digit. Med. 2018;1:54. doi: 10.1038/s41746-018-0061-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 139.Baltscheffsky H. Major “anastrophes” in the origin and early evolution of biological energy conversion. J. Theor. Biol. 1997;187:495–501. doi: 10.1006/jtbi.1996.0384. [DOI] [PubMed] [Google Scholar]