Harnessing non-destructive 3D pathology

Abstract

Recent technological advances have demonstrated the feasibility of achieving high-throughput slide-free three-dimensional (3D) pathology of biopsy and surgical specimens. In comparison to conventional slide-based pathology, 3D pathology has the potential to provide a transformative improvement in diagnostic performance for a number of reasons: (1) vastly greater (multiple log orders) sampling of tissue specimens, (2) volumetric imaging of cell distributions and tissue structures that are prognostic and predictive, (3) nondestructive imaging, which allows valuable biopsy specimens to be used for downstream molecular assays, and (4) a simplified process with cost benefits for pathology laboratories. However, due to the immense size of these feature-rich datasets, new challenges exist in terms of data management and computer-aided interpretation. In this forward-looking Perspective, we first provide a brief overview of the imaging technologies that can enable nondestructive 3D pathology, including computational tools needed to support these 3D methods. We then provide a roadmap for how machine learning, which is already being developed within the context of 2D digital pathology, should be leveraged and refined for 3D pathology. Finally, we discuss future challenges and opportunities for the clinical validation, regulatory approval, and clinical adoption of this new paradigm for precision medicine, including for reducing health disparities across populations. This includes learning from, and integrating with, other diagnostic modalities such as radiology and genomics.

One Sentence Summary:

In this Perspective article, Liu et al. motivate and summarize the optical methods and related computational technologies being developed to enable a new diagnostic paradigm of slide-free nondestructive 3D pathology, as well as a vision for its clinical adoption.

1. Introduction and summary of clinical vision and impact

1.1. Motivation and vision for nondestructive 3D pathology

Disease diagnosis and characterization depend upon the rapid and accurate pathological analysis of biopsies and surgically excised tissues. This careful interrogation of the morphological and molecular characteristics of the tissue plays a key role in determining which treatments are most appropriate for individuals. For many cancers, the biopsy-determined grade of the disease provides the basis for stratifying patients for clinical management, and can result in dramatically different treatment paths ^1,2. As an example, for prostate cancer, patients with low-grade cancer (Gleason score < 7) are candidates for active surveillance whereas patients with higher-grade cancer (Gleason score ≥ 7) are candidates for curative therapy (radiation and/or surgery), often in conjunction with neoadjuvant therapy for those with the highest-grade cancer (Gleason scores 8 to 10). Unfortunately, interobserver variance amongst pathologists is high, with kappa values ranging from 0.3 and 0.8 ^3-6. This ambiguity can, in some cases, result in the overtreatment of certain patients with indolent disease ^7,8, which results in unnecessary side effects and financial toxicity to both patients and the healthcare system. Likewise, the undertreatment or nonoptimal treatment of patients with aggressive disease leads to preventable morbidity and mortality ⁹, along with dramatically increased costs of care for patients with late-stage diseases. For example, largely due to delayed diagnosis and undertreatment of disease, uninsured men are roughly 5 times more likely to have incurable metastatic prostate cancer at initial diagnosis than insured men ¹⁰, and have 2.5 times the risk of prostate cancer-specific mortality ¹¹.

Another example to illustrate the limitations of conventional pathology relates to the use of immune-checkpoint inhibitors (targeting PD-1 and PD-L1) as a treatment for patients with non-small-cell lung cancers (NSCLC), which account for 80 – 85% of lung cancers. Compared to standard chemotherapy, these drugs can decrease the risk of progression by up to 60% but are only effective in approximately 20% of patients ¹². With treatment costs ranging from $150K to $1M per patient, as well as the possibility of auto-immune side effects, reliable assays are needed to predict which patients will respond to these treatments. Current companion diagnostics and complementary diagnostics for predicting PD-1/PD-L1 treatment response are all based on immunohistochemistry (IHC) of PD-L1 expression, but have been criticized for their unreliability ^13,14. A multitude of reasons have been offered to explain the shortcomings of these assays, including the variability in pathologist interpretation of PD-L1 IHC ¹⁵, the spatial and temporal heterogeneity of PD-L1 expression within the tumor microenvironment, and the general complexity of the immune response, in which multiple cell types must interact in a coordinated fashion to deliver a durable immunomodulatory response.

One contributing factor for the unreliability in diagnosing and grading malignant tissues, and for developing predictive assays, is that pathology laboratories handle biopsies and surgical specimens via a decades-old analog imaging process that only samples a small fraction of the specimens in 2D. In terms of reliability, imaging a small percentage of a heterogeneous tissue specimen is bound to introduce errors. For example, a 5-micron thick section of a 1-mm thick biopsy represents only 0.5% of the biopsy. This limited sampling is due, in part, to the complexity and time-consuming nature of traditional slide-based pathology, but also by its inherent destructiveness. In an era of molecular medicine where increasing numbers of assays (e.g. genomics and proteomics) are desired for personalized care (i.e. “precision medicine”), there is an ever-increasing need to preserve valuable tissue specimens for such downstream assays ¹⁶, further constraining the number of tissue sections that can be analyzed via destructive sectioning. In addition to sampling limitations, the thin slide-mounted tissue sections that are currently viewed by pathologists provide only a 2D view of tissue structures and molecular targets, which can be challenging to interpret accurately and reproducibly (with high interobserver concordance) ^17-19. Subjective visual interpretation of these 2D images is another source of uncertainty, which digital pathology with machine learning strives to remedy ²⁰.

As a complement to traditional slide-based 2D pathology, there has been a re-invigorated interest in nondestructive 3D pathology, especially in light of recent technological advancements in optical clearing, high-throughput microscopy, and computational tools (including artificial intelligence or AI). At the heart of this movement is a belief that nondestructive volumetric microscopy can improve concordance and accuracy (diagnostic, prognostic, and predictive power) in the analysis of tissue specimens, thus resulting in superior patient outcomes. Here, diagnosis refers to the identification of a disease condition, including sub-type, whereas prognosis refers to the anticipated trajectory and outcome of the condition. In the context of this article, the term “predictive” refers to the anticipated effects of a specific treatment. Technical benefits of 3D pathology over traditional pathology include: (1) improved sampling of large volumes of tissue rather than sparse sampling with thin slide-mounted sections; (2) volumetric imaging of diagnostically relevant structures; (3) non-destructive imaging, which allows intact tissue specimens (e.g. core-needle biopsies) to be made fully available for downstream proteomic/genetic assays; (4) a simpler slide-free imaging process, which has the potential to save time and costs. A summary of some of these points is provided in Fig. 1.

Figure 1. Conventional pathology vs. nondestructive 3D pathology.

(A) The conventional histology workflow was developed over a century ago and involves the use of harsh fixatives and dehydration reagents (e.g. xylene) followed by wax embedding, destructive sectioning, and staining of slide-mounted sections with chromogens such as H&E. In addition to being time consuming and destructive, only a small fraction of a clinical specimen is viewed in 2D. (B) Recent advances in optical clearing and fluorescence labeling, along with high-throughput volumetric microscopy, enable entire specimens (e.g. core-needle biopsies) to be imaged in 3D with minimal tissue processing or mounting requirements. This method provides rich 3D structural (and molecular) information of large intact specimens, and preserves valuable clinical specimens for downstream assays (e.g. DNA and RNA sequencing).

While the full value of 3D pathology data has yet to be determined, three illustrative examples are provided in Fig. 2. The first example deals with convoluted structures, such as the branching-tree vascular and glandular networks known to be disrupted during disease progression. When viewing such complex 3D structures as 2D cross sections on glass slides, artifacts and ambiguities are unavoidable. This is particularly problematic for Gleason grading of prostate cancers, which is currently based on glandular architecture alone. For example, what appears in 2D to be a poorly formed gland (a variant of Gleason pattern 4) might be a tangential section of a well-formed gland (Gleason pattern 3) when viewed in 3D. Consequently, this cancer could be re-categorized from the 2D-determined grade (Gleason score 3+4=7) to a lower-grade Gleason score (3+3=6) when viewed in 3D ^21,22. The second example is of complex distributions of cells, such as within the tumor-immune microenvironment. Here, the spatial relationships and interactions between cell types – e.g. the extent and proximity of PD-1 expressing cytotoxic T cells to PD-L1 expressing cancer cells ²³, or the balance between immunosuppressive neutrophils and cytotoxic T cells ^24,25 – could be more ideally characterized and quantified in 3D. Furthermore, the ability to interrogate larger volumes of tissue would be beneficial in light of the spatial heterogeneity of most diseases at microscopic length scales ^26-28. The third example is of rare cells that can play critical roles in the initiation, treatment resistance, and/or dissemination of various diseases. Examples of rare events that are difficult to identify on thin tissue sections include tumor-progenitor cells ^29-31, minimal residual disease ^32,33, and aggressive tumor subclones that invade the lymphovascular network ^34-36 and metastasize ³⁷. Note that the second and third categories mentioned above, of “complex distributions” and “rare events,” are related. In particular, as we begin to understand tumors as hierarchical collections of cells with significant intratumoral heterogeneity, one must recognize the pivotal role of rare cell populations, such as tumor-initiating cells (i.e. cancer stem cells), in some of the largest challenges/opportunities in oncology such as disease progression, treatment resistance, and recurrence ^38-40.

Figure 2. Examples in which 3D pathology could outperform 2D pathology.

(A) For convoluted 3D structures, 2D cross sectional views can be misleading. (B) For distributions of cells and other structures, 2D cross sectional views might preclude accurate quantification of complex spatial relationships. (C) Finally, for rare cells and microstructures, 2D sections might not provide adequate sampling to identify and quantify such targets. (D) Clinical examples are provided corresponding to the three categories above.

1.2. Current status of 3D pathology

Three-dimensional microscopy / pathology is a technology that has been embraced and driven by life scientists for a number of decades. This has been catalyzed primarily by the invention of confocal microscopy, followed by multiphoton microscopy and most-recently light-sheet microscopy. In section 2, we will provide a cursory overview of these imaging technologies that are foundational to nondestructive 3D microscopy. In the realm of clinical diagnostics, however, 3D pathology is in its infancy. Much of this is due to technical adoption barriers, in terms of microscopy, tissue preparation, and computational methods (see section 2).

The past few years have witnessed early signs of a renewed interest in 3D pathology for clinical diagnostics. For example, in 2016, van Royen et al. ⁴¹ demonstrated the basic feasibility to image archival formalin-fixed paraffin-embedded (FFPE) and fresh prostate tissues in 3D by employing optical clearing and confocal microscopy. That same year, Olson et al. described the use of multiphoton microscopy with optical clearing to generate 3D histology images of kidney biopsies ⁴². The following year, Tanaka et al. ⁴³ utilized optical clearing and light-sheet microscopy to examine FFPE specimens (bladder), showing the ability to quantify certain histomorphometric features, such as vascular density and tortuosity. Here, they showed that 3D pathology is superior to 2D pathology for staging cancers and predicting progression-free survival. This same group also imaged and analyzed the lymphatics within bladder tumors in 2018 ⁴⁴. In 2017, Glaser et al. ²¹ described an open-top light-sheet (OTLS) microscopy system that was specifically designed for high-throughput 3D pathology of large clinical specimens. This report included an anecdotal study demonstrating that 3D pathology could improve the grading of prostate core-needle biopsies by mitigating some of the ambiguities and artifacts encountered when viewing 2D tissue sections of prostate carcinoma. A follow-up study by Reder et al. ²², using the OTLS technology, provided additional insights into the potential value of 3D pathology, including the high variability in glandular morphology throughout an entire core-needle biopsy, which could have a dramatic influence on treatment decisions. More recently, Lee et al. ⁴⁵ performed 3D confocal microscopy of immune infiltrates in preclinical and human core needle biopsies (head & neck cancer), showing that the spatial distribution of tumor-infiltrating cytotoxic T cells (CTLs) is correlated with other hallmarks of disease progression such as the tumor microvasculature. In addition, Verhoef et al. ⁴⁶ performed a detailed investigation of the 3D structure of prostate cancers, revealing two major architectural subgroups of growth patterns that could potentially be of prognostic value. Note that the previous examples are not intended to constitute an exhaustive list of 3D pathology studies, but serve to illustrate the growing interest and progress in this field, as well as the many unknowns that remain. In addition, while there is some overlap in technologies, we are not discussing intraoperative applications of optical microscopy, as described in many other articles for rapid ex vivo ^21,47-56 and in vivo ^57-61 examination of excised tissues and wound cavities to guide surgical decisions.

1.3. Major challenges in nondestructive 3D pathology

A number of challenges will need to be overcome in order to translate nondestructive 3D pathology into mainstream clinical practice:

• The ease-of-use, technical capabilities, and throughput of tissue-preparation protocols (clearing and labeling), imaging devices, and data-processing routines, as described in section 2. This includes a discussion of big-data challenges that 3D pathology datasets will introduce.

• Computational analysis and visualization techniques to assist clinicians with the interpretation of large 3D pathology datasets within reasonable time frames, as discussed in section 3. This includes developing and applying new classes of computational pathology algorithms for detection, segmentation, visualization and characterization of 3D morphology and histologic primitives.

• Strategies for regulatory approval and acceptance by clinicians and patients, including a staged approach for introducing new technologies such as machine learning, and issues related to cost and time (workflow). These issues will be discussed in section 4.

• Integration with other diagnostic technologies, such genomics and radiology, for delivering holistic decision support for patients/clinicians across disparate populations. These issues will be discussed in section 5.

2. Enabling technologies for nondestructive 3D pathology

2.1. Tissue-clearing and labeling protocols

In recent decades, numerous protocols have been developed to improve the transparency of excised biological tissues for the purposes of enabling deep optical imaging at high resolution. A number of review articles have been written that describe the principles and popular approaches ^62,63. In short, optical clearing seeks to homogenize the refractive index profile within tissues, primarily through dehydration and replacement of the water (n ~ 1.33) with a high-index solvent that is better-matched to the remaining tissue components (mostly proteins and lipids). Certain protocols also perform lipid removal with detergents, in some cases accelerated with active electrophoretic transport, and/or decalcification to enable optical imaging through bone ^64,65. Early protocols were often developed for clarification of brain tissue for neuroscience applications with limited optimization for other organs ^66-69. Such protocols were often complex, including the use of hydrogel embedding, in order to preserve the structure of delicate brain tissues as well as to maintain the brightness of genetically encoded fluorescent proteins ^68,70. Recent efforts have focused on clearing whole organisms and various types of human tissues ^64,65,71,72. While an in-depth analysis and comparison of these approaches is beyond the scope of this article, one relatively simple method that has gained popularity for clearing a variety of preclinical and clinical tissues has been iDISCO ⁷¹, including a more-recent variant that utilizes a food-grade cinnamon oil (ethyl cinnamate, ECi) ⁷² for final index matching rather than the corrosive and carcinogenic dibenzylether (DBE) utilized in the original iDISCO protocol. Note that different clearing protocols exhibit varying levels of compatibility with fluorescent labeling approaches, including small-molecule fluorescent probes and large antibodies ^62,73.

For clinical applications of 3D pathology, unique requirements and constraints exist for tissue processing in preparation for imaging. First and foremost, until these techniques become the standard of care in future decades, they should ideally not interfere with current methods in histopathology, such as slide-based H&E staining and immunohistochemistry. For most clearing methods, the reagents are relatively gentle in comparison to formalin fixation. In other words, once the tissue is fixed in formalin – as is currently required within strict time frames upon clinical biopsy or resection – subsequent steps for most 3D pathology pipelines are less harsh and damaging to the tissue and its molecular constituents (e.g. proteins, DNA, and RNA). Nonetheless, studies are needed to rigorously demonstrate that 3D pathology methods are compatible with standard pathology techniques. For example, initial studies by us and others have shown that fixed and deparaffinized tissues that are labeled with fluorescent stains and cleared with methods such as iDISCO / ECi can be subsequently embedded in paraffin and subjected to standard H&E and IHC methods with no apparent degradation in quality ^22,43,56.

Beyond demonstrating that 3D pathology methods do not interfere with current histopathology methods, the advent of 3D pathology offers the potential to modernize tissue-preservation methods. In particular, as genomics and transcriptomics assays rapidly improve and gain clinical acceptance, a major performance bottleneck is the damaging effects that formalin fixation has on DNA and RNA integrity. Alternative tissue-preservation methods, such as RNAlater ^74,75 and PAXgene ^76,77, have been developed to maintain the fidelity of nucleic acids. A tissue-preservation protocol, relying upon intra- and intermolecular crosslinking of biomolecules, has also been developed to enable effective optical clearing and fluorescent labeling of molecular targets (including RNA transcripts) ⁷⁸. The ability to incorporate and standardize new tissue-preservation strategies within a clinical pipeline for 3D pathology, for high-quality volumetric imaging and optimization of nucleic acids, could be transformational for precision medicine. Section 5 provides additional discussions on these forward-looking topics.

2.2. Technical requirements for translation of 3D pathology into clinical practice

While there are many factors to consider when designing any 3D optical imaging system, key attributes to consider include resolution, contrast (signal to background ratio), speed/throughput, ease of use, and cost/complexity. In terms of importance, characteristics such as ease-of-use and speed/throughput are of higher priority for applications in anatomic pathology in comparison to most research applications – while life-science researchers often seek to resolve increasingly finer structures in order to gain novel biological insights, pathologists generally do not image tissues at the high levels of resolution that biologists are accustomed to (e.g. with 1.4 NA oil-immersion objectives). With the exception of a few specialties such as renal pathology, which routinely employs electron microscopy, the vast majority of anatomic pathology cases are viewed at low resolution (standard 5x to 10x objectives, NA ~ 0.1 to 0.3) and occasionally at moderate resolution (standard 20x to 40x objectives, NA ~ 0.4 to 0.8). Rather, for pathologists, the ability to view large areas/volumes of tissue is often of paramount importance, especially in light of the spatial and molecular heterogeneity of neoplastic lesions ^28,79. Therefore, in general, the requirements for spatial resolution are relaxed for clinical versus research applications of 3D microscopy.

In terms of achieving high image contrast, which directly impacts imaging depth, a key distinction is that while biologists often desire to visualize living organisms, pathologists tend to examine excised and preserved specimens. With most living tissues, the ability of an optical-sectioning microscope to deliver high contrast (signal to background ratio, SBR) is important for high-quality imaging at reasonable depths (e.g. up to ~0.5 to 1 mm for confocal and multiphoton microscopy in most tissue types). However, recent advances in optical-clearing approaches enable nearly all excised tissues (including bone) to be rendered highly transparent ^{63,69-71,80-82}, thereby allowing for imaging depths of several millimeters, and even centimeters (see section 2.1). This ability to make tissues transparent to light has greatly relaxed the requirements for contrast (rejection of background light) for 3D microscopy of excised specimens.

2.3. Destructive 3D microscopy techniques

While the concept of 3D pathology is not new, the technologies have evolved considerably over the years. Early studies relied upon destructive serial sectioning techniques ⁸³, which required considerable expense and labor for imaging large numbers of sectioned tissues, and their subsequent 3D reconstruction. Automated serial-sectioning approaches, such as knife-edge scanning (KESM) and micro-optical sectioning tomography (MOST) ^84,85, have greatly improved throughput, and have been commercialized for the purposes of delivering 3D pathology data, but are destructive of tissue specimens and introduce sectioning artifacts (Fig. 3A).

Figure 3. A comparison of selected imaging methods for 3D pathology.

(A) Knife-edge scanning microscopy (KESM) and micro-optical sectioning tomography (MOST) are destructive methods in which 2D images are acquired as a specimen is serially sectioned. These stacks of adjacent 2D images are used to reconstruct a 3D image of the specimen. (B) With confocal and multiphoton laser-scanning microscopy, a single point is typically imaged within a thick specimen, and is spatially scanned in three directions to nondestructively generate a 3D image over time. (C) With light-sheet microscopy, a 2D “optical section” within a transparent thick specimen is illuminated. Fluorescence generated within that light sheet is imaged in the orthogonal direction onto a sensitive high-speed camera. Scanning the 2D light sheet through the sample (or vice versa) allows for rapid generation of a 3D image in a plane-by-plane fashion.

2.4. Confocal and multiphoton laser-scanning microscopy

In terms of nondestructive high-resolution 3D imaging of tissues, the traditional work-horse technologies have been confocal and nonlinear/multiphoton microscopy ^49,50,86-88. As mentioned previously, several recent studies on 3D pathology have utilized confocal microscopy ^41,46,89, and it remains the most prevalent volumetric microscopy technique in academic and industrial labs. Multiphoton microscopy has also been widely adopted in academic research due to the ability to provide enhanced imaging depths (contrast) when imaging nontransparent tissues, and also the ability to achieve label-free imaging by relying upon a number of linear and nonlinear contrast mechanisms such as autofluorescence generation ^{50,54,87,90-93}, second harmonic generation (SHG) ^94-96, coherent four-wave mixing (e.g. CARS and SRS) ^{51,52,87,97,98}, and pump-probe methods ^99,100. A number of groups have recently explored the use of multiphoton microscopy for clinical applications of anatomic pathology such as surgical guidance and diagnosis ^{42,50,87,93,101}.

While confocal and multiphoton microscopy have become ubiquitous in biomedical research labs, offering exquisite contrast (i.e. rejection of background) and spatial resolution, there are certain challenges to overcome for clinical applications. For example, confocal and nonlinear microscopy typically generate an image in a point-by-point fashion, which requires spatial scanning in all three dimensions to create a volumetric image (Fig. 3B). This tends to add mechanical complexity and is often slow. While methods do exist to accelerate this process, such as the use of spinning disks for confocal microscopy ^102,103, and temporal-focusing ^104,105 or multifocal methods^106-108 for nonlinear microscopy, speed is often still a significant constraint when trying to image large 3D volumes. Furthermore, these technologies, which rely upon complex high-NA optics, high-speed laser scanning, and pulsed lasers (for multiphoton microscopy), are often expensive both in terms of equipment and technical support staff.

2.5. Light-sheet microscopy

Over the past decade, light-sheet microscopy, also known as selective plane illumination microscopy (SPIM), has emerged as the technique of choice for rapid 3D fluorescence microscopy of relatively transparent specimens (e.g. embryos and optically cleared tissues) ^109-117. In light-sheet microscopy, a thin excitation beam is incident upon the specimen perpendicular to detection axis such that only one localized focal plane (“optical section”) of interest is excited within the sample (Fig. 3C). Light-sheet systems utilize highly sensitive and fast sCMOS detector arrays to obtain 2D images from a specimen, from which a 3D dataset may be rapidly generated by scanning the specimen (or light sheet). A well-appreciated feature of light-sheet microscopy is its highly efficient geometry that only excites fluorescence within the detection plane, thereby minimizing photobleaching and photodamage compared with alternative 3D microscopy techniques ^111,112,114. Thus, light-sheet microscopy has been referred to as a “gentle” form of 3D microscopy ¹¹⁸.

Early light-sheet microscopes were constrained to image small non-clinical specimens (e.g. embryos, mouse brains, etc.) that were often embedded in agarose and/or mounted in specialized mounts (e.g. for sample rotation), thus limiting the size/geometry of the specimens and the ease-of-use of the systems ^{112,114,117,119}. More recently, a number of light-sheet microscope systems have been developed to accommodate larger specimens with less physical constraints and simpler mounting requirements ^{21,115,117,120-130}. Many of these systems have utilized an inverted architecture in which one or more tissue samples may be conveniently placed on a platform and imaged from above or below. In particular, imaging from below the specimen(s) allows for “open-top” light-sheet (OTLS) microscopy in which tissues, transparent sample holders, and accessories of arbitrary size can be accommodated ^{21,123,125,130,131}. One downside of OTLS or other forms of inverted light-sheet microscopy is that the imaging depth is limited by the working distance of the angled imaging objectives. Since working distance typically trades off with NA, resolution is usually in the low to moderate range (NA < 0.8) for such inverted systems. However, as mentioned in section 2.2., this level of resolution is adequate for the vast majority of clinical diagnostic applications. In order to maximize imaging depth and resolution, some research groups have explored the use of a single high-NA objective, oriented perpendicular to the sample, which is used to generate both an oblique illumination light sheet, and to image the light-sheet-generated fluorescence signal onto a detector array (camera) ^128-130.

In its simplest form, light-sheet microscopy utilizes relatively low-NA illumination such that the light sheet has a long depth of focus at the expense of a thick beam waist. While higher-NA collection ensures that adequate (i.e. micron to sub-micron-scale resolution) is achieved in two dimensions within the plane of the light sheet, most light-sheet systems exhibit anisotropic resolution in which resolution in markedly worse in the direction axial to the light sheet (due to the thick beam waist of the low-NA light sheet). This is similar to conventional slide-based histology, in which paraffin-embedded tissue sections are generally cut at a thickness of 4 to 5 microns, but are imaged in the orthogonal direction with micron- to sub-micron scale resolution. However, with 3D imaging, isotropic resolution can potentially be of importance and several strategies have been developed to achieve this with light-sheet microscopy. A popular approach has been to rapidly swap the imaging and collection paths of a light-sheet microscope such that two volumetric datasets of the specimen are ultimately obtained, but with different low-resolution axes. A “fusion” de-convolution algorithm can then be applied to generate a 3D image with isotropic resolution, albeit with large demands for alignment, calibration, and computational processing ^117,132,133. A more-direct approach for achieving isotropic resolution employs axial scanning of a high-NA light sheet (i.e. one in which the thickness of the beam waist matches the lateral resolution of the collection path) ^127,134. Synchronizing the beam waist with the “rolling shutter” of a sCMOS camera ensures that an image is generated in a line-by-line fashion from signal generated at the beam waist (isotropic resolution) as it is scanned. Specially engineered illumination strategies, such as “propagation-invariant beams,” have also been employed to achieve isotropic resolution without sacrificing depth of focus, albeit with some trade-offs in terms of image contrast and both hardware and computational complexity ^114,135,136. For clinical diagnostic applications of 3D pathology, the value of achieving isotropic resolution is currently unclear and studies are especially needed to determine its value for computer-assisted analysis of datasets.

2.6. Data-handling challenges

For all of the previously mentioned 3D microscopy techniques, digital reconstruction of large 3D images requires stitching and fusing of large numbers of 2D image tiles. Once generated, these 3D datasets are often on the terabyte (TB) scale, where challenges exist in terms of storage, low-loss compression, quality control, and visualization. In order to streamline progress in the years to come, standard open-source informatics tools and annotated datasets should ideally be established while this field is within a nascent but logarithmic stage of growth. As shown in Fig. 4, 3D microscopy generates vastly more data compared to conventional 2D microscopy approaches. However, data-generation speeds (~800 MB/sec) are similar to current whole-slide imaging (WSI) devices since they utilize similar sCMOS camera technologies – this should facilitate adoption by institutions that are already accommodating a digital pathology workflow. Figure 4 outlines various strategies for processing raw data from 3D microscope systems (e.g. local vs. cloud-based pipelines), as well as general strategies for machine-learning-based analysis of these massive feature-rich datasets.

Figure 4. Example data-processing and image-analysis workflows.

(A) In contrast to conventional microscopes, light-sheet microscopes acquire data at up to 1 GB/sec and require specialized hardware, such as a local 10 Gbit networked server, or a cloud-based storage and analysis solution. (B) Machine-learning tools will be necessary to assist with the analysis of large 3D pathology datasets for clinical decision support. Strategies include a multi-stage “hand-crafted-feature”-based approach, in which intuitive and well-understood microstructures are segmented and quantified as inputs for prognostic and predictive classifiers. Alternatively, an “end-to-end” approach can be used for direct classification based on raw 3D pathology images through a deep-learning model. Note that deep-learning techniques can also be utilized for certain steps within the multi-stage hand-crafted approach, for example to assist with segmentation tasks as described in section 3.2. These topics will be further examined in sections 3 and 4.

2.6.1. 3D image stitching.

A key step in the acquisition pipeline for 3D microscopy/pathology is efficiently assembling large numbers of 2D image tiles into seamless volumetric datasets. A number of commercial software packages, such as Imaris Stitcher and Volocity, have been developed to address this challenge, as well as popular open-source tools, such as TeraStitcher ^137,138 and BigStitcher ¹³⁹. Some software tools, such as BigStitcher ¹³⁹, are designed to correct for deformation and registration artifacts through affine transformations, including chromatic shifts between wavelength channels. Such operations can be computationally expensive, and should ideally be tailored for specific imaging methods, and parallelized for maximum throughput. For example, for the open-source community, efforts are underway to upgrade the popular hierarchical (multi-resolution) HDF5 file format with a similar format, N5, which will allow for parallel writing (https://github.com/saalfeldlab/n5). For the clinical implementation of 3D pathology, quality-control algorithms will ultimately be needed to ensure the performance of these, and other, image-processing tasks (see section 3.3).

2.6.2. Compression of 3D imaging data.

The detectors of choice for camera-based 3D microscopy approaches, including light-sheet microscopy, are currently 16-bit sCMOS cameras that generate ~800 MB of data per second. Having a large 16-bit dynamic range has practical benefits for avoiding detector saturation when imaging bright tissue regions while also ensuring that bit noise (i.e. digitization noise) is not an issue when imaging dim tissue regions. However, the signal-to-noise ratio (SNR) at any tissue region is generally much lower than 16 bits, due to a combination of detector noise and/or shot noise (signal- and background-induced). This means that a significant level of “lossless” compression can be achieved by windowing the dynamic range of the data to remove noise (at the low end) and unused pixel well capacity (at the high end). For example, unlike lossy compression schemes like JPEG, which down-sample the data in the Fourier or wavelet space, a recently developed B³D compression scheme estimates the noise level of every pixel within an image (based on neighboring pixels) and limits the compression such that pixel deviations remain within that noise level ¹⁴⁰. With B³D, fully “lossless” compression can be routinely achieved with a ~10X reduction in file size for the 16-bit imaging data generated by sCMOS-camera-based light-sheet microscopy systems. In terms of speed, B³D is built on the CUDA framework to enable GPU-based compression of imaging data at high speeds (> 1 GB/sec), which surpasses the data rate of standard sCMOS cameras ¹⁴⁰. Depending upon the image-analysis task at hand, the use of more advanced 3D compression methods that factor in the inherent redundancy of a 3D dataset could yield significant data-compression results. For optical microscopy, most compression work has been in 2D, even when applied to 3D, as each slice is considered independently. However, other methods such as Fourier or wavelet-based compression are likely to provide superior compression results for 3D datasets with acceptable tradeoffs ¹⁴¹. Additional studies are needed to examine the effects of these various compression schemes on both manual and computerized image-analysis routines.

2.6.3. Visualization.

Human observers may choose to visualize 3D pathology datasets as volume renderings or as a series of 2D cross-sectional views, depending upon the diagnostic problem of interest and the complexity/density of the image. In some cases, it may be necessary to segment (i.e. extract) a subset of tissue structures using machine-learning techniques (see section 3) in order to visualize the 3D structure in an informative way. For example, the ability to visualize a 3D mesh model of a vascular or glandular branching-tree network could yield key insights including tortuosity, branching, and other features that are not easily inferred from 2D images. On the other hand, it is also helpful for pathologists to visualize 2D cross sections of the same vessels or glands within the context of the surrounding cellular and stromal milieu, as is the case with standard H&E histology. In particular, since pathologists are currently trained to interpret 2D cross sections, and a vast body of pathology literature exists for characterizing diseases with 2D images, it is desirable to have both 2D and 3D visualizations. Note that the amount of time required to review large volumes of 3D data is often impractical. For example, in our experience, a prostate biopsy, which can be evaluated in 2D within a few minutes, can require 15-20 minutes for a thorough 3D evaluation ²². Thus, in many cases, the full 3D dataset may only be necessary and practical to visualize in the case of diagnostic ambiguity, where it will have the largest clinical impact. In fact, early clinical implementations of 3D pathology may be as an adjunct to standard 2D pathology in situations when diagnostic ambiguity arise or critical treatment decisions must be made, for example for Gleason 6 – 7 prostate cancer patients needing to decide between active surveillance, surgery, or radiation. In summary, a variety of visualization techniques will likely need to be developed and standardized in order for pathologists to extract maximal utility from 3D pathology datasets (see section 4.4). A number of examples are shown in Fig. 5 of volume-rendered vs. cross-sectional views of 3D pathology.

Figure 5. Examples of nondestructive 3D pathology of clinical specimens.

(A) 12 core-needle biopsies from the prostate of a single patient are imaged comprehensively in 3D with an open-top light-sheet (OTLS) microscopy system ¹²³. A fluorescent analog of H&E staining is used to label the specimen, and is false-colored to mimic the appearance of standard H&E histology. (B) Benign and malignant glands are easily identified, with signification variations in appearance as a function of depth, which suggests that 3D pathology may improve diagnosis and grading of prostate carcinoma ^21,22. (C) A bladder cancer specimen (FFPE) is deparaffinized, cleared, fluorescently labeled for nuclei and N-cadherin, and then imaged with light-sheet microscopy ⁴³. Scale bars: 80 μm (yellow) and 1,600 μm (cyan). (D) A number of vascular features (tortuosity, kurtosis, and density) are plotted for 45 bladder specimens (human), showing significant differences between normal patients, those with non-muscle-invasive tumor (<pT2) and those with muscle-invasive tumor (>pT2). These quantitative vascular features were obtained after segmenting out the vessel network (E). Scale bars: 80 μm. (F) An ROC analysis was performed for the ability to detect muscle-invasive vs. noninvasive tumor, showing that 3D vascular features outperform 2D features, and that combining all 3D features yields the best performance. (G) Multiplexed 3D immunofluorescence imaging, with confocal microscopy, of intact core-needle biopsies of cancer ⁴⁵. Scale bar at the top left: 500 μm. (H) Normalized densities of CD3+CD8+ cytotoxic lymphocytes (CTLs), and CD31+ microvasculature in EGFR+ parenchyma, are used to cluster human tumors into inflamed and noninflamed phenotypes. (I) 3D spatial distance mapping of an inflamed patient sample reveals that over 54% of CD3+CD8+ CTLs are located within 10 μm from microvessels.

The ability to render 3D datasets with color palettes that mimic conventional slide-based H&E histology and IHC will likely be important for pathologists to trust and adopt 3D pathology methods in the near future, as well as to validate 3D pathology datasets. While fluorescence images from light-sheet and other forms of 3D microscopy are typically acquired using monochrome cameras (with the appropriate filters in place), it is possible to false-color the datasets to mimic the appearance of standard chromogenic stains that are visualized by standard bright-field pathology microscopes. For example, it is relatively straightforward to utilize the formulae for Beer-Lambert-law absorption of light to convert two-channel images of tissues (labeled with a fluorescent analog of H&E) into H&E-like representations ¹⁴². Likewise, 3D immunofluorescence images can be rendered to mimic conventional chromogenic IHC ¹²³. This ability to create images that are familiar to pathologists is crucial for clinical adoption, so that pathologists can continue to rely upon existing disease-classification schemes, while also learning how to improve their diagnostic determinations with the added insights that 3D pathology offers.

2.6.4. Value of open-source tools for data processing, visualization, and analysis.

A key element to utilizing and deploying any imaging method is to identify the optimal informatics and computational workflows. In the life cycle of all image datasets, be it research or clinical, software is used to acquire, visualize, analyze and disseminate results. Currently, in the research imaging world, there are a number of both open- and closed-source software tools. Popular open-source platforms include the Fiji / ImageJ environment ^143-145 and related applications such as MicroManager¹⁴⁶, BigDataViewer / BigStitcher ^139,147, and Napari (https://github.com/napari/napari). Closed-source analysis platforms for 3D microscopy data include Imaris, Amira, Arivis, and Aivia. There is a growing movement in the imaging field advocating for the value and need for open-source workflows to ensure reproducibility, transparency, and broad dissemination ^148-151. Since software tools undergo continuous developments over rapid timelines, and obtaining patent protection for software is difficult, many commercial entities have chosen open-source platforms, especially when used to support other technologies and devices that enjoy a stronger IP position. The need for accuracy, transparency and reproducibility is particularly relevant in 3D pathology due its emerging nature, and the large diversity of analysis parameters and metrics that will be developed. Future tools are likely to be developed by many independent researchers and companies with access to well-curated cloud-hosted 3D pathology datasets, in which a common open-source software platform could accelerate progress and standardization.

3. Challenges / opportunities for artificial intelligence in 3D pathology.

3.1. Overview

Recent adoption of whole slide imaging (WSI) scanners by multiple hospitals and health care institutions, which have started digitizing their entire pathology workflows, complemented by rapid increases in computational power, have led to the proliferation of digital pathology approaches in oncology and other areas ^20,152-154. The moniker of digital pathology has now become associated with artificial intelligence (AI), including machine-learning approaches to quantitatively examine whole-slide images to address clinical challenges in early detection, diagnosis, prognosis and treatment response. While AI approaches have so far been developed mostly for analysis of 2D pathology images, the advent of 3D pathology techniques has provoked interest in 3D AI and feature-learning approaches. High-quality comprehensive 3D representations of tissue micro-architecture over large regions of interest offer a potential opportunity for AI analyses, due to the large amounts of data that can be generated from each patient specimen in a nondestructive fashion.

AI approaches in oncology primarily aim to develop a machine classifier for clinical decision support, such as identifying patients with aggressive disease who would benefit from aggressive therapies such as chemotherapy, or identifying patients who are likely to respond to specific forms of therapy such as immune-checkpoint blockade. Two commonly used classification approaches include “end-to-end” and “multi-stage” feature-extraction (i.e. “hand-crafted”) approaches ¹⁵⁵. The first strategy involves training a deep-learning model to directly classify a lesion/patient based on the imaging data. Such strategies have been shown to be extremely powerful and accurate, but suffer from lack of interpretability in many cases and the need for very large numbers of well-curated patient datasets in order to train a reliable algorithm. Advances are being made on data-efficient techniques as applied to digital pathology, such as multiple instance learning, semi-supervised learning, and transfer learning approaches ^156-158. However, the variance on account of pre-analytic factors in the context of pathology often makes it difficult to create generalizable algorithms without sufficiently large and diverse datasets. In addition to being sensitive to minute and often-imperceptible variations in image quality, it is often difficult to spot errors when they occur, or to determine the source of the errors ¹⁵⁹. Alternatively, traditional or deep-learning-based image-processing methods can also be used to segment out tissue structures that are already well-known and trusted by pathologists (e.g. cells, glands, collagen, etc.), from which quantitative “hand-crafted features” can be extracted (e.g. density, tortuosity, fractal dimension, angular disorder, etc.). These quantitative histomorphometric features can then be used in a multi-stage approach to train a clinical classifier. An advantage of such an approach is that domain experts (i.e. pathologists) can verify the accuracy of the intermediate segmentation steps of this process, which engenders trust and enables error-checking. Another advantage of a “hand-crafted” approach is that each digital pathology dataset typically contains large numbers of morphological exemplars (i.e. features) that can contribute to the training of an algorithm. Therefore, for low-level tasks (e.g. segmentation of well-conserved micro-architectural features), a modest number of patient specimens or training sets is often sufficient ^159-161. The caveat with these approaches, however, is that they require a greater amount of domain-specific information for model training and might be more challenging to train compared to end-to-end approaches. Finally, hybrid approaches are possible, such as using hand-crafted features as intermediate operators within a DL network, and/or in conjunction with DL-derived data-driven features, to develop an optimal classifier ^162,163.

3.2. Unique challenges for AI in 3D pathology

While there has been a sharp growth in the use of deep learning (DL)-based approaches for prognostic and predictive classification with digital pathology data, these have been largely confined to 2D pathology images. In training these networks, the images are typically broken up into smaller 2D patches and introduced to a DL network for training. The challenge with 3D pathology images is that traditional 2D networks are not equipped to handle large 3D datasets. This will necessitate 3D CNNs, which have been previously used in the context of 3D radiographic images ¹⁶⁴, albeit at much smaller file sizes (low-resolution radiography).

Beyond the computational expense of training 3D CNNs, an additional challenge is definition of the training set and annotating regions or targets of interest. The need for manual annotation of structures and primitives in 3D by a domain expert (i.e. pathologist) will necessitate efficient and user-friendly interactive software tools. These tools will need to be flexible enough to allow the user to navigate through the 3D volume and refine and update annotations of primitives and regions of interest. While tools like Sedeen (Pathcore Inc.)¹⁶⁵ and QuPath¹⁶⁶ allow for object annotation of 2D pathology images, these toolkits will require substantial modifications to accommodate the annotation of structures in large 3D pathology datasets. Alternatively, in order to bypass the requirements for laborious manual 3D annotations by domain experts, which is somewhat subjective, certain features can be labeled via molecular biomarkers such that traditional intensity- and morphology-based segmentation methods can be employed. Examples include keratin 8 (KRT8) to identify the luminal epithelial cells that line all prostate glands, and CD31 (PECAM-1) to highlight the lymphovasculature. While this molecular-labeling approach is objective and does not rely on manual annotations, the downside is that antibody-based labeling of thick 3D tissues can be very slow and expensive. However, if a DL model can be trained to predict the appearance of 3D immunofluorescence images based on images generated with inexpensive and fast small-molecule fluorophores (e.g. an H&E analog), or even without the use of labels ^160,161,167, it could be possible to develop an annotation-free segmentation algorithm based on “synthetic” immunolabeling of specific features. Such a method would be objective (no human annotations), fast (small-molecule labeling of thick tissues), and inexpensive (antibody-free).

3.3. The need for quality control for AI analysis

In addition to data size, one of the most important factors that influences performance of AI approaches in digital pathology is the inherent quality of the data fed into the algorithm. AI analysis of 2D pathology is often limited by pre-analytic sources of variations, such as with staining quality (e.g. hue, saturation, intensity), sectioning artifacts, out-of-focus regions, and other subtle variations due to the different scanners and scanning parameters used ^20,168. While the sources of image variability are different for nondestructive 3D pathology approaches, there is a similar need to control such variations when it comes to implementing AI approaches. For example, factors that can influence the quality of 3D pathology include the degree of fixation, cold and warm ischemia times, deparaffinization quality (if FFPE blocks are being used), variations in staining and clearing protocols, fluctuations in laser intensities, optical alignment, and software post-processing routines (e.g. section 2.6.1). Note that with 3D microscopy approaches, images are always “in focus” provided the alignment of the microscope is maintained. Image quality is therefore most influenced by the many tissue-preparation issues mentioned above. As with slide-based histology, the automation of these various steps is of value for reducing variability, but quality-control metrics are still needed to ensure that 3D pathology datasets are of adequate quality to ensure reliable AI-based results.

Recent work has involved building automated and semi-automated quality-control methods for 2D whole-slide images, which can be translated to 3D pathology. These routines aim to automatically detect variations in color, staining, and common artifacts present in histology images. One such tool is HistoQC ¹⁶⁹, which interrogates digital tissue slide images and provides a quantitative score of the overall quality of the image. Additionally, the tool enables the precise identification of localized regions that have been compromised by artifacts such as cracks in the glass, hair shafts, tissue folds, and pen markings. Similar tools will need to be extended for quality control of 3D pathology images, in which image artifacts include stitching defects and regions of poor staining/clearing.

Ultimately, some level of variation in image quality is unavoidable for both 2D and 3D pathology, regardless of the degree of automation and standardization of processes. As mentioned in the previous section, a hand-crafted feature-based AI approach towards image interpretation offers potential advantages. First, histomorphometric features can be identified that are relatively insensitive to image-quality variations, as shown in a recent 2D pathology example, where a sensitivity analysis was used to identify a subset of quantitative features that could be integrated into robust clinical classifiers ¹⁷⁰. Second, segmentation algorithms (for extraction of quantitative features) can be more-easily trained with diverse datasets such that they are robust against image-quality variations. While this is also possible with an end-to-end classification approach if sufficient numbers of patient datasets are available for training, the use of a hand-crafted approach allows pathologists to visually inspect the results at a critical intermediate step (feature segmentation) and to rely on smaller patient cohorts in which each 3D pathology dataset often contains hundreds to thousands of diverse “exemplars” to accurately train algorithms to segment well-conserved low-level structures (e.g. glands, collagen, broad classes of cells).

4. Translational challenges and clinical acceptance

4.1. Clinical studies with archived tissues

In contrast to clinical trials involving therapeutic agents, where new patients must be recruited and followed for response, new technologies in pathology can often be validated using already-established slide repositories or tissue biobanks with detailed follow-up data. Examples include The Cancer Genome Atlas ¹⁷¹, the Prostate Cancer Biorepository Network (http://prostatebiorepository.org/), and many more. These biobanks permit the use of a “prospective-retrospective” study design. For instance, the Oncotype Dx Recurrence Score (Genomic Health), a 21-gene expression assay using FFPE tissue, was validated using archived specimens from NSABP trials B14 and B20 ^172,173. This randomized clinical trial was completed more than a decade before the validation of Oncotype Dx, enabling the use of 10-year recurrence-free survival as the primary endpoint, thus reinforcing the value of well-characterized archival tissue in translating promising diagnostic technologies to the clinic ¹⁷⁴.

Banked tissue is not without its problems. Pre-analytic factors, including formalin fixation time, cold ischemia time, and freezing methodology, all affect the quality of tissue stored in a biobank ¹⁷⁵. If the effects of these pre-analytic factors are severe, the assay can be negatively impacted, or samples/patients must be removed from the study, introducing bias. An additional concern with the use of archival tissue is that standard-of-care treatments for the target population could have changed over time, which could potentially confound studies in which the primary endpoints are clinical outcomes rather than biologic measurements.

The gold standard for validation of a diagnostic test is a prospective, randomized trial that is evaluated in a well-defined target population, where evaluation of the diagnostic test is the primary purpose of the trial ¹⁷⁶. This type of study design minimizes bias, and is considered the highest level of evidence - LOE 1. To further extend the example of Oncotype Dx Recurrence Score, the assay was further validated by a prospective, randomized trial (TAILORx), which reinforced the value of the test ¹⁷⁷. The cumulative body of evidence, including prospective-retrospective studies and prospective, randomized studies, led to the categorization of Oncotype Dx Recurrence Score as the only “preferred” test in the 2018 NCCN breast cancer treatment guidelines for a specific subset of breast cancer patients facing a chemotherapy decision, which is the highest level of clinical acceptance.

4.2. Regulatory strategies

Regulatory approval is a challenging issue that requires close collaboration with relevant regulatory stakeholders (e.g. the FDA in the United States) to develop appropriate strategies for various implementations of 3D pathology. In this section, we provide a high-level overview of potential regulatory scenarios, along with examples from devices and diagnostics that are currently in the market. There are two primary regulatory pathways for diagnostic tests in pathology – FDA approval and laboratory developed tests (LDTs) ¹⁷⁸.

Diagnostic tests and kits are classified as medical devices by the FDA, and are subject to the same regulatory processes as other such devices. Medical devices are categorized according to their perceived level risk, from Class I to Class III. Class III devices are of highest risk, including implanted devices and those that are used to sustain life. Class II devices are considered moderate to high risk, and have a predicate device that can be used for comparison. A whole-slide scanner for 2D digital pathology, a technology that was first approved by the FDA in 2017 after an extensive clinical study, is an example of a Class II device ^179,180. Class I devices are the lowest risk devices, which include the analog light microscopes used within pathology laboratories. A recent ex vivo tissue microscopy device, the Caliber I.D. Vivascope 2500, has been classified as a Class I device.

An alternative regulatory pathway for many diagnostics tests, if provided as a “medical service” by a single laboratory, is the LDT pathway. Tests designated as LDTs must be performed in a CLIA-certified laboratory with appropriate analytic and clinical validation documentation. The FDA does not currently regulate LDTs, but did express its intentions to issue an oversight framework for LDTs in recent years ^181,182.

The regulatory strategy for 3D pathology will be dependent on the paradigm by which the technology is disseminated. If the technology is developed as a device that is marketed to pathology laboratories, FDA approval will be necessary for use of the device, and potentially for decision-support algorithms based on the device. If individual laboratories develop “homebrew” 3D pathology tests, the LDT pathway could be a viable option. Regardless of the regulatory pathway for 3D pathology, a strong base of evidence proving clinical utility will be crucial for adoption.

4.3. Financial and workflow (process time) considerations for clinical integration

The workflows for conventional pathology and 3D pathology are similar for the accessioning, grossing, and initial tissue-processing steps (i.e. fixation and dehydration). The differences begin after the tissue is embedded in paraffin (conventional pathology) or placed in the clearing solution. In the conventional pathology workflow, the tissue must be physically sectioned (by hand), placed on a glass slide, stained, cover-slipped, and scanned using a whole-slide imager to produce a digital 2D dataset. Using a set of 12 prostate biopsies as an example, this process requires ~40 minutes of hands-on histotechnologist time, ~4 hours of total time, and ~$300,000 in capital equipment costs. In the 3D pathology workflow, automated tissue scanning using light-sheet microscopy, with similar resolution to what is provided by a 10X – 20X objective (NA ~ 0.4) in a standard pathology microscope, requires ~30 minutes (10% of sample digitized) to ~4 hours (100% of sample digitized) ¹²³. Since manual sectioning is not required, the 3D pathology workflow can be entirely automated, which means it is not dependent on labor shift timing or histotechnologist availability. The capital equipment costs are likely to be similar, although no FDA-approved devices are currently available for 3D pathology. Thus, the 3D pathology workflow can reduce the labor burden on histotechnologists while nondestructively generating orders-of-magnitude more data than a whole-slide imager.

4.4. A staged approach for clinical translation

While a fully-automated diagnostic and decision-support workflow could become a reality, the integration of AI will likely occur in a step-wise fashion (Fig. 6). For integration of 3D pathology within clinics, at least three stages can be foreseen. The first stage, which requires the most time and human effort, is also the least risky, in which a pathologist directly interacts with 3D pathology datasets. Datasets would be rendered to mimic the current standards of H&E histology and IHC, and would be viewed much like how 2D whole-slide images are currently viewed, albeit with the ability to scroll through the depth of a specimen. Early studies have shown anecdotally that such a strategy could be highly informative, such as for preventing the over-grading (and over-treatment) of disease ²¹ and for identifying regions of malignancy that could be missed or misinterpreted with slide-based 2D histology ²². Likewise, the sampling limitations and ambiguity of 2D histology are known to lead to the misdiagnosis and under-grading of certain patients ^183-185, which presents an opportunity for 3D pathology to provide significant improvements. Since 3D assessments of tissues (entire biopsies, as an example) are more time-consuming than 2D assessments of sparse 2D sections, a second stage of AI integration would be to automatically triage specimens that are diagnostically unequivocal, and to provide pathologists with only cases that are ambiguous. For example, a recent study, based on 2D digital pathology images, showed that it may be possible to triage up to 75% of cancer cases based on computational analysis, while retaining 100% detection sensitivity ¹⁸⁶. Finally, as clinical studies prove their effectiveness, fully-automated computational 3D pathology workflows will emerge. A challenge to achieving these goals will be to integrate 3D pathology seamlessly into laboratory information management systems (LIMS) such that various sources of patient data can be integrated both for clinician- and AI-based treatment decisions.

Figure 6. Staged approach for translation of 3D pathology into clinical practice.

3D pathology datasets, generated by reference labs or in-house pathology labs, may initially provide additional visual information for pathologists as they seek to improve their diagnostic determinations. Early incorporation of AI analysis will likely be for triaging unequivocal cases, in order to reduce pathologist workloads, and to guide their efforts towards regions of ambiguity and/or diagnostic importance. As AI algorithms are increasingly validated and trusted by clinicians, they may eventually be utilized for fully automated analysis of 3D pathology datasets, with pathologist oversight if necessary. The vision for 3D pathology is to provide clinical decision support (prognostication and prediction) to guide treatment decisions, likely in conjunction with other molecular and imaging assays.

5. Future directions

5.1. Integration with lab medicine / molecular assays

It has become increasingly clear that precision medicine will benefit from adopting a multiplexed multi-omics approach that combines diverse diagnostic technologies for patient stratification and clinical decision support. In particular, the complexities of the tumor-immune microenvironment, and various factors that contribute to an individual’s “cancer-immune setpoint,” is becoming well-appreciated by the multitudes working to improve the efficacy and precision delivery of recent immuno-oncology approaches ¹⁸⁷. For example, various factors that correlate with response to immune-checkpoint blockade include tumor mutational burden (TMB) ¹⁸⁸, microbiome profiles ^189,190, patterns of biomarker expression (e.g. PD-1 / PD-L1, CTLA, chemokine receptors, etc.) ^14,191,192, and the spatial distribution of tumor and immune cells (e.g. tumor infiltrating lymphocytes) ¹⁹³.

In order to facilitate the integration of 3D pathology with other clinical assays such as DNA/RNA sequencing, a first step is to ensure that 3D pathology methods do not interfere with standard laboratory methods. Recent studies have indicated that certain clearing and fluorescence labeling methods are relatively gentle, and often utilize reagents that are already part of the standard histology workflow (e.g. xylene, ethanol, etc.). Once tissues have been formalin fixed per standard clinical practice, most clearing reagents are less harsh than the initial fixation step, and standard FFPE processing and histology assays can still be performed (see section 2.1). Furthermore, as described in section 2.1, the advent of nondestructive 3D pathology offers the opportunity to modernize a pathology workflow that dates back nearly a century, for example by developing formalin-free tissue-preservation and labeling/clearing protocols that can maintain RNA integrity.

3D analogs to existing pathology practices, such as laser-capture micro-dissection and manual slide-based macro-dissection of tissues, would also be of value. Such tissue-enrichment techniques could generate orders of magnitude more material for downstream assays than slide-based tissue shavings and laser-captured regions, which could result in a significant leap in sensitivity and accuracy for detecting rare mutations. For rare cell types, 3D micro-aspiration of individual cells or cell contents within thick specimens is also a possibility, as previously demonstrated in the context of neuroscience investigations ^194-196. The ability to extract lysates from tens to hundreds of individual rare cells, such as aggressive tumor cells exhibiting lymphovascular invasion, followed by low-input sequencing of those lysates, could enable the discovery of new biomarkers of aggressive disease (for prognostication), druggable targets, and mechanistic insights.

5.2. Radiomics as a model and precedence for the value of 3D imaging

Radiomics or the quantitative interrogation and subsequent mining of pixel-level, sub-visual data from standard medical imaging methods (ultrasound, CT, PET, MRI) has recently been shown to be a non-invasive method to answer clinically relevant questions pertaining to diagnosis, prognosis and treatment response across the oncology spectrum ^197,198. Radiomics has traditionally relied on analyzing multiple 2D images across a region of interest, with interpolation being used to extend the analysis in 3D. 3D pathology resembles radiomics, but differs in a few key aspects. Unlike a conventional CT slice which is 512x512 pixels, a whole slide image at 20x magnification is often 40,000 x 40,000 pixels, or 4 orders of magnitude larger. 3D pathology can add an additional 4 orders of magnitude, which introduces a big-data problem but also an opportunity since it provides abundant fuel for data-hungry AI approaches. In addition, unlike reconstructed radiographic images, 3D pathology images contain large numbers of highly-repetitive micro-scale features, such as cells, glands and stromal structures, which can allow for efficient training and use of AI methods (both end-to-end and multi-stage hand-crafted approaches, as discussed in section 3.1). As one example, radiomic and pathomic features from 2D pathology images were used to create a unified predictor of early recurrence in early stage lung cancers ¹⁹⁹.

Pathology is often utilized to validate or follow up on an observation that is first seen with radiology (e.g. mammography leading to breast biopsy). However, except for a few specialized multiscale animal studies ^200,201, there have been few efforts to gain a multiscale spatial understanding of tissue composition in disease. Modern 3D pathology, when deployed with the type of computational tools that radiography has pioneered, could provide a cell-to-organ view not possible with any other currently available clinical imaging strategy. This need for both microscopic and “mesoscopic” imaging (over larger fields of view) is starting to be appreciated by researchers ²⁰². As an example, light-sheet microscopes have now been developed for highly diverse applications ranging from single-cell investigations with advanced high-resolution systems to large-volume studies of intact organs (e.g. mouse brains) ^126,203-205.

5.3. Co-registration of multi-scale imaging data (pathology and radiology)

Extensive research has been carried out to co-register conventional 2D pathology with medical images for improved characterization of diseases over a large range of spatial scales ^206-209. A driving motivation has been to use the gold standard method of pathology to elucidate changes seen at more macroscopic tissue scales with standard medical imaging methods ²¹⁰. For example, in current clinical practice, radiology techniques are often used for the early detection of various diseases, where histopathology is subsequently relied upon to provide diagnostic, prognostic, and predictive insights. Pathology and radiology approaches are also often used for cross validation. For example, numerous recent studies have correlated PET imaging results with histologically determined metrics such as proliferative index (Ki-67 expression), micro-vessel density, and immune phenotype (e.g. CD8 expression) ^211-213. However, these correlative studies would benefit from increased standardization between medical imaging and histopathology. Prior efforts to standardize the co-registration of radiology and pathology approaches include placing fiducial markers to guide ex vivo mold-based tissue slicing ²¹⁴. 3D reconstructions of 2D pathology data have also been generated to improve registration accuracy ²⁰⁷. The emergence of nondestructive 3D pathology, which can enable visualization of large tissue volumes, has the potential to greatly facilitate and improve co-registration with radiographic images in addition to improving genomics assays. In addition, intermediate imaging modalities such as preclinical micro-CT/MRI/PET, may be helpful to bridge the large gap in spatial resolution and volumetric field between microscopy and whole-body imaging techniques ²⁰².

5.4. Opportunity to study and mitigate health disparities across populations

Recent work has suggested that 2D computational pathology features of cancer from H&E-stained whole slide images exhibit population-level differences. In particular, Bhargava et al ²¹⁵ showed that stromal features of prostate cancer differed significantly between African American (AA) and Caucasian American (CA) men, in which a computational prognostic model trained with these stromal features was strongly associated with risk of recurrence in two validation datasets of AA men. Interestingly, this prognostic model for prostate cancer, which was trained with data from AA men alone, was nearly twice as accurate compared to a model trained with a combination of AA and CA men. Considering that significant morphometric differences have been identified in the stroma between AA and CA men with prostate cancer, there is optimism that the computational interrogation of 3D stromal and epithelial morphology can further improve the treatment of disparate populations. This includes those who have traditionally been underserved, such as AA men with prostate cancer, whose mortality rate is nearly 2.5 times higher than CA men ²¹⁶.

5.5. Developing holistic decision-support algorithms (imaging, sequencing, health records, etc.)

Recent research has expanded from interrogating a single source of information (e.g. radiology, pathology, genomics, metabolomics) towards multi-modality approaches to improve prognostication and prediction. This is especially attractive when it comes to AI-based methods that thrive on large amounts of orthogonal and complementary data. AI approaches integrating these multiple data types have already been used in cancers of the lung ^199,217, breast ^218,219, brain ²²⁰, and prostate ²²¹, as well as for cardiovascular ²²² and neurological diseases ²²³. These prior examples foreshadow the use of holistic decision-support systems (Fig. 6) that combine data from multiple diagnostic modalities and patient records to build accurate models for diagnosis, prognosis and prediction of therapeutic response.

6. Summary and outlook

Unlike radiology, in which many of the imaging technologies have been developed and refined over the last half-century, the field of pathology is rooted in over a century of tradition. This extensive history behind current pathology practices will make change difficult. However, the fact that pathology is still regarded as the “gold standard” for clinical diagnosis attests to the wealth of insight that microscopy offers, even when obtained from small numbers of thin 2D sections. The recent FDA approval of digital pathology solutions indicates that pathology is entering into a phase of modernization and change that will likely evolve over the next half century. In many ways, this recent advent of 2D slide-based digital pathology will pave the way for nondestructive 3D pathology, for example by establishing the clinical IT infrastructure and resources needed to support a digital 3D pathology workflow. An optimistic viewpoint is that nondestructive 3D pathology would represent the ultimate fulfillment of the vision for digital pathology that is currently being pursued with 2D whole-slide imaging (WSI). For example, a significant factor in the success of digital X-ray imaging at the end of the 20^th century was the economic benefits of replacing analog films with reusable X-ray panels ²²⁴. Likewise, nondestructive 3D pathology offers the potential to bypass glass slides, as well as associated tissue-sectioning processes, in favor of a simple digital approach that requires less human labor and consumables.

As with any diagnostic method, clinical value must be demonstrated through large-scale randomized studies. The ability to perform studies on 3D pathology with archived FFPE tissue specimens will facilitate and accelerate this process, but studies with prospective patient specimens will be of value as well. Convenience and cost savings for pathologists may not be sufficient to drive rapid adoption of 3D pathology, but studies showing clear advantages for improving clinical outcomes will cause oncologists and patients to demand these new services. As discussed in this article (Fig. 2), initial studies will likely focus on the ability of 3D pathology to improve disease prognostication and prediction of treatment response through: (1) the characterization of complex structures (e.g. vessels, glands, collagen); (2) the quantification of complex spatial relationships (e.g. the tumor-immune microenvironment); and (3) the analysis of rare cells that are challenging to identify on 2D sections. A more-concrete example of the first scenario would be to show that 3D characterization of glandular architecture in prostate biopsies provides superior classification of indolent vs. aggressive prostate cancers in comparison to traditional Gleason grading (2D pathology). For moderate-risk patients, this could help urologists and prostate oncologists to determine which patients to place on active surveillance (watchful waiting) versus curative treatments (surgery or radiation), the latter of which carries risks for significant side effects (e.g. incontinence and impotence). An example of the second scenario would be to show that 3D quantification of the tumor-immune microenvironment (e.g. lymphocytes, neutrophils, tumor cells expressing specific checkpoint receptors, etc.) within large volumes of biopsy or surgical tissue is superior to traditional 2D histology and IHC for predicting patient response to various immuno-oncology regimens, especially when combined with complementary assays such as genomics. Again, the goal is to assign the right patients (likely responders) to appropriate pharmacologic treatments while sparing others (including insurers) from the financial costs and side effects of ineffective treatments. Finally, an example of the third scenario would be to show that identification and quantification of rare events such as lymphovascular invasion (LVI) in large surgical specimens (e.g. prostatectomies or breast lumpectomies) is feasible with high-throughput 3D pathology methods – due to orders of magnitude increased sampling compared to traditional 2D pathology – and is of value for prognostication and guiding adjuvant therapies.

In this Perspective article, we have reviewed some of the key technologies that will underlie the emergence of nondestructive 3D pathology as a clinical diagnostic method in the decades to come. Rapid advances in high-throughput 3D microscopy, data-processing, and AI-based image interpretation are currently being coordinated, along with clinical-validation studies to demonstrate the value of this diagnostic paradigm. We have discussed how 3D pathology has the potential to form a bridge between anatomic pathology and other diagnostic disciplines such as genomics and radiology, all of which should ideally be integrated to support accurate clinical decision-making.