Non-destructive 3D pathology with analysis of nuclear features for prostate cancer risk assessment

Abstract

Prostate cancer treatment decisions are heavily reliant upon subjective visual interpretation (assigning Gleason patterns, or International Society of Urological Pathology (ISUP) grade groups) of limited numbers of 2D histology sections. Under this paradigm, inter-observer variance is high, with ISUP grades not correlating well with outcome for individual patients, and which contributes to the over- and under-treatment of patients. Recent studies have demonstrated improved prognostication of prostate cancer outcomes based on computational analyses of glands and nuclei within 2D whole slide images. Our group has also shown that the computational analysis of 3D glandular features, extracted from 3D pathology datasets of whole intact biopsies, can allow for improved recurrence prediction compared to corresponding 2D features. Here we seek to expand on these prior studies by exploring the prognostic value of 3D shape-based nuclear features in prostate cancer (e.g., nuclear size, sphericity). 3D pathology datasets were generated using open-top light-sheet (OTLS) microscopy of 102 cancer-containing biopsies extracted ex vivo from the prostatectomy specimens of 46 patients. A deep learning-based workflow was developed for 3D nuclear segmentation within the glandular epithelium versus stromal regions of the biopsies. 3D shape-based nuclear features were extracted, and a nested cross-validation scheme was used to train a supervised machine classifier based on 5-year biochemical recurrence (BCR) outcomes. Nuclear features of the glandular epithelium were found to be more prognostic than stromal cell nuclear features (AUC = 0.72 versus 0.63). 3D shape-based nuclear features of the glandular epithelium were also more strongly associated with risk of BCR than analogous 2D features (AUC = 0.72 versus 0.62). The results of this preliminary investigation suggest that 3D shape-based nuclear features are associated with prostate cancer aggressiveness and could be of value for the development of decision-support tools.

Introduction

Prostate cancer (PCa) accounts for 1 in 8 of the newly detected cancer cases in the United States, affecting nearly 250,000 patients annually [1]. Currently, grading of PCa for prognosis and treatment planning relies on 2D histology, where a set of core-needle biopsies is formalin-fixed and paraffin-embedded (FFPE), sectioned, and mounted on glass slides and stained for microscopic analysis. To assess the aggressiveness of the cancer, pathologists use the International Society of Urological Pathology (ISUP) grade group system [2–4] which involves visual examination and interpretation of complex glandular morphologies in a limited number of 2D histological tissue sections (4-μm thick) cut from each biopsy. In this workflow, typically ~1% of the volumetric extent of each biopsy is analyzed. In part due to this limited “sampling” of the biopsies in 2D, grading of prostate cancer suffers from high levels of inter- and intra-observer variability [5–8]. A related issue is that ISUP grade groups do not precisely correlate with individual patient outcomes, especially for moderate-risk cases (ISUP Grade Group 2) that account for approximately 50% of the newly detected PCa cases in the USA. These problems contribute to the overtreatment of indolent cases (with surgery or radiation), which are often accompanied by life-changing side effects (incontinence or impotence) and potentially avoidable healthcare costs. Conversely, they also contribute to the undertreatment of aggressive cases, which can lead to potentially avoidable metastasis and death [9].

By combining recent advances in tissue-clearing techniques [10–12] and high-throughput three-dimensional microscopy [13–19], there is now the ability to generate non-destructive 3D pathology datasets of large clinical specimens (surgical excisions or whole biopsies). In particular, it is possible to rapidly label thick tissues with a small-molecule fluorescent analog of H&E, and to false color the datasets so that they closely mimic the appearance of standard slide-based H&E histology (e.g. supplementary material, Figure S1) [14,18,20]. Unlike conventional histology, non-destructive 3D pathology achieves orders-of-magnitude greater microscopic sampling of tissues than conventional slide-based histology and enables volumetric quantification of diagnostically significant microarchitectures. The non-destructive imaging process also preserves tissue specimens for downstream molecular assays [21]. However, since a single 3D pathology dataset of a biopsy can be tens to hundreds of gigabytes in size, there are significant challenges for the human interpretation of such datasets. Manual examination of these datasets can be tedious, which motivates the development of computational methods for analysis. Here, we employ interpretable “hand-crafted” features, computationally derived from segmentation masks of tissue structures (e.g., glands and nuclei), to facilitate the clinical acceptance of 3D pathology as a new diagnostic approach.

A recent study by our group demonstrated the benefits of the computational analysis of 3D glandular features for PCa risk stratification [19]. In that study, ex vivo biopsies were extracted from archived radical prostatectomy specimens obtained from patients with known 5-year biochemical recurrence (BCR) outcomes. These patients were originally diagnosed with low- to moderate-risk PCa (Grade Groups 1 – 3), with 50% of cases exhibiting BCR within 5 years of prostatectomy [22]. Biopsies were stained with small-molecule fluorescent analogs of hematoxylin and eosin (H&E), optically cleared with a dehydration and solvent-immersion protocol, and then non-destructively imaged with open-top light-sheet microscopy (OTLS) to generate whole-biopsy 3D pathology datasets (Figure 1A). For 3D segmentation of prostate glands (lumen, epithelium, and stromal compartments), our group developed a computational pipeline called “image-translation-assisted segmentation in 3D” (ITAS3D). 3D glandular features (e.g., gland curvature, eccentricity, volume ratios) were quantified from the cancer-containing regions of the biopsies. We showed that these quantitative features are more strongly associated with BCR outcomes than analogous 2D glandular features.

Figure 1. Workflow to evaluate the prognostic value of 3D nuclear features for prostate cancer risk assessment.

(A) Archived (FFPE) radical prostatectomy specimens were obtained from a 46-patient cohort with known 5-year biochemical recurrence (BCR) outcomes, from which simulated (ex vivo) biopsies were extracted (6 biopsies per case, per sextant biopsy protocol). The biopsies were labeled with a fluorescent analog of H&E, optically cleared to render them transparent to light, and comprehensively imaged in 3D via OTLS microscopy. (B) 3D nuclear segmentation masks of cancer-containing regions were generated using the deep-learning based cellpose model and 3D glandular segmentations were generated by the previously published ITAS3D pipeline. This enabled us to distinguish between nuclei in the glandular epithelium (blue) versus the surrounding stroma (yellow). (C) 3D shape-based features of epithelial and stromal nuclei were extracted and passed into a multiparameter classifier (LASSO) to stratify patients based on 5-year BCR outcomes.

In addition to glandular morphologies, which are currently the sole basis for Gleason grading (nuclear morphologies are not looked at for standard-of-care Gleason grading), studies using 2D whole-slide images (conventional 2D pathology) have shown that there is prognostic value in computationally extracted nuclear features within PCa [22–33]. Note that prior to Gleason grading becoming the widely accepted standard for grading PCa, 2D cytology, including nuclear morphologies, were also used by pathologists in various grading schemes [30–35]. However, the prognostic value of 3D (volumetric) nuclear features has not been previously investigated. Therefore, we sought to quantify a preliminary set of 3D shape-based nuclear features and their 2D counterparts to directly compare their ability to predict BCR outcomes. Our underlying hypothesis is that 2D cross sections of a nucleus (as seen with conventional histology) cannot fully elucidate certain complex changes in 3D nuclear morphology and therefore may be limited for patient risk stratification. For example, supplementary material, Figure S2 provides a simple illustration of how an irregularly shaped 3D nucleus may appear quite regular and circular when viewed as 2D cross sections.

For 3D nuclear and glandular segmentation, we used cellpose [36] and our previously reported ITAS3D pipeline [19], respectively, which allowed us to distinguish between nuclei within the gland epithelium versus surrounding stromal compartments (Figure 1B). 3D shape-based nuclear features and 2D analogs were extracted from each biopsy, such that the prognostic value of 3D versus 2D nuclear features could be compared. These 3D and 2D features were used to train a multi-parameter machine classifier to stratify patients based on their BCR outcomes, which serves as a proxy for aggressive versus indolent PCa (Figure 1C).

Materials and methods

Tissue preparation

This study was approved by the institutional review board (IRB) of the University of Washington (Seattle, WA, USA; Study 00004980), where research specimens were previously obtained from patients with informed consent. Archived FFPE prostatectomy specimens from a prior case-cohort research study (Canary TMA) [22] were collected from 46 patients with PCa, all of which were identified as ISUP Grade Group 1–3 (Gleason Grade 3+3, 3+4, or 4+3). These patients were followed for at least 5 years after radical prostatectomy as part of the prior study [22], in which the primary study endpoints were 5-year BCR outcomes and time to BCR, which were also used as endpoints for this study. FFPE specimens were identified for each case corresponding to the six regions of the prostate targeted in standard biopsy procedures (Figure 1A). Specimens were deparaffinized by heating them at 75 °C for 1 h until the paraffin wax melted. The specimens were then placed in 65 °C xylene for 48 h. Next, a simulated core-needle biopsy (~ 1 mm in width) was taken from each of the six deparaffinized blocks per case, resulting in 300 total biopsy cores (Figure 1A).

We used a previously developed “T&E” staining protocol (nuclear dye TO-PRO3 and Eosin) and imaged each biopsy core using OTLS microscopy [15,19]. Simulated biopsies underwent two 1-h washes in 100% ethanol to remove excess xylene and were partially rehydrated in 70% ethanol for 1 h. Individual biopsies were stained for 48 h in 70% ethanol at pH 4 with a 1:200 dilution of Eosin-Y (catalog no. 3801615, Leica Biosystems) and a 1:500 dilution of TO-PRO3 (catalog no. T3605, Thermo Fisher Scientific) at room temperature with gentle agitation. After staining was complete, the biopsies were dehydrated in 100% ethanol for 2 hours. Samples were optically cleared by individually placing them in ethyl cinnamate (catalog no. 112372, Sigma-Aldrich) (refractive index of n = 1.56) for 8 h before imaging using OTLS microscopy.

For imaging, ethyl cinnamate was used as the immersion medium and samples were mounted within a custom HIVEX (n = 1.55) biopsy holder, as described previously [15]. A four-channel digitally controlled laser module (Skyra, Cobolt Lasers, Hübner Photonics, Kassel, Germany) was used for illumination. TO-PRO3 fluorescence was excited at 638 nm and Eosin fluorescence was excited at 561 nm. Fluorescence bandpass filters at 721 nm ± 65 nm and 618 nm ± 50 nm were used for collection of TO-PRO3 fluorescence and Eosin fluorescence, respectively. The lateral and axial resolution of the OTLS microscope used for this study was ~0.95 μm and ~3.5 μm, respectively (full width at half maximum of the point spread function). Tissues were imaged at an isotropic sampling pitch of ~0.44 μm/pixel. The volumetric imaging time was approximately 0.5 minute per cubic mm of tissue for each illumination wavelength.

Data preparation

Two-channel OTLS microscopy datasets were stored on disk in the HDF5 format with microscopy metadata in an XML file. A custom compression filter (B3D) was used to provide 10x lossless compression. Raw OTLS images were reviewed in BigDataViewer [37] by board-certified pathologists to identify cancer-containing biopsies (Figure 1A). Of the initial 276 biopsy cores extracted, 102 contained cancer. Visual examination of the 3D datasets enabled us to identify regions of the biopsies in which most of the glands were cancer. Continuous (stitched and fused) 3D volumes were created of these cancer regions using the BigStitcher plugin for ImageJ [38]. H&E-like false coloring was performed on our OTLS datasets using FalseColor-python, a previously published method that mimics Beer–Lambert absorption of light as a function of staining concentration [20].

Nuclear segmentation and feature extraction

Given the memory requirements necessary for 3D segmentation, fused 3D images were broken into discrete blocks for processing. Each block was 2048 × 1432 × 500 voxels in size, or approximately 1024 by 716 by 200 μm in the axial direction (along the length of the needle core) (Figure 2A,B). Prior to segmentation, data blocks underwent median filtering with a circular structuring element (r = 2 voxels) from scikit-image [39] followed by contrast-limited adaptive histogram equalization (CLAHE) (Figure 2C, supplementary material, Figures S3,S4). After preprocessing, data blocks were passed into the cellpose [36] 3D nuclear segmentation model using the following parameters: an average diameter of 17 voxels, net averaging, and a batch size of 6 (Figure 2D, Supplementary materials and methods). All processing was done on a desktop computer equipped with 512 GB of RAM and a Nvidia Quadro 5000 RTX GPU.

Figure 2. 3D nuclear segmentation pipeline for biopsies imaged with OTLS microscopy.

(A) Nuclear channel (To-Pro-3) of a PCa biopsy imaged by OTLS microscopy with the cancerous region outlined with a dashed red box. (B) The cancerous region is broken up into discrete data blocks before processing. (C) Each data block is passed into a two-step preprocessing procedure before segmentation (see text for details). (D) Preprocessed data blocks are passed into cellpose to generate 3D nuclear segmentation masks, where each segmented nucleus is assigned a unique label.

To avoid quantifying the properties of nuclei that were artificially fragmented/cropped at the edges of each data block, segmented nuclei touching the boundaries of each data block were removed using the clear_border method from the skimage.segmentation python library [39]. Before feature extraction, small objects within each segmentation mask measuring less than the 1^st volume percentile were removed as noise (Supplementary materials and methods, supplementary material, Figure S5). Quantitative features (e.g. volume, sphericity, major/minor axis length) were extracted from the segmentations using the regionprops method from the skimage.measure python library. A full set of 3D nuclear features (and 2D counterparts) used is listed in supplementary material, Figure S6. Glandular masks of each cancerous region, generated via our ITAS3D pipeline [19], were used to classify segmented nuclei as belonging to epithelial or stromal compartments based on the centroid location of each segmented nucleus within the glandular mask (Supplementary materials and methods). Biopsy-level feature sets were collected by averaging the extracted features from all data blocks within a biopsy.

To compare the prognostic value of 2D and 3D nuclear features, 2D nuclear features were extracted from the 3D dataset at three cross-sectional levels separated by 20 μm, and then averaged (supplementary material, Figure S7). This replicates the standard-of-care pathology practice at many institutions in which histology sections are cut (and viewed) at three such levels [40].

Statistical analysis

Patients who experienced BCR within 5 years after radical prostatectomy were labeled as belonging to the ‘BCR’ group, and all other patients as ‘non-BCR.’ Biochemical recurrence (BCR) is defined as a rise in prostate-specific antigen (PSA) levels of >0.2 ng/ml after treatment (>8 weeks after radical prostatectomy). To assess the ability of different nuclear features to distinguish between BCR and non-BCR groups, we applied ROC analysis from which the area-under-the-curve (AUC) could be extracted.

Using the binary 5-year BCR category as the endpoint, a multiparameter classifier was developed to stratify patient risk. The least absolute shrinkage and selection operator (LASSO) linear regression model was used [41]. LASSO minimizes overfitting by making use of an L1 regularization term and identifies a subset of the most predictive input features. The LASSO tuning parameter $\lambda$ was optimized using an internal 4-fold cross validation (CV) scheme, where the dataset was randomly separated into four equally sized groups: three training datasets, and one to validate the model’s performance. Due to the lack of an external validation cohort, a nested CV scheme was used to develop multiparameter classifiers based on 3D and 2D features, and to determine the performance of the multiparameter models without bias or data leakage between parameter estimation and validation. Hyperparameter tuning was performed during each iteration of the outer CV, and LASSO regression was applied on the training set of the outer CV once an optimal $\lambda$ was identified in the inner CV. AUC values were calculated from the validation dataset of the outer CV. This nested CV was performed 200 times to determine an AUC (average and SD) [42].

Kaplan–Meier (KM) analysis was carried out to compare BCR-free survival rates for high- versus low-risk groups of patients. This analysis used a subset of 45 cases for which time-to-recurrence data was available. The performance of the models based on 2D or 3D features for epithelial nuclei was quantified with p-values (via log-rank test), hazard ratios (HR), and concordance index (C-index) metrics. For the multiparameter classification model used for KM analysis, the outer 4-fold CV in our nested CV schema was replaced by a leave-one-out approach, where one case was left out each iteration (45 total iterations, i.e. one iteration for each of the cases) to calculate the probability of 5-year BCR for that patient [43]. The samples were labeled as low- or high-risk via a posterior class probability threshold of 0.5. MATLAB was used for KM analysis and all other statistical analyses was performed in Python using the scipy and scikit-learn libraries [44].

Results

3D nuclear segmentation

Generating ground-truth annotations to train a 3D nuclear-segmentation algorithm can be tedious. Therefore, we chose cellpose as our segmentation framework for two reasons: first, as a “generalist” segmentation algorithm, cellpose requires minimal retraining or annotations to operate on unseen data, and secondly, cellpose is compatible with 3D datasets and can efficiently segment densely packed nuclei within large tissue volumes. To assess segmentation quality, we generated 3D ground truth annotations of nuclei within small sub-volumes (n = 6) of PCa biopsies imaged with OTLS microscopy and found that cellpose successfully generated 3D nuclear segmentations with an average dice coefficient of 0.81 (supplementary material, Figure S8). When evaluating cellpose segmentation quality with our moderate-resolution OTLS datasets (~0.9-micron lateral resolution), the best segmentation results were achieved with the native-resolution datasets without any down sampling (supplementary material, Video S1).

At full resolution, each individual (single biopsy) 3D pathology dataset was too large to segment at once. Thus, cancerous biopsy regions were divided into smaller blocks for 3D segmentation (Figure 2A,B). Each data block underwent a two-step preprocessing routine (Figure 2C) (Supplementary materials and methods, supplementary material, Figures S3,S4) that improved segmentation quality. This process decreased the rate of over-segmentation and improved the detection of nuclei in low-signal regions of the tissue. Images were then passed into the cellpose nuclear-segmentation model to generate a final 3D segmentation mask (Figure 2D, supplementary material, Videos S2 and S3). An image atlas of example 3D nuclear-segmentation results is shown in Figure 3, including the stratification of nuclei into epithelial and stromal compartments by using ITAS3D-generated gland-segmentation masks.

Figure 3. Image atlas of 3D nuclear segmentation results.

(A) Non-destructive 3D pathology dataset of a PCa biopsy and segmented sub-volume. Sagittal (B), coronal (C), and axial (D) views of 3D nuclear segmentations of the segmented sub-volume. During segmentation, each identified nucleus is given a unique integer label, which is represented by a color for visualization purposes (scale bar, 100 μm). (E) Larger field of view showing cancerous glands and surrounding stroma (left). colorized segmentation masks overlaid onto H&E false-colored OTLS images (middle). Segmented nuclei colored by their location within the prostate microarchitecture (right) (scale bar, 50 μm).

Preliminary clinical study

To evaluate the prognostic value of 3D versus 2D nuclear features (see materials and methods), our study consisted of 46 PCa cases in which the patients were followed for a minimum of 5 years after radical prostatectomy as part of the Canary TMA case-cohort study, which recruited primarily low- to intermediate-risk patients [22]. The primary endpoints of the Canary TMA study were 5-year biochemical recurrence (BCR) outcomes and time to BCR, which were also used as endpoints in our study. Roughly half of the cases in the Canary TMA study had BCR within 5 years.

Six simulated core needle biopsies were extracted from the 46 cases and imaged with OTLS (see materials and methods). Review of the 3D pathology datasets by genitourinary pathologists (L.D. True and N.P. Reder) identified the cancer-enriched regions within each of the 276 biopsies (i.e. regions in which most glands were cancerous), in which only 102 of the biopsies contained appreciable amounts of cancer. The nuclear-segmentation and ITAS3D (gland-segmentation) pipelines were applied to the cancer-enriched regions of these 102 biopsies. By combining the 3D nuclear segmentations (instance segmentations) with the glandular segmentation masks (semantic segmentation), nuclei were identified as contained in either the glandular epithelium or stromal tissue compartments (see Materials and Methods).

2D and 3D shape-based nuclear features were quantified and analyzed in terms of their association with BCR outcomes (supplementary material, Figure S6). For 2D analysis, average values from a total of 3 cross-sectional levels were calculated, in which the three levels were separated by 20 μm (mimicking clinical practice at many institutions, supplementary material, Figure S7). A total of 18 shape-based nuclear features were used as the inputs into a multiparameter risk classifier based on 5-year BCR outcomes. Separate risk-classification models were trained for the nuclei in the glandular epithelium versus stromal compartments. The average area-under-the-ROC-curve (AUC) was used as a measure of the ability of the combined nuclear features to predict 5-year BCR outcomes as a proxy for indolent versus aggressive disease.

3D shape-based nuclear features consistently outperformed their analogous 2D features, both for epithelial nuclei and stromal nuclei (Figure 4A,B). From comparison of Figure 4A and Figure 4B, it was apparent that there was better performance with models trained on shape-based nuclear features from the glandular epithelium than models trained with nuclei from the stromal regions. 3D shape-based nuclear features in the stroma were found to be weakly prognostic with an average AUC = 0.63 ± 0.06, while 2D shape-based nuclear features in the stroma were not prognostic with an average AUC = 0.50 ± 0.05.

Figure 4. Clinical analysis comparing the prognostic value of 3D & 2D nuclear features.

(A) ROC curves are shown for multiparameter models trained on 3D (red) and 2D (blue) nuclear features in epithelial regions. (B) ROC curves of multiparameter models trained on 3D (red) and 2D (blue) nuclear features in stromal regions. (C,D) Kaplan–Meier curves are shown for BCR-free survival, showing that the model trained on epithelial 3D nuclear features (C) can better stratify patients into low- and high-risk categories than the model trained on epithelial 2D nuclear features (D). (E,F) Violin and box plots are shown for two examples of epithelial 3D nuclear features, along with their analogous 2D features, for cases in which BCR occurred within 5 years of radical prostatectomy (“BCR”) and for cases in which there was no BCR within 5 years of radical prostatectomy (“non-BCR”). For both example features, “Mean epithelial nucleus-to-convex-hull ratio” in (E), and “epithelial nucleus-to-convex-hull variance” in (F), the 3D feature shows improved patient risk stratification compared to its 2D counterpart.

Kaplan–Meier (KM) curves of BCR-free survival were constructed for a subset of cases in which time-to-recurrence (BCR) data was available (Figure 4C,D). Compared with models based on 2D shape-based nuclear features, models based on 3D shape-based nuclear features are associated with a higher HR and C-index, along with a significant P value (P < 0.05). This suggests a stronger association with outcome for 3D shape-based nuclear features compared to corresponding 2D features.

For this early-stage study, 14 out of 18 of the shape-based nuclear features were independently associated with BCR outcomes (p < 0.05). Of these 14 features, 5 were prognostically significant in 3D but not in 2D. As one example, Figure 4E,F shows the mean and variance of the nucleus-to-convex hull ratio of epithelial nuclei (3D versus 2D). The convex hull is the minimum polyhedron (or polygon in 2D) that encloses a nucleus. Figure 4E shows that the nuclei in BCR cases (i.e., aggressive PCa) typically had a higher mean nucleus-to-convex-hull ratio (P = 1.1 × 10⁻⁴). Physically, this means that the nuclei were more spherical or ovaloid rather than irregular and/or curved in shape. However, this relationship is not apparent in the 2D case (P = 0.24). Similarly, Figure 4F demonstrates that the variance of the nucleus-to-convex-hull ratio was greater for recurrent versus non-recurrent cases when examined in 3D with P = 8.6 × 10⁻⁶, while this difference was less in 2D (P = 0.47).

Overall, our analysis of a variety of nuclear shape features consistently reveals less heterogeneity (variance) in aggressive (BCR) cancers compared to non-aggressive cases (supplementary material, Figure S9). This parallels the results of a previous study in which higher-grade PCa cases had smaller and more homogeneous glandular structures (less spatial variance) [19].

Discussion

Improvements in resolution, scale, and multiplexing capacity for non-destructive 3D imaging technologies are leading to disease insights that can inform treatment decisions [21]. However, given the growing size of datasets generated by modern “spatial biology” techniques, computational tools must also be developed to enable pathologists and oncologists to efficiently comprehend such large datasets. An attractive initial approach is to rely on intuitive features familiar to pathologists (e.g., glandular and cytologic features), which will improve interpretability and facilitate clinical acceptance. As in our previous study examining 3D glandular features [19], our goal in this study has been to demonstrate the value of 3D pathology by providing a direct comparison of intuitive 3D versus 2D nuclear features analyzed computationally. Our work leverages the fact that computational 2D pathology has already been demonstrated to improve disease prognostication [29,45,46], and explores the additional value that computational 3D pathology can offer for certain applications such as risk stratification of PCa. We have avoided comparing our computational results with risk classifiers or nomograms that rely on human interpretation of histology images [43,47,48]. Our motivation for this is that by directly comparing 3D versus 2D computational pathology, we remove the subjectivity introduced by human interpretation. Such human-observer studies, to be performed in the future, will require a significantly larger cohort of patients and a large panel of pathologists to mitigate interobserver discordance.

Previous studies have used 3D imaging of in vitro cancer models to examine tumorigenesis, drug response, and cancer-associated alterations in cellular development [49–51]. However, to our knowledge, this is the first report to analyze the prognostic significance of nuclei within their native 3D context in human cancer specimens. Given the relatively small number of cases in this preliminary analysis, we limited the number of 3D nuclear features to those that we deemed would most likely have prognostic significance based on previous studies [23–29,52–58]. Shape-based nuclear features are an attractive choice for several reasons: they are the most frequently used features for prognostication based on 2D whole slide images [18–23, 42, 43], they are intuitive for clinical and biological interpretation [59], they have been considered in the past for PCa grading [30–35], and there are relatively straightforward analogs between 2D and 3D shape-based features. Rather than exhaustively mining all possible nuclear features to maximize classifier performance, we have deliberately based our classifiers on a limited set of intuitive features to show that 3D nuclear features have obvious prognostic value. In other words, with some effort, we believe that better 3D features and models can be developed in the future.

Our analysis shows that for PCa, epithelial nuclei hold the most prognostic significance (versus stromal nuclei) for stratifying patients based on known clinical outcomes (Figure 4A). Given that PCa is typically a disease of epithelial cells that form glands, this result is not surprising. However, we find that 3D shape-based features of stromal nuclei are also somewhat prognostic (Figure 4B), which is supported by previous studies using 2D histology [4,28,29,60–62]. Since these stromal nuclei are from a mixture of cell types, it is not surprising that they are less prognostic than the epithelial nuclei in the cancer glands. Most importantly, for both epithelial and stromal models, our results demonstrated that 3D shape-based nuclear features were more strongly associated with BCR compared to their 2D counterparts for risk stratification (Figure 4A–D). Even with the limited number of samples and extracted features in this preliminary analysis, certain differences between nuclei in indolent and aggressive cases were only statistically significant when examined in 3D (Figure 4E,F). These findings support the value of 3D digital pathology methods for clinical management of PCa. Overall, the prognostic value of the 3D nuclear features analyzed in this study is less than the prognostic value of the 3D glandular features we previously explored¹⁹. This is expected and consistent with the fact that PCa glandular structures are the current basis for clinical Gleason grading. Therefore, the ideal clinical classifier should holistically combine both glandular and cytologic/nuclear features in the future.

In addition to combining cytologic and architectural features (e.g., nuclei and glands), there are clear opportunities to extend our work. For example, adding 3D features from additional tissue structures, such as vasculature, could reveal novel signatures of aggressiveness. Having a comprehensive spatial and molecular view of tumors in 3D would also be of obvious clinical value^63,64. Finally, obtaining higher-resolution datasets with the latest generations of OTLS microscopy systems^16,17 should allow for more accurate quantification of 3D nuclear features as well as the extraction of sub-nuclear features (e.g. chromatin texture, nucleoli). This preliminary study motivates future exploratory directions using computational 3D pathology, as well as larger-scale clinical studies to guide treatment decisions, such as deciding which PCa patients should be placed on active surveillance versus treated with surgery/radiation, or which patients should receive adjuvant therapies after surgery/radiation. Ultimately, we aim to demonstrate that computational 3D pathology can improve the long-term outcomes and quality of life for patients with PCa and many other diseases.

Supplementary Material

supinfo

Figure S1. Example 2D images of benign (top) and cancerous (bottom) prostate tissues imaged with 2nd-generation OTLS microscopy versus slide-based H&E histology

Figure S2. Morphological complexities of nuclei cannot be fully captured in 3D

Figure S3. Effect of median filter kernel size on segmentation results

Figure S4. Effect of preprocessing procedure on cellpose segmentation results in low signal regions

Figure S5. Sensitivity of nuclear properties and features with respect to volume threshold

Figure S6. Set of shape-based nuclear features used for prognostic analysis

Figure S7. 2D segmentation pipeline

Figure S8. Comparison of nuclear segmentation methods on manually annotated OTLS volumes

Figure S9. Comparison of the epithelial nuclear features (variance) between BCR groups

video_S1

Video S1. Effect of down sampling on segmentation results

video_S2

Video S2. 3D segmentation results

video_S3

Video S3. Colorized segmentation results for a PCa biopsy

Data availability statement

The original 3D pathology datasets used in this study may be found at: https://doi.org/10.7937/44MA-GX21 [68]

The base code for the cellpose segmentation framework may be found at: https://github.com/mouseland/cellpose

The code base for the segmentation, feature extraction, and analysis pipelines may be found at: https://github.com/serrob23/OTLS-cellpose