Automatic cancer detection on digital histopathology images of mid-gland radical prostatectomy specimens

Abstract.

Purpose: Automatic cancer detection on radical prostatectomy (RP) sections facilitates graphical and quantitative surgical pathology reporting, which can potentially benefit postsurgery follow-up care and treatment planning. It can also support imaging validation studies using a histologic reference standard and pathology research studies. This problem is challenging due to the large sizes of digital histopathology whole-mount whole-slide images (WSIs) of RP sections and staining variability across different WSIs.

Approach: We proposed a calibration-free adaptive thresholding algorithm, which compensates for staining variability and yields consistent tissue component maps (TCMs) of the nuclei, lumina, and other tissues. We used and compared three machine learning methods for classifying each cancer versus noncancer region of interest (ROI) throughout each WSI: (1) conventional machine learning methods and 14 texture features extracted from TCMs, (2) transfer learning with pretrained AlexNet fine-tuned by TCM ROIs, and (3) transfer learning with pretrained AlexNet fine-tuned with raw image ROIs.

Results: The three methods yielded areas under the receiver operating characteristic curve of 0.96, 0.98, and 0.98, respectively, in leave-one-patient-out cross validation using 1.3 million ROIs from 286 mid-gland whole-mount WSIs from 68 patients.

Conclusion: Transfer learning with the use of TCMs demonstrated state-of-the-art overall performance and is more stable with respect to sample size across different tissue types. For the tissue types involving Gleason 5 (most aggressive) cancer, it achieved the best performance compared to the other tested methods. This tool can be translated to clinical workflow to assist graphical and quantitative pathology reporting for surgical specimens upon further multicenter validation.

1. Introduction

Radical prostatectomy (RP), the removal of the prostate gland, is the most common treatment for organ-confined prostate cancer (PCa), which is performed on $\sim 40\%$ of PCa patients annually in the United States.¹ Approximately 17% to 29% of patients experience cancer recurrence after surgery, portended by serum prostate-specific antigen relapse.^2,3 Adjuvant and salvage therapy, including radiation therapy to the prostate bed, can be life-saving for patients with recurrence, to prevent or delay mortality due to metastatic disease.³

The surgical pathology report provides valuable information for postsurgery prognosis, recurrence risk management, and selection and guidance of adjuvant therapy. The primary and secondary tumors in the specimen are reported in terms of location, volume, and the aggressiveness of differentiation based on the Gleason grading system, which classifies the tumors into five grades based on their morphological patterns, considering Gleason grade 1 to 3 as low grade and 4 to 5 as high grade.³

Advances in pathology have included establishing standards for important report elements and a move toward synoptic summaries.⁴ However, current clinical pathology reporting is still primarily text-based, qualitative, and subject to interobserver variability, which leads to challenges in terms of quantitative and repeatable interpretation of lesion size, location, and spread. Methods of measuring and reporting tumor volume vary, and no generally accepted standard has been established.⁵ Large interobserver variability has been reported for identifying extra prostatic extension (EPE) (i.e., where the tumor extends outside of the prostate into the surrounding region) without anatomic landmarks.⁶ In addition, assessing the differentiation degree of the tumor using Gleason score⁷ (i.e., assigning a total Gleason score using the sum of Gleason grades of primary and secondary tumors) has been established for decades, but reporting the total Gleason score remains problematic. For cases with multiple nodules, reporting the overall score may underestimate the tumor aggressiveness.⁸

Currently, pathology reports include or seek to incorporate accurate and detailed information to maximize clinical utility.^9,10 Bettendorf et al.⁹ proposed a hand-drawn tumor map of the prostatectomy specimen for pathology reporting, which provides a way for visual estimation of tumor size and location. To quantify the hand-drawn loci on the anatomical maps in prostatectomy specimens, Eminaga et al.¹⁰ developed an extensible markup language-based document architecture. However, this method requires substantial effort from the pathologist, with potential to slow the workflow and increase fatigue. Also the quantitative reporting based on these approaches is derived from approximate hand-drawn representations. Thus the clinical challenges remain. Annotating each cancerous region of interest (ROI) at high resolution ($20\times$ or higher) on whole-mount RP sections enables quantitative reporting, giving measurements of tumor size, location, and grade. This would resolve the clinical challenges mentioned above and would also benefit research studies investigating the relationship between quantitative pathologic parameters, such as tumor volume and clinical outcomes. Moreover, PCa is challenging to detect and localize on imaging, such as multiparametric magnetic resonance imaging. This has motivated the undertaking of imaging validation studies that use annotated prostatectomy histology as the reference standard.^11,12 Adequate target delineation of the tumor through such studies can potentially improve disease control by allowing safe boosting of radiation dose (or targeting other ablative therapies) to corresponding areas of the prostate bed and reduction of side effects by reducing the treatment margins.¹³

The time required to conduct such contouring manually precludes its use within a clinical pathology workflow and adds substantial time and expense to imaging validation studies. Therefore, there is an unmet need for a software tool that can provide accurate and fast automatic contouring of cancerous regions on whole-mount digital histopathology images of RP specimens.

Distinguishing cancer from noncancer tissue on histology is challenging since their appearance is visually similar [e.g., in Fig. 1, Gleason grade 3 (G3), small gland G4, and benign prostatic hyperplasia (BPH) (noncancer), are visually similar]. Previous studies have demonstrated the potential for computational approaches to address this problem (see the recent review article¹⁴). Most studies involved training and validation on a small subset (typically hundreds) of selected ROIs. In comparison, detecting cancerous regions throughout entire whole-mount whole-slide images (WSIs) of RP sections from different patients is more challenging for three primary reasons. First, the high resolution of the digital histopathology images leads to a large number of pixels in each WSI (e.g., a WSI of a mid-gland prostate section often includes $>4\text{gigapixels}$), requiring an efficient approach. Second, there is substantial heterogeneity of appearance in cancerous and noncancerous prostate tissue. Noncancerous tissue includes normal parenchyma, high-grade prostatic intraepithelial neoplasia (PIN), cancer-mimicking atrophy, and BPH, all of which having different appearances and some of which (e.g., PIN) sharing many common features with cancerous tissue. For cancerous tissue, there are different morphological patterns across different Gleason grades (e.g., different appearances for cancer samples G3 and G4 and noncancer samples benign, PIN, and BPH in Fig. 1). Even within the same grade, the tissue appearance can be quite heterogeneous.³ For instance, Gleason grade 4 has several subtypes that have different morphological patterns (e.g., small glands, large cribriform glands, mucinous glands, poorly formed glands, etc.) (Fig. 1). Testing a system on all of the tissue throughout the WSIs thus introduces more heterogeneity into the samples, resulting in a more challenging classification problem. Third, staining variability is a substantial issue, with contributing factors such as slide preparation and batch effects.^15,16 Although many color normalization methods have been proposed,^17–19 color normalization on all images to a template did not reduce instability of texture features and therefore has not completely resolved the problem.¹⁵ Other challenges arise from factors including the presence of tissue marking dye due to the inking of the surfaces as part of pathology processing, artifacts (e.g., tissue folding, out-of-focus regions, and tissue tearing), presence of red blood cells (RBCs), and within-ROI heterogeneity (i.e., ROIs contain both cancerous and noncancerous tissue). Validation using all available tissue avoids bias from ROI selection and tests the system against the full variability in staining and tissue appearance.

Fig. 1.

Tissue samples: (a) cancer (G3) and noncancer tissue samples (benign, PIN, and BPH) and (b) subtypes of cancer (G4).

This paper presents a system for cancer detection and localization on WSIs of mid-gland RP sections, yielding state-of-the-art accuracies using three different methods with fast processing times. To the best of our knowledge, this is the first study that developed and validated a system for detecting cancerous regions on whole-mount WSIs using all tissues throughout the slides, including all clinically relevant grade groups, and demonstrating a practical processing time. Also we have validated the system on the largest reported annotated data set (i.e., we used the largest amount of tissue from the largest set of whole-mount tissue sections), with highly detailed reference standard contours provided by a pathologist. This is very important for translating the technology toward clinical application. Although many previous studies have demonstrated the capability of machine learning methods for distinguishing cancer tissue from noncancer tissue, most studies used preselected ROIs or biopsy WSIs, which contain a much smaller amount of tissue than whole-mount prostatectomy WSIs. Because PCa is highly heterogeneous, such studies do not demonstrate that machine learning methods can be implemented on whole-mount prostatectomy WSIs from large data sets for cancer detection for clinical translation. Only two studies^20,21 used materials from whole-mount prostatectomy WSIs without preselected ROIs, but they reported a limitation of their methods regarding detection of high-grade cancer tissues, which are of high importance in clinical practice.

In addition, our study provided the first direct comparison of cancer detection performance using conventional and deep learning approaches. We also identified that the three-class tissue components provided the major cues for cancer detection and better detection performance for detecting high-grade cancer using pretrained AlexNet.

The primary contributions of this paper are as follows.

• (1)A calibration-free adaptive thresholding algorithm for fast nuclei segmentation, which yields consistent three-class tissue component maps (TCMs) despite staining variability across WSIs for PCa detection on whole-mount WSIs.
• (2)Validation of a conventional machine learning approach with three classifiers and a deep learning approach using the inputs of raw images and TCMs for cancer detection on 286 WSIs of mid-gland RP sections from 68 patients (4 to $5\mathrm{WSIs}/\text{patient}$). Cross validation (CV) was performed, grouping data on per-patient basis (i.e., tissues from a single patient never appeared in both the training and testing sets).
• (3)Direct comparisons of the five machine learning methods using our proposed pipeline for PCa detection on whole-mount WSIs.
• (4)Identified 14 TCM-based texture features for effective cancer detection from the conventional machine learning approach, yielding better than state-of-the-art performance compared to other nondeep learning-based methods.
• (5)Achieved the best overall performance across all methods using deep learning approaches and the use of the three-class TCM demonstrated more robust performance across different types of prostate tissue.

2. Related Work

Substantial work has been published on the problem of PCa detection on digital histology images of hematoxylin and eosin (H&E)-stained specimens.¹⁴ Most previous research has focused on the classification of cancerous versus noncancerous using preselected ROIs, with a few previous studies focused on cancer identification and localization on whole-mount tissues (biopsy and RP).

Studies^22–30 performed cancer versus noncancer classification using preselected ROIs. Their overall positive results are valuable in that they point to the potential utility of the methods employed. However, the total processed tissue amount in each study was $<4{\mathrm{cm}}^2$, which is about half of the size of a single WSI of a mid-gland tissue section. Considering the heterogeneity of PCa tissue and staining variability among WSIs, the limited extent of the processed tissue points to the need for a study with more comprehensive validation throughout entire WSIs. Also these studies generally did not prioritize computational cost and the scalability to larger tissue samples as would be encountered in clinical practice.

Several proposed approaches have demonstrated the potential for detection and localization of cancerous regions on WSIs of biopsy tissues. Doyle et al.³¹ used a multiresolution approach to detect PCa with a boosted Bayesian classifier, and reported areas under the receiver operating characteristic curve (AUCs) of 0.84, 0.83, and 0.76 at the lowest, intermediate, and highest resolution levels, respectively, on 100 biopsy WSIs from 58 patients with an average processing time of $\sim 3\min$ per $1000\times 1000\text{pixel}$ region. Litjens et al.³² used a super-pixel-based approach and reported an AUC of 0.96 at the per-slide level with a sensitivity of 1.0 and specificity 0.4, and an AUC of 0.92 at the super-pixel patch level using 10-fold CV (data were stratified on a per-patient basis) using 204 WSIs of biopsy tissues from 163 patients. Approximately 4 min of processing time per WSI was reported. Their more recent work³³ used a convolutional neural network and reported an AUC of 0.99 (90th percentile) and 0.98 (median percentile). The processing time was not reported. Overall, it is not clear that these systems would scale up to the data sizes involved in RP specimens; for instance, a computation time of 4 min per WSI of biopsy tissue³² would result in a requirement of $\sim 17\mathrm{days}$ of computation per mid-gland RP WSI. Recently, Campanella et al.³⁴ demonstrated that a weakly supervised approach using multiple instance learning was able to classify prostate biopsy slides, at the slide level, as containing cancer (or not) with an AUC of 0.99, 0.96, and 0.93, testing on 1784, 1274, and 12,727 biopsy WSIs, respectively. Those data sets were from the same scanner, a different scanner, and different institutes, respectively. Their work showed that when the classification of interest is at the whole-slide level, curated data sets with detailed ground-truth contours are not necessary for training.

Several studies reported system designs for detecting PCa on WSIs of prostate tissue sections. Monaco et al.²⁰ and Rashid et al.²¹ classified each individual gland as malignant or benign by extracting gland features. Monaco et al. reported a sensitivity and specificity of 0.87 and 0.90, respectively, using 40 RP tissue sections from 20 patients at low resolution ($8\mu \mathrm{m}/\text{pixel}$) with average processing time of 2.75 min per whole-mount tissue section. Rashid et al. reported a sensitivity of 90%, a specificity of 93%, and an accuracy of 93%, validating on 20 WSIs from 11 patients with a training data set of 50 WSIs from 19 patients without reporting system processing time. Both works reported a limitation that the system was unable to detect poorly differentiated cancer due to the dependence of the method on gland classification. This is an issue for Gleason grade 5 (G5) cancer, in which the glands are disrupted. In their data set, no G5-inclusive tissue samples (i.e., pure G5 and mixed grades including G5, such as $\mathrm{G}4+5$) were used. However, the presence of G5 cancer is highly negatively prognostic;³⁵ and therefore it is critical to detect G5 cancer.

DiFranco et al.³⁶ used a tile-based approach, classifying each $512\times 512\text{pixel}$ tile of the WSI from RP specimens and reported an AUC of 0.95 using 15 WSIs from 14 different patients. Texture features were calculated at different color channels of RGB and CIE L*a*b color space. The experiments conducted in the study used a 16-core server without reporting computational time, and the need for parallel computing was described.

Nguyen et al.³⁷ reported a false positive rate (FPR) of 6% with 78% sensitivity using 6 images (approximate size of $4000\times 7000\text{pixels}$) for training and 11 WSIs (approximate size of $5000\times 23000\text{pixels}$) for testing, incorporating nucleus-related cytology features with texture features without reporting the processing time.

Nir et al.³⁸ used multiple pathologists’ annotations with hand-crafted features and reported an AUC of 0.85 with an accuracy of 90.5%, a sensitivity of 91.5%, and a specificity of 85.2% at the optimal operating point. They tested on 333 tissue microarray (TMA) cores sampled from 231 RP specimens for classifying TMA patches as cancerous versus noncancerous. The validation was conducted using leave-one-patient-out (LOPO) CV with a processing time of $\sim 14\mathrm{h}$ for feature extraction for 333 TMAs, and $\sim 4.5\mathrm{h}$ for 231 LOPO simulations for training and classification using parallel computing via two 12-core computers. They reported an overall AUC of 0.75 when testing their system, trained using all of the TMAs, on an external dataset of 230 WSIs of RP sections from 56 patients using all of the cancerous tissues, and 10% of the noncancerous tissue by random sampling.

To the best of our knowledge, no system for PCa detection has been reported and validated for performance throughout all tissues on RP WSIs including all clinically relevant grade groups, with practical processing time.

3. Methods

Figure 2 shows a block diagram depicting the training of conventional and deep learning-based classifiers for automatic cancer detection on WSIs of RP sections, building on our previously reported prototype system.³⁹ In training, cancerous and noncancerous ROIs were determined using gold-standard histopathology annotations. We computed a three-class TCM for each ROI using our proposed segmentation method. The three machine learning approaches used in our study were as follows. (1) We computed 14 selected first- and second-order statistical features^40,41 from TCMs and trained three classifiers [a Fisher linear discriminant classifier (FisherC), a logistic linear classifier (LoglC), and a support vector machine (SVM) classifier]. (2) Pretrained AlexNet⁴² (pretrained on the ImageNet database⁴³) was fine-tuned using the TCMs. (3) Pretrained AlexNet was fine-tuned using raw image ROIs without any image processing. During testing, the trained system classified all of the ROIs covering each WSI as cancerous or noncancerous. We validated the classification results against gold standard histopathology annotations using CV, with data grouped on a per-patient basis, ensuring that samples from the same patient never appeared in both the training and testing sets in any fold. We performed the validation using each of the trained systems, and the results were collected for analysis and comparison. Our implementation used MATLAB 2018a (The Mathworks, Natick, MA), OpenCV 3.1 for SVM implementation, and PRtools 5.0 (Delft Pattern Recognition Research, Delft, The Netherlands) for the implementation of FisherC and LoglC, and MATLAB 2018a for AlexNet. The code for tune-AlexNet-TCM is publicly available in the Github repository: [https://github.com/WenchaoHanSRI/WenchaoProstate].

Fig. 2.

Method overview for system training using three different machine learning methods. (a) WSIs with our expert annotations. Different colored annotations represent different types of tissue based on the Gleason grading system. (b), (c) Zoomed views from the black square regions in (a) and (b), respectively.

3.1. Data

3.1.1. Manual annotation

This study was approved by our institutional Human Subjects Research Ethics Board with informed consent of all patients. We obtained 299 WSIs of H&E-stained, $4\text{-}\mu \mathrm{m}$ thick, paraffin-embedded mid-gland tissue sections from 71 RP specimens from patients with biopsy-confirmed PCa (clinical stage T1 or T2). All tissue sections were prepared and scanned in our hospital pathology laboratory following the same protocol.²⁸ Two different types of scanners were used: an Aperio ScanScope GL (Leica Biosystems, Wetzlar, Germany) for sections from 46 patients and an Aperio ScanScopeAT Turbo (Leica Biosystems, Wetzlar, Germany) for sections from the other 25 patients. Sections were scanned at $20\times (0.5\mu \mathrm{m}/\text{pixel})$ in bigtiff pyramid format without compression. The resulting 24-bit RGB color images have a pixel size of $0.5\mu \mathrm{m}/\text{pixel}$.

Each WSI was contoured and graded by a trained physician (Gaed) at $20\times$ magnification using different colors for different Gleason grades, and verified by one of two genitourinary pathologists (Moussa or Gomez). Contouring was performed using a Cintiq 12WX pen-enabled display (Wacom Co. Ltd., Saitama, Japan) with the ScanScope ImageScope v11.0.2.725 image viewing software (Aperio Technologies, Vista, CA, USA). Contouring was conducted at high precision [Figs. 2(b) and 2(c)], which takes about 70 h per case. Where Gleason grades were intermingled to a degree where they could not be readily separated, foci were given a grade such as $4+3$, indicating that the majority of the focus contained grade 4 cancer, and the remainder contained grade 3 cancer.

3.1.2. Ground truth ROI labeling

Each WSI was separated into a set of square $960\times 960\text{pixel}$ ROIs. ROIs containing at least 50% cancerous tissue according to the manual pathology annotations were considered cancerous; all other ROIs were considered noncancerous.

3.1.3. Data separation for system tuning and feature selection

We performed classifier hyper-parameter tuning and feature selection on a separate tuning data set comprising 13 WSIs from 3 different patients. We did not use the patients in the tuning data set for CV and used only the 68 remaining patients for CV. The tuning and CV data sets each have a mixture of WSIs from both scanners.

3.2. Tissue Component Mapping

The purpose of tissue staining is to assist in identifying different types of tissue components, which have semantic meaning to the pathologist for identifying abnormalities. We developed an algorithm that assigns a label to each image pixel to generate a TCM for further analysis. We labeled each pixel as one of three tissue components: nuclei, lumen, and stroma/other tissue. The main steps of our TCM generation algorithm are: (1) segmentation of nuclei using color deconvolution and a proposed adaptive thresholding method, (2) segmenting luminal areas by global thresholding in the red-green-blue (RGB) color space, and (3) designating other pixels as stroma/other tissue. These steps are described in more detail as follows.

3.2.1. Nucleus mapping

We segmented cell nuclei by adaptive thresholding of the hematoxylin channel after color deconvolution.⁴⁴ We then used morphological operations to eliminate RBC pixels that were falsely labeled as nuclei, by relabeling them as stroma/other tissue. The details of these steps are described below.

Color deconvolution. We used a color deconvolution algorithm⁴⁴ to separate the H&E stains into three image channels corresponding to the hematoxylin stain, eosin stain, and the background.

We used the standard deconvolution matrix used by Ruifrok and Johnston⁴⁴ and applied this algorithm to each ROI independently. This separated each ROI into three gray-level images representing the amount of hematoxylin [e.g., Fig. 3(b), darker region corresponding to larger amount of hematoxylin stain], eosin, and background, respectively. Since most substances within nuclei are basophilic, they bind to hematoxylin. We, therefore, used the hematoxylin channel to segment nuclei.

Fig. 3.

Plot of number of connected components and the binarized hematoxylin channel at the corresponding thresholds. (a) Sample ROI. (b) Gray-level image representation of the hematoxylin channel after color deconvolution. (c)–(g) Binary maps after thresholding using the thresholds where the blue triangle, purple circle, black square, red arrow, and blue square labeled in the plot, respectively. The red square highlighted region in a zoomed-in view shows in (h). (i), (j) Images of zoomed in view highlighted by the red squares in (e) and (f), respectively.

Cell nucleus segmentation using adaptive thresholding: staining variability results in variations in hematoxylin channel intensity across different WSIs [see the middle images for the two sample ROIs in Fig. 4(a)]. The left case has much more darkly hematoxylin-stained stroma/fibromuscular tissue than the right, where the nuclei are more prominent. This makes global thresholding inapplicable for nucleus segmentation. We, therefore, propose an adaptive thresholding method.

Fig. 4.

Tissue component segmentation. Top row: two cases of H&E stained WSIs with their TCMs to the right. (a) Left: ROI samples from each of the cases, middle: gray-level hematoxylin channel images, right: nuclei map after adaptive thresholding. (b) Left: ROI samples from each of the cases, right: computed TCM using our segmentation method. (c) Left: ROI samples from the two cases with RBCs included, right: TCM computed using our segmentation method.

Our proposed algorithm is based on the observation that as the threshold on the hematoxylin channel increases, causing more hematoxylin-stained tissue to be excluded in the thresholded image [Figs. 3(c)–3(g)], initially the components of background tissue are excluded as a sharp decreasing number of connected components [Figs. 3(c) and 3(d)], resulting in a nuclei map with some background tissue. With a further increase of the threshold, background tissue was excluded to form a nuclei-only map [Fig. 3(e)]. At this point of inflection, the threshold can separate nuclei from other tissue components [Fig. 3(e)]. With further increasing of the threshold, the number of connected components increases slightly [shown qualitatively in Figs. 3(e) and 3(f) and quantitatively in Fig. 3(h) between the black square and the red arrow] because pixels from the same full nucleus disappear to separate the nucleus to form multiple independent connected components [Figs. 3(e), 3(f), 3(i), and 3(j)].

To detect this inflection point, we first fit a high-order polynomial curve to the data depicting the number of connected components as a function of threshold. The hyper parameter of curve order was tuned manually to 20 using our tuning data set, with two objectives in mind: (1) making the polynomial order as low as possible to avoid noise generated from oscillation based on the Runge phenomenon⁴⁵ and (2) minimizing the squared error between the curve and the original points. The resulting continuous and differential plane curve was fit using least squares approximation, and the coefficients were calculated using the Vandermonde method⁴⁶ with the chosen curve order.

Using the fit curve, we computed the desired inflection point in three steps. First, we computed the threshold $\tilde{x}$ giving the largest number of connected components [blue triangle in

$$\tilde{x}=\underset{x\in X}{\arg \max }F(x)$$ (1)

Figs. 3 and 5(a)] as denoted, where $x$ is a threshold on the hematoxylin channel, $X$ is the domain of thresholds on the hematoxylin channel, and $F$ is the fit polynomial curve. Second, we computed the threshold $\ddot{x}>\tilde{x}$ corresponding to the sharpest decrease in connected components [purple circle in Figs. 3 and 5(a) and 5(c)] as denoted

$$\ddot{x}=\underset{x\in X,x>\tilde{x}}{\arg \max }{F}^{\prime \prime }(x).$$ (2)

Third, the threshold $x_T\in X$ is defined as the closest rising inflection point to $\ddot{x}$ [black square in Figs. 3 and 5(a) and 5(b)]:

$$x_T=\underset{x\in X_T}{\arg \min }x-\ddot{x},$$ (3)

$$X_T=\{x_T>\ddot{x}|(\exists \delta ϵR^{+})(\forall ϵ<\delta )[F^{\prime }(x_T)>F^{\prime }(x_T+ϵ)\wedge F^{\text{'}}(x_T)>F^{\prime }(x_T-ϵ)]\},$$ (4)

where $X_T$ is the set of isolated local maxima of the first derivative (therefore the set of rising inflection points) of curve $F$ such that $x_T>\ddot{x}$.

Fig. 5.

Plots of (a) number of connected components and amount of hematoxylin stain with fitted curve in red, (b) first derivative of curve in (a), and (c) second derivative of curve in (a).

This threshold was found for each WSI via a cumulative assessment of 2000 randomly selected $120\mu \mathrm{m}\times 120\mu \mathrm{m}$ samples lying within the prostate (i.e., avoiding clear slide areas) and not containing tissue marking dye (i.e., avoiding areas of artifact).

RBC removal. Since RBCs [Figs. 6(a) and 6(f)] also stain with hematoxylin, the adaptive thresholding process erroneously labels them as nuclei [Fig. 6(b)]. However, RBCs have higher saturation than nuclei in a red-pink hue, allowing us to distinguish them from nuclei. We selected and applied hue–saturation–intensity RBC thresholds ($\text{hue}\ge 0.95/1$, $\text{saturation}\ge 0.72/1$, and $\text{intensity}\ge 0.6/1$) based on a cumulative histogram from 100 $40\mu \mathrm{m}\times 40\mu \mathrm{m}$ RBC ROIs selected from our tuning data set [see a sample thresholding result in Fig. 6(c)]. We then applied morphological dilation with a disk-shaped structuring element of $\text{radius}=4\mu \mathrm{m}$ (approximate radius of human RBCs) [Fig. 6(f)]. This resulted in an RBC mask [Fig. 6(e)] that was subtracted from the nucleus map [Fig. 6(b)] to eliminate falsely detected RBCs [Fig. 6(d)].

Fig. 6.

(a) RBC removal for an example ROI. (b), (d) Nuclei maps before and after RBC removal, respectively. (c) Binary mask covering the RBCs, generated by thresholding in HSV color space. (e) A binary mask created from (c) after morphological operation with a disk shaped structuring element of $\text{radius}=4\mu \mathrm{m}$ [approximate radius of human RBC (f)].

3.2.2. Lumen and stroma/other tissue component segmentation

Luminal regions are consistently nearly white on each WSI. We used global thresholding to segment luminal pixels with threshold values of $\text{red}\ge 0.86/1$, $\text{green}\ge 0.71/1$, and $\text{blue}\ge 0.82/1$. (Threshold values were chosen using a cumulative histogram calculated from luminal ROIs sampled from the tuning data set.) All pixels that were not labeled as nuclei or lumen were labeled as stroma/other.

3.2.3. Tuning ROI size and down-sampling ratio

The ROI size used in the experiment affects the resolution of the resulting cancer map, and the down-sampling ratio of the TCM affects computation time. We selected an ROI size of $480\mu \mathrm{m}\times 480\mu \mathrm{m}$ ($960\times 960\mathrm{pixel}$) and nearest-neighbor down-sampling ratio of 0.25 by experimentation with our tuning data set. We tested ROI sizes from $120\mu \mathrm{m}\times 120\mu \mathrm{m}$ to $720\mu \mathrm{m}\times 720\mu \mathrm{m}$ in $120\mu \mathrm{m}$ steps. We tested down-sampling ratios of 0.25 to 1 in increments of 0.25. The selection of those two parameters was manually performed based on evaluating the performance (AUC) for cancer detection (using FisherC) in leave-one-patient-out cross validation (LOPO CV) on the tuning data set.

3.3. Feature Extraction and Selection

We calculated 24 first-order and 132 second-order statistical features^40,41 from the TCM of each ROI, giving a total of 156 features. The second-order statistical features were based on the gray-level co-occurrence matrix (GLCM)⁴⁰ and gray-level run length matrix (GLRLM).⁴¹ GLCMs and GLRLMs were calculated using neighbors in four directions [(0, 1) denoted direction 1 in Table 1, ($-\mathrm{1,1}$) denoted 2, ($-\mathrm{1,0}$) denoted 3, and ($-1,-1$) denoted 4] without aggregation over the directions. In total, we calculated 156 features: (22 GLCM + 11 GLRLM) × 4 directions + 24 first-order features = 156.

Table 1.

Selected features used in cross validation.

Mean gradient value
GLRLM short-run low gray-level emphasis-1
GLRLM short-run low gray-level emphasis-3
GLRLM short-run low gray-level emphasis-4
GLCM energy-1
GLCM energy-2
GLCM information measure of correlation-1
GLCM information measure of correlation-2
GLCM inverse difference moment-2
GLCM inverse difference moment-3
GLCM cluster shade-3
GLCM correlation-1
GLCM entropy-2
GLRLM short-run emphasis-3

We selected the 14 top ranked features using backward feature selection via ranking the AUCs from LOPO CV of a Fisher linear classifier with the TCMs on the tuning dataset. The number of features was chosen by iterative experiments from 1 to 156 using feature selection as described. The chosen texture features are listed in Table 1.

3.4. Cancer Detection Using Machine Learning

We classified each ROI as cancerous versus noncancerous using the calculated features. We performed supervised machine learning on the TCMs using (1) FisherC, (2) LoglC, and (3) SVM (NU-SVC with a radial basis function kernel, parameters tuned as $\text{cost}=12.5$, $\text{gamma}=0.50625$ using our tuning dataset). Each of these classifiers is henceforth denoted as: (1) TCM-texture-FisherC, (2) TCM-texture-LoglC, and (3) TCM-texture-SVM, respectively.

We also used transfer learning by fine-tuning pretrained AlexNet with our TCMs and raw image ROIs, which are denoted as: (4) tune-AlexNet-TCM and (5) tune-AlexNet-RawIM. The methodological details of using transfer learning are described in the following paragraphs.

We used TCMs as input images to fine-tune the pretrained AlexNet. The TCMs were converted into RGB color images, in which red, green, and blue represent nuclei, stroma/other, and lumen, respectively. All the ROIs of size of $240\times 240\times 3$ were resized to $227\times 227\times 3$ (to conform to the necessary input size for AlexNet) using bilinear interpolation. For comparison, the same method as that used for tune-AlexNet-TCM was repeated using the “raw” unmodified H&E images instead of TCMs.

We replaced the final fully connected layer of AlexNet with a fully connected layer, which has a two-way output followed by the two-way softmax algorithm with a two-class label output (cancerous versus noncancerous). We calculated the loss function using cross entropy. The weights and biases of the replaced layers were initialized with random numbers. We set the initial learning rate $\alpha =0.0001$ for all the other layers, and $\alpha =0.002$ for the output layer to make the weights and biases from other layers almost unchanged while those from the output layer learn faster. We used the adaptive moment estimation (“Adam”) optimizer⁴⁷ for gradient descent. The following hyper-parameters were set as: ${\beta }_1=0.9$, ${\beta }_2=0.9999$, and $\epsilon ={10}^{-8}$.⁴⁷ Other training parameters were set as: $\text{minibatch size}=200$ and $\text{maximum epoch}=10$, which (including $\alpha$) were chosen using our tuning dataset.

3.5. Experimental Design and Evaluation Methods

3.5.1. Cross validation

We performed LOPO CV for each classifier to classify each ROI as cancerous versus noncancerous covering each WSI. In each CV, data were grouped on a per-patient basis, such that no same-patient samples were used in both the training and testing sets. During training, the positive (cancerous) and negative (noncancerous) samples were balanced by random subsampling of the negative samples. We performed testing on all ROIs covering each WSI in our 68-patient set (i.e., all tissue on all slides was classified; we did not use only selected ROIs in our experiments).

We calculated the error metrics (using a fixed operating point at the confidence level of 0.5) of error rate, FPR, false negative rate (FNR), AUC, and area under the precision recall curve (AU-PRC)⁴⁸ by validating against our expert manual annotations. For the AU-PRC, we also calculated the baseline as a reference, which reflects the classification results that would result from random guessing.⁴⁸ The overall performance was measured by averaging each of the error metrics across all the patients. The sample sizes for each tissue type are shown in Table 2. We also calculated the AUCs for each tissue type using each of the methods from LOPO CV.

Table 2.

Sample size for each tissue type ($480\mu \mathrm{m}\times 480\mu \mathrm{m}$).

Cancerous tissue types									Noncancerous tissue types
G3	G4	G5	EPE	$\mathrm{G}3+4$	$\mathrm{G}4+3$	$\mathrm{G}4+5$	$\mathrm{G}5+4$	$\mathrm{G}5+3$	Atrophy	PIN	Healthy
14,718	3949	37	272	6008	3839	727	8216	16	5433	26,449	1,178,814

We did statistical tests to compare the AUCs and AU-PRCs obtained from the different methods. We grouped AUC and AU-PRC values into three groups: group 1 (TCM-texture-Fisher, TCM-texture-LoglC, and TCM-texture-SVM); group 2 (tune-AlexNet-TCM); and group 3 (tune-AlexNet-RawIM). We tested the AUCs for normality using the Shapiro–Wilk test. We compared the AUCs for the three groups (i.e., group 1 versus group 2 versus group 3) using the Kruskal–Wallis test. We then compared each pair of groups using the Wilcoxon rank sum test. We did the same tests for the AU-PRCs.

We also performed a two-split validation on data from the two different scanners, where the WSIs from 43 patients were from scanner 1, and the WSIs from the remaining 25 patients were from scanner 2. We used tune-AlexNet-TCM and tune-AlexNet-RawIM to perform this experiment.

3.5.2. Training sample size experiment

We conducted an experiment to investigate how the number of patients in the training set influences the system performance for each machine learning approach. We randomly selected 34 patients as the testing set, then iteratively trained the classifiers using training set sizes ranging from 1 patient to 33 patients. At each iteration, we trained all of the classifiers described in Sec. 3.4, tested the trained systems on the entire 34-patient testing set, and computed error metrics as described in Sec. 3.5.1.

4. Results

4.1. Tissue Component Segmentation

The average adaptive threshold on the hematoxylin channel for WSIs within each patient ranged from 0.5 to 1.1. The standard deviation ranged from 0 to 0.4, with 42% of patients having a standard deviation $\ge 0.1$. The $\text{average}\pm \text{standard deviation}$ threshold for all patients was $0.7\pm 0.2$.

Qualitative results are shown in Fig. 4, where staining variability was observed for the two WSIs shown [Fig. 4(a)]. The hematoxylin channel thresholds are different for the samples after color deconvolution [Fig. 4(a), middle image for each case]. The nuclei were segmented by our adaptive thresholding method to generate nuclei maps [Fig. 4(a), right image for each case].

Figure 4 illustrates TCMs for the two WSIs right beside each case. Note the consistency of the maps despite the observed staining variability. Figures 4(b) and 4(c) show zoomed views for samples taken from the two cases, and their TCMs after segmentation. Also note that the confounding RBCs in the oval in Fig. 4(c) left and indicated by the yellow arrows in Fig. 4(c) right were correctly classified into the “stroma/other” category and not as nuclei.

4.2. Cancer versus Noncancer Classification

The quantitative results for cancer versus noncancer classification from our LOPO CV are reported in Table 3.

Table 3.

$\text{Average}\pm \text{standard deviation}$ of error metrics for cancer versus noncancer classification from LOPO CV.

	Error rate (%)	FNR (%)	FPR (%)	AUC [0, 1]	AU-PRC	Baseline
TCM-texture-FisherC	$13.7\pm 6.7$	$12.6\pm 16.9$	$13.6\pm 7.1$	$0.94\pm 0.05$	$0.36\pm 0.26$	$0.03\pm 0.26$
TCM-texture- LoglC	$12.3\pm 6.0$	$12.4\pm 15.6$	$13.6\pm 7.1$	$0.95\pm 0.05$	$0.38\pm 0.26$	$0.03\pm 0.26$
TCM-texture-SVM	$8.5\pm 4.6$	$13.1\pm 13.7$	$8.2\pm 4.6$	$0.96\pm 0.04$	$0.44\pm 0.25$	$0.03\pm 0.26$
Tune-AlexNet-TCM	$6.2\pm 4.0$	$10.7\pm 12.6$	$6.0\pm 4.2$	$0.98\pm 0.03$	$0.52\pm 0.24$	$0.03\pm 0.26$
Tune-AlexNet-RawIM	$5.5\pm 6.9$	$8.5\pm 13.8$	$5.2\pm 6.7$	$0.98\pm 0.02$	$0.52\pm 0.23$	$0.03\pm 0.26$

From the normality test, neither AUCs nor AU-PRCs were normally distributed. The Wilcoxon rank sum test results for each of the two groups are shown in Table 4. For AUCs, group 1 and 2, group 1 and 3, and group 2 and 3 were significantly different. For AU-PRCs, group 1 and 2, group 1 and 3 were significant. Group 2 and 3 were not significant.

Table 4.

Wilcoxon rank sum test results for each of the two groups.

Testing groups			$p$ values
AUC
Group 1	Group 2		$<\mathbf{0.0001}$
Group 1		Group 3	$<\mathbf{0.0001}$
	Group 2	Group 3	$<\mathbf{0.006}$
AU-PRC
Group 1	Group 2		$<\mathbf{0.0002}$
Group 1		Group 3	$<\mathbf{0.0004}$
	Group 2	Group 3	$<0.8903$

Note: $p$ values that are significant ($p<0.05$) are bolded. The group that had better performance within each pair is bolded.

The results from the two-split validation using data from two different scanners are reported in Table 5.

Table 5.

$\text{Average}\pm \text{standard deviation}$ of error metrics for cancer versus noncancer classification for WSIs from each scanner.

	Error rate (%)	FNR (%)	FPR (%)	AUC [0, 1]	AU-PRC	Baseline
Results for WSIs from scanner 1
Tune-AlexNet-TCM	$14.0\pm 6.4$	$5.1\pm 7.0$	$14.4\pm 7.0$	$0.97\pm 0.03$	$0.37\pm 0.20$	$0.03\pm 0.06$
Tune-AlexNet-RawIM	$3.7\pm 3.7$	$35.8\pm 25.9$	$2.3\pm 2.1$	$0.96\pm 0.04$	$0.39\pm 0.23$	$0.03\pm 0.06$
Results for WSIs from scanner 2
Tune-AlexNet-TCM	$7.8\pm 3.4$	$8.0\pm 8.1$	$7.8\pm 3.4$	$0.97\pm 0.02$	$0.50\pm 0.24$	$0.02\pm 0.02$
Tune-AlexNet-RawIM	$4.4\pm 2.6$	$6.4\pm 5.8$	$4.4\pm 2.6$	$0.99\pm 0.01$	$0.50\pm 0.18$	$0.02\pm 0.02$

From the LOPO CV experiments, we calculated the AUCs for cancerous and noncancerous tissue types. The results are shown in Table 6.

Table 6.

AUCs for cancer versus noncancer classification for each tissue type from the LOPO CV. The highest AUCs in each row are bolded in this table.

	TCM-texture-FisherC	TCM-texture-LoglC	TCM-texture-SVM	Tune-AlexNet-TCM	Tune-AlexNet-RawIM
AUCs for each cancerous tissue type
G3	$0.89\pm 0.07$	$0.90\pm 0.07$	$0.90\pm 0.07$	$0.95\pm 0.04$	$\mathbf{0.96}\pm \mathbf{0.03}$
G4	$0.93\pm 0.07$	$0.93\pm 0.07$	$0.92\pm 0.11$	$0.94\pm 0.12$	$\mathbf{0.95}\pm \mathbf{0.09}$
G5	$0.86\pm 0.00$	$0.85\pm 0.00$	$0.80\pm 0.00$	$\mathbf{0.88}\pm \mathbf{0.00}$	$0.87\pm 0.00$
$\mathrm{G}3+4$	$0.93\pm 0.07$	$0.93\pm 0.06$	$0.93\pm 0.07$	$0.97\pm 0.03$	$\mathbf{0.98}\pm \mathbf{0.03}$
$\mathrm{G}4+3$	$0.94\pm 0.05$	$0.94\pm 0.05$	$0.95\pm 0.05$	$\mathbf{0.97}\pm \mathbf{0.03}$	$\mathbf{0.97}\pm \mathbf{0.03}$
$\mathrm{G}4+5$	$0.88\pm 0.00$	$0.87\pm 0.00$	$0.84\pm 0.00$	$\mathbf{0.93}\pm \mathbf{0.00}$	$0.90\pm 0.00$
$\mathrm{G}5+4$	$0.92\pm 0.00$	$0.91\pm 0.00$	$0.87\pm 0.00$	$\mathbf{0.94}\pm \mathbf{0.00}$	$0.90\pm 0.00$
$\mathrm{G}5+3$	$\mathbf{1.00}\pm \mathbf{0.00}$	$\mathbf{1.00}\pm \mathbf{0.00}$	$0.99\pm 0.00$	$\mathbf{1.00}\pm \mathbf{0.00}$	$\mathbf{1.00}\pm \mathbf{0.00}$
EPE	$0.87\pm 0.06$	$0.82\pm 0.11$	$0.73\pm 0.22$	$\mathbf{0.86}\pm \mathbf{0.20}$	$\mathbf{0.86}\pm \mathbf{0.14}$
AUCs for each noncancerous tissue type
Atrophy	$0.97\pm 0.04$	$\mathbf{0.98}\pm \mathbf{0.03}$	$0.94\pm 0.05$	$0.96\pm 0.07$	$0.96\pm 0.06$
PIN	$0.80\pm 0.119$	$0.83\pm 0.11$	$0.84\pm 0.11$	$0.92\pm 0.07$	$\mathbf{0.93}\pm \mathbf{0.06}$
Healthy	$0.91\pm 0.06$	$0.92\pm 0.06$	$0.92\pm 0.05$	$0.96\pm 0.03$	$\mathbf{0.97}\pm \mathbf{0.03}$

Confusion matrices are shown in Table 7 for each classification method, using the fixed operating point corresponding to the confidence level of 0.5.

Table 7.

Confusion matrix for each method from LOPO CV; each sample is a $480\mu \mathrm{m}\times 480\mu \mathrm{m}$ ROI.

	System predicted label	Pathologist annotation
		Cancerous	Noncancerous
TCM-texture-FisherC	Cancerous	32,134	162,974
	Noncancerous	5647	1,047,722
TCM-texture-LoglC	Cancerous	31,385	145,563
	Noncancerous	6396	1,065,133
TCM-texture-SVM	Cancerous	30,260	99,523
	Noncancerous	7521	1,111,173
Tune-AlexNet-TCM	Cancerous	32,092	70,530
	Noncancerous	5689	1,140,166
Tune-AlexNet-RawIM	Cancerous	30,537	66,983
	Noncancerous	7244	1,143,713

Figure 7 shows the quantitative results from the training sample size experiment.

Fig. 7.

Plots for error metrics at each training patient number for cancer detection. Red and black dashed lines: reference points using 12 and 32 patients for training, respectively.

Figure 8 illustrates the capability of our system to map cancer throughout entire WSIs. The upper and lower cases in Fig. 8 depict average and below average performance of the methods (see error metrics below each map in Fig. 8 in comparison to Table 3) for LOPO CV using each of the three classifiers. Our implementation is not optimized for speed and consists primarily of MATLAB code.

Fig. 8.

Cancer maps for two example WSIs. (a), (e) Example WSIs; (b), (f) cancer maps from TCM-texture-SVM; (c), (g) cancer maps from tune-AlexNet-TCM; (d), (h) cancer maps from tune-AlexNet-RawIM. Color contours in each image are the pathologist’s annotations. The error metrics are below each cancer map. Labels in the cancer maps are: dark gray: true positives, light grey: true negatives, black: false positives, white: false negatives, and ER: error rate.

The non-deep learning methods require $\sim 45\min$ to map cancer throughout an entire WSI using an Intel i5 workstation at 3.10 GHz with 24 GB of random access memory. For the deep learning methods, the tune-AlexNet-TCM method requires 15 min, and the tune-AlexNet-RawIM method requires 3 min.

5. Discussion

In this paper, we proposed an approach to tissue component mapping that finds the loci of nuclei, luminal regions, and other tissue components (including stroma) based on our adaptive thresholding algorithm, which compensates for staining variability across the WSIs. This algorithm is amenable to fast implementation and yields consistent TCMs supporting cancer detection using machine learning algorithms. We validated conventional and deep learning-based approaches for classifying $480\mu \mathrm{m}\times 480\mu \mathrm{m}$ ROIs as cancer or noncancer throughout digitized mid-gland WSIs of RP sections. We did not subsample the tissue available in our data set. We cross validated using all of the tissue on every slide, ensuring that tissues from the same patient never appeared in both the training and testing sets. For CV, we used 286 WSIs from 68 patients including 1.3 million $480\mu \mathrm{m}\times 480\mu \mathrm{m}$ ROIs, which is $3588{\mathrm{cm}}^2$ of prostate tissue in total. To the best of our knowledge, this represents the largest validation data set presented thus far in the literature for this problem. All of the validated methods achieved similar or better performance with respect to comparable previously published approaches. The necessary processing times suggest the potential of an optimized, parallel implementation of the algorithms to yield processing speeds compatible with the clinical pathology workflow, upon further multicenter validation.

5.1. Tissue Component Mapping

The proposed nuclei segmentation method demonstrated robustness to staining variability. Even in our single-center data set where robotic staining systems were used, we observed substantial staining variability across and within patients. This was evidenced by the substantial WSI-to-WSI variability in the optimal thresholds on the hematoxylin channel for nuclei segmentation. This suggests that the concentration of hematoxylin stain for nuclei in WSIs from different patients varies substantially. This is illustrated qualitatively with two example WSIs that were stained very differently (Fig. 4) and the corresponding hematoxylin channel images [Fig. 4(a) middle image sets for both cases]. The hematoxylin channel intensity of the nuclei in the more lightly stained image [Fig. 4(a) right case] is similar to that of the stroma in the more darkly stained image [Fig. 4(a) left case]. Figure 4 shows the proposed method’s segmentation of nuclei from the background tissue despite the staining variation; we compare the loci and shapes of the nuclei in the example output label map images with the original stained images shown in Figs. 4(a) and 4(b). In addition, our method successfully assigned RBCs as stroma/other tissue [Fig. 4(c)] for the samples in Fig. 4. Although we did not evaluate the segmentation algorithm via quantitative measurement, validation results for cancer detection (Table 3) indirectly reflect the utility of the segmentation algorithm.

Compared to machine learning or normalization based approaches,^21,28,49 our proposed algorithm does not use other images for calibration/training. Therefore, by using this approach, differences between validation performance and performances for an unknown population could be reduced. In addition, this algorithm, whose implementation has not been optimized for speed, has a processing time of $45\min /\text{whole-mount}$ section using conventional machine learning and $15\min /\text{whole-mount}$ section using AlexNet on a standard workstation for cancer detection throughout entire slides. In clinical practice, it is important for the system to have the ability to process whole-mount sections within a reasonable time frame.

5.2. Cancer versus Noncancer Classification

5.2.1. Overall system evaluation

Our system yielded state-of-the-art overall performance (all metrics except for AU-PRC in Table 3) compared to literature-reported performance on this problem. Since our data set is highly imbalanced (0.037 million cancerous ROIs versus 1.2 million noncancerous ROIs), we also computed the AU-PRC to evaluate system performance. Considering we have a large number (1.2 million) of negative samples, our FPRs (Table 3) imply a large number of false positives in detection. However, the FPR values (Table 3) are small, which also indicates that the number of false positives is small relative to the total number of negatives. In addition, our FNRs (Table 3) imply high recall. More intuitively, in interpreting the confusion matrices (Table 7), it is important to note that the total number of negatives in our data set is far larger than the total number of positives. This leads to what appears to be, on first impression, a large number of false positive classifications. However, this concern is tempered when comparing the number of false positives to the generally much larger number of true negatives in each case (i.e., there is a large amount of negative tissue, and the classifiers are correctly identifying it as such most of the time). This is helpful to the clinical scenario where this system is providing assistance in reviewing slides to the pathologist since a high-recall system minimizes the chance that a cancerous region will be missed and improves pathologist efficiency by drawing attention to most of the cancer on each slide. Since all previous studies, in this area as we discussed in the previous work, did not use the AU-PRC metric, direct comparison of our work to previous studies using this metric is not possible. However, the system performance using this metric can be evaluated by comparing system AU-PRC to the baseline,⁴⁸ which reflects the expected AU-PRC arising from random guessing.⁴⁸ All of the methods yielded AU-PRCs much higher than the baselines, which indicates superior performance to random guessing.

This is the first study that used all tissues covering WSIs of whole-mount RP sections including all clinically relevant grade groups. This performance suggests the potential to use our proposed pipeline for cancer detection in a clinical setting after multicenter validation. Deep learning approaches overall outperformed the conventional machine learning-based approaches (AUCs and AU-PRCs in Table 3, and the statistical tests in Table 4). Also tune-AlexNet-RawIM yielded superior overall performance compared to tune-AlexNet-TCM using the AUC metric (Tables 3 and 4), but no significant difference was observed using AU-PRC metric (Tables 3 and 4). The significant differences given by the Wilcoxon rank sum test indicate that the AUC distributions are different, even though their means are similar. Tune-AlexNet-RawIM yielded higher AUCs for patients who do not have G5 cancer involved, whereas tune-AlexNet-TCM yielded higher AUCs for patients who have G5 cancer involved.

Our proposed pipeline using deep learning approaches was not substantially affected by the scanner types as the overall results from the two-split validation (AUCs in Table 5) were close to the LOPO CV overall results (AUCs in Table 3). Although testing on the data from scanner 1 yielded inferior performance compared to the LOPO CV using the AU-PRC metric (AU-PRC in Tables 3 and 5), we used a much smaller sample size for training (i.e., training used 25 patients from scanner 2). In particular, we have a much smaller sample size for each cancer tissue type in training (baseline for data from scanner 2 in Table 5).

All the methods were affected by tissue type and the corresponding sample size of each tissue type (Tables 2 and 6). Higher AUCs were often associated with larger sample sizes for the different tissue types, whereas lower AUCs were associated with smaller sample sizes, with one anomaly. Although $\mathrm{G}5+4$ has a relatively large sample size of 8216, its training sample size was small because most of the G5 cancer was concentrated in very few patients, and no tissue from the same patient was used in both training and testing.

The classifier performance for different tissue types varies according to the machine learning method used. We speculate this may relate to the different sample sizes of the different tissue types. Tune-AlexNet-RawIM and TCM-texture-SVM are the most sensitive to sample size, performing worse than the other methods for G5, $\mathrm{G}4+5$, $\mathrm{G}5+4$, and EPE, all of which having relatively small sample sizes (Tables 2 and 6; bottom case in Fig. 8). Because G5 is the most aggressive cancer, it is important to not miss any G5 involved tissue in clinical practice. In comparison, the performances of tune-AlexNet-TCM and TCM-texture-FisherC were the least sensitive to the smaller sample sizes of these tissue types. Despite the small sample size, tune-AlexNet-TCM demonstrated the best performance for these tissue types (Tables 2 and 6; bottom case in Fig. 8). This is reinforced by the observation that in the training sample size experiment, the FNRs of tune-AlexNet-RawIM and TCM-texture-SVM decreased much more with increasing sample sizes, compared to the other methods (Fig. 7, FNR metric). Tune-AlexNet-TCM and tune-AlexNet-RawIM showed larger fluctuations in FNR compared to the TCM-texture-based methods, when the training sample size is small (Fig. 7 FNR before red dashed line). We speculate that this may arise due to the inherent randomness associated with fine-tuning AlexNet. As the number of training patients increases, the amplitude of the fluctuation reduces (all error metrics in Fig. 7, especially FNR), suggesting that larger training sample sizes may increase the stability in performance of the tune-AlexNet methods. In addition, we found that the amplitude of the FNR fluctuations was larger than the amplitude of the FPR fluctuations. This may be due to the heterogeneity of PCa tissue, such that we have many different cancerous tissue types with relatively smaller samples sizes, compared to the larger samples of noncancerous tissue.

In the LOPO CV, we also found that, for PIN, the FPRs were smaller when using more complex models (tune-AlexNet-TCM and -RawIM) (Table 6). We speculate that this could be due to PIN’s resemblance to cancerous tissue, requiring more complex models or more image information (i.e., raw images or TCMs rather than 14 calculated texture features) to differentiate it from other cancerous tissue. From a clinical perspective, high-grade PIN is considered as a putative precursor lesion, as PIN shares features with cancer tissue.⁵⁰

5.2.2. Performance comparison by methods

Tune-AlexNet-RawIM yielded the highest AUC and AU-PRC in the LOPO CV and in the training sample size experiment, but it is sensitive to sample size. Although Shin et al.⁵¹ demonstrated efficient (i.e., smaller sample size) training by fine-tuning pretrained AlexNet and this method performs better than or equal to tune-AlexNet-TCM and the TCM-texture-based methods in general (all the error metrics in Fig. 7 and LOPO CV in Table 3), tune-AlexNet-RawIM yielded inferior performance in terms of FNR with training sets containing fewer than 12 patients (Fig. 7 after red dotted line). This may be because of the heterogeneity of cancerous tissue samples; 12 patient data sets may not have provided enough positive samples of each cancer tissue type with which to train the system. This is also reflected by the inferior performance of tune-AlexNet-RawIM on cancerous tissue types involving G5 (see AUCs for tissue types of G5, $\mathrm{G}4+5$, and $\mathrm{G}5+4$ in Table 6). This could be due to the fact that only two patients had G5 cancer in our data set, combined with the fact that we performed LOPO CV. To illustrate, note the large portion of $\mathrm{G}5+4$ cancerous regions that were missed by tune-AlexNet-RawIM [white regions in Fig. 8(h)]. This is of particular concern because missing G5 cancer could result in failure to apply adjuvant therapy after surgery or could increase the pathologist’s necessary editing time to correct the cancer maps. Although tune-AlexNet-RawIM had an error rate of 0% on $\mathrm{G}5+3$ using a large training sample size of G5 from tissues of G5, $\mathrm{G}4+4$, $\mathrm{G}4+5$, the sample size being tested was too small (9 ROIs from one patient) to be considered representative.

In comparison, tune-AlexNet-TCM yielded more stable performance in the training sample size experiment and demonstrated superior performance on tissue types with smaller sample sizes. With fewer than 12 training patients, tune-AlexNet-TCM yielded a lower FNR than tune-AlexNet-RawIM in nearly every case (Fig. 7 before the red dotted line). In LOPO CV, even though the samples of G5 involved tissue types were restricted to two patients, the highest AUCs were achieved by tune-AlexNet-TCM (Table 6). More intuitively, for example, in Fig. 8(g), we can see most of the cancerous regions of $\mathrm{G}4+5$ and $\mathrm{G}5+4$ were captured by tune-AlexNet-TCM. This suggests that higher-order tissue features (e.g., TCMs) can enhance the performance of tune-AlexNet when sample size is small. Also comparing the results between using tune-AlexNet-TCM and tune-AlexNet-RawIM in CVs (Table 3), the performance differences are negligible. Considering the huge dimensionality reduction from the raw image (227 pixels × 227 pixels × 3 color channels × 256 intensities per channel) to TCM (227 pixels × 227 pixels × 3 tissue component labels), the computed TCM can effectively reduce the dimensionality and make salient the key visual cues for the cancer detection problem.

Comparable results between using TCM-texture-SVM and tune-AlexNet-TCM and -RawIM in CVs (Table 3) suggest that extracting features from TCMs can effectively reduce the high-dimensional image information to 14 TCM-based texture features, and the resulting feature set is appropriate for our problem. This could support better understanding of the key visual cues for our problem. For TCM-texture-based methods, the SVM was more sensitive to sample size than FisherC and LoglC because (1) its performance monotonically and substantially improved as the training sample size increased and exceeded the performance of the FisherC and LoglC classifiers with a training sample size of 33 in the training sample size experiment (Fig. 7 after the black dotted line) and (2) higher FNRs and FPRs were found for tissue types having smaller sample sizes (G5 involved tissue types, excepting PIN as discussed previously) in LOPO CV (Tables 2 and 6).

Finally, to compare our method to more recently published methods by Campanella et al.,³⁴ our work is complementary to theirs, such that our classifiers map the locations of cancer throughout the slides with tile-level classifications, on RP slides. This enables quantification of tumor sizes and locations, and coupled to future work automating the segmentation of the surgical margin and prostatic capsule, would enable the quantification of negatively prognostic factors such as extraprostatic extension and positive surgical margins. An interesting avenue of future work fusing these two approaches would be to evaluate the accuracy of the weak labels on each tile from the method described by Campanella et al. against our manual annotations, to discover the accuracy of their method for whole-slide cancer mapping.

6. Limitations

Our results should be interpreted in the context of several limitations of our study. All of our tissues were processed in the same clinical pathology laboratory and the cancerous annotations were done by one physician and verified by one of the two pathologists. These aspects of our study limit the variability of the material (i.e., the tissues and resulting images) and observers’ contours. Although our adaptive thresholding algorithm demonstrated the capability of compensating for staining variability, the performance of this algorithm needs to be further tested on an external data set, in which the WSIs are processed and collected differently. However, this algorithm is based on finding an inflection point in the plot of the number of connected components versus the threshold. In theory, this inflection point may not always exist, although this situation did not happen in any of our cases. This inflection point does not exist in the extreme case where the background tissue is so darkly stained by hematoxylin that it is similar in appearance to the nuclei. However, this situation can be detected by our algorithm because the solution of our function set does not exist. Also validations were conducted at the $480\mu \mathrm{m}\times 480\mu \mathrm{m}$ ROI level with regions containing more than 50% cancer considered as positive. All the conventional machine learning-based approaches used feature selection on an independent tuning data set, such that the feature may be overfit to the tuning data set. Those factors may lead to suboptimal performance of the system. Future work may include using classifiers that do not require feature selection (e.g., L1 or L2 regularized linear classifiers, random forest, and gradient boosting algorithms). We only used one type of deep learning approach with pretrained AlexNet; there are many deep learning architectures (e.g., ResNet and GoogleNet) demonstrating superior performance to AlexNet in other image classification tasks, and those methods may yield better performance for our problem and could be important avenues of future work. Although we used an adaptive threshold for nuclei segmentation, it was a global threshold for each WSI. Although our informal experiments (not reported here) suggested that locally adaptive thresholding did not improve performance, such an approach could be straightforwardly adapted from our methods if needed. Finally, it must be acknowledged that all CV studies may be subject to positive bias in their results; validation using an external data set is required to support clinical translation of this tool.

7. Conclusion

In conclusion, our proposed methods demonstrated state-of-the-art performance for automatically mapping PCa on whole-mount WSIs, with validation on the largest high-precision pathologist-annotated data reported thus far. Transfer learning with pretrained AlexNet demonstrated performance superior to the tested conventional machine learning approaches. Compared to using raw image ROIs as inputs, the use of TCMs yielded similar overall performance and more stable performance across different tissue types with various sample sizes, and substantially superior performance for the G5, the most aggressive cancer tissue type. Upon successful multicenter validation, this system could support imaging validation studies using annotated histopathology as the gold standard and could also facilitate quantitative and graphical pathology reporting after RP, which has the potential to support better prognosis, recurrence risk management, and adjuvant therapy planning.

Biographies

Wenchao Han received his BSc degree in optical and electronic information from Huazhong University of Science and Technology in 2010 and his MEng degree in communication systems and data networking from Western University in 2014. He is a PhD candidate in the Department of Medical Biophysics, Western University, Canada. He is also a research assistant at Baines Imaging Laboratory, London Regional Cancer Program, Canada. He is a member of SPIE.

Carol Johnson received her BSc degree in physics from the University of Toronto. She is a research associate at Baines Imaging Laboratory, London Regional Cancer Program, Canada.

Andrew Warner received his MSc degree in epidemiology and biostatistics from the Western University. He is a research associate in the Department of Radiation Oncology, London Regional Cancer Program, Canada.

Mena Gaed received his bachelor of medicine and bachelor of surgery degrees from Cairo University and his MSc degree in pathology and laboratory medicine from the Western University, Canada. He is a research associate in the Department of Pathology and Laboratory Medicine at the Western University.

Jose A. Gomez received his MD from the Pontifical Xavierian University, Colombia. He is an associate professor in the Department of Pathology and Laboratory Medicine at the Western University. He is also an anatomical pathologist specialized in the genitourinary tract, head and neck, and breast at London Health Sciences Center.

Madeleine Moussa received her MBBCh degree from Ain Shams University, Egypt, and her master’s degree in anatomy from Queen’s University in 1991. She is a professor in the Department of Pathology and Laboratory Medicine at the Western University. She is also a genitourinary pathologist at London Health Sciences Center.

Joseph Chin received his MD from the University of Toronto and did an urologic oncology fellowship at Roswell Park Cancer Institute. He is a professor in the Departments of Urology and Oncology at the Western University, London, Ontario, Canada. He is a urologic oncologist at London Health Sciences Center and is active in clinical trials and translational research. He has published 250 research papers and book chapters.

Stephen Pautler received his MD from the University of Calgary in 1995. He completed his urology residency at the Western University. He is an associate professor in the Departments of Surgery and Oncology at the Western University. He is a surgeon at St. Joseph’s Health Care, London, Canada.

Glenn Bauman is a radiation oncologist and a professor in the Departments of Oncology and Medical Biophysics, Western University, London, Ontario, Canada, specializing in genitourinary and central nervous system malignancies. His primary areas of research interest are multimodality image guided radiotherapy and cancer imaging.

Aaron Ward received his PhD in computing science from Simon Fraser University, Canada. He is an associate professor in the Departments of Medical Biophysics and Oncology, Western University, Canada. He is also a scientist at Lawson Health Research Institute, Canada. His research interests focus on computer-assisted diagnosis and therapy guidance based on medical imaging and digital pathology, with applications in oncology.

Disclosures

No conflicts of interest, financial or other potential conflicts of interest, are declared by the authors.