Deep Recurrent Neural Networks for Prostate Cancer Detection: Analysis of Temporal Enhanced Ultrasound

Abstract

Temporal enhanced ultrasound (TeUS), comprising the analysis of variations in backscattered signals from a tissue over a sequence of ultrasound frames, has been previously proposed as a new paradigm for tissue characterization. In this paper, we propose to use deep recurrent neural networks (RNN) to explicitly model the temporal information in TeUS. By investigating several RNN models, we demonstrate that long short-term memory (LSTM) networks achieve the highest accuracy in separating cancer from benign tissue in the prostate. We also present algorithms for in-depth analysis of LSTM networks. Our in vivo study includes data from 255 prostate biopsy cores of 157 patients. We achieve area under the curve, sensitivity, specificity, and accuracy of 0.96, 0.76, 0.98, and 0.93, respectively. Our result suggests that temporal modeling of TeUS using RNN can significantly improve cancer detection accuracy over previously presented works.

I. Introduction

Prostate cancer (PCa) is amongst the most commonly diagnosed forms of cancer in men. The American Cancer Society estimates 161,360 new cases to be diagnosed in 2017 in the United States. Early stage PCa detection followed by treatment results in a five-year survival rate of above 95% [1]. Core needle biopsy, the current gold standard for PCa detection, is performed under transrectal ultrasound (TRUS) guidance where ultrasound is used for anatomical navigation rather than for targeting PCa.

In the past four decades, ultrasound (US) imaging has been increasingly used for tissue characterization and non-invasive detection of disease. Ultrasound-based tissue typing has been investigated extensively to enable targeted biopsy [2]. Most of the research in this field has focused on the analysis of texture and spectral features [3] of a single ultrasound image. Additionally, elastography [4] and Doppler imaging [5] have been explored to determine changes in tissue stiffness and blood flow associated with PCa, respectively. Fusion of TRUS with multi-parametric MRI (mp-MRI) has also been used for PCa detection [6]–[8], where lesions identified on mp-MRI are registered to TRUS images and used for targeted biopsy.

Temporal Enhanced Ultrasound (TeUS), comprising a time series of ultrasound images, has been reported as an alternative tissue characterization technique (Fig. 1) [2], [9], [10]. TeUS utilizes an underlying machine learning framework to extract information from the backscattered ultrasound data obtained in a time span of few seconds without any imposed mechanical excitation. We demonstrated that ultrasound-induced thermal effects [11] and tissue micro-vibrations [12] influence the backscattered ultrasound data. Furthermore, we independently analyzed the effects of tissue elasticity and micro-vibration [12], [13], and presented an analytical model linking elasticity to TeUS measurements, which showed that TeUS signature is both dependent on the scattering function of the medium and the inverse of tissue elasticity. Since 2007, TeUS has been successfully used for characterization of prostate cancer in ex vivo [2], [14] and in vivo [15]–[19] studies. Comparison of the TeUS framework with the analysis of a single Radio Frequency (RF) ultrasound frame indicates that the two approaches are complementary [2], [16], with TeUS showing higher overall area under the curve (AUC).

Fig. 1.

A schematic diagram of Temporal Enhanced Ultrasound (TeUS) data generation. A time series of a fixed point in an image frame, shown as a red dot, is analyzed over a sequence of ultrasound frames to determine tissue characteristics [38].

Previously, manually engineered feature representations extracted from RF TeUS data [16], [20], [21] have been used with shallow discriminant models such as Support Vector Machines (SVMs) [2], Random Forest [22], [23], Bayesian classifiers [24] and multi-layer feed-forward perceptron networks [14], to differentiate tissue types [25]. Moradi et al. [14], [24] used fractal dimension with Bayesian classifiers and feed-forward neural networks for ex vivo characterization of prostate tissue. More recently, Imani et al. [15], [16] combined extracted features from Discrete Fourier and Wavelet transforms of TeUS data and fractal dimension to characterize in vivo prostate tissue using SVMs. Ghavidel et al. [23] used spectral features of TeUS data and performed feature selection using Random Forests to classify lower grade prostate cancer from higher cancer grades.

Deep neural networks [26], [27] have been recently established as a powerful machine learning paradigm. Our group has applied deep networks for automatic feature selection from TeUS spectrum [17], [18] to address the so-called “cherry picking” of the features [21]. This framework exploited Deep Belief Networks (DBN) [28] to automatically learn a high-level latent feature representation from TeUS data. We subsequently extended our work from detecting PCa to its grading [29]. We also examined Convolutional Neural Networks (CNNs) to combine temporal and spatial information from TeUS data to detect high-risk prostate cancer. Additionally, we utilized deep-learning-based features to examine the physical phenomenon governing TeUS [12], [17], [30]. These features pointed to the presence of a strong backscattered ultrasound signal in frequency range of 0 − 2 Hz. This observation led to a hypothesis that tissue micro-vibration, possibly due to pulsation in major blood vessels surrounding the prostate, at the heartbeat frequency (∼ 1.2 Hz) is a key contributor to TeUS. To-date, our efforts have mainly focused on spectral analysis as the key pre-processing step for feature extraction from TeUS data [17]. Most recently, analysis of TeUS in the temporal domain using probabilistic methods, such as Hidden Markov Models (HMMs), have shown significant promise [31], [32].

In this paper, we propose to use Recurrent Neural Networks (RNNs) [33]–[35] to explicitly analyze TeUS data in temporal domain. Specifically, we use Long Short-Term Memory (LSTM) networks [33] and Gated Recurrent Units (GRUs) [36], [37], the classes of RNNs, to effectively learn long-term dependencies in the data. In an in vivo study with 157 patients, we analyze data from 255 suspicious cancer foci obtained during MRI-TRUS fusion biopsy. We achieve AUC, sensitivity, specificity and accuracy of 0.96, 0.76, 0.98, and 0.93, respectively. Our results indicate that RNN can identify temporal patterns in the data that may not be readily observable in spectral analysis of TeUS [17], leading to significant improvements in detection of PCa.

II. Background: Recurrent Neural Networks

RNNs are a category of neural networks that are “deep” in temporal dimension and have been used extensively in time-sequence modeling [39]. Unlike a conventional neural network, RNNs are able to process sequential data points through a recurrent hidden state whose activation at each step depends on that of a previous step. Generally, given sequence data x = (x₁,, …, x_T), an RNN updates its recurrent hidden state h_t by:

$$h_t=\{\begin{array}{ll}0,\hfill & ift=0\hfill \\ \phi (h_{t-1},x_t),\hfill & otherwise\hfill \end{array}$$ (1)

where x_t and h_t are data values and the recurrent hidden state at time step t, respectively, and φ(.) represents the nonlinear activation function of a hidden layer, such as a sigmoid or hyperbolic tangent. Optionally, the RNN may have an output y = (y₁,, …, y_T). In the traditional RNN model aka vanilla, the update rule of the recurrent hidden state in (1) is implemented as:

$$h_t=\phi (\mathbf{W}{x}_t+\mathbf{U}{h}_{t-1}),$$ (2)

where W and U are the coefficient matrices of the input at the present step and the recurrent hidden units activation at the previous step, respectively. We can further expand Equation (2) to calculate the hidden vector sequence h = (h₁,, …, h_T):

$$h_t=\phi ({\mathbf{W}}_{ih}{x}_t+{\mathbf{W}}_{hh}{h}_{t-1}+b_h),$$ (3)

where t = 1 to T, W_ih denotes the input-hidden weight vector, W_hh represents the weight matrix of the hidden layer, and b_h is the hidden layer bias vector.

It has been observed that using the traditional RNN implementation, gradients decrease significantly for deeper temporal models. This is referred to as the vanishing gradient, and makes learning of long-term sequence data a challenging task for RNNs. To address it, other types of recurrent hidden units such as LSTM and GRU have been proposed. As shown in Equations (2) and (3), traditional RNN simply applies a transformation to a weighted linear sum of inputs. In contrast, an LSTM-based recurrent layer creates a memory cell c at each time step whose activation is computed as:

$$h_t=o_t\phi (c_t),$$ (4)

where o_t is the output gate that determines the portion of the memory cell content in time step t (c_t) to be exposed at the next time step [33]. The recursive equation for updating o_t is:

$$o_t=\sigma ({\mathbf{W}}_{oi}{x}_t+{\mathbf{W}}_{oh}{h}_{t-1}+{\mathbf{W}}_{oc}{c}_{t-1}+b_o),$$ (5)

where σ(.) is the logistic sigmoid function, W_oi is the input-output weight matrix, W_oh is the hidden layer-output weight matrix, and W_oc is the memory-output weight matrix. The memory cell, c_t, is updated by adding new content, ${\overline{c}}_t$, and discarding part of the present memory:

$$c_t={\mathbf{i}}_t\odot {\overline{c}}_t+{\mathbf{f}}_t\odot c_{t-1},$$ (6)

where ⨀ is an element-wise multiplication and ${\overline{c}}_t$ is calculated as:

$${\overline{c}}_t=\phi ({\mathbf{W}}_{ci}{x}_t+{\mathbf{W}}_{ch}{h}_{t-1}+b_c),$$ (7)

In this equation, the W term represents weight matrices; e.g., W_ci is the input-memory weight matrix. Input gate i, and forget gate f determine the degree that new information is to be added and current information is to be removed, respectively, as follows:

$${\mathbf{i}}_t=\sigma ({\mathbf{W}}_{ix}{x}_t+{\mathbf{W}}_{ih}{h}_{t-1}+{\mathbf{W}}_{ic}{c}_{t-1}+b_i);$$ (8)

$${\mathbf{f}}_t=\sigma ({\mathbf{W}}_{fx}{x}_t+{\mathbf{W}}_{fh}{h}_{t-1}+{\mathbf{W}}_{fc}{c}_{t-1}+b_f).$$ (9)

All weight matrices, W, and biases, b, are free parameters that are shared between cells across time. Figure 2 shows a graphical model of an LSTM cell. A slightly different version of LSTMs are GRUs [36] which have fewer number of parameters to avoid over-fitting in the lack of sufficient training samples. GRUs combine the forget and input gates into a single update gate, u, and merge the cell memory and hidden state to a reset gate, r. The activation of h_t of the GRU at time t is a linear interpolation between the previous activation, h_t−1, and the updated activation, ${\overline{h}}_t$:

$$h_t=(1-u_t)h_{t-1}+{\overline{h}}_t{u}_t,$$ (10)

where u_t, the update gate at time step t, determines how much the unit updates its activation or content. The update gate can be calculated as follows:

$$u_t=\sigma ({\mathbf{W}}_{ui}{x}_t+{\mathbf{W}}_{uh}{h}_{t-1}),$$ (11)

where W_ui is the input-update weight matrix and W_uh denotes the update-hidden weight matrix. The updated activation, ${\overline{h}}_t$, is computed similarly to the traditional RNN in Equation (2) as follows:

$${\overline{h}}_t=\phi ({\mathbf{W}}_{pi}{x}_t+{\mathbf{W}}_{rh}(r_t\odot h_{t-1})).$$ (12)

Finally, the reset gate, r_t, is computed as:

$$u_t=\sigma ({\mathbf{W}}_{ri}{x}_t+{\mathbf{W}}_{rh}{h}_{t-1}).$$ (13)

Fig. 2.

Overview of the proposed method. We use two layers of RNNs with LSTM cells to model the temporal information in a sequence of TeUS data. x⁽ⁱ⁾ = (x₁,, …,x_T), T = 100 indicates the i^th sequence data, while x_t refers to the t^th time step.

III. Materials

A. TeUS Data Acquisition and Histopathology Labeling

TeUS data was acquired from 157 subjects during fusion prostate biopsy. All subjects provided informed consent to participate, and the study was approved by the institutional research ethics board. Data from 255 biopsy targets of the subjects, that were identified as suspicious for cancer in preoperative mp-MRI examination, were used. Through a consensus, two radiologists assigned an overall MR suspicious level score of “low”, “moderate” and “high” to each target [6]. These scores are based on findings on each mp-MRI sequence using previously described criteria [6], which indicate both the presence of prostate cancer and tumor grade. The standardized PI-RADS criteria were not used for this study. Subsequently, subjects underwent MRI-guided ultrasound biopsies using UroNav (Invivo Corp., FL) MR-US fusion system [40]. Prior to biopsy sampling from each target, the ultrasound transducer was held steady freehand for 5 seconds to obtain T = 100 frames of beam-formed RF data (i.e., prior to envelope detection and compression). For this purpose, an endocavity curvilinear probe (Philips C9–5ec) with frequency of 6.6 MHz was used. This procedure was followed by firing the biopsy gun to acquire a tissue sample. Histopathology information of each biopsy core is used as the gold-standard for generating a label for that core. Histopathology reports include the length of cancer in the biopsy core and a Gleason Score (GS) [41]. The GS is reported as a summation of the Gleason grades of the two most common cancer patterns in the tissue specimen. Gleason grades range from 1 (resembling normal tissue) to 5 (aggressive cancerous tissue).

In our dataset, 83 biopsy cores are cancerous with GS 3+3 or higher, where 31 of those are labeled as clinically significant cancer with GS ≥ 4+3. The remaining 172 cores are non-cancerous and include benign or fibromuscular tissue, chronic inflammation, atrophy and Prostatic Intraepithelial Neoplasia (PIN) [29].

B. Preprocessing and Data Augmentation

The biopsy needle path is mechanically constrained by the biopsy guide attached to the ultrasound transducer. For each biopsy target, we analyze an area of 2 mm × 10 mm around the target location, along the projected needle path (Fig. 3). We divide this region to 80 equally-sized Regions of Interest (ROIs) of 0.5 mm ×0.5 mm. For each ROI, we generate a sequence of TeUS data, ${\mathbf{x}}^{(i)}=(x_1^{(i)},\dots ,x_T^{(i)})$, T = 100 by averaging over all time series values within a given ROI of an ultrasound frame. We augment the training data by creating ROIs using a sliding window of size 0.5 mm × 0.5 mm over the target region, which results in 1,536 ROIs per target (see Fig. 3). The number of averaged time series within an ROI depends on the axial and lateral resolution of the system at the ROI location.

Fig. 3.

Preprocessing and ROI selection: the target region is divided to 80 ROIs of size 0.5 mm × 0.5 mm and then a sliding window is used for the data augmentation.

IV. Method

A. Proposed Discriminative Method

Our overarching objective is to develop a deep learning model to discriminate cancer and benign prostate regions in TeUS data. Let $\mathcal{D}={\{({\mathbf{x}}^{(i)},{\mathbf{y}}^{(i)})\}}_{i=1}^{|\mathcal{D}|}$ represent a collection of all labeled ROIs, where x⁽ⁱ⁾ is the i^th TeUS sequence and y⁽ⁱ⁾ indicates the corresponding label. An individual TeUS sequence of length T, ${\mathbf{x}}^{(i)}=(x_1^{(i)},\dots ,x_T^{(i)})$, is composed of signal-amplitude values $x_t^{(i)}$ for each time step, t, and is labeled as yⁱ ∈ {0, 1}, where zero and one indicate benign and cancer outcome, respectively in histopathology (see Fig. 1). We aim to learn a mapping from x⁽ⁱ⁾ to y⁽ⁱ⁾ in a supervised framework by using RNNs to explicitly model the temporal information in TeUS. Our sequence classification approach is built with connected RNN layers followed by a softmax layer to map the sequence to a posterior over classes. Each RNN layer includes T = 100 homogeneous hidden units (i.e., traditional/vanilla RNN, LSTM or GRU cells) to capture temporal changes in TeUS data. The model learns a distribution over classes P(y|x₁,, …, x_T) given a time-series sequence x₁,, …, x_T rather than a single, time independent input. Figure 2 shows an overview of the proposed architecture with LSTM cells.

Given the input sequence x = (x₁,, …, x_T), RNN computes the hidden vector sequence h = (h₁,, …, h_T) in the sequence learning step. As discussed in Section II, h is a function of the input sequence x, model parameters, Θ, and time, t: φ(x; Θ, t). $\mathrm{\Theta }=\{\mathcal{W},\mathcal{B}\}$ denotes the sequence learning model parameters, where $\mathcal{W}$ is the set of weights and $\mathcal{B}$ is the set of biases in Eq. (3) for vanilla RNN cells, in Eq. (4) for LSTM cells, and in Eq. (10) for GRU cells through time steps, t = 0 to t = T. All weight matrices, $\mathcal{W}$, and biases, $\mathcal{B}$, are free parameters that are shared across time. The final node generates the posterior probability for the given sequence:

$${\mathbf{z}}^{(i)}={\mathbf{w}}_s^T\mathbf{h}+b_s;$$ (14)

$${\overline{\mathbf{y}}}^{(i)}\equiv P({\mathbf{y}}_j^{(i)}\mid \mathbf{x})=\mathcal{S}({\mathbf{z}}_j^{(i)})=\frac{e^{{\mathbf{z}}_j^{(i)}}}{e^{{\mathbf{z}}_0^{(i)}+e^{{\mathbf{z}}_1^{(i)}}}},j\in \{0,1\},$$ (15)

where $\mathcal{S}$ is the softmax function, which in our binary classification case is equivalent to the logistic function, and ${\overline{\mathbf{y}}}^{(i)}$ indicates the predicted label. The optimization criterion for the network is to maximize the probability of the training labels or equivalently, to minimize the negative log-likelihood defined as a the loss function. This function is the binary cross-entropy between y⁽ⁱ⁾ and ${\overline{\mathbf{y}}}^{(i)}$ over all training samples, ${\mathcal{D}}_{train}={\{({\mathbf{x}}^{(i)},{\mathbf{y}}^{(i)})\}}_{i=1}^N$:

$$\mathcal{L}(\overline{\mathbf{y}},\mathbf{y})=-\frac{1}{N}\sum _{i=1}^N\left[{\mathbf{y}}^{(i)}\log {\overline{\mathbf{y}}}^{(i)}+(1-{\mathbf{y}}^{(i)})\log (1-{\overline{\mathbf{y}}}^{(i)})\right],$$ (16)

where $N=\left|{\mathcal{D}}_{train}\right|$. During training, the loss function is minimized through a proper gradient optimization algorithm like stochastic gradient descent (SGD), root mean square propagation (RMSprop) or adaptive moment estimation (Adam) [42].

B. Cancer Classification

The RNN models learn a probability distribution over classes, P(y|x₁,, …, x_T), given a time-series sequence, x₁,, …, x_T. Let $\mathcal{C}={\{({\mathbf{x}}^{(i)},{\mathbf{y}}^{(i)})\}}_{i=1}^{|\mathcal{C}|}$ represent the collection of all labeled ROIs surrounding a target core, where $\mathcal{C}\in {\mathcal{D}}_{test}$, $|\mathcal{C}|=80$, x⁽ⁱ⁾ represents the i^th TeUS sequence of the core, and y⁽ⁱ⁾ indicates the corresponding label. Using the probability output of the classifier for each ROI, we assign a binary label to each target core. The label is calculated using a majority vote based on the predicted labels of all ROIs surrounding the target. For this purpose, we define the predicted label for each ROI, ${\overline{y}}^{(i)}$, as 1, when P(y⁽ⁱ⁾|x⁽ⁱ⁾) ≥ 0.5, and as 0 otherwise. The probability of a given core being cancerous based on the cancerous ROIs within that core is:

$$P_{\mathcal{C}}=\frac{{\sum }_i^{\left|\mathcal{C}\right|}I({\overline{y}}^{(i)}=1)}{\left|\mathcal{C}\right|}.$$ (17)

A binary label of 1 is assigned to a core, when $P_{\mathcal{C}}\ge 0.5$.

C. Network Analysis

To better understand the temporal information in TeUS, we examine the LSTM gates. For this purpose, following training, we use the learned weights and biases to regenerate the network behavior for any given sequence of length T. First, the state of each cell is set to zero. Then, the full learning formula (Eqs. (4)–(9)) along with the model parameters $\mathrm{\Theta }=\{\mathcal{W},\mathcal{B}\}$ are recursively applied for T = 100 time steps. A summary of the steps is presented in Algorithms 1 and 2. Finally, the on-and-off behavior of the hidden activation in the last layer of the network is used to analyze the high level learned features.

Algorithm 1.

Examination of the LSTM Gates

Input: Trained model parameters “$\mathrm{\Theta }=\{\mathcal{W},\mathcal{B}\}$”, input data “X ”, number of time-steps “T”, number of input sequence “N”.

Output: States activation “$\mathcal{S}$”, gates activation “$\mathcal{G}$”

Initialization: Set the state of each cell “inStates” to zero.

1: for i = 0 to N do

2:   for t = 0 to T do

3:    x ←X(i, :)

4:    $\{\mathcal{S}(i,t),\mathcal{G}(i,t)\}\leftarrow \mathrm{Step}(x,\mathit{\text{inStates}}(i,t),\mathcal{W},\mathcal{B})$

5:    $inState(i,t)\leftarrow \mathcal{S}(i,t)$

6:   end for

7: end for

8: return $\mathcal{S}$, $\mathcal{G}$

Algorithm 2.

Recurrent Step Function of the LSTM

Input: Trained model parameters “$\mathrm{\Theta }=\{\mathcal{W},\mathcal{B}\}$”, input sequence “x”, input states of time step (t − 1) “$\mathcal{S}$”.

Output: States activation of the current time step (t) “${\mathcal{S}}_t$”, gates activation of the current time step (t) “${\mathcal{G}}_t$”

1: procedure $\mathrm{Step}(x,{\mathcal{S}}_{t-1},\mathcal{W},\mathcal{B})$

2: Woi, Woh, Woc, Wci, Wch, Wix, Wih, Wf x, Wf h, $W_{\mathit{fc}}\leftarrow \mathcal{W}$

3: bo, bc, bi, $b_f\leftarrow \mathcal{B}$

4: ht−1, $c_{t-1}\leftarrow {\mathcal{S}}_{t-1}$

5: it ← σ(Wix x + Wihht−1 + Wicct−1 + bi)

6: ft ← σ(Wf x x + Wf hht−1 + Wf cct−1 + b f)

7: ot ← σ(Woi x + Wohht−1 + Wocct−1 + bo)

8: $\overline{c}\leftarrow \phi (W_{ci}x+W_{ch}{h}_{t-1}+b_c)$

9: $c_t\leftarrow {\mathbf{i}}_t\odot {\overline{c}}_t+f_t\odot c_{t-1}$

10: ht ← otφ(ct)

11: ${\mathcal{S}}_t\leftarrow \{h_t,c_t\}$

12: ${\mathcal{G}}_t\leftarrow \{i_t,f_t,o_t,c_t\}$

13: return ${\mathcal{S}}_t$, ${\mathcal{G}}_t$

V. Experiments

A. Data Division

Data is divided into mutually exclusive training, ${\mathcal{D}}_{train}$, and test sets, ${\mathcal{D}}_{test}$. Training data is made up of 84 cores with the following histopathology label distribution: benign: 42 cores; GS 3+3: 2; GS 3+4: 14; GS 4+3: 3; GS 4+4: 18; and, GS 4+5: 5 cores. Training cores were selected from those with homogeneous tissue regions (See [38] for more details). Therefore, the training data are selected from biopsy cores with at least 4.0 mm of cancer in a typical core length of 18.0 mm, 26 of which are labeled as clinically significant cancer with GS ≥ 4+3. Benign cores are randomly selected from all available non-cancerous cores.

The test data consists of 171 cores, where 130 cores are labeled as benign, 29 cores with GS ≤ 3+4, and 12 cores with GS ≥ 4+3. Given the data augmentation strategy in Section III-B (Fig. 3), we obtain a total number of 84 × 1,536 = 129, 024 training samples $(N=\left|{\mathcal{D}}_{train}\right|=129,024)$.

B. Hyper-Parameter Selection and Network Structure

The performance of deep RNNs, similar to other deep learning approaches, are affected by their hyper-parameters. In practice, hyper-parameter selection can be constrained as a generalization-error minimization problem. Solutions are often based on running trials with different hyper-parameter settings, and choosing the setting that results in the best performing model (Fig. 4). We optimize the hyper-parameters through a grid search, which is an exhaustive search through a prespecified subset of the hyper-parameter space of the learning algorithm.

Fig. 4.

Comparison between optimizer performance for different RNN cells: Each curve corresponds to an RNN network structure with two hidden layers, batch size of 128 with dropout rate of 0.2 and regularization term of 0.0001. (a) LSTM. (b) GRU. (c) Vanilla RNN.

The grid search starts with randomly partitioning the selected training dataset, ${\mathcal{D}}_{train}$, into training (80%) denoted by ${\mathcal{D}}_{tr}$ and held-out validation sets (20%) denoted by ${\mathcal{D}}_{val}$. This partitioning results in $N_{tr}=\left|{\mathcal{D}}_{tr}\right|=103,219$ training samples and $N_{val}=\left|{\mathcal{D}}_{val}\right|=25,805$ held-out validation samples. To guide the grid search algorithm, we track the loss, accuracy and AUC on both ${\mathcal{D}}_{tr}$ and ${\mathcal{D}}_{val}$. Loss is defined using Eq. (16) as the binary cross-entropy between the predicted label and the true label, while accuracy is the percentage of the correctly predicted labels. To stabilize learning and prevent the model from over-fitting on the training data, we use regularization and dropout, two of the most effective proposed strategies [43]. Regularization adds a penalty term to the loss function (Equation (16)) to prevent the coefficients from getting too large. Here, we use L2 regularization in the form of $\lambda \Vert \omega {\Vert }_2^2$, where we search for λ as the hyper-parameter. Dropout prevents co-adaptations on training data. In each step of training, a dropout layer removes some units of its previous layer from the network, which means the network architecture changes in every training step. These units are chosen randomly based on the probability parameter of the dropout layer as another hyper-parameter. We perform a grid search over the number of RNN hidden layers, n_h ∈ {1, 2}, batch size, b_s ∈ {64, 128}, and initial learning rate, lr ∈ {0.01 − 0.0001} with three different optimization algorithms, SGD, RMSprop and Adam [42]. We also experiment with various levels of dropout rate, d_r ∈ {0.2, 0.4} and L2-regularization term (λ), l_reg ∈ {0.0001, 0.0002}. These result in 96 different hyper-parameter settings for the proposed approach. All models are trained with the same number of iterations and training is stopped after 100 epochs. Models benefit from reducing the learning rate by a factor once learning stagnates [43]. For this purpose, we monitor the validation loss and if no improvement is observed over 10 epochs, the learning rate is reduced by l_rnew = l_r × factor, where factor = 0.9.

C. Model Training and Evaluation

Once the optimum hyper-parameters are identified, the entire training set, ${\mathcal{D}}_{train}$, is used to learn the final model. Loss is used as the performance measure for early stopping to avoid over fitting. Training is stopped if the loss as we defined in Eq. (16) increases or if it does not decrease after 10 epochs. An absolute change of less than δ = 0.0004 is considered as no improvement in loss. We also record the model performance for ${\mathcal{D}}_{val}$ to track its behavior in a random subset of training data.

To assess the performance of our method, we report its sensitivity, specificity, and accuracy in detecting cancerous tissue samples in the test data, ${\mathcal{D}}_{test}$. All cancerous target cores are considered as the positive class (labeled as 1), and non-cancerous cores as the negative class (labeled as 0). Sensitivity or recall is defined as the percentage of cancerous cores that are correctly identified, while specificity is the proportion of non-cancerous cores that are correctly classified. Accuracy is the ratio of the true results (both true positives and true negatives) over the total number of cores. The overall performance of the models are reported using AUC. The curve depicts a relative trade-off between sensitivity and specificity. The maximum value for AUC is 1, where higher values indicate better classification performance.

D. Implementation

We implement the RNNs in Keras [44] using the Tensorflow [45] back-end. Training is performed on a GeForce GTX 980 Ti GPU with 6 GB of memory, hosted by a machine running Ubuntu 16.04 operating system on a 3.4 GHz Intel Core™ i7 CPU with 16 GB of memory. Training of vanilla RNN, LSTM and GRU network structures with 100 epochs takes 1.1, 8.1 and 3.3 hours, respectively. Early stopping and calculation of additional performance metrics are implemented using Keras callbacks and the Tensorflow back-end to evaluate internal states and statistics of the model during training. The proposed network analysis method in Algorithms 1 and 2, is implemented independently of Keras or Tensorflow in Python 2.7, executed on CPU.

VI. Results and Discussion

A. Model Selection

Results from hyper-parameter search demonstrate that network structures with two RNN hidden layers outperform other architectures. Furthermore, for the vanilla RNN, b_s = 128, lr = 0.0001; for LSTM, b_s = 64, lr = 0.0001; and for GRU, b_s = 128, lr = 0.01 generate the optimum models. For all models, d_r = 0.2 and l_r eg = 0.0001 generate the lowest loss and the highest accuracy for both ${\mathcal{D}}_{tr}$ and ${\mathcal{D}}_{val}$. The learning curves for different optimization algorithms and initial learning rates on ${\mathcal{D}}_{tr}$ are shown in Fig. 4. Each curve corresponds to an RNN network structure with two hidden layers, the batch size of 128 with the dropout rate of 0.4 and regularization term of λ = 0.0001, where the vertical axis is the loss and the horizontal axis is the number of iterations. It is clear that RMSprop substantially outperforms SGD optimization for all of the RNN cell types while RMSprop and Adam have similar performance for GRU and LSTM cells. RMSprop leads to a better performance on our data.

Learning curves of the different RNN cells using the optimum hyper-parameters are shown in Fig. 5. The right-vertical axis represents the loss value while the left-vertical axis shows the accuracy and AUC, and the horizontal axis is the number of iterations. We observe that all models converge after 65±7 epochs, and GRU and LSTM cells outperform vanilla RNN cells in terms of accuracy. Comparing Fig. 5(a) and (b) demonstrates that the network with GRU cells has a steeper learning curve and converges faster than the network with LSTM cells. One possible reason could be fewer number of parameters to be learned in GRU cells compared to LSTM cells. Fig. 5 shows that the network with LSTM cells leads to a lower loss value and a higher accuracy.

Fig. 5.

Learning curves of different RNN cells using the optimum hyper-parameters in our search space. All of the models use the RMSprop optimizer and converge after 65 ± 7 epochs. (a) LSTM. (b) GRU. (c) Vanilla RNN.

B. Model Performance

Table I shows the classification results in the test dataset, ${\mathcal{D}}_{test}$, including 171 target cores. Models with LSTM and GRU cells consistently achieve higher performance compared to vanilla RNN and the spectral method proposed in [17]. A paired t-test shows statistically significant improvement in AUC (p < 0.05) with LSTM and GRU cells. Moreover, the LSTM configuration has the highest performance for detection of cancer. Using the LSTM model as the best configuration, we achieve specificity and sensitivity of 0.98 and 0.76, respectively, where we classify 31 out of 41 cancerous cores correctly. Table II shows performance of models for classification of cores in ${\mathcal{D}}_{test}$ for different MR suspicious levels as explained in Section III. For samples with moderate MR suspicious level (70% of all cores), we achieve AUC of 0.97 using the LSTM-RNN structure. In this group, our sensitivity, specificity, and accuracy are 0.78, 0.98, and 0.95, respectively. For samples with high MR suspicious level, we consistently achieve higher sensitivity result compared to those with moderate MR suspicious level.

TABLE I.

Model Performance for Classification of Cores in the Test Data (N = 171)

Method	Specificity	Sensitivity	Accuracy	AUC
LSTM	0.98	0.76	0.93	0.96
GRU	0.95	0.70	0.86	0.92
Vanilla RNN	0.72	0.69	0.75	0.76
Spectral [17]	0.73	0.63	0.78	0.76

TABLE II.

Model Performance for Classification of Cores in the Test Data for Different MR Suspicious Levels. N Indicates Number of Cores in Each Group

MR suspicious levels	Moderate MR suspicious level (N = 115)				High MR suspicious level (N = 20)
	Specificity	Sensitivity	Accuracy	AUC	Specificity	Sensitivity	Accuracy	AUC
LSTM	0.98	0.78	0.95	0.97	0.86	0.92	0.90	0.97
GRU	0.95	0.70	0.89	0.91	0.80	1.00	0.90	0.92
Vanilla RNN	0.82	0.70	0.82	0.73	0.85	0.67	0.70	0.68
Spectral [17]	0.70	0.78	0.80	0.80	0.83	0.90	0.85	0.95

C. Comparison with Other Methods

The best results to-date involving TeUS are based on spectral analysis as proposed in [17]. We compare our RNN-based approach with this method as the most related work. As reported in Table I and Table II, the LSTM-RNN models consistently outperform spectral analysis of TeUS with a paired t-test showing statistically significant improvement in sensitivity, specificity, accuracy, and AUC (p < 0.05) for LSTM-RNN. Datasets used in a few other studies are different from the current work so a detailed comparison is not feasible. However, the proposed network architectures outperform Imani et al. [46], who used cascade CNNs, and Uniyal et al. [22], who applied random forests for TeUS based PCa detection, both on data from 14 patients. While analysis of a single RF ultrasound frame is not possible in the context of our current clinical study, previously, we have shown that TeUS is complementary to this information and generally outperforms analysis of single RF frames [2], [11]. The best results reported using a single RF frame analysis [47] involve 64 subjects with an AUC of 0.84, where they used PSA as an additional surrogate. In a recent study, Nahlawi et al. [31], [48] used Hidden Markov Models (HMMs) to model the temporal information of TeUS data for prostate cancer detection. In a limited clinical study including 14 subjects, they identified cancerous regions with an accuracy of 0.85.

Furthermore, we qualitatively compare the cancer likelihood colormap resulting from LSTM-RNN with the spectral analysis of TeUS [17]. Figure 6 shows two examples of the cancer likelihood maps from the test dataset. There is an observable improvement in identifying cancerous regions around the biopsy target that match the gold-standard label.

Fig. 6.

Cancer likelihood maps overlaid on B-mode ultrasound images, along the projected needle path in the TeUS data, and centered on the target. Red indicates predicted labels as cancer, and blue indicates predicted benign regions. The boundary of the segmented prostate in MRI is overlaid on TRUS data. The arrow points to the target location. The top row shows the result of LSTM and the bottom row shows the result of spectral analysis [17] for benign targets (a), and cancer targets (b) and (c).

D. Network Analysis

To analyze the network behavior and identify LSTM cells that contribute most to differentiating between benign and cancerous tissue, we examine the final high-level feature representation for cancerous and benign samples. By generating the difference map between the final activation of the network (h_t at t = 100) for TeUS data from benign and cancerous samples, we identify 20 cells with the highest activation difference. This refers to cells whose activation level change the most between benign and cancer samples.

In these cells, we observe the evolution of high-level feature representations. Specifically, as per Algorithm 1, input TeUS data from benign and cancerous ROIs in ${\mathcal{D}}_{test}$ are forward propagated in the models. Activation of the input gate (i(t)), the output gate (o(t)) and the cell state (c(t)) for the top 20 active cells are studied.

We observe cell states c(t) evolve over time and gradually learn discriminative information. Moreover, the input gate i(t) evolves so that it attenuates parts of the input TeUS sequence and detects the important information from the sequence. Interestingly, the input gate reduces the contribution of TeUS data to model learning around time step 50, for both cancerous and benign samples. The evolution and attenuation patterns of c(t) and i(t) suggest that the most discriminative features for distinguishing cancerous and benign tissue samples are captured within the first half of TeUS sequence. These finding match those reported by Nahlawi et al. [31], [48].

To further examine this, we evaluate the evolving behavior of LSTM-RNN by training and testing the RNN models with different TeUS sequence lengths. Figure 7 shows the performance of the models evaluated by AUC for different TeUS sequence lengths. For each case, using the training procedure explained in Section V-C, we trained an RNN-based deep network with 10–100 RNN cells corresponding to TeUS length. Similar to previous observations, the vanilla RNN-based model has the lowest performance compared to GRU and LSTM based models. By increasing the length of input TeUS sequence, the performance of the models improve. However, for TeUS sequence length more than 50, the improvement saturates. Using a paired t-test, the DeLong test, and bootstrapping, we demonstrate that for sequence length of over 50, there is no statistically significant improvement in performance using the LSTM-RNN model (p > 0.05).

Fig. 7.

Sequence length effect.

VII. Conclusion

In this paper, we present an accurate approach for detecting PCa from TeUS data collected during MRI-TRUS guided biopsy. We utilize deep RNNs to explicitly model the temporal information of TeUS. Our investigation of several RNN structures shows that LSTM-based RNN can efficiently capture temporal patterns in TeUS data with statistically significant improvement in accuracy over our previously proposed spectral analysis approach [17]. In a large clinical study including 255 suspicious cancer foci obtained from 157 patients, we achieve AUC, sensitivity, specificity and accuracy of 0.96, 0.76, 0.98, and 0.93, respectively. We also presented algorithms for in-depth analysis of high-level latent features of LSTM-based RNN. A transformational finding, achieved through this analysis, is that the most discriminative features for detection of PCa can be learned from a fraction of the full TeUS time series. Specifically, in our data, less than 50 ultrasound frames were required to build models that accurately detect PCa. This information can be used to optimize TeUS data acquisition for clinical translation [38], [49]. Cancer likelihood maps can be used to guide systematic biopsies and potentially increase the yield of high grade cancer. For fusion biopsies, the map can help compensate for small mis-registrations between MRI and ultrasound. Eventually, the approach can enable monitoring of active surveillance patients by detecting changes in prostate tissue over time.