Deep Volumetric Segmentation of Murine Cochlear Compartments from Micro-Computed Tomography Images

Abstract

Local drug delivery to the inner ear via micropump implants has the potential to be much more effective than oral drug delivery for treating patients with sensorineural hearing loss and to protect hearing from ototoxic insult due to noise exposure or cancer treatments. Designing micropumps to deliver appropriate concentrations of drugs to the necessary cochlear compartments is of paramount importance; however, directly measuring local drug concentrations over time throughout the cochlea is not possible. Recent approaches for indirectly quantifying local drug concentrations in animal models capture a series of magnetic resonance (MR) or micro computed tomography (μCT) images before and after infusion of a contrast agent into the cochlea. These approaches require accurately segmenting important cochlear components (scala tympani (ST), scala media (SM) and scala vestibuli (SV)) in each scan and ensuring that they are registered longitudinally across scans. In this paper, we focus on segmenting cochlear compartments from μCT volumes using V-Net, a convolutional neural network (CNN) architecture for 3-D segmentation. We show that by modifying the V-Net architecture to decrease the numbers of encoder and decoder blocks and to use dilated convolutions enables extracting local estimates of drug concentration that are comparable to those extracted using atlas-based segmentation (3.37%, 4.81%, and 19.65% average relative error in ST, SM, and SV), but in a fraction of the time. We also test the feasibility of training our network on a larger MRI dataset, and then using transfer learning to perform segmentation on a smaller number of μCT volumes, which would enable this technique to be used in the future to characterize drug delivery in the cochlea of larger mammals.

I. Introduction

Among the 466 million people affected by hearing loss [1], up to 90% of the cases are sensorineural, meaning that they are related to the inner ear/cochlea or the vestibulocochlear nerve. Future methods for treating sensorineural hearing loss involve delivering drugs to the inner ear either orally or via injection. A key challenge with these methods are non-therapeutic concentrations within the cochlea due to poor permeability of the blood-labyrinth barrier for the case of systemic injections or low concentrations due to limitations of diffusion through the round window membrane.

Emerging inner ear therapeutics for treating sensorineural hearing loss and genetic disorders, and for providing protection, can overcome these limitations by site-directed delivery via micropumps [2], [3]. Traditional techniques to quantify drug delivered by these micropumps involve infusing the rodent cochlea with compounds and measuring the concentrations throughout cochlear compartments. The measurements can be carried out directly (e.g., with ion-selective micro-electrodes [4] or perilymph sampling [5][6]), but direct measurement techniques are susceptible to cerebrospinal fluid (CSF) contamination and yield results that have limited spatio-temporal resolution.

Auditory measures based on distortion product otoacoustic emissions (DPOAE) and auditory brainstem response (ABR) provide indirect approximations of concentration along the length of the cochlea, but they fail to yield any information on relative concentrations within the major scala (ST, SM, SV). MRI resolution (1.5 to 2 mm) is insufficient for imaging small cochlear compartments such as in the mouse, where scala length can range from 5 to 7 mm. In contrast, μCT provides sufficient spatial resolution with direct measurement of concentration of an infused Iodine-based contrast agent (Fig. 1) within all cochlear compartments, enabling quantification of local drug delivery at micron resolution [7], [8].

Fig. 1.

Top-left: Baseline cochlear scan (no contrast agent), followed by scans at 10, 40, 70, 100, and 130 minutes after infusion with iodine-based contrast agent.

To measure drug/contrast agent concentration from different compartments of the cochlea, it is essential to perform accurate segmentation. Previous works have used atlas-based segmentation to identify the scala tympani (ST), scala media (SM), and scala vestibuli (SV) across the series of cochlear scans [7], [8]. Although atlas-based segmentation techniques can have good performance, they rely on computationally expensive deformable registration techniques that can require minutes to hours to run on a CPU.

An alternative to atlas-based segmentation that avoids deformable registration of an atlas to each new animal model is to employ modern segmentation techniques based on convolutional neural networks (CNNs) [9], [10], [11]. CNNs for segmentation typically use a fully convolutional encoder-decoder architecture [9]. Importantly, due to the generalization ability of CNNs [12], the encoder portion of a learned CNN can be used as a general feature extractor [13] and can be subsequently fine-tuned to perform segmentation on a different but related dataset (for example, cochlear scans of larger rodents).

In this work, we propose a CNN architecture for segmenting murine cochlear compartments from μCT images, based on simplifications of a V-Net [14], which has been shown to be highly effective for segmenting the prostate in MRI scans to isolate its region boundaries for radiotherapy [14], segmenting the brain into its individual brain structures [15], and separating nodules from CT scans of the lung [16]. We train and test our simplified V-Net on subsets of a dataset comprising 137 μCT images of the cochlea of 21 mice, and we compare training from scratch versus fine-tuning a network trained on the brain tumor segmentation BraTS ‘18 dataset [17], [18], [19]. We then show that the simplified V-Net can be used to extract contrast agent concentrations that are similar to those extracted following an atlas-based segmentation procedure, but in just a fraction of the time.

II. Methods

This section describes our dataset of μCT scans of the murine cochlea, our proposed simplified V-Net architecture for segmenting cochlear compartments from these scans, and our methods for training our simplified V-Net.

A. Dataset

Our dataset consists of μCT scans of the heads of 21 mice [8] scanned on a Siemens Inveon PET/CT scanner. Each μCT scan has resolution 16 × 16 × 16μm, which is high enough to resolve the ST, SM, and SV compartments of the cochlea. For each mouse, a baseline scan is captured while continuous infusion of artificial perilymph (e.g., Fig. 3), and then a contrast agent is infused into the left cochlea at a constant flow rate through a cochleostomy that is drilled at the base of ST. The artifical perilymph and contrast agent are separated by a 10 nl air bubble to prevent mixing in the infusion tube as well as to monitor the flow rate. After infusion, the mouse is scanned from 4 to 6 times [8]. The basal region of the left cochlea becomes saturated with the contrast agent after the baseline scan, making compartment boundaries indiscernible. This yields one volume useful for segmentation from the left cochlea, and four to six volumes from the right cochlea. For our dataset of 21 mice, we have a total of 137 volumes.

Fig. 3.

Left: 2-D slice from μCT volume of the left mouse cochlea at baseline. Right: corresponding slice of reference masks of ST, SM, and SV.

Reference masks of ST, SM, and SV are constructed for each cochlear volume by automatic B-spline deformable registration [20] of a cochlear atlas [21] using Insight Toolkit (ITK) [22] followed by visual verification to eliminate any obvious errors in the masks. In all cases, normalized image correlation after deformable registration was −0.85±0.05 (best = −1). In some cases, extra tissue and bone was included in the periphery of the cochlea, preventing the registration metric from reaching −1.

B. Network Architecture

The V-Net [14] is a CNN architecture that was designed for segmenting volumes, and it has shown good performance on datasets with less than 100 samples. V-Net is comprised of an encoding and a decoding branch. The output is collected after the decoding branch by applying soft-max to a convolutional layer that has a kernel size of 1 and has 4 output channels (i.e., three channels for each segmentation mask and one channel for background). We use a simplified version of V-Net, as shown in Fig. 2. The pooling operations for downsampling between blocks are replaced by strided convolutions to improve the memory footprint of the network. Additionally, we include batch normalization layers after every convolutional layer in order to avoid over-fitting.

Fig. 2.

Simplified V-Net architecture. Block 1 and 9: single convolution layer. Block 2 and 8: 2 convolution layer, rest of the blocks have 3 convolution layers. Red dotted lines represent connections that transfer information from encoder to decoder so that location information lost during encoding can be retrieved.

To further improve segmentation performance on the cochlear μCT scans, we explore the following modifications to V-Net, which are illustrated in Fig. 4:

• Network 1: We downsample the input by a factor of two using trilinear interpolation before feeding it to the network. In addition, we remove the fourth of the five stages of the encoder network to ensure that the dimensions of the input are not reduced by too much. This ensures that the network sees more of the volume at every convolution step and reduces the memory required for processing. The output convolution layer is upsampled to the original volume size before passing it to the soft-max layer.

• Network 2: At every encoding and decoding stage in V-Net, the convolutional layers are changed to have kernel size of three, with dilations of one, two, and three in the first, second and third convolutional layers. This idea was previously proposed for 2D dense segmentation networks to increase the receptive field of the kernels [23], and it yielded significant performance improvement in that setting. Although V-Net already has a theoretical receptive field of 551 × 551 × 551 at the output, our modification makes the receptive fields larger in the initial layers of the network, enabling the network to learn better features.

• Network 3: Most fully convolutional networks have millions of parameters and require a GPU to evaluate in a reasonable time. To ensure that our V-Net can run in real time on a CPU in a clinical setting, we reduce the number of channels in the convolutional layers by a factor of four, which reduces the number of trainable parameters from 43.7 million to 1.05 million.

Fig. 4.

Network 1: Downsampling input and upsample the output, with removal of a block to ensure proper encoded input size. Network 2: Dilated convolutions at every block. Network 3: Decreased number of convolutions, incidentally decreasing the number of channels in each block.

C. Network Training

All networks were implemented and trained using the Pytorch framework. The dataset was divided into 70% training, 20% validation, and 10% testing on a per-mouse basis. Care was taken to maintain distribution of left and right cochlea among the split. All volumes in the dataset were scaled to a uniform size of 128 × 128 × 128 voxels. The dataset was augmented to randomly include elastic transformations [24] with elasticity coefficient, σ = 6, and intensity of deformation, α, chosen randomly between 0 (no transformation) and 50. Due to possible class imbalance, the generalized Dice metric [25] was used for the loss function.

We performed training on a workstation with 64GB of system memory, Intel(R) Core(TM) i7–7700K 4.2 GHz CPU, and a NVIDIA Titan Xp having 12 GB of memory. We used Adaptive moment estimation (Adam) [26] for optimizing the network after every mini-batch, and we used a batch size of 2 due to limited data and GPU memory constraints. The training was started with a learning rate of 5e⁻⁴, and tuned between 1e⁻³ to 1e⁻⁵ for different networks. The weight decay term was adjusted to 0.1×learning rate. Each model was trained for approximately 10 hours for 250 epochs. Test volumes were segmented from trained models in less than two seconds on GPU. And in the case of Network 3 with significantly reduced number of parameters, each volume was segmented within a reasonable time of 9.4±0.52 seconds on just the CPU.

We initialized our training in two ways: (1) from scratch, and (2) by fine-tuning weights learned by training the stock VNet on BraTS 2019 T1 weighted contrast enhanced scans. Although the BraTS dataset is meant for tumor segmentation, the general features learnt by the encoder may be useful for segmenting cochlear compartments. We transferred the complete BraTS pre-trained network as it is, fine tuned by training for less than quarter of the time (50 epochs) on the cochlea dataset, and predicted the segmentations on the test set.

III. Results

Fig. 5 shows 2-D and 3-D visualizations of segmentation results as compared to reference masks. As can be seen in the figure, there are some minor segmentation errors. The narrow section as seen inside the first oval is not complete; this might be due to the section disappearing during encoder downsampling, resulting in the network not learning features of that particular section of the scan. The second oval shows an area where region boundaries are smoother than what is expected versus the reference masks.

Fig. 5.

Reference masks vs. predicted segments, ST, SM and SV in cyan, yellow and maroon respectively. Effect of network downsampling on fine structures and smoother region boundaries predicted by V-Net can be seen in red ovals 1 and 2 respectively.

For each network, performance was evaluated using Jaccard index (Intersection Over Union - IOU) on the validation set, which contained 14 volumes. Results are shown in Table I. To prove the feasibility of transfer learning using this learned model, the same network was trained using BraTS ‘18 dataset, and fine tuned on cochlea dataset. The training time was reduced by 75%, but performance was reduced to 70.64% (IOU).

Table I:

Performance metrics for V-Net: Stock, Network 1 (Net 1), Network 2 (Net 2) and Network 3 (Net 3) in terms of Jaccard Index averaged over all segments (in %) and dice loss

Architecture	Avg. Jaccard Index	Final Dice Loss
Stock	79.345%	0.1709
Net 1	81.5%	0.1471
Net 2	83.382%	0.1482
Net 3	76.486%	0.2069

It is important to note that the reference segmentations were constructed by visually verifying the results of atlas-based segmentation, rather than by manually labeling every cochlear compartment in every volume. In this sense, it can be argued that our V-Net is actually learning how to mimic the performance of atlas-based segmentation. We argue, however, that this approach still has merit if: (a) segmentation of new (unseen) scans can be performed much more rapidly, and (b) concentrations measured from the scalae segmented by our proposed networks are consistent with the concentrations measured from the scalae segmented by atlas-based segmentation methods.

Since test volumes can be segmented through our V-Net architectures in less than two seconds (on a GPU), and atlas-based segmentation requires minutes to hours of effort (often with user intervention) to perform the necessary deformable registration step, our V-Net segmentation strategy is certainly much more rapid than atlas-based segmentation. Having observed only minor translations between baseline and other timepoint scans, we transfer baseline CNN segmentation using fast rigid registration.

To compare concentrations, consider Fig. 6, which illustrates an example of concentrations extracted from scala tympani using atlas based segmentation versus using V-Net based segmentation. Concentrations are obtained from a seven-voxel diameter window around the medial axis of the segment. Note the dip in concentration seen in scan 2 due to the air bubble introduced during infusion as explained in section II-A. There are no significant visual differences in the plots. The average relative error in ST, SM, and SV between the concentration curves generated using atlas-based segmentation versus V-Net segmentation is 3.37%, 4.81%, and 19.65%, respectively. The average is calculated over all the timepoints. In terms of absolute concentration difference, these are equal to 1.02 mg/ml, 0.91 mg/ml and 1.1mg/ml in ST, SM and SV. Concentration throughout the length of SV is lower than 50 mg/ml, hence small change in concentration gives large relative error. However, we can see from absolute concentration values in Fig. 6 that these correspond to small change in concentration. These amounts of error over all scans have no significant effect on further processing steps.

Fig. 6.

Concentration from ST at 10, 40, 70, 100, 130 mins. Top: From atlas based method. Bottom: From CNNs. x-axis represents the length of ST from base to apex in percentage and y-axis represents the concentration of the contrast agent in mg/ml.

The mean relative error between atlas-based segmentation versus V-Net segmentation remains low in all the mice can be seen in the errorbar plot in Fig. 7. The lower concentration of the contrast agent due to it diffusing slowly towards the apex in the first 20 minutes of the experiment, may be the cause of the impulse error values seen in the errorbar of scan 2.

Fig. 7.

Errorbar representing relative error between atlas-based segmentation vs. V-Net segmentation. x-axis represents the length of ST from base to apex in percentage and y-axis represents the relative error averaged over all scans in percentage.

IV. Conclusions and Future Work

In this paper, we showed that a V-Net style architecture can be used to segment μCT scans of murine cochlear compartments in a manner that performs similarly to atlas-based segmentation but within seconds as opposed to minutes-to-hours. Concentrations of a contrast agent measured in each segment were similar. As opposed to stock V-Nets, modified networks with dilated convolutions are seen to yield the best performance. The low absolute error concentration values between atlas-based method implies that V-Net based method can be adopted for segmentation of cochlear regions. Future work will involve exploring the use of V-Net style architectures on cochlear scans from larger rodent models, including guinea pigs.