Deep learning based automatic segmentation of the placenta and uterine cavity on prenatal MR images

Abstract

Magnetic resonance imaging (MRI) has potential benefits in understanding fetal and placental complications in pregnancy. An accurate segmentation of the uterine cavity and placenta can help facilitate fast and automated analyses of placenta accreta spectrum and other pregnancy complications. In this study, we trained a deep neural network for fully automatic segmentation of the uterine cavity and placenta from MR images of pregnant women with and without placental abnormalities. The two datasets were axial MRI data of 241 pregnant women, among whom, 101 patients also had sagittal MRI data. Our trained model was able to perform fully automatic 3D segmentation of MR image volumes and achieved an average Dice similarity coefficient (DSC) of 92% for uterine cavity and of 82% for placenta on the sagittal dataset and an average DSC of 87% for uterine cavity and of 82% for placenta on the axial dataset. Use of our automatic segmentation method is the first step in designing an analytics tool for to assess the risk of pregnant women with placenta accreta spectrum.

INTRODUCTION

During pregnancy, the human placenta serves as the lifeline of the fetus, providing endocrine, metabolic, and immune functions during fetal development [1]. It has a major role in fetal and maternal health. Proper placental development can have significant impact on future wellbeing [2]. Abnormal placenta growth and development results in adverse pregnancy complications, such as preeclampsia, fetal growth restriction, and stillbirth [3]. A growing placental condition is placenta accreta spectrum (PAS), an abnormal invasion of the placenta into the uterus, which can result in hysterectomy, multisystem organ failure, and death due to the placenta not detaching from the uterus following delivery [4]. The current standard for assessing the placenta is 2D ultrasound, which requires the physician to create 3D images in their mind and suffers from several limitations including approximation of the shape of the placenta and localizing features. An addition of 3D magnetic resonance imaging (MRI) may provide improved soft tissue contrast and wider field of view needed for placental investigation [5]. Studies have shown that MRI can highlight features such as intraplacental bands which correspond to abnormal placentation [6,7].

Various segmentation methods have been developed for the placenta. Shahedi et al. proposed a minimally interactive semi-automatic segmentation tool which combined deep learning and region growing to segment the uterine cavity and placenta in MR images [10]. Shahedi et al. further improved their network and performed fully automatic segmentation of the uterine cavity and placenta in MR images using a 3D U-Net model [8,9]. Han et al. utilized the U-Net for automatic placenta segmentation and modified it to be thinner and less computationally expensive using separable convolution [14]. Looney et al. developed a two-pathway, hybrid convolutional neural network model using transfer learning to automatically segment the placenta [15]. Hu et al. applied a novel layer into the segmentation U-Net which was weighted by automated acoustic shadow detection to recognize artifacts specific to ultrasound [16]. Alansary et al. addressed the issue of motion corrupted prenatal MRI segmentation via a 3D multi-scale convolutional neural network and a conditional random field method [17]. Wang et al. developed a semi-automatic placenta segmentation tool called Slice-Seg, which used slice-by-slice propagation to provide a minimally interactive segmentation of the placenta from sparse and motion-corrupted prenatal MRI [11]. Recently, Wang et al. proposed another deep learning-based semi-automatic segmentation method called DeepIGeoS, which combined automatic segmentations with user interactions through geodesic distance transforms to further refine segmentation results [18]. Zhang et al. designed a multi-region saliency-aware learning method for cross-domain ex vivo placenta image segmentation, which guides the translation between domains by enforcing attention and saliency consistency constraints [19]. Most recently, Zimmer et al. proposed a multi-task learning approach that combines the classification of placental location and semantic placenta segmentation in a fully convolutional neural network for ultrasound to achieve a high-quality segmentation with reduced image artifacts [20].

This study is a continuation of our previous work in devising a fully automatic segmentation platform for delineating the placenta and uterine cavity in pregnant women using MRI. In previous work, we demonstrated the advantages of end-to-end deep neural networks for placenta and uterus segmentation over semi-automatic methods and computerized algorithms [8–12]. Deep learning brings the advantages of speed of computation time and improved segmentation accuracy [13]. In this study, our objective is to present a fully automatic deep learning segmentation model for fast, accurate, and repeatable 3D segmentation of the placenta and uterine cavity in sagittal and axial T2-weighted MRI of pregnant women with and without placental complications. We used a 3D U-Net 3+ architecture to implement our end-to-end prediction model and achieved an average Dice similarity coefficient (DSC) of 95.3% for uterine cavity and of 87.7% for placenta on the sagittal dataset and an average DSC of 91.8% for uterine cavity and of 82.7% for placenta on the axial dataset. A clinical obstetrics application of the automatic segmentation method can provide a powerful analytical tool to assess the risk of pregnant women with placenta accreta spectrum.

METHODS

Data

Our dataset consisted of 101 normal women with 7-mm axial placental acquisitions as well as 5-mm sagittal acquisitions and 143 women with suspected PAS, all who had 7-mm axial acquisitions. The MR images were taken in an interleaved fashion. In total there were 244 axial datasets. Each axial image volume contained between 28 and 62 2D axial slices spaced 7 mm apart. The slices were sized at 256 × 256 pixels except for three cases with smaller sizes. The physical sizes of the image pixels in the axial dataset ranged from 1.051 × 1.051 mm² to 1.953 × 1.953 mm². Each sagittal volume had 27–80 2D sagittal slices scanned with the patient lying on her side, with each slice sized at 256 × 256 pixels. Corresponding physical sizes of the image pixels ranged from 1.172 × 1.172 mm² to 1.758 × 1.758 mm². For both datasets, manually segmented labels of the uterine cavity and the placenta were provided by an expert radiologist.

Data preparation and preprocessing

To prepare the data, cases in the axial dataset with MR image slice sizes of less than 256 × 256 pixels were first zero-padded across the volume to be 256 × 256 pixels. Following this, the data preparation and preprocessing steps for the sagittal and axial datasets were the same. The MR image volumes and labels were cropped along the imaging axis based on the manual segmentations to contain only those slices containing uterine cavity. This step narrowed the minimum and maximum number of MR image volume slices in the sagittal dataset from (26 to 80) to (24 to 61). Similarly, the range of the slices in the axial dataset was reduced from (28 to 62) to (24 to 61). A 2D median filter with a kernel size of 3 × 3 was applied to each image slice to reduce image noise. Additionally, intensity-level slicing and normalization were applied to the MR image volumes according to the following equation:

$${\widehat{I}}_i(x,y,z)=\{\begin{array}{c}0\mathit{\text{if}}{I}_i(x,y,z)<p_5\left(I_i\right)\\ 1\mathit{\text{if}}{I}_i(x,y,z)>p_{99}\left(I_i\right)\\ \frac{I_i(x,y,z)-p_5\left(I_i\right)}{p_{99}\left(I_i\right)-p_5\left(I_i\right)}\mathit{\text{otherwise}}\end{array}$$

where I_i(x, y, z) represents the original intensity of a given voxel in an image volume, ${\widehat{I}}_i(x,y,z)$ represents the new intensity of the voxel, and p₅(I_i) and p₉₉(I_i) represent the 5^th and 99^th intensity percentiles across the image volume, respectively.

Because the placenta was always entirely within the uterine cavity, there was a complete overlap between the manual segmentation label of the placenta and the uterine cavity. To convert the label volumes into one -hot encoded tensors, the placenta label was subtracted from the uterine cavity label to create a new uterus channel without the intersection. The placenta channel was not modified, and the background channel was generated by inverting the original uterine cavity label. Finally, the channels were stacked along the fourth axis to create image volumes of shape 256 × 256 × N × 3. To keep the network inputs homogeneous, blocks of five consecutive slices were extracted from the training image volumes, with a separate 256 × 256 × 5 sized contiguous block for each possible position within the volume for a total of N-4 training cases per image. The image blocking procedure is detailed in Figure 1, where the identification for each block extracted from an MR image volume is given as $B_i^l$, with i representing the number of the initial slice of the block as part of the volume sequence, and l representing the number of the last slice in the block.

Figure 1.

Process to turn 3D MR image volumes into blocks of five slices each for inputs into the neural network.

Both datasets were divided randomly using a 70-10-20 training, validation, testing split. The 101 sagittal image data were randomly split into groups of 71 training, 10 validation, and 20 testing images. The 241 axial image data gave 169 training, 24 validation, and 48 testing images. The two datasets were used to train and test two different neural networks as described below. When feeding image blocks into the network, we applied a simple data augmentation of left-right flipping across the block at a 50% chance.

Neural network architecture

For the automatic segmentation task, we used the state-of-the-art U-Net 3+ architecture, which makes an improvement on the normal U-Net and U-Net++ by employing full-scale skip connections to increase positional and boundary awareness in segmentation while keeping the model size relatively small [21]. U-Net 3+ adds skip connections from each encoder layer to each decoder layer of equal or deeper depth, as well as intra-skip connections between all decoder layers. These full-scale skip connections are used to integrate low-level details with high-level positional information from feature maps in different scales, which is where the U-Net and U-Net++ fall short by failing to explicitly learn positional and bounding details from organs in segmentation tasks. The U-Net 3+ combines this holistic hierarchical data from the low- and high-level features of the image via max pooling each low-level encoded layer with a larger resolution than the output decoder layer and bilinear up-sampling each high-level encoded layer with a smaller resolution than the output decoder layer [21]. Huang et al. also proposed a module called a classification-guided module (CGM), which applies an extra classification task for predicting whether the input image has organ or not to reduce the number of false-positives, but this was not utilized in our work due to the preprocessing steps making the CGM unnecessary.

We hoped the incorporation of the U-Net 3+ would yield improvements in segmentation performance as compared to our previous U-Net model. The original U-Net 3+ proposed by Huang et al. is a 2D network with five resolution levels. We modified the U-Net 3+ to accept 3D inputs of size 256 × 256 × 5 × 1 and output segmentation predictions of size 256 × 256 × 5 × 3 with the background, placenta, and uterine cavity as the three channels. Additionally, we reduced the number of resolution levels from five to four to greatly lighten the memory load. The convolutional neural network (CNN) architecture is detailed in Figure 2. As an example of data flow in the 3D U-Net 3+, the decoder D2 as shown in Figure 2 was fed through the concatenation of all four encoder layer outputs. Encoder layer E1, having a two times larger resolution than D2, was max pooled by a factor of two and passed through 64 filters. Encoder layer E2 with the same resolution as D2 was simply filtered and concatenated. Encoder layer outputs E3 and E4 were up-sampled using bilinear interpolation at two and four times the resolutions, respectively, before filtering and concatenation. The filters at the decoder input at each encoder scale provide crucial learning parameters for enhancing the organ segmentation performance of the U-Net 3+ model.

Figure 2.

Convolutional neural network architecture (3D U-Net 3+). The network accepts inputs of size 256×256×5 and outputs channels corresponding to background, placenta, and uterine cavity for each voxel for a total output size of 256×256×5×3.

The chosen hybrid loss function was given as:

$$L=\alpha L_{pl}+\beta L_{ut}$$

where L_pl was the placenta label Dice similarity coefficient (DSC) and L_ut was the uterine cavity label DSC. The loss weights were chosen as α = 0.6 and β = 0.4 to give slightly greater weight to the placenta.

Implementation details

The 3D U-Net 3+ architecture was written in Python using TensorFlow and Keras. We trained the network for up to 500 epochs and saved weights based on best validation loss. Our machine was a high-performance computer with four NVIDIA RTX A6000 GPUs, each with 48 GB of memory. Due to the large size of the network, the batch size was initially restricted to eight 3D input blocks on one GPU. We artificially expanded this to 32 blocks per batch by employing the TensorFlow mirrored distribution strategy to duplicate the model onto the four available GPUs. Training was done using the Adam optimizer with an initial learning rate of 1e-4.

Postprocessing and evaluation

Using the 20 testing images from the sagittal dataset and 48 from the axial dataset, each testing image set was fed into the trained network with a single five-slice block at a time with a four-slice overlap with the next block. The blocks were also flipped from left-right and separate predictions were made on the original and mirrored blocks. The mirrored predicted label was flipped again and was averaged with the original predicted label. With a SoftMax output layer, the network output probabilities for each voxel across the three channels summing to 1. All calculated voxel probabilities were added to an empty array with the size of the label volume with block overlap, which was finally collapsed into a ternary output label using TensorFlow’s argmax function, which collapses a tensor along a specified axis by replacing the values across the axis with the index of the largest value.

Resultant predicted uterine cavity and placenta segmentation labels were further postprocessed in MATLAB using a four-step segmentation smoothing and filling process proposed by [23]. First, a 3D erode operator was applied to the predicted labels with a cubic structuring element of size 3 × 3 × 3 to remove small protrusions from the label. Next, the islands in the label volumes were removed by keeping only the largest 26-connected component of the labels. In the third step, the labels were dilated using a cubic structuring element of size 3 × 3 × 3. Lastly, the holes, or regions in the segmentations labeled as 0 unreachable from the surface of the volume, were filled in.

We compared the predicted labels to the manually segmented labels using several error metrics: Dice similarity coefficient (DSC), Hausdorff distance (HD), signed volume difference, and percent signed volume difference. The Dice similarity coefficient can be calculated as follows:

$$\mathit{\text{ Dice }}=\frac{(2\ast \mathit{\text{ Area of Overlap }})}{\mathit{\text{ Total pixels combined }}}$$

The Hausdorff distance is calculated as the greatest of all the distance from a point in one image to the closest point in the other image [22]. The volume difference and percent volume difference can be calculated as follows:

$$\begin{gathered} \Delta V=V_{\mathit{\text{pred}}}-V_{\mathit{\text{ref}}} \\ \mathrm{\Delta \%}V=\frac{V_{\mathit{\text{pred}}}-V_{\mathit{\text{ref }}}}{V_{\mathit{\text{ref }}}}\ast 100 \end{gathered}$$

where ΔV in represents the raw volume difference scaled to the real world using the corresponding physical sizes of the image pixels, V_pred represents the total 3D volume of the uterine cavity or placenta predicted label, and V_ref represents the total 3D volume of the uterine cavity or placenta ground truth label.

RESULTS

Figure 3 shows the training and validation curves for the sagittal and axial 3D U-Net 3+ models. The best validation Dice score was achieved at epoch 169 for the sagittal model and epoch 157 for the axial model.

Figure 3.

Training curves given as a) overall Dice coefficient of the sagittal model, b) weighted Dice loss of sagittal model, c) overall Dice coefficient of the axial model, and d) weighted Dice loss of the axial model. Red dotted lines indicate the epoch at which the best training weights were achieved.

The hold-out test data of the sagittal and axial datasets were segmented using the best training models for each dataset. The end-to-end process of forming a prediction label on an entire image volume, including all postprocessing steps, took approximately thirty seconds on average. Figure 4 displays a boxplot for the DSC and percent volume difference on all test data segmentations as compared to the ground truth labels for the sagittal and axial models.

Figure 4.

Boxplots for placenta and uterine cavity segmentation performance represented as a) DSC for the sagittal model, b) percent volume difference for the sagittal model, c) DSC for the axial model, and d) percent volume difference for the axial model. The sagittal model was evaluated on hold-out test data with n=20 and the axial model was evaluated on hold-out test data with n=48.

In the sagittal dataset, we achieved a DSC of 0.953 ± 0.014 and 0.877 ± 0.031 for the uterine cavity and placenta, respectively. In the axial dataset, we achieved a DSC of 0.918 ± 0.033 and 0.827 ± 0.051 for the uterine cavity and placenta. Tables 1 and 2 show the full results of testing the best models on the sagittal and axial data using the four selected error metrics. We trained our old 3D U-Net network on the data with the same hyperparameters and postprocessing steps to compare performances. Each evaluation metric is a comparison between the predicted label and manually segmented label from the same MR image.

Table 1.

Automatic segmentation performance of the proposed 3D U-Net 3+ network and previous 3D U-Net network on the 20 test images of the sagittal MRI dataset. Error metrics are presented as mean ± std for the placenta and uterine cavity labels.

	Placenta				Uterine cavity
	DSC (%)	HD (mm)	VD (cm3)	VD (%)	DSC (%)	HD (mm)	VD (cm3)	VD (%)
U-Net 3+ (new)	87.7 ±3.1	20.6 ± 10.6	−16.6 ± 109	−1.24 ±9.25	95.3 ± 1.4	13.0 ± 11.4	−146 ±226	−2.91 ±4.50
U-Net (previous)	83.4 ±5.7	24.7 ±21.5	−23.4 ± 151	−1.75 ± 12.8	88.6 ±3.4	25.2 ± 10.1	−172 ± 419	−3.43 ± 8.34

Table 2.

Automatic segmentation performance of the proposed 3D U-Net 3+ network and previous 3D U-Net network on the 48 test images of the axial MRI dataset with n=16 for number of hysterectomy patients and n=32 for non-hysterectomy patients. Error metrics are presented as mean ± std for the placenta and uterine cavity labels.

	Placenta				Uterine cavity
	DSC (%)	HD (mm)	VD (cm3)	VD (%)	DSC (%)	HD (mm)	VD (cm3)	VD (%)
U-Net 3+ (new)	82.7 ±5.1	19.3 ±9.2	−90.5 ± 159	−11.2 ± 17.9	91.8 ±3.3	17.2 ± 14.2	374 ±337	10.3 ±9.3
U-Net (previous)	81.0 ±7.2	25.5 ±20	−84.2 ±207	−10.4 ±23.3	87.2 ±5.6	26.1 ±9.8	−391 ±433	−10.8 ± 12

Sample slices of predicted labels of the placenta and uterine cavity for four cases overlaid onto the original test image slices are shown in Figure 5. Another four example cases of 3D surface overlaps of predicted placenta and uterine cavity labels with the ground truth labels are shown in Figure 6.

Figure 5.

Predicted labels for four patient’s MR images (rows a, b, c, and d) with five selected 2D slices across the volume for each. a) and b) are sagittal images. c) and d) are axial images. Labels are overlaid over the original image slices in magenta and green for uterine cavity and placenta, respectively.

Figure 6.

Superimposed 3D surfaces of predicted placenta (left column) and uterine cavity (right column) segmentation labels and ground truth labels for four test cases, with each row of images from the same patient. a), b), c) and d) are sagittal image cases. e), f), g), and h) are axial image cases. Predicted labels are shown in maroon and ground truths are shown in green.

On the sagittal dataset consisting of MR images in women with suspected PAS, we achieved an improvement of 3.1% in Dice score on the placenta (81.5% to 84.6%) and 1% on the uterine cavity (92.1% to 93.1%). The 3D U-Net 3+ model demonstrated superior performance to the 3D U-Net model for all evaluation metrics. The network seemed to predict the uterine cavity label quite accurately but under-segmented the placenta.

On the axial dataset consisting of MR images of both normal and suspected abnormal pregnancies, we achieved an improvement of 1.2% in Dice score on the placenta (80.3% to 81.5%) and 2% on the uterine cavity (86.5% to 88.5%). As with the sagittal data, the network under-segmented the placenta but tended towards over-segmentation of the uterine cavity. This could be attributed to the large variability between patients with placental complications.

In contrast with the sagittal dataset, the network under-segmented the placenta but tended towards over-segmentation of the uterine cavity. Figure 7 depicts an example of a poor segmentation in an axial image case. In Figure 7, the network severely under-segmented both the placenta and the uterine cavity. Because over a third of the axial dataset consisted of patients who eventually underwent a hysterectomy, the underperformance of the axial network compared to the sagittal network could possibly be attributed to the large variability in placenta shape between patient s with placental complications.

Figure 7.

Poor segmentation performance case in an axial image, shown as superimposed 3D surfaces of predicted and ground truth labels of the a) placenta and b) uterine cavity. Predicted labels are shown in maroon and ground truths are shown in green.

The 3D U-Net 3+ model demonstrated superior performance to the 3D U-Net model for all evaluation metrics. On the sagittal dataset consisting of MR images of only normal pregnancies, we achieved an improvement of 4.3% in Dice score on the placenta (83.4% to 87.7%) and 6.7% on the uterine cavity (88.6% to 95.3%). The network segments the uterine cavity and placenta accurately, with high average DSC and low average SD for both. On the axial dataset consisting of MR images of both normal and abnormal pregnancies, we achieved an improvement of 1.7% in Dice score on the placenta (81.0% to 82.7%) and 4.6% on the uterine cavity (87.2% to 91.8%).

DISCISSION AND CONCLUSION

We proposed a new 3D U-Net 3+ architecture which can automatically segment the placenta and uterine cavity on MR image volumes with relatively high accuracy in a rapid way. Performance of automatic segmentation algorithms remains limited heavily by the amount of data, especially in abnormal placenta cases, resulting in inter-patient variability in uterus and placenta shapes. The results are promising for comparative performance to current MRI segmentation methods on the axial slices and brings automated processing of the MRI data to facilitate diagnosis of maternal and fetal health in clinical obstetrics.

Training of the deep neural network was hindered by memory limitations and the amount of data. With the goal of 3D segmentation of the placenta and uterine cavity in mind, the network accepted image blocks as inputs of size 256 × 256 × 5. Allowing for training on larger input blocks than five slices could further improve segmentation performance by widening the field of view. However, with the size of the 3D U-Net 3+ model and number of parameters, the amount of available computer memory limited the size of the 3D blocks and posed a challenge for training optimally. As with other deep learning-based medical image segmentation tasks, the network would benefit from additional data.

In the future, we plan specific, quantifiable insights on the advantages or disadvantages of sagittal MRI over axial MRI for prenatal imaging, as well as apply additional augmentations, post -processing techniques, and binning methods to improve upon our current segmentation results.