Automatic lung segmentation in dynamic thoracic MRI using two-stage deep convolutional neural networks

Abstract

Lung segmentation in dynamic thoracic magnetic resonance imaging (dMRI) is a critical step for quantitative analysis of thoracic structure and function in patients with respiratory disorders. Some semi-automatic and automatic lung segmentation methods based on traditional image processing models have been proposed mainly for CT with good performance. However, the low efficiency and robustness of these methods and inapplicability to dMRI make them unsuitable to segment the large numbers of dMRI datasets. In this paper, we present a novel automatic lung segmentation approach for dMRI based on two-stage convolutional neural networks (CNNs). In the first stage, we utilize the modified min-max normalization method to pre-process MRI for increasing the contrast between the lung and surrounding tissue and propose a corner-points and CNN based region of interest (ROI) detection strategy to extract the lung ROI from sagittal dMRI slices, which can reduce the negative influence of tissues located far away from the lung. In the second stage, we input the adjacent ROIs of target slices into the modified 2D U-Net to segment the lung tissue. The qualitative and quantitative results demonstrate that our approach achieves high accuracy and stability in terms of lung segmentation for dMRI.

1. INTRODUCTION

Dynamic thoracic magnetic resonance imaging (dMRI) has been employed to evaluate lung function in adolescent idiopathic scoliosis (AIS) [1], to analyze diaphragmatic motion [2], to quantify regional dynamic thoracic function in thoracic insufficiency syndrome (TIS) [3], and to understand developmental changes in normal respiratory dynamics [4]. In these applications, lung tissue segmentation in MRI plays a crucial role in quantitative analysis. Manual delineation of the lung is extremely time-consuming, labor-intensive, and suffers from inter-reader variability, particularly since dMRI acquired over several respiratory cycles can include 100s of slices. As such, there is an urgent need for clinicians and researchers to have access to a robust automatic and accurate lung segmentation method to improve work efficiency. To solve this problem, traditional image processing models based on semi-automatic and automatic methods [5–8] have been proposed with good performance. However, semi-automatic methods are not appropriate to process large numbers of dMRI datasets due to their low efficiency, complex shape variability, low contrast, other nearby confounding tissues of similar appearance, and poor signal-to-noise-ratio of lung tissue in dMRI. These challenges seriously hamper the robustness of the automatic methods.

In this paper, we propose an automated lung segmentation system based on two-stage deep convolutional neural networks (CNNs) for dMRI, achieving a high agreement with reference standard manual segmentations. The main innovations of our study include: 1) A fully automatic, robust, and accurate lung segmentation system for dMRI using a two-stage deep CNNs model; 2) An accurate and adaptive lung ROI detection strategy based on the corner-points concept and CNN; 3) An ingenious design of network architecture combing pixel-wise classification and regression to detect the location of the corner-points of the lung ROI; 4) Demonstration of as close to the highest possible lung segmentation performance considering the quality of the lung ROI.

2. MATERIALS & METHODS

Figure 1 is a schematic depiction of the main stages in our method. In the first stage, we proposed an automatic corner-points detection model based on U-Net [9] to locate the top-left and bottom-right points of the bounding boxes for the lung tissue in sagittal 2D MRI slices. To enhance the contrast between the lung and surrounding tissues, we modified the min-max normalization method to pre-process the MRIs. Then, we extracted the ROIs of lung from MRI slices as the input data for the lung segmentation model according to the location of the detected two corner points, reducing the negative effect of the tissues located far away from the lung. To ensure that the generated ROI can cover the lung region, we enlarged the size of the ROI by 1.2 times. Encouraged by our previous work [10], the adjacent slices of the target slice were inputted into the segmentation network as the context information to improve lung segmentation performance. Finally, the output probability map of the segmentation model was transformed into a binary image using the thresholding method with a threshold of 0.5. To restore the spatial information of segmentation results, we utilized zero-padding to embed the segmentations in the full image outside the ROI.

Figure 1.

Illustration of two-stage CNN structure for automatic lung segmentation from dynamic thoracic MRI.

2.1. Image datasets

All MRI studies were acquired under an IRB-approved research protocol at the Children’s Hospital of Philadelphia along with Health Insurance Portability and Accountability Act waiver. The de-identified MRI data sets were subsequently analyzed at the University of Pennsylvania. The data set (Table 1) consists of sagittal bright-blood dMRI scans from 63 subjects acquired over several natural breathing cycles using the following parameters: True-FISP sequence, TR=3.82 ms, TE=1.91 ms, and flip angle=76°. 103 3D frames of these scans were annotated by individuals with a background and appropriate training in human anatomy and the radiological appearance of the relevant structures, through use of the open-source software CAVASS [11].

Table 1.

Summary of the dynamic thoracic MRI data set.

Data set	# of Subjects	# of 3D Frames
Training set	36	59 (2225 slices)
Validation set	6	11 (414 slices)
Testing set	21	33 (1254 slices)
Total	63	103 (3893 slices)
Voxel size (mm3)	1.46 × 1.46 × 6.00

2.2. Modified min-max normalization method

As shown in Figure 1, the discrepancy of intensity between the lung and surrounding tissues is not obvious, increasing the difficulties for lung ROI detection and segmentation. The main reason is that the high contrast between the lung and tissue located far away from the lung suppresses the local contrast between the lung and surrounding tissue. To solve this problem, we propose a pre-processing method based on the min-max normalization model and lung intensity statistics. Assuming that I(x) represents the intensity of pixel x, the pre-processed image I_N(x) is obtained as follows:

$$I_N(x)=\left\{\begin{array}{rr}\hfill I_{min},& \hfill \text{ if }I(x)<I_{min}\\ \hfill I_{max},& \hfill \text{ if }I(x)>I_{max}\\ \hfill I(x),& \hfill \text{otherwise}\end{array}\right..$$ (1)

Subsequently, I_N(x) is rescaled.

$$I_N(x)=\frac{I_N(x)-I_{min}}{I_{max}-I_{min}},$$ (2)

where I_min and I_max represent the minimum and maximum intensity of lung tissue in MRI, respectively. The first equation is used to map the intensity range of I(x) into the new range [I_min, I_max] to increase the contrast between the lung and surrounding tissue, and the second equation rescales the intensity range of I_N(x) into [0, 1] for accelerating the convergence speed for the deep learning models. In our experiments, the parameters I_min and I_max were set to 0 and 400, respectively, after MRI signal intensity standardization [12].

2.3. Corner-points and CNN based ROI strategy

In [10], we developed a generalized ROI strategy for upper airway in static and dynamic MRI by exploiting the spatial information of the annotated dataset, in which the bounding box is determined by the location of the center point (x_c, y_c) and window size (2d_h, 2d_v), as shown in Figure 2(a). However, this method is not appropriate to produce the ROI of the lung in dMRI, due to the complex variability of lung location. To solve this problem, we propose an adaptive lung ROI detection method based on U-Net, named Corner-points based ROI, in which the ROI is determined by the location of the top-left (x_TL, y_TL) and bottom-right (x_BR, y_BR) points as well as the horizontal and vertical distance offset (Δx_TL, Δy_TL, Δx_BR, Δy_BR), as shown in Figure 2(b). Firstly, we utilized the manual lung segmentation results of training data to automatically generate the bounding boxes and extracted the coordinate information of the top-left and bottom-right corner-points. Then, we exploited the coordinates of two corner-points as the centers to draw two solid circles with a radius of 5 pixels. The background, top-left, and bottom-right circles were annotated with labels “0”, “1” and “2”, respectively. Next, we utilized these annotation data to train the deep CNN for segmenting the top-left and bottom-right circles from MRI and extracted the location of the local-maximum in the possibility map as the detected top-left (x_TL, y_TL) and bottom-right (x_BR, y_BR) corner-points. To reduce the spatial error of corner-points detection, we computed the horizontal and vertical distances between pixel x and the center of the circles as additional annotation data, training the deep CNN to predict the distance offset (Δx_TL, Δy_TL, Δx_BR, Δy_BR). The refined location of the top-left and bottom-right corner-points was expressed as follows:

$$\left(x_{TL}^{*},y_{TL}^{*},x_{BR}^{*},y_{BR}^{*}\right)=\left(x_{TL},y_{TL},x_{BR},y_{BR}\right)+\left(\Delta x_{TL},\Delta y_{TL},\Delta x_{BR},\Delta y_{BR}\right).$$ (3)

Figure 2.

Illustration of ROI strategies: (a) Generalized ROI. (b) Corner-points based ROI.

Finally, the detected top-left and bottom-right corner-points were employed to produce the lung ROI for improving the segmentation accuracy and training speed of the lung segmentation model in the second stage.

2.4. Network architecture

We designed a novel deep CNN based on 2D U-Net to detect the corner-points of the lung ROI, as shown in Figure 3. The network consists of four modules: i) the feature learning module (left side) for extracting multi-scale features from the input images including 16 convolutional layers with 3×3 kernels, 4 max-pooling layers with stride 2, 3 dropout layers with rate 0.6, and 5 batch-normalization layers, ii) the feature fusion module (right side) for up-sampling the feature maps and integrating the multi-scale features as a feature pyramid including 12 convolutional layers with 3×3 kernels, 4 up-sampling layers based on bilinear interpolation, and 4 concatenation layers, iii) the pixel-wise classification module for classifying the feature map into three categories (background, top-left, and bottom-right corner-points) including 2 dropout layers with rate 0.6, 2 convolutional layers with 3×3 kernels, and 1 soft-max function, and iv) the regression module for predicting the distance offset including 2 dropout layers with rate 0.6, and 2 convolutional layers with 3×3 kernels. The main differences between our network and U-Net include replacing the de-convolutional layer with an up-sampling layer for reducing the number of parameters in our model and using the convolutional layer with 3×3 kernels and 32 output channels to reduce the redundancy of the feature pyramid. In the second stage of our framework, a similar network was employed to segment the lung tissue in the extracted ROIs. However, the distance offset regression module is not needed for the lung segmentation task.

Figure 3.

The architecture of the corner-points detection network.

2.5. Loss function

We combined the cross-entropy and L1 loss functions together to define the loss function of the corner-points detection network as:

$$L\left(S_G,O_G;W\right)=-\frac{1}{N}{\sum }_{x\in \Omega }\text{log}\left(P\left(l=S_G(x)\mid x\right)\right)+\frac{{\lambda }_1}{N}{\sum }_{x\in \Omega }{‖O_G(x)-O(x)‖}_1+{\lambda }_2\Vert W{\Vert }_1,$$ (4)

where S_G(x) and O_G(x) denote the ground truth of the category and distance offset at pixel x, P(l = S_G(x)|x) represents the possibility value of pixel x classified as ground truth S_G(x), O(x) denotes the predicted distance offset at pixel x, W and Ω represent the parameters of the network and image domains, ‖·‖₁ is the L1-norm, and λ₁ and λ₂ serve as trade-off parameters among the three terms. To improve the lung segmentation performance, we utilized the FP&FN+DICE loss function [10] to train the lung segmentation network, which has been shown to efficiently reduce the false-positive and false-negative rates.

3. RESULTS AND DISCUSSION

We utilized the testing data to test our two-stage CNNs and calculated the 2D Dice index (2D DICE) for evaluating the lung ROI detection performance and the 3D Dice index (3D DICE) and 3D average Hausdorff distance (HD) for evaluating the lung segmentation performance in dMRI. To investigate the effect of ROI strategy on lung segmentation performance, we conducted three comparison experiments by removing the ROI strategy (Non-ROI) and replacing the corner-points and CNN based ROI strategy with the ground truth of ROI and generalized ROI method [10]. Figure 4 illustrates lung detection results, indicating that the proposed corner-points and CNN based ROI strategy obtains good agreement with the manual ROI and that the offset correction can efficiently increase the degree of overlap between automatic and manual ROIs. Figure 5 shows three lung segmentation results and Table 2 summarizes the quantitative results. We observe that the proposed two-stage deep CNNs model achieves a high mean value of 0.97 and a low standard deviation of 0.02 for 3D DICE, indicating excellent segmentation accuracy and stability. A 2D DICE value of 0.92 shows that the corner-points and CNN based ROI strategy achieves an acceptable accuracy of lung detection. The comparison between the performance values in the Non-ROI row of Table 2 with others demonstrates that the ROI detection strategies can efficiently improve the lung segmentation accuracy. In addition, the lung segmentation accuracy improves with increasing the lung ROI detection accuracy. These experimental data demonstrate the importance of the object detection strategy for object segmentation.

Figure 4.

Examples of lung detection in dynamic thoracic MRI. The green and red colors represent manual and automatic detection results, respectively. (a) Generalized ROI strategy. (b) Corner-points and CNN based ROI strategy without offset correction. (c) Corner-points and CNN based ROI strategy.

Figure 5.

Examples of lung segmentation in dynamic thoracic MRI. The green and orange colors represent manual and automatic segmentation results, respectively. (a) Original sagittal MR images; (b) manual segmentation; (c) automatic segmentation using modified U-Net; (d) automatic segmentation using generalized ROI and modified U-Net; (e) automatic segmentation using corner-points and CNN based ROI and modified U-Net; (f) automatic segmentation using ground truth of ROI and modified U-Net.

Table 2.

Lung ROI detection and segmentation performance on testing data. All values are expressed as “mean/standard deviation.”

First stage	Second stage	ROI detection	Lung segmentation
		2D DICE	3D DICE	3D HD (pixels)
Non-ROI	Modified U-Net	-	0.83/0.13	4.18/2.54
Generalized ROI		0.23/0.12	0.86/0.06	5.64/5.18
Corner-points and CNN based ROI		0.92/0.13	0.97/0.02	3.11/4.89
Ground truth of ROI		1.00/0.00	0.98/0.01	1.30/0.52

4. CONCLUSIONS

In this paper, we utilize deep learning techniques to construct a fully automatic system for lung segmentation in dynamic thoracic magnetic resonance imaging (dMRI) for the quantitative analysis of lung function. Our method can be divided into two stages: 1) extraction of the lung ROI from sagittal dMRI slices using the corner-points and CNN based ROI strategy for reducing the negative influence of the tissue located far away from the lung, 2) segmentation of the lung tissue in the lung ROIs using the modified 2D U-Net. Experimental results demonstrate that our approach achieves excellent agreement with manual detection and segmentation of lung tissue in dMRI and can be utilized routinely for segmenting lungs in dMRI of TIS patients where the thorax is often highly distorted, adding to segmentation challenges coming from poor and confounding intensity appearance.

Considering that the ROI detection accuracy is positively correlated with the segmentation performance of our approach, we are developing an interactive ROI strategy in order to ensure that each ROI detection result in the dMRI acquisition sequences is sufficiently accurate, even in the presence of extreme distortions of the anatomy of the chest as observed in severe cases of TIS.