Boundary Restored Network for Subpleural Pulmonary Lesion Segmentation on Ultrasound Images at Local and Global Scales

Abstract

To evaluate the application of machine learning for the detection of subpleural pulmonary lesions (SPLs) in ultrasound (US) scans, we propose a novel boundary-restored network (BRN) for automated SPL segmentation to avoid issues associated with manual SPL segmentation (subjectivity, manual segmentation errors, and high time consumption). In total, 1612 ultrasound slices from 255 patients in which SPLs were visually present were exported. The segmentation performance of the neural network based on the Dice similarity coefficient (DSC), Matthews correlation coefficient (MCC), Jaccard similarity metric (Jaccard), Average Symmetric Surface Distance (ASSD), and Maximum symmetric surface distance (MSSD) was assessed. Our dual-stage boundary-restored network (BRN) outperformed existing segmentation methods (U-Net and a fully convolutional network (FCN)) for the segmentation accuracy parameters including DSC (83.45 ± 16.60%), MCC (0.8330 ± 0.1626), Jaccard (0.7391 ± 0.1770), ASSD (5.68 ± 2.70 mm), and MSSD (15.61 ± 6.07 mm). It also outperformed the original BRN in terms of the DSC by almost 5%. Our results suggest that deep learning algorithms aid fully automated SPL segmentation in patients with SPLs. Further improvement of this technology might improve the specificity of lung cancer screening efforts and could lead to new applications of lung US imaging.

Introduction

Subpleural pulmonary lesions (SPLs) are focal opacities with a very high pretest probability of malignancy [1]. SPLs can be primary malignant or benign pulmonary lesions, including cancer, pneumonia, and tuberculosis. Lung cancer is one of the most frequently diagnosed tumors globally and has the highest morbidity and mortality rates [2]. Pulmonary tuberculosis is the most commonly diagnosed disease in developing countries [3, 4]. In addition, the incidence of pneumonia is high in working-age patients and is also the leading infectious disease in that population [3]. Features of pulmonary lesions, such as their shape, margin, and orientation, are fundamental for predicting treatment outcomes as well as in distinguishing benign and malignant lesions [5–7]. Thus, precise and reliable segmentation is required for quantification in clinical practice.

Previously, the main diagnostic methods for lung disease were radiography, computed tomography (CT), and bronchoscopy [8]. Compared with other imaging modalities, ultrasound (US) devices and techniques provide noninvasive, radiation-free, in vivo cross-sectional view [8–10],thus it is increasingly used in the context of pulmonary diseases [9, 10]. When pathological changes such as exudation, hyperplasia, and necrosis occur in the lung tissue and involve the pleura, unique US images can be acquired due to the destruction of the normal lung structure related to gas exchange, which includes subpleural nodules [11], atelectasis [12], lung consolidation [11], pleural thickening [11], lung cavity [4], calcification [4], and pleural effusion [12]. These important anatomical pathological conditions suggest the use of US in investigating SPLs.

As the manual interpretation of US images is time consuming and limited by experience, recent studies have developed computer-aided diagnosis systems to provide reliable and reproducible information [13]. However, three main challenges significantly impede precise lesion segmentation: (1) difficulty in imaging structures around the air in the lungs, which limits the use of prior knowledge; (2) irregular and blurred contours by unexpected speckle noise and undesirable abnormalities impeding precise delineation; (3) intense inhomogeneity within the lesions that hinders accurate segmentation. Thus, the homogeneity among the neighboring tissues together with the heterogeneity within US images poses difficult challenges for the automated segmentation of lung lesions.

Segmentation on biomedical images can be classified as conventional non-DNN approaches and DNN approaches. Conventional methods usually design artificial features (such as oriented gradients [14, 15], Haar features [14, 15], curvature [16, 17], and Haralick texture features [18]) on images and then construct various segmentation models (such as graph models [6, 19] and shape models [20–23]) to differentiate abnormal regions. For example, Soliman et al. [23] integrated two visual appearances of lungs to form their shape model for lung segmentation on CT chest images. Thanh et al. [21] and Yang et al. [22] used edge detectors to solve segmentation problem. Kumar et al. [6] proposed to combine optimal threshold selection and reconstruction for lung parenchyma segmentation. However, the conventional approaches are usually impeded by the limitation of artificial features, particularly on complicated scenarios (e.g., blurred boundary and imaging noise).

More recently, DNNs have been widely studied for biomedical image segmentation, which benefit from high-level features. The encoder–decoder network is now becoming a standard structure for many DNN-based segmentation models [24–27]. For example, Gerard et al. [25] used encoder–decoder network to extract multi-resolution features from CT images for segmentation of acutely injured lungs. Souza et al. [26] firstly pre-segmented an image as patches and then used DNN to determine if the patches contain lung. Hu et al. [24] combined DNN with conventional approaches (such as SVM, K-means, and Gaussian Mixture Model) to improve lung segmentation from CT images.

Some segmentation approaches have been developed for the purpose of lesion segmentation on US images, including methods based on intensity thresholds, level sets, active contours, and canonical correlation analysis [28–30]. Intensity-based approaches are sensitive to noise and image quality. Active contours, level sets, and canonical correlation analysis require initialization, which can affect the results. To avoid these obstacles, deep neural networks (DNNs), in which the learned features are highly convolved to encode the intrinsic structures of the image, have been applied to sonographic images and have successfully segmented in US images of other tissues, such as the breast [31], carotid artery [32], prostate [33], and thyroid nodules [34].

Although there exist many studies using DNNs for segmentation, the challenges of automated detection and segmentation for subpleural pulmonary lesion has not been addressed. Many investigators designed their DNN models on the basis of U-Net. For example, Gruetzemacher et al. [35] extended U-Net as a 3D model to perform detection and segmentation for pulmonary tumor. Kamal et al. [36] and Veronica [37] incorporated dense connection and fuzzy model with U-Net, respectively. However, a weakness of using U-Net structure is that the performance is highly reliant on the shallowest features where the deeper features are not fully used for segmentation, which causes inaccurate prediction for the location and shape of lesions.

In this study, we propose a novel DNN-based framework to automatically segment SPLs in ultrasound images. As shown in Fig. 1, most FCN-based networks feed DNN features to their neighbored shallower layer to produce the segmentation map. However, their performance is highly reliant on the shallowest features and the advantages of deeper features are not fully used for segmentation. In contrast, our model feeds DNN features not only to their neighbored shallower layer but also to all other shallower layers. Our approach ensures that the segmentation map is obtained from multi-level features of all DNN layers, so that the advantage of deeper features can be used for segmentation. Our contributions can be summarized as follows.

1. To address the issue of blurred boundary of subpleural pulmonary lesion (SPL) for segmentation, we propose a dual-stage framework to exploit multi-level contexts of ultrasound images. In the first stage, our global DNN extracts holistic features of whole image to recognize the location of SPL; then, in the second stage, our local DNN focuses on upsampled lesion-oriented context, which could capture subtle changes of image intensities around SPL boundary for more precise delineation.

2. Although the proposed dual-stage framework could be implemented with existing DNN models (e.g., FCN [38] and U-Net [39]), these DNN models are still limited by heterogeneity problem. To address this issue, we innovate a new DNN model that optimizes the usage of multi-level features from different DNN layers. The features of deeper layers (deep-features) are highly convolved and abstract to represent the intrinsic structures of SPL, which are less affected by heterogeneity problem. However, as deep-features are abstract, they are weak to capture the intensity changes around SPL boundary. Therefore, we use the features of shallower layers (shallow-features) to complement deep-features in our boundary restored network (BSN).

Fig. 1.

Conceptual structure of FCN, FCN-based network, and our BRN. The gray nodes represent DNN features from different layers, where the leftmost and rightmost nodes are the features from the deepest layer and shallowest layer, respectively

Methods

We propose a DNN-based framework to segment SPLs on both the global and local scales. Our DNN model is capable of restoring boundary information from high-level abstract features, thus producing more precise segmentation for SPLs than conventional DNN models such as fully convolutional networks (FCNs) [38]. In this section, we first present our DNN model, called the boundary-restored network (BRN), which accurately locates the SPLs and delineates the corresponding boundaries. We then propose a novel framework using BRN to segment SPLs on the global and local scales.

Difference of our network and other FCN-based networks are illustrated in Fig. 1. As shown in Fig. 1, most FCN-based networks feed DNN features to their neighbored shallower layer to produce the segmentation map. However, their performance is highly reliant on the shallowest features and the advantages of deeper features are not fully used for segmentation. In contrast, our model feeds DNN features not only to their neighbored shallower layer but also to all other shallower layers. Our approach ensures that the segmentation map is obtained from multi-level features of all DNN layers, so that the advantage of deeper features can be used for segmentation.

Boundary-Restored Network

As shown in Fig. 2, the BRN is based on the structure of an FCN, which is a general DNN model for image segmentation. In an FCN, there are five convolutional blocks to encode the input image or intermediate feature maps as higher-level representations. Each convolutional block consists of sequential convolutions defined as:

$$Y_{i\text{'}\text{'}j\text{'}\text{'}d\text{'}\text{'}}=b_{d\text{'}\text{'}}+\sum _{i\prime =1}^{H\prime }\sum _{j\prime =1}^{W\prime }\sum _{d\prime =1}^D{w}_{i\prime j\prime d}\times X_{S\left(i\text{'}\text{'}-1\right)+i\prime -p,S\left(j\text{'}\text{'}-1\right)+j\prime -P,d\prime ,d\text{'}\text{'}}$$ 1

where X ∈ ℝ^{H × W × D} is the input (either the image or intermediate feature maps) of the convolution, w is a filter with the size of H^′ × W^′ × D × D^″, and b is the bias term. S is the stride of filter and P is the padding size. In a DNN model, the filter slides along the input data to produce the convolved data using Eq. (1). Training the DNN model involves learning the proper filter to address the specific type of images. The size of the filter is usually small (e.g., 3×3, 5×5, or 7×7) so that the filter can capture local changes in the image, and the third dimension of the filter must be identical to that of the corresponding input data. Pooling operations are usually inserted between the convolutional blocks to make the DNN model less sensitive to input shifts and distortions [40]. As the pooling operation decreases the resolution of the input data, the feature maps become more intrinsic and abstract representations of the image. As has been noted in many papers [40, 41], the obtained features from the convolutional blocks and pooling operations are more meaningful and semantically rich than handcrafted features and can be used to understand the image and to locate the target objects.

Fig. 2.

Basic structure of an FCN and our BRN

As shown in Fig. 2, the output of an FCN is derived directly from the feature maps of the last convolutional block. However, with the inclusion of the pooling operations, the feature maps are spatially sparse and lose many local details; thus, the boundaries of objects cannot be captured from the feature maps. To address this issue, we infer the segmentation maps from both low- and high-level features so that they can complement each other. High-level features with high downsampling ratios are more intrinsic and abstract and are better at locating the object than low-level features. Low-level features with low downsampling ratios preserve the boundary details and are better for delineating the object. Therefore, we propose the use of high-level features for object detection and then progressively adopt low-level features to delineate the boundary of the object.

As shown by the red arrows in Fig. 2, the feature maps are progressively concatenated with the features from the next lowest level:

$$Y^s={\left[\delta \left(Y^{s-1},{\overline{Y}}^{5-s}\right),{\overline{Y}}^{5-s}\right]}^T$$ 2

where Y^s denotes the feature maps obtained in the s-th (s ∈ {1, 2, 3, 4}) concatenation step, ${\overline{Y}}^l$ denotes the feature maps obtained in the l-th level of the FCN, and δ(x, y) is the upsampling operation that makes the resolution of x identical to that of y. Following many existing methods, we adopt deconvolution to implement δ(x, y). Deconvolution can be calculated by

$$y_{i^{″}{j}^{″}{d}^{″}}=\sum _{d^{\prime }=1}^D\sum _{i^{\prime }=0}^{q\left(H^{\prime },S_h\right)}\sum _{j^{\prime }=0}^{q\left(W^{\prime },S_w\right)}{f}_{1+S_{i^{\prime }}+m\left(i^{″}+P,S\right),1+S_{j^{\prime }}+m\left(j^{″}+P,S\right),d^{″},d^{\prime }}\times x_{1-i^{\prime }+q\left(i^{″}+P,S\right),1-j^{\prime }+q\left(j^{″}+P,S\right),d^{\prime }}$$ 3

where $m\left(k,S\right)=\left(k-1\right)modS,q\left(k,n\right)=\frac{k-1}{S}$. x ∈ ℝ^{H × W × D}is the input data of the deconvolution, w is the filter with the size of H^′ × W^′ × D × D^″, S is the stride of filter, and P is the padding size. In Eq. (2), Y⁰ is the shallowest features of BRN, which is directly obtained from the encoder of DNN. The location information of the object is encoded in Y^s − 1, and the boundary information can be restored from ${\overline{Y}}^{5-s}$. In addition to the connections between neighboring layers, we also adopt short connections (shown as the blue, green, magenta, and yellow arrows in Fig. 2) to connect more layers in our BRN and extract more comprehensive features that preserve the location and boundary information. Considering the limited computational capacity, we only added short connections between the features that are on the top of each convolution block. Afterwards, the final segmentation map can be obtained by a convolutional layer (filter size, 1×1; stride, 1; pad, 0; channel, 1) that adopts Y⁴ as the input (details in Fig. 3). The pseudo code of our method is presented as Algorithm 1.

Fig. 3.

Flowchart of our framework for peripheral pulmonary lesion segmentation

To train our BRN, the segmentation map is compared to the ground truth using a loss function. The loss is then backpropagated through the network to update the filters of Eq. (1) via stochastic gradient descent [42]. In this paper, we adopt the sigmoid cross-entropy loss to train the BRN, which is defined as:

$$E=-\frac{1}{N}\sum _n\left[\mathit{GT}\left(n\right)log\left(\mathit{SM}\left(n\right)\right)+\left(1-\mathit{GT}\left(n\right)\right)log\left(1-\mathit{SM}\left(n\right)\right)\right]$$ 4

where GT is the ground truth, SM is the segmentation map, and N is the number of training images.

Peripheral Pulmonary Lesion Segmentation

We propose a dual-stage framework consisting of two BRNs, as shown in Fig. 3. For peripheral pulmonary lesion segmentation using our BRN, the global BRN (stage 1) adopts the whole ultrasound image as input and produces the corresponding coarse segmentation map (denoted by SM¹) at the global level. We can obtain the region of interest (RoI) on SM¹ by binarizing SM¹ with a threshold of 0.5, which is used to construct the lesion-oriented local context. The lesion-oriented local context is defined as the area of the bounding box approximating the RoI. To address oversegmentation in the global BRN, we enlarge the bounding box by 35 pixels to enclose more of the image in the lesion-oriented local context. Then, we input the lesion-oriented local context into another BRN (the local BRN) (stage 2) to finely delineate the boundaries of the lesions on the local level. We use the generated probability maps obtained from the local BRN (denoted by SM²) as the final segmentation results of our framework.

Results

Data and Ethics

US scans of patients diagnosed with SPLs were obtained with a Logiq E9 scanner (General Electric (GE) Healthcare, Wauwatosa, WI, USA) between June 2017 and February 2018. The frequency ranged from 1 to 6 MHz. The tenets of the Declaration of Helsinki were followed, and the Institutional Review Board of Shanghai Pulmonary Hospital approved the study. Informed consent was obtained from all the subjects. To evaluate the layer segmentation results, one specialist manually labeled the SPL RoIs. The ground truth was defined as the RoIs generated by the specialist.

In total, 255 patients aged from 14 to 85 years with an average age of 59.17 ± 16.01 years were enrolled in the study: 183 males (71.76%) and 72 females (28.23%) were included. Three to eight ultrasound cross-section slices from the same patient were obtained randomly from the SPLs. In total, 1612 ultrasound images were retained. The pathological types are shown in the table below (Table 1).

Table 1.

Demographic information of our dataset

Pathological type	Patients (N, %)	Images (N, %)
Squamous cell carcinoma	56 (21.96%)	452 (28.04%)
Adenocarcinoma	85 (33.33%)	564 (34.99%)
Small cell lung cancer	8 (3.14%)	48 (2.98%)
Neuroendocrine carcinoma	7 (2.75%)	29 (1.8%)
Large cell lung cancer	1 (0.39%)	8 (0.5%)
Poorly differentiated cancer	6 (2.35%)	36 (2.23%)
Sarcoma	1 (0.39%)	9 (0.56%)
Pleural mesothelioma	1 (0.39%)	3 (0.19%)
Solitary fibroma	1 (0.39%)	5 (0.31%)
Pulmonary tuberculosis	68 (26.67%)	336 (20.84%)
Non-tuberculosis mycobacteria	1 (0.39%)	5 (0.31%)
Pneumonia	14 (5.49%)	84 (5.21%)
Pulmonary abscess	5 (1.96%)	30 (1.86%)
Inflammatory granuloma	1 (0.39%)	3 (0.19%)

Evaluation Metrics

We used Dice similarity coefficient (DSC), Accuracy, Sensitivity, Specificity, Precision, Matthews correlation coefficient (MCC), Jaccard similarity metric (Jaccard), Average Symmetric Surface Distance (ASSD), and Maximum symmetric surface distance (MSSD) to evaluate our framework [17, 43–45]. These evaluation metrics are defined as follows:

$$\mathit{DSC}=2\times \frac{\left|A_R\cap A_G\right|}{\left|A_R\right|+\left|A_G\right|}.$$ 5

$$\mathit{\text{Accuracy}}=\frac{\mathit{TP}+\mathit{TN}}{\mathit{TP}+\mathit{TN}+\mathit{FN}+\mathit{FP}}.$$ 6

$$\mathit{\text{Sensitivity}}=\frac{\mathit{TP}}{\mathit{TP}+\mathit{FN}}.$$ 7

$$\mathit{\text{Specificity}}=\frac{\mathit{TN}}{\mathit{TN}+\mathit{FP}}.$$ 8

$$\mathit{\text{Precision}}=\frac{\mathit{TP}}{\mathit{TP}+\mathit{FP}}.$$ 9

$$\mathit{MCC}=\frac{\mathit{TN}\times \mathit{TP}-\mathit{FN}\times \mathit{FP}}{\sqrt{\left(\mathit{TP}+\mathit{FP}\right)\times \left(\mathit{TP}+\mathit{FN}\right)\times \left(\mathit{TN}+\mathit{FN}\right)\times \left(\mathit{TN}+\mathit{FP}\right)}}.$$ 10

$$\mathit{\text{Jaccard}}=\frac{\mathit{DSC}}{2-\mathit{DSC}}.$$ 11

$$\mathit{\text{ASSD}}=\frac{1}{\left|B_G\right|+\left|B_R\right|}\times \left(\sum _{x\in B_R}\underset{y\in B_G}{min}d\left(x,y\right)+\sum _{y\in B_G}\underset{x\in B_R}{min}d\left(y,x\right)\right)$$ 12

$$\mathit{\text{MSSD}}=max\left(\underset{x\in B_R}{max}\left(\underset{y\in B_G}{min}d\left(x,y\right)\right),\underset{y\in B_G}{max}\left(\underset{x\in B_R}{min}d\left(\mathrm{y},x\right)\right)\right)$$ 13

where A_R and A_G are the areas of the segmentation results and the ground truth, respectively. TP, TN, FP, FN denote true positive area, true negative area, false positive area, and false negative area, respectively. B_G and B_R are the border pixel sets of the ground truth and the segmented results, respectively. d(x,y) is the Euclidean distance of between pixels x and y in real-world coordinates. A good algorithm should have higher values for the all the metrics except for MSSD and ASSD. The value of MSSD and ASSD is 0 mm for a perfect segmentation.

Implementation

Our framework was implemented in Caffe [46] on a PC equipped with a 3.6-GHz CPU and a 12-GB TITAN-X GPU. We fine-tuned the global and local BRNs using a pretrained Visual Geometry Group (VGG) model [47]. To train the global and local BRNs, the momentum and weight decay were set to 0.9 and 5e−4, respectively. The learning rate was initialized as 1e−10 and was dynamically decayed to lr × (1 + γ × t)^−δ, where lr is the initial learning rate and t is the training iteration. γ and δ control the speed of the learning rate decay and were set to 1e−4 and 0.75, respectively. We fine-tuned the two BRNs until they converged. We only adjusted the above parameters for training purpose and these parameters are not used in the testing phase of our model. The average processing time for one image was 0.35 s. The computational complexity of our method and other methods were listed in Table 2.

Table 2.

Running time of our method¹ and other methods on an ultrasound slice image²

Method	FCN	U-Net	PSP	Amulet	Global BRN (stage 1)	Local BRN (stage 2)	dual-stage BRN (whole)
Time (s)	0.05	0.11	0.35	0.10	0.17	0.17	0.35

¹Our method was dual-stage BRN

²Abbreviations: BRN Boundary-Restored Network, FCN fully convolutional network, PSP Pyramid Scene Parsing Network

State-of-the-Art Comparison Models

We compared our framework with two representative DNN models: an FCN [38] and U-Net [39]. An FCN is a general segmentation DNN model with the inclusion of fully convolutional layers and is widely used in many computer vision applications. The high-level features of the FCN are extracted via VGGNet [47]; then, these features are decoded to produce the segmentation maps via a deconvolution operation, which is the inverse of a convolution operation. U-Net, which progressively decodes the feature maps, was proposed for biomedical image segmentation. Similar to our BRN, short connections are also inserted in U-Net to restore the local details of the images. Detailed structures of the models for comparison are referred to [38, 39]. We also selected an extension model of FCN (Pyramid Scene Parsing Network, PSP [48]) and an extension model of U-Net (Amulet [49]) as comparison methods. All models (including our model and comparison models) were trained and tested with the same data.

Experimental Results

The experimental results of our framework and the comparison models are listed in Table 3. Our model consistently outperformed the state-of-the-art models with a mean ASSD of 5.68 mm and mean MSSD of 15.61 mm, which indicates the superiority of our framework. The accuracy and specificity of all methods were over 95%. The sensitivity and precision of dual-stage BRN were over 85%, nearly 10% higher than comparison models. The calculated mean ratios of DSC, MCC, and Jaccard were 83.45%, 0.8330, and 0.7391, respectively. As shown in Fig. 4, our framework also performed visually better than the selected comparison models.

Table 3.

Experimental results of our method¹ and comparison methods²

Method	FCN	U-Net	PSP	Amulet	Global BRN	Dual-stage BRN
DSC (%)	72.09±17.41	76.68±16.83	76.38 ± 25.61	80.85±18.57	78.29±14.40	83.45±16.60
Accuracy (%)	97.47±5.50	97.74±5.54	96.32 ± 8.74	97.13±7.81	97.88±5.47	98.52±5.48
Sensitivity (%)	73.89±21.97	76.31±21.46	74.93 ± 27.72	79.53±23.02	76.50±19.96	85.17±17.94
Specificity (%)	98.52±5.61	98.81±5.60	98.95 ± 5.60	99.21±5.44	98.99±5.52	99.11±5.50
Precision (%)	73.89±18.93	82.29±15.11	86.07 ± 20.90	86.61±13.88	85.18±12.97	88.07±18.03
MCC	0.7208±0.1676	0.7685±0.1547	0.7619 ± 0.2542	0.8070 ± 0.1843	0.7851±0.1309	0.8330±0.1626
Jaccard	0.5863±0.1767	0.6462±0.1893	0.6676 ± 0.2486	0.7089 ± 0.2057	0.6611±0.1635	0.7391±0.1770
ASSD (mm)	21.99±6.265	5.75±5.69	15.46 ± 16.53	22.84 ± 17.36	25.10±6.06	5.68±2.70
MSSD (mm)	72.65±14.45	32.42±23.11	36.91 ± 22.11	73.01 ± 14.73	72.55±14.29	15.61±6.07

¹Our method was dual-stage BRN

²Abbreviations: ASSD Average Symmetric Surface Distance, BRN Boundary-Restored Network, DSC Dice similarity coefficient, FCN fully convolutional network, Jaccard Jaccard similarity metric, MCC Matthews correlation coefficient, MSSD maximum symmetric surface distance, PSP Pyramid Scene Parsing Network

Fig. 4.

Views of the segmentation results of our framework and other methods on an image. The red and yellow regions are the segmentation contours of the standard reference and the algorithmic result, respectively

Ablation Analyses

We further removed the local BRN and directly tested the global BRN in ablation analyses. The performance of the global BRN by itself is shown in Table 3. Our dual-stage framework enhanced the single DNN model with improvements of 5.16% in the DSC, 0.048 in MCC, and 0.078 in Jaccard. The calculated mean ratios of ASSD and MSSD of dual-stage BRN were 19.42 mm and 56.94 mm lower than global BRN in our dataset. It is worth noting that even the global BRN by itself outperformed the other state-of-the-art models in terms of the DSC, MCC, and Jaccard.

Discussion

The clinical manifestations of lung diseases are diverse. The phenomenon in which the same disease has different characteristics is often encountered in chest imaging [8]. Some lesions are difficult to classify as benign or malignant; sometimes, the two can be present in the same scan. For example, the incidence of lung cancer in tuberculosis patients is 11 folders higher than healthy people [50]. Moreover, lung cancer can also easily lead to tuberculosis infection or recurrence, and lung cancer with obstructive pneumonia is also common in clinical practice [51]. Thus, subpleural pulmonary lesion segmentation in lung diseases was a sophisticated problem [1].

We propose the use of BRN, the basic component of our segmentation framework, for detecting SPLs. The BRN restored more boundary information on the segmentation maps than the FCN. We attribute this result to the utilization of the low-level DNN features in the BRN. As shown in Fig. 5, the low-level DNN features complement the high-level features, which optimizes the segmentation ability of the model. The segmentation maps obtained from U-Net contain many false positives because U-Net does not connect the different levels of DNN features. Thus, the inferences of SPLs primarily rely on the features of the top DNN layer, which contains many local details, which distract the DNN from correctly localizing lesions.

Fig. 5.

Evolution of the segmentation maps in our BRN

An advantage of using our framework for SPL segmentation is multiscale detection and segmentation. As shown in Fig. 6, while the global BRN can correctly locate lesions in the original ultrasound images, the local BRN finely delineates the boundaries of the lesions from the corresponding RoIs. The superior performance of the dual-stage framework compared to the single DNN model is as follows: as the local BRN focuses on the details in the local image, more boundary information can be analyzed for precise delineation, which is crucial for morphology analysis of SPLs. SPL extraction could help identify the perilobular region and centrilobular structures and highlight perilobular fibrosis, especially in interstitial lung disease [52]. The segmentation of SPL could also improve the ability to differentiate interstitial lung abnormalities including cancer in smokers [52, 53].

Fig. 6.

Intermediate outputs of our framework. Red: manual segmentation; yellow: our final delineated contour; blue: contour of the segmentation map from the global BRN

The images used in this study were lung ultrasound images of various kinds of diseases including pneumonia tuberculosis and even cancer. Though X-rays and CT scans are the main imaging examination methods for lung diseases, as they can clearly show slight changes in the lesions, they involve ionizing radiation, which will cause differing degrees of damage to the patients’ bodies [3, 8]. Compared with CT and X-ray examinations, US scans had high sensitivity, low cost, and minimal trauma, which are obvious advantages for the long-term follow-up of the disease. Furthermore, lung ultrasound can be used for early detection and management of respiratory complications in ICU [12]. It can display not only the characteristics of lung semiotics (i.e., effusions and consolidations) but also the blood flow to adherent lung lesions [10].Thus, it is vital to identify the lesions like SPLs from these images for further disease management.

By applying our dual-stage BRN framework, we were able to accurately identify SPLs and the edges of the handcrafted features. Next, we will further analyze the grayscale distribution and textures of the lesions and improve the recognition speed so that the proposed technique can be run in real time to help doctors identify lesions and make diagnoses [54]. In addition, contrast-enhanced ultrasonography is an important research topic in the diagnosis of SPLs [55]. Based on our image recognition results, we will analyze contrast-enhanced images to achieve accurate judgments of the phase information and find image features that are difficult to identify manually, which will further improve the application value of US imaging in the diagnosis of SPLs.

Future Work

A major aim of our manuscript and many other relevant papers is to provide radiologists with desirable and robust segmentation results, which helps radiologists to interpret ultrasound images. However, pathology is not considered in this manuscript as the intrinsic image structure and criteria of differentiating tumor on pathological images are quite different, which makes it difficult to directly and simply translate the algorithm from radiology to pathology. We will explore the translation between radiology and pathology as our future work.

Conclusion

Our dual-stage BRN framework can be applied to segment multiple types of lung lesions in US images, and it outperformed existing methods. In this study, dual-dimensional images of SPLs were identified and analyzed by a computer, which lays the foundation for further benign and malignant tumor identification, color flow imaging, and contrast-enhanced imaging analysis. The dual-stage BRN, suitable for large database segmentation, can provide clinically relevant information for better disease management.

Author Contributions

All authors contributed to the planning of the work described. Y.Z., K.B., L.X. and M.S. conducted the data collection. Y.X., Z.N., G.D., and Y.W. analyzed and interpreted the results. All authors critically revised the content and approved the final manuscript.

Funding

This study received funding from 2018 Supporting Project of Medical Guidance (Chinese and Western Medicine) of Science and Technology Commission of Shanghai Municipality (18411966700), 2019 Technical Standard Project of Shanghai "Science and Technology Innovation Action Plan" of Science and Technology Commission of Shanghai Municipality (19DZ2203300), Clinical Research Foundation of ShangHai Pulmonary Hospital (fk1940), Shanghai Sailing Program (No. 19YF1439300 & 19YF1440000) , Medical-Engineering Funding of Shanghai Jiao Tong University (No. ZH2018QNA24&ZH2018QNA20),National Key R&D Program of China (No. 2016YFC0904800). The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Compliance with Ethical Standards

Conflicts of Interest

The authors declare that there are no conflicts of interest related to this article. This research was not sponsored by any company. The authors have full control of the data.