COVID‐19 CT image segmentation based on improved Res2Net

Abstract

Purpose

Corona virus disease 2019 (COVID‐19) is threatening the health of the global people and bringing great losses to our economy and society. However, computed tomography (CT) image segmentation can make clinicians quickly identify the COVID‐19‐infected regions. Accurate segmentation infection area of COVID‐19 can contribute screen confirmed cases.

Methods

We designed a segmentation network for COVID‐19‐infected regions in CT images. To begin with, multilayered features were extracted by the backbone network of Res2Net. Subsequently, edge features of the infected regions in the low‐level feature f ₂ were extracted by the edge attention module. Second, we carefully designed the structure of the attention position module (APM) to extract high‐level feature f ₅ and detect infected regions. Finally, we proposed a context exploration module consisting of two parallel explore blocks, which can remove some false positives and false negatives to reach more accurate segmentation results.

Results

Experimental results show that, on the public COVID‐19 dataset, the Dice, sensitivity, specificity, $S_{\alpha }$, $E_{\varnothing }^{mean}$, and mean absolute error (MAE) of our method are 0.755, 0.751, 0.959, 0.795, 0.919, and 0.060, respectively. Compared with the latest COVID‐19 segmentation model Inf‐Net, the Dice similarity coefficient of our model has increased by 7.3%; the sensitivity (Sen) has increased by 5.9%. On contrary, the MAE has dropped by 2.2%.

Conclusions

Our method performs well on COVID‐19 CT image segmentation. We also find that our method is so portable that can be suitable for various current popular networks. In a word, our method can help screen people infected with COVID‐19 effectively and save the labor power of clinicians and radiologists.

1. INTRODUCTION

Corona virus disease 2019 (COVID‐19) is acute respiratory infection caused by novel coronavirus. The pandemic COVID‐19 is still to have a devastating impact on the people health and well‐being of the global population. A key step in fighting against COVID‐19 is effective screening of infected patients, which allows the isolation of infected individuals for immediate treatment to mitigate the spread of the virus. Nowadays, the primary screening method for detecting COVID‐19 cases is the reverse transcription polymerase chain reaction (RT‐PCR) ¹ testing but is a time‐consuming, laborious, and complicated manual process. Furthermore, the sensitivity of RT‐PCR testing is highly variable ² , ³ and the positive rate of RT‐PCR decreases over time. ⁴ , ⁵

Another method that has been used to screen for COVID‐19 is radiography examination (e.g., chest X‐ray [CXR] or computed tomography [CT] imaging). Guan et al. found that COVID‐19 infectors have imaging radiographic abnormalities in CXR and CT images (e.g., ground‐glass opacity [GGO], bilateral abnormalities, and interstitial abnormalities). ⁶ In contrast to CXR, CT screening is very popular due to its three‐dimensional view of the lung. On CT images, areas of COVID‐19 infection can be distinguished by pulmonary GGO in the early stages of COVID‐19, and by solid lung lesions in the later stages. ⁷ Many research argued that the sensitivity of CT is higher than and RT‐PCR for COVID‐19 detection. ⁸ , ⁹ , ¹⁰ Therefore, CT was an effective supplementary method for RT‐PCR to detect COVID‐19.¹¹ However, in the clinicians’ practice community, CT method routinely relies on the radiologist depiction of the infection areas; this is a highly subjective task, influenced by clinician bias and experiences. Fortunately, automatic segmentation technology can reduce the labor intensity of radiologists, improve the accuracy of COVID‐19 diagnosis, and save precious time for the patients. Therefore, it is necessary to assist radiologists in labeling lung infections by using automatic segmentation techniques.

Nowadays, some of CT image segmentation methods have emerged, and convolutional neural network (CNN) method has been a hot topic. Keshani et al. utilized the support vector machine classifier to detect the lung nodule from CT slices. ¹¹ Wang et al. ¹² proposed a data‐driven model CF‐CNN, and it can extract different sensitive features from 3D and 2D images for CT image segmentation. Jiang et al. ¹³ put forward two multi‐resolution residual networks, which can combine image resolution and image characteristics to segment tumors. Tian et al. ¹⁴ introduced the background and development of instance segmentation technology and commonly used datasets in this field. Undoubtedly, these CT image segmentation methods mentioned earlier improve the segmentation performance of specific images to a certain extent.

Recently, some methods ¹⁵ , ¹⁶ , ¹⁷ , ¹⁸ , ¹⁹ for COVID‐19 have been proposed. Wang et al. ¹⁹ proposed a modified inception neural network for classifying COVID‐19 patients and normal ones. Chen et al. ²⁰ collected 46 096 CT image slices from COVID‐19 patients and control patients with other diseases; these CT images were used to train U‐Net++ ²¹ to identify COVID‐19 patients. Fan et al.22 put forward the COVID‐19 pneumonia infection area segmentation network, Inf‐Net: leveraging parallel partial decoder to aggregate high‐level features, and exploiting reverse attention to enhance expressiveness. Saood et al. ²³ utilized two known deep learning networks (SegNet ²⁴ and U‐Net ²⁵ ) for image tissue classification. Though these previous methods perform well in COVID‐19 CT image segmentation, there are still many incorrectly predicted regions (false positive and false negative regions) and many unclear edges. This is because there are several challenging issues in COVID‐19 CT images segmentation as follows:

1. The infected regions in COVID‐19 CT images are usually very small; and extremely similar to its background area (see Figure 1).

2. COVID‐19 training datasets, especially CT images with ground truth (GT), are often too small to train a high‐quality deep learning network.

FIGURE 1.

Examples of corona virus disease 2019 (COVID‐19)‐infected regions in computed tomography (CT) (a) and infected regions ground truth (GT) map (b)

To address issues mentioned earlier, we take the Res2Net as the backbone and elaborate a network Ref‐Net to implement automatic COVID‐19 CT image segmentation. Literatures ²⁶ , ²⁷ pointed out that the low‐level convolutional layer contains those edge spatial information features, whereas the high‐level convolutional layer retains those global semantic information features. Therefore, in the lower layer of the backbone network, we employ edge attention module (EAM) to extract edge features and generate a prediction map with clearer boundaries; in the higher layer, we well‐designed attention position module (APM) to acquire the global prediction map, which can detect the whole object more accurately. It is difficult to train a high‐quality network because there are few labeled data during training. This leads to false negative and false positive areas in the prediction results. We proposed a multi‐scale context exploration module (CEM) based on the context feature learning mechanism to gradually remove false positives and false negatives. Therefore, CEM can solve the problem of insufficient accuracy of network model caused by less training data. All this can ensure that our COVID‐19 CT image segmentation results have not only clearer boundaries but also less false predictions.

2. MATERIALS AND METHODS

The network framework of our COVID‐19 CT image segmentation network (Ref‐Net) is shown in Figure 2. In Figure 2, we take Res2Net as the backbone network to extract multilayered features f ₁,f ₂,f ₃,f ₄,f ₅. Because low‐level features retain the edge information of the target object, we extended f ₂ feature by using an EAM to generate the edge feature map $M_e$. High level is rich in the global positioning semantic information of the target object, so we input f ₅ into the attention positioning module (APM) that is composed of channel attention and spatial attention to get the preliminary prediction map M ₅. The three CAMs gradually combine the current features and upper‐layer features. In the end, CAM can eliminate false positives and false negatives in the prediction results, and the refined final prediction map is thus obtained.

FIGURE 2.

Network structure diagram of Ref‐Net

2.1. Backbone (Res2Net)

Bottleneck block (see Figure 3a) is the basic building block of many modern backbone CNNs architectures, for example, ResNet, ²⁸ ResNeXt. ²⁹ Res2Net ³⁰ block (see Figure 3b) has expanded the range of the receptive field in each layer and can represent multi‐scale features at a finer granularity. Especially, Res2Net replaces the 3 × 3 filters of Figure 3a with smaller filters, while connecting different filter groups in a hierarchical residual‐like style.

FIGURE 3.

The difference between bottleneck block and Res2Net block

As shown in Figure 3b, the feature maps are uniformly divided into s subsets of feature maps after $1\times 1$ convolution, denoted by $x_i$, where $i\in \{1,2,\text{…},\mathrm{s}\}$; each $x_i$ (except for x ₁) has a corresponding 3 × 3 convolution, denoted by $k_i$; $y_i$ represents the output of $k_i$, and it can be expressed by the following equation:

$$y_i=\left\{\begin{array}{c}{x}_{i}i=1\hfill \\ k_i\left(x_i\right)i=2\hfill \\ k_i\left(x_i+y_{i-1}\right)2<i\le s\hfill \end{array}\right..$$ (1)

Finally, in order to better fuse the information of different scales $y_i$, all of them are connected to the Res2Net block via a $1\times 1$ convolution.

2.2. Edge attention module (EAM)

From several existing literatures, ³¹ , ³² , ³³ , ³⁴ we can see that the edge feature of the infected regions is not taken part in the COVID‐19 image segmentation, which is actually the main cause to the blur boundaries. Hence, we introduced the EAM to focus on the edge features of the target. High‐level features retain semantic information that can provide abstract descriptions, but they have less verbosely information. Low‐level features pay more emphasis on spatial information that can construct object boundaries. Therefore, adding the EAM to the lower layer can better pay attention to the edge features.

We add the EAM after a more appropriate lower layer f ₂ to lay emphasis on edge features. Specifically, the EAM is composed of a convolutional layer with one convolution kernel, via which the edge feature map ($M_e$) can be obtained. At the same time, the true‐value edge feature map ($M_G$) can be generated by using the derivation of the GT. Therefore, the final clearer boundary can be extracted by comparing $M_e$ with $M_G$ to correct $M_e$ iteratively. In detailed implementation, we measured the dissimilarity between the edge feature map ($M_e$) and the true‐value edge feature map ($M_G$), which is constrained by a standard binary cross‐entropy loss function $L_e$. The loss function could be computed from the following equation:

$$\begin{array}{c}\hfill L_e=-\sum _{x=1}^w\sum _{y=1}^h\left[M_G\log \left(M_e\right)+\left(1-M_G\right)\log \left(1-M_e\right)\right],\end{array}$$ (2)

where w and h represent the width and height of the feature map, respectively.

2.3. Attention position module (APM)

As is well known, attention mechanism can selectively highlight the important areas of an image. Channel attention lays emphasis on which channel features are more meaningful ³⁴ , ³⁵ ; spatial attention pay attention to which area features are more meaningful. ³⁶ , ³⁷ Combining channel attention and spatial attention can make the network focus on more meaningful features and locate targets more accurately. The structure diagram of APM is shown in Figure 4.

FIGURE 4.

Attention position module structure diagram

From Figure 4, we can see that the structure diagram of APM is composed of channel attention and spatial attention, and its purpose is to enhance high‐level global semantic information and then obtain preliminary segmentation results. The high‐layer output f ₅ is exactly the input feature F of the APM. The channel number, height, and width of feature F are denoted as C, H and W, respectively. First, the input feature F is reshaped to obtain Q, K, and V (K, Q, and V represent for query, key, and query, respectively, and K, Q, V are abstract concepts used to calculate attention ³⁷ ), where $\{Q,K,V\}\in R^{C\times N}$, and N refers to the number of pixels. Matrix multiplication (MM) is used between Q transposed and K, and the result passes through the softmax layer to generate the channel attention map $X\in R^{C\times C}$. The influence of the channel j on the channel i in Figure X is specifically expressed as the following equation:

$$x_{ij}=\exp \left(\frac{\exp \left(Q_{i:}·K_{j:}\right)}{{\sum }_{j=1}^C\exp \left(Q_{i:}·K_{j:}\right)}\right),$$ (3)

where $Q_{i:}$ represents the row i of matrix Q, and $K_{j:}$ represents the row j of matrix K.

Thus, we performed MM between V and X after transposing and resized the resulting feature shape into $R^{C\times H\times W}$. In addition, we also introduced a proportional parameter γ, which is learnable, with an initial value of 1 and constantly learning to update the weight. Finally, the final channel attention output feature $F^{\prime }$ is obtained by jumping connection. Any line of the output feature $F^{\prime }$ can be expressed by the following equation:

$$F_{i:}^{\prime }=\gamma \sum _{j=1}^C\left(x_{ij}{V}_{j:}\right)+F_{i:},$$ (4)

where $F_{i:}^{\prime }$ represents the row i of the channel attention output feature $F^{\prime }$; γ is the proportional parameter; $V_{j:}$ represents the row j of the value V matrix; and $F_{i:}$ indicates the channel attention enter the row i of the feature.

The specific process of spatial attention is similar to channel attention. $F^{\prime }$ is the input of spatial attention, and it undergoes three 1 × 1 convolutions and changes the shape to obtain a new query ($Q^{\prime }$), key ($K^{\prime }$), and value ($V^{\prime }$). It is worth noting that $\{Q^{\prime },K^{\prime }\}\in R^{C_1\times N}$, $C_1=C/8$, and $V\in R^{C\times N}$. MM exists between the transposition of $Q^{\prime }$ and $K^{\prime }$, then it is normalized to get the spatial attention map $X^{\prime }\in R^{N\times N}$. Different from channel attention, the influence of the position j on the position i in spatial attention is calculated from the following equation:

$$x_{ij}^{\prime }=\exp \left(\frac{\exp \left(Q_{:i}^{\prime }·K_{:j}^{\prime }\right)}{{\sum }_{j=1}^N\exp \left(Q_{:i}^{\prime }·K_{:j}^{\prime }\right)}\right),$$ (5)

where $Q_{:i}^{\prime }$ represents the column i of query $Q^{\prime }$, and $K_{:j}^{\prime }$ represents the column j of key $K^{\prime }$.

The following process is similar to channel attention. The transposition of $X^{\prime }$ and $V^{\prime }$ is subjected to MM to change the shape of the result. Introduce the proportional parameter ${\gamma }^{\prime }$ and then jump and connect to obtain the final output $F^{\prime \prime }$.

2.4. Contextual exploration module (CEM)

Contextual feature learning plays an important role in many computers vision tasks, and many works have exploited contextual information that can enhance feature representation. ³⁸ , ³⁹ , ⁴⁰ , ⁴¹ , ⁴² , ⁴³ , ⁴⁴ That is to say, in the task of image segmentation, context information has strong spatial constraints. Namely, in the process of image segmentation, some ambiguity regions in the prediction results can be removed by context feature information. As is mentioned before, the infected regions of COVID‐19 are small and highly similar to the background (see Figure 1a), which always results in false positive predictions and false negative predictions in the image segmentation. To solve this problem, we designed a CEM composed of two parallel explore blocks (EBs), so it can find and eliminate both false positive and false negative regions and reach a more accurate segmentation result.

The structure diagram of the CEM is illustrated in Figure 5. As shown in Figure 5 (left), First, we up‐sampling the high‐level prediction maps $M_{high}$ and normalize it with a sigmoid layer. Subsequently, we use this normalized map to multiply the current level feature $F_c$. Thus, the foreground feature can be further expanded to extract the attention foreground feature $F_{fore}$. Similarly, we reverse the normalized map to multiply the current level feature $F_c$ to extract the attention background feature $F_{back}$. $F_{fore}$ and $F_{back}$ are sent into two parallel EB to detect the false positive regions $F_{fp}$ and false negative regions $F_{fn}$ from the prediction results. After detecting the false positive regions $F_{fp}$ and the false negative regions $F_{fn}$, we can perform as follows:

$$F_1=U\left(CBR\left(F_h\right)\right),$$ (6)

$$F_2=BR\left(F_1-\alpha F_{fp}\right),$$ (7)

$$F_{out}=BR\left(F_2+\beta F_{fn}\right),$$ (8)

where $F_h$ is the upper‐level input feature; $F_{out}$ is the output feature after false positives and false negatives are eliminated; C, B, and R represent convolution, normalization, and ReLU, respectively; U represents upsample; both α and β are learnable parameters. The element‐wise subtraction of F ₁ and $\alpha F_{fp}$ can eliminate false positives, and the element‐wise addition of F ₂ and $\beta F_{fn}$ can eliminate false negatives. ⁴³ , ⁴⁴

FIGURE 5.

Contextual exploration module structure diagram

As shown in Figure 5 (right), the EB consists of four branches, and the structure of the four branches is similar: 3 × 3 convolution is used for channel reduction; $K_i\times K_i$ convolution is adopted for local feature extraction ($K_1=1,K_2=3,K_3=5,K_4=7$). The size of the convolution kernel of the dilated convolution is 3 × 3, and the expansion rate is $r_i$, which is used for context awareness. It should be noticed that batch normalization (BN) and ReLU must be performed after each convolution. Each branch must be entered into the next branch for further processing. Finally, the output results of the four branches are superimposed in the channel dimension. The structure of the EB can enrich the context exploration capabilities to discover false positives and false negatives in the prediction results.

2.5. Loss function

The weighted IOU loss function $L_{IoU}^w$ is different from other IOU loss functions. In order to highlight the importance of difficult samples, the weight w of difficult sample points is increased. The weighted binary cross‐entropy $L_{BCE}^w$ focuses on the difficult sample points to assign weights to them in order to highlight their salience, instead of assigning weights to all pixels in the difficult samples. The definitions of these losses are the same as those ones in Refs. [45, 46], and their effectiveness has been verified. We introduce a loss function to combine $L_{IoU}^w$ and $L_{BCE}^w$ as follows:

$$L_s=L_{IoU}^w+\mu L_{BCE}^w,$$ (9)

where μ represent the weight, and we set it to 1; $L_{IoU}^w$ and $L_{BCE}^w$ provide global supervision and local supervision, respectively, and the segmentation results are more accurate.

We also supervise the output map M ₅ by using APM and the outputs M ₄,M ₃,and M ₂ of the three CEMs. The loss function could be computed from the following equation:

$$L_i=L_s\left(GT,M_i^{up}\right),$$ (10)

where GT is the true value map of the infection area segmentation; $M_i^{up}$ represents the up‐sampling of the picture $M_i$ to the same size as the true value map GT.

In addition, we use $L_e$ (see Equation 2) to monitor the output of the EAM. Therefore, the total loss function is calculated from the following equation:

$$L_{total}=\sum _{i=2}^{i=5}{L}_i+L_e.$$ (11)

2.6. Evaluation metrics

We use three popular metrics to evaluate our model: the Dice similarity coefficient, sensitivity (Sen), and specificity (Spec). We also introduce three golden metrics for object detection: structure measure ($S_{\alpha }$), enhance‐alignment measure ($E_{\phi }$), and mean absolute error (MAE). During the evaluation, we take M ₂ upsample as the final prediction map $M_f$ and compare it with the segmentation truth map $M_{GT}$.

Dice similarity coefficient (Dice) is used to calculate the similarity of two samples. For the best segmentation result, it is 1 and the worst is 0. It can be calculated as follows:

$$Dice=\frac{2TP}{FP+2TP+FN},$$ (12)

where TP represents the area that is predicted to be infected and actually infected; TN means the area that is predicted to be the background and it is actually the background; FP indicates the part that it is predicted to be an infected area but is in fact the uninfected area; FN stands for the part that is predicted to be the background, which is actually the infected area.

Sensitivity (Sen) is the proportion of the predicted infected area to all infected areas. Sen is calculated as follows:

$$Sen=\frac{TP}{TP+FN}.$$ (13)

Specificity (Spec) is the proportion of the recognized background versus the total background. It can be calculated as follows:

$$Spec=\frac{TN}{FP+TN}.$$ (14)

Structure measure ($S_{\alpha }$) can measure the structural similarity between the prediction map ($M_f$), and the truth map ($M_{GT}$), and be calculated from Equation (15):

$$S_{\alpha }=\left(1-\alpha \right)\times S_o+\alpha \times S_r,$$ (15)

where α refers to the balance coefficient, which is set to 0.5 by default; $S_o$ represents the object level similarity, and $S_r$ denotes the regional level similarity.

Enhance‐alignment measure ($E_{\phi }$) measures the local and global similarity between $M_f$ and $M_f$ at the same time, and it can be expressed as follows:

$$E_{\phi }=\frac{1}{w\times h}\sum _x^w\sum _y^h\varnothing \left(M_f\left(x,y\right),M_{GT}\left(x,y\right)\right),$$ (16)

where w and h are the width and height of the truth map $M_{GT}$, respectively; (x, y) means the pixel coordinates, and ∅ represents the enhanced alignment matrix. We report the mean value of all $E_{\phi }$ in the experiment, denoted as $E_{\varnothing }^{mean}$.

MAE indicates the average of the absolute errors between the predicted value and the observed value. We employ MAE to measure the error between $M_f$ and $M_f$ at the pixel level, which is defined as

$$MAE=\frac{1}{w\times h}\sum _x^w\sum _y^h\left|M_f\left(x,y\right),-\left(x,y\right)\right|.$$ (17)

3. RESULTS

Our model is implemented under the PyTorch framework on the Ubuntu V20.04 distribution. The hardware environment includes CPU, Intel i7‐4790; GPU, NVIDIA GTX TITAN X (12G); RAM, 32G.

3.1. Experimental details

3.1.1. Dataset

The dataset we used consists of 100 labeled CT slices from the COVID‐19 CT segmentation dataset. ⁴⁷ All the CT images were from more than 40 COVID‐19 patients and collected by the Italian Society of Medical and Interventional Radiology. A radiologist segmented the CT images using different labels for identifying lung infections. It is the first open‐access COVID‐19 dataset for lung infection segmentation, but with a small size. Inspired by literature, ²¹ we divided 100 labeled CT images into train, validation and test datasets, which consists of 45 CT images randomly selected as training samples, 5 CT images for validation, and the remaining 50 images for testing. CT slices do not have a uniform resolution. Before training, we uniformly resize the resolution of all CT slices to 352 × 352.

3.1.2. Training parameters

Because these CT images do not have a uniform resolution, we have to resize them to 352 × 352 before training. We train Inf‐Net using a multi‐scale strategy. ³³ Specifically, we first resample the training images using different scaling ratios, that is, (0.75, 1, 1.25), and then train Inf‐Net using the resampled images, which improves the generalization of our model. In addition, the learning rate is set to 1e − 4. The batch size is 2, and the training epoch is 100.

3.2. Comparative experiment

To evaluate our method comprehensively, we compared it with the state of the arts: U‐Net, ²⁵ U‐Net++, ²² Attention U‐Net, ⁴⁸ Gated‐UNet, ⁴⁹ Dense‐UNet,⁵⁴ and Inf‐Net. ²¹ In order to compare the performance of these related networks fairly, all of them are trained with the same network parameter settings. In addition, we evaluate our method on six golden evaluation metrics mentioned earlier: The smaller the average absolute error (MAE) value is, the better the performance is; the larger the other index values is, the better the performance is. The segmentation results of the COVID‐19 infection area are listed in Table 1, and the best results of each metrics are marked in bold.

TABLE 1.

Quantitative results of infection area segmentation

Methods	Backbone	Dice	Sen	Spec	$S_{\alpha }$	$E_{\varnothing }^{mean}$	MAE
U‐Net	VGG16	0.439 ± 0.007	0.534 ± 0.006	0.858 ± 0.007	0.622 ± 0.004	0.625 ± 0.002	0.186 ± 0.015
Attention U‐Net	VGG16	0.583 ± 0.008	0.637 ± 0.01	0.921 ± 0.003	0.744 ± 0.004	0.739 ± 0.003	0.112 ± 0.01
Gated‐UNet	VGG16	0.623 ± 0.005	0.658 ± 0.008	0.926 ± 0.006	0.814 ± 0.003	0.814 ± 0.002	0.102 ± 0.013
Dense‐UNet	DenseNet161	0.515 ± 0.006	0.594 ± 0.005	0.840 ± 0.002	0.662 ± 0.001	0.662 ± 0.005	0.184 ± 0.01
U‐Net++	VGG16	0.581 ± 0.01	0.672 ± 0.007	0.902 ± 0.003	0.720 ± 0.002	0.720 ± 0.004	0.120 ± 0.015
Inf‐Net	Res2Net	0.682 ± 0.008	0.692 ± 0.012	0.943 ± 0.004	0.781 ± 0.002	0.838 ± 0.002	0.082 ± 0.008
Ref‐Net (ours)	Res2Net	0.755 ± 0.005	0.751 ± 0.009	0.959 ± 0.002	0.795 ± 0.001	0.919 ± 0.01	0.060 ± 0.006

Abbreviation: MAE, mean absolute error.

From Table 1, it can be seen that our proposed network (Ref‐Net) performs well. Compared with the latest COVID‐19 segmentation model Inf‐Net, the Dice similarity coefficient of our model has increased by 7.3%; the sensitivity (Sen) has increased by 5.9%, and the other evaluation indicators: Spec, $S_{\alpha }$, $E_{\varnothing }^{mean}$ have increased by 1.6%, 1.4%, and 8.1%, respectively; meanwhile, the MAE has dropped by 2.2%.

The PR (precision‐recall) curve of U‐Net, U‐Net++, Inf‐Net, and our Ref‐Net is shown in Figure 6a. If the PR curve of one network X surrounds the PR curve of another network Y, it can be concluded that the performance of network X is better than Y. Our algorithm in Figure 6a is represented by a red bold line. It can be seen that the PR curve of ours surrounds other networks, so our network is better than others. Figure 6b shows the evaluation result of another evaluation metric: F‐measure. F‐Measure weighs the precision rate and the recall rate and averages them to consider them comprehensively, because the precision rate and recall rate sometimes conflict. Obviously, the most commonly used evaluation method is F‐measure, sometimes. Therefore, we use both PR curve and F‐measure curve to show the outperformance of our algorithm. From Figure 6b, we can also see that the F‐measure curve of ours still surrounds other networks, so our network is better than others indeed.

FIGURE 6.

Precision–recall curve (a) and F‐measure (b)

To further evaluate our method, some representative results of the COVID‐19 CT image segmentation are shown in Figure 7. From Figure 7, we can see that our method has clearer boundaries and has fewer false negatives and false positives. From Figure 7, it is very intuitive to see that our method is better, and the segmentation results are more accurate.

FIGURE 7.

Visual comparison of infection area segmentation result

We also found that our algorithm has good portability and is suitable for various current popular networks. To prove this viewpoint, we further compare our network with Inf‐Net. We replaced Inf‐Net and Ref‐Net with other backbone networks on the same dataset, and the COVID‐19 CT image segmentation results are shown in Table 2.

TABLE 2.

Compares the performance of Inf‐Net with ours (Ref‐Net) on different backbone networks

Methods	Backbone	Dice	Sen	Spec	$S_{\alpha }$	$E_{\varnothing }^{mean}$	MAE
Inf‐Net	VGG16	0.695 ± 0.006	0.705 ± 0.01	0.930 ± 0.003	0.760 ± 0.003	0.824 ± 0.005	0.075 ± 0.006
Ref‐Net	VGG16	0.748 ± 0.004	0.719 ± 0.008	0.961 ± 0.002	0.782 ± 0.002	0.910 ± 0.007	0.059 ± 0.004
Inf‐Net	ResNet	0.680 ± 0.007	0.695 ± 0.01	0.940 ± 0.002	0.784 ± 0.003	0.835 ± 0.006	0.073 ± 0.007
Ref‐Net	ResNet	0.754 ± 0.005	0.743 ± 0.007	0.960 ± 0.001	0.791 ± 0.001	0.914 ± 0.005	0.059 ± 0.004
Inf‐Net	Res2Net	0.682 ± 0.008	0.692 ± 0.012	0.943 ± 0.004	0.781 ± 0.002	0.838 ± 0.002	0.082 ± 0.008
Ref‐Net	Res2Net	0.755 ± 0.005	0.751 ± 0.009	0.959 ± 0.002	0.795 ± 0.001	0.919 ± 0.01	0.060 ± 0.006

Abbreviation: MAE, mean absolute error.

In Table 2, to verify the superiority of Res2Net as the backbone network, we replaced the backbone network of ours with VGG16 or ResNet by using the same network parameter settings. We can see that the result of Res2Net as the backbone network is the best. Furthermore, we carried out a comparative experiment of Ref‐Net and Inf‐Net using different backbone networks. We can see that our method has a significant improvement in Dice, $E_{\varnothing }^{mean}$, and MAE compared to Inf‐Net. Therefore, we can conclude that our algorithm is still superior to Inf‐Net when using a different backbone network.

3.3. Ablation experiment

In order to verify the effectiveness of the attention positioning module (APM), we discuss the location of APM in this subsection. On the basis of Ref‐Net, we add the attention positioning module to the feature layers f ₂, f ₃, f ₄, and f ₅, respectively, and try to find the best position of the APM.

The working procedures of ablation experiments are as follows (take the addition of the attention positioning module in f ₄ as an example): Without changing the position and number of other modules, the attention positioning module after the feature layer f ₅ is deleted at first. The feature layer f ₅ and its directly generated prediction map are used as the input of the first CEM, which replaces the feature layer and prediction of f ₅ after the attention location module in the original model. Subsequently, f ₄ is input into the APM, and its accordingly output is regarded as the low‐level input of the first context exploration. By following the previous process, attention positioning modules are added to f ₂ and f ₃. The results are shown in Table 3 (the best results are marked in red). It can be seen from Table 3 that adding the attention positioning module (APM) at a higher level (f ₅) has the best performance. This is because that the high level of the CNN is rich in the global positioning semantic information of the object and can locate the target object more comprehensively and accurately.

TABLE 3.

Ablation experiment of attention position module

Module Setting	Dice	Sen	Spec	$S_{\alpha }$	$E_{\varnothing }^{mean}$	MAE
f 2 + APM	0.747	0.720	0.948	0.788	0.871	0.071
f 3 + APM	0.743	0.725	0.953	0.787	0.913	0.066
f 4 + APM	0.746	0.736	0.951	0.790	0.915	0.068
f 5 + APM	0.755	0.751	0.959	0.795	0.919	0.060

Abbreviation: MAE, mean absolute error.

We also conducted an ablation experiment to ensure that the effectiveness of each module. The experimental results are shown in Table 4.

TABLE 4.

Ablation experiment of network model

Module setting	Dice	Sen	Spec	$S_{\alpha }$	$E_{\varnothing }^{mean}$	MAE
(No. 1) Backbone	0.442	0.570	0.825	0.651	0.569	0.207
(No. 2) Backbone + EAM	0.548	0.727	0.765	0.673	0.661	0.231
(No. 3) Backbone + APM	0.551	0.504	0.943	0.679	0.723	0.113
(No. 4) Backbone + CEM	0.626	0.608	0.953	0.736	0.790	0.086
(No. 5) Backbone + EAM + APM	0.605	0.618	0.933	0.711	0.807	0.098
(No. 6) Backbone + EAM + CEM	0.751	0.727	0.962	0.782	0.907	0.058
(No. 7) Backbone + APM + CEM	0.626	0.592	0.969	0.769	0.794	0.082
(No. 8) Backbone + EAM + APM + CEM	0.755	0.751	0.959	0.795	0.919	0.060

Abbreviations: APM, attention position module; CEM, context exploration module; EAM, edge attention module; MAE, mean absolute error.

In Table 4, we take Res2Net as the backbone, whose evaluation metrics are shown in row 1. In order to evaluate the effectiveness of a single module, the EAM, attention positioning module (APM), and CEM were added to the backbone network Res2Net, respectively, whose corresponding results are listed in rows 2–4 of Table 4, respectively. It can be seen that the performance of adding any module could have been improved. That is to say, the EAM, APM, and CEM are all effective actually. Then we arrange and combine the various modules to find the best collocation model. The results are shown in rows 5–8. From Table 4, it can be concluded that the network we designed (no. 8) performs best.

4. DISCUSSION

CT is considered a low‐cost, accurate, and efficient diagnostic tool for screening and diagnosis of COVID‐19. It can assess the severity of lung infection. Therefore, the accurate segmentation of infected areas from CT images by automatic segmentation technology can help doctors to quickly screen COVID‐19 patients and reduce the workload of doctors.

We designed CT image segmentation network using Res2Net as the backbone network, and EAM, APM, and CEM were used to further improve the segmentation results. According to literature 26, low‐level features in shallower convolutional layers preserve spatial information for representing edges, whereas deep convolutional layers preserve semantic information for locating objects. Therefore, we send the low‐level features f ₂ into the EAM, while feeding the high‐level features f ₅ into our designed APM. But, the COVID‐19 infection area is small and blurred resulting in false negatives and false positives in the segmentation results. Fortunately, context can enhance the ability of feature expression and can eliminate false negatives and false positives to great extent. Therefore, we carefully design a parallel CEM to gradually output accurate segmentation results. Experimental results in Table 1 and Figure 7 both demonstrate the superiority of our method.

From Figure 7, we can easily see that the segmentation result of Inf‐Net is better than U‐Net and U‐Net++, because Inf‐Net has own reverse attention. But there are still many ambiguities in the prediction results of Inf‐Net; moreover, it does not pay attention to the low‐level boundary information that leads to unclear boundaries appeared. By contrary, the segmentation results of our method are clearer than those of Inf‐Net. On the one hand, the results of our method remove the ambiguity area in the Inf‐Net prediction result. On the other hand, some uninfected areas are segmented (false positives), whereas some infected areas are not detected (false negatives) in the Inf‐Net segmentation results. But our algorithm Ref‐Net eliminates these false positive regions and false negative regions to the greatest extend, which is undoubtedly contributed to the more accurate final results. In our viewpoint, this is because there are three CEMs in our model to remove false positives and false negatives altogether.

In future work, we plan to use semi‐supervised learning ⁵¹ , ⁵² , ⁵³ to further improve the accuracy of COVID‐19 CT image segmentation. In addition, the trust of doctors and patients is required to finally apply deep models to clinical practice. Therefore, the research on the interpretability ⁵⁴ of deep models is also essential.

5. CONCLUSION

We propose a segmentation network for segmenting COVID‐19‐infected regions in CT images. On the public datasets, our network performs very well. Our method can automatically segment infected areas in COVID‐19 CT images effectively and efficiently, helping clinicians to screen infected patients, and reducing their burden.

Liu S, Tang X, Cai T, Zhang Y, Wang C. COVID‐19 CT image segmentation based on improved Res2Net. Med Phys. 2022;1‐13. 10.1002/mp.15882