Fully feature fusion based neural network for COVID-19 lesion segmentation in CT images

Abstract

Coronavirus Disease 2019 (COVID-19) spreads around the world, seriously affecting people’s health. Computed tomography (CT) images contain rich semantic information as an auxiliary diagnosis method. However, the automatic segmentation of COVID-19 lesions in CT images faces several challenges, including inconsistency in size and shape of the lesion, the high variability of the lesion, and the low contrast of pixel values between the lesion and normal tissue surrounding the lesion. Therefore, this paper proposes a Fully Feature Fusion Based Neural Network for COVID-19 Lesion Segmentation in CT Images (F3-Net). F3-Net uses an encoder–decoder architecture. In F3-Net, the Multiple Scale Module (MSM) can sense features of different scales, and Dense Path Module (DPM) is used to eliminate the semantic gap between features. The Attention Fusion Module (AFM) is the attention module, which can better fuse the multiple features. Furthermore, we proposed an improved loss function $Los{s}_{Covid-BCE}$ that pays more attention to the lesions based on the prior knowledge of the distribution of COVID-19 lesions in the lungs. Finally, we verified the superior performance of F3-Net on a COVID-19 segmentation dataset, experiments demonstrate that the proposed model can segment COVID-19 lesions more accurately in CT images than benchmarks of state of the art.

1. Introduction

COVID-19 spreads fast and is highly contagious, so the early detection and diagnosis of COVID-19 can help prevent the disease. Currently, COVID-19 detection mainly uses real-time reverse transcriptase-polymerase chain reaction technology (RT-PCR) [1]. Although this technology has become the “gold standard” for diagnosing COVID-19, the strict test environment limits its test speed. More importantly, due to the high mutation rate and latency of COVID-19, resulting in high false negatives in the detection of this technology [2], [3]. And yet, in another respect, RT-PCR has always been used in screening stage and confirmed diagnosis, which does not work in the clinical treatment, especially for diagnosis of disease severity and subtype. In today’s clinical field, radiology imaging technology has become an important complementary role to RT-PCR [4]. Computed tomography (CT) images contain rich image semantics, which contains essential features such as the texture, shape, and graphics of the lesion [5]. Understanding the evolution of COVID-19 in CT images can be an essential basis of the early prevention, control, diagnosis, and treatment of COVID-19. Nowadays, most of the work marking the lesions in CT images is manually performed by doctors or professionals. Repetitive work has brought a massive workload to the staff [6]. In addition, manual operation is easily affected by the annotator’s clinical experience and subjective consciousness, so it is difficult to ensure the accuracy of the annotation. Therefore, there is an urgent need to use artificial intelligence technology to automatically segment the COVID-19 lesion area from the CT image, which is conducive to further judgment and evaluation of the subtypes of lesion, and can relieve the pressure of doctors on medical image recognition [7].

In recent years, convolutional neural network (CNN) has proved its good performance in image processing. Therefore, it is feasible to realize automatic segmentation of COVID-19 lesions in CT based on the CNN model. Moreover, there has been much work recently applying CNN to radioactive images for the diagnosis of COVID-19 [7]. For example, Wang et al. [8] established a public dataset based on X-ray images of COVID-19 and designed the first COVID-19 detection network COVID–Net. Wang et al. [9] firstly uses the pre-trained U-Net to segment the lung segment from the CT image and then input the lung segment into the weakly-supervised network for COVID-19 infection prediction.

Although many CNN networks have been proposed to solve the diagnosis problem of COVID-19, only a few of them focus on the segmentation of the COVID-19 lesion area in CT images. Because COVID-19 lesions have some difficulties that are not conducive to segmentation on CT images among different stages in the different patients:

• 1.The scales of lesions are not uniform. The area of the lesion is quite different. For example, the smaller lesion area is less than $1{\mathrm{cm}}^2$, and the largest lesion area is close to half of the lung lobe.
• 2.The morphology of lesions is variable. For example, the shape of a ground-glass lesion is close to a sponge, while the shape of a cord-like lesion is like a cord.
• 3.The contrast of CT images is unclear. The mutation period of the lesion is short, and the speed is fast. In the CT image, the pixel value of the lesion area and the organ area of the human body is almost equal, the position is similar, and the contrast is low.

Those problems have undoubtedly brought great difficulties to the segmentation task of COVID-19, but the existing solutions rarely focus on such problems. For example, weakly-supervised frameworks are used to train the model to solve the problem of fewer samples in the data set [9], [10]. Although some modules such as multi-scale and attention mechanism are added to the CNN model [11], [12], they may have overlooked the above issues. So, these questions for the segmentation of COVID-19 lesions are still a challenging task.

To solve the above problems, we firstly construct a dataset of CT images in this paper. The dataset contains 4472 CT images of COVID-19 patients. Each CT image has its corresponding saliency label of the lesion area. A new deep learning model F3-Net is designed to accurately segment the COVID-19 lesion area from the patients’ CT images. In addition, we also designed a loss function $Los{s}_{Covid-BCE}$ that pays more attention to the lesion area according to the distribution characteristics of the lesion pixels. We divide the constructed dataset into the training set, validation set, and test set, then put the training set to F3-Net, set the loss function to $Los{s}_{Covid-BCE}$ for training, and use the validation set to adjust the original network. Finally, the trained F3-Net is tested on the test set. The experimental results show that F3-Net has achieved good results on our dataset. In general, the contribution of this article lies in three aspects:

• 1.We designed a new COVID-19 lesion segmentation network F3-Net. F3-Net adopts the encoder–decoder architecture. In the encoder stage, a Multiple Scale Module (MSM) is used to extract different scales’ features. Because decoder features need to introduce encoder features to compensate for the loss of information, we designed a Dense Path Module (DPM) to reduce the semantic gap. To further realize the fusion of the two, we also designed an Attention Fusion Module (AFM). The AFM can dynamically adjust the weight of each channel of the feature and enhance the distinguishability. It is worth mentioning that both MSM and AFM can be added to the existing network architecture as independent units.
• 2.Based on the distribution characteristics of COVID-19 lesions in patients’ CT images, this work designed a new $Los{s}_{Covid-BCE}$ loss function. $Los{s}_{Covid-BCE}$ defines a weight function $q_{i,j}$ according to the distribution of the lesion pixels in the training set. By $q_{i,j}$, the points in the non-lesion area can be given a coefficient value greater than $1$. The label value of the points in the non-lesion area is $0$. During the training process, points in the non-lesion area can obtain a larger loss value to pay more attention to the lesion area and achieve more accurate segmentation.
• 3.Through experimental research on a dataset constructed from CT images of 358 COVID-19 patients, the results show that the $Los{s}_{Covid-BCE}$ loss function proposed in this paper can increase the model’s attention to the lesion area, and F3-Net can segment COVID-19 lesions more accurately in CT images than the benchmarks of state of the art. F3-Net can reduce the burden on clinicians to comment on the infected area of the lesion and increase the possibility of further research on the mechanism of the COVID-19 lesion.

The remaining paper is organized as follows. In Section 2, we briefly summarize the related works. We introduce the implementation details of F3-Net. Extensive experiments are conducted in Section 4 to evaluate the performance of F3-Net on COVID-19 datasets, with in-depth analysis. Section 5 concludes this work.

2. Related work

This section introduces some classic semantic segmentation methods, then reviews current COVID-19 lesion segmentation methods using the CNN models.

2.1. Semantic segmentation

The main task of semantic segmentation is the prediction of each pixel in the image and identifying the pixel area of different objects. It is an important part of the image processing field [13]. With the rise of the DCNN, many semantic segmentation models have been proposed in recent years. Fully Convolutional Networks (FCN) [14] is the most typical model among them. FCN uses multiple convolution operations to reduce the size of features and increase the number of channels. In the final process, the feature is mapped to the same size as the image through the convolution operation, and the number of channels is the number of classifications. Wang proposed a convolutional neural network and incorporate it into a semi-supervised learning framework [15]. Yang designed a multi query attention module which fuse the multi-scale feature [16]. DeepLab is another convolution neural network model which used the atrous convolution in the convolutional layer. Later, there are other versions of Deeplab were proposed [17], [18].

In the field of medical imaging, Ronneberger et al. [19] designed the U-Net which is used image segmentation. To U-Net adopted encoder–decoder architecture, and took skip connections to eliminate the semantic gap between features. Because a lot of medical data in clinical diagnosis is three-dimensional, Milletari et al. [20] developed a V-Net by using 3D convolution and residual block based on U-Net.

Zhou et al. [21] modified an U-Net＋＋, they built multiple U-Net networks with different layers, then skip-connected them, and finally determined the optimal architecture through pruning. In addition, U-Net had also been expanded to 3D U-Net [22], Attention U-Net [23], MultipeResUNet [24] for medical image segmentation.

2.2. Artificial intelligence for COVID-19

There have been many attempts to apply the CNN model to the detection and segmentation of COVID-19 lesions [7]. In terms of typical models, Xu et al. [25] employed ResNet [26] to distinguish COVID-19 from other cases. Wang et al. [27] connects two parallel 3D-ResNet branches to construct a prior-attention residual learning (parl) module, one for detection and the other one for classification during the module, then stack multiple parl modules to build an end-to-end model for COVID-19 detection. Cao et al. [28] employed U-Net [19] to segment the lungs from CT scans. Chen et al. [29] respectively used U-Net＋＋ for detection and segmentation of the infection lesions from CT scans. Han et al. [30] applied the attention mechanism to multiple instance learning complete the automatic segmentation of COVID-19 and increased interpretability of the network. Wu et al. [31] proposes a Sequential Region Generation Network (SRGNet), to jointly detect and segment the lesion areas of COVID-19. Hu et al. proposes a segmentation scheme by enhancing supervised information and fusing multi-scale feature maps of different levels based on the encoder–decoder architecture [32].

In terms of innovative models, Fan et al. [33] designed a COVID-19 lesion segmentation network Inf-Net based on CT images. The Edge Attention module (EA) was added to Inf-Net to make full use of the rough information of the lesion boundary. A Reverse Attention module (RA) was constructed to provide details of the lesion area. Zhao proposes a method that integrates a 3D V-Net with shape priors was proposed for medical image segmentation [34]. Ouyang et al. [35] used two different sampling strategies, uniform sampling (US) and size-balanced sampling (SS), for the dataset, and then inputted the sampled results into two branches with the same architecture. Each branch consists of a 3D ResNet34 followed by an attention module, then the results of two branches are applied to ensemble learning. Li et al. [36] developed the COVNet to detect COVID-19, CAP, and Non-Pneumonia. COVNet gets a set of CTs into multiple branches, each of which is ResNet50 and shares parameters, and then the results of each branch are merged using max pooling. Wu et al. [11] adopted the encoder–decoder architecture and used VGG16 as the backbone of the encoder, and then an Enhanced Feature Module (EFM) was added in the final stage of the encoder. An Attentive Feature Fusion (AFF) that payed more attention to small-size feature maps was constructed as a basic block in the decoder stage. In addition, to solve the problem of few high-quality annotation samples in CT images, Wang et al. [37], and Wang et al. [9] designed a semi-supervised framework for scans of COVID-19. And several CT scan diagnosis systems had been established for testing the suspected COVID-19 cases [38], [39].

3. Method

3.1. F3-Net framework

The overall framework of F3-Net is shown in Fig. 1. As we can see from Fig. 1, the overall architecture is U-shaped, which is divided into two parts, namely Encoder and Decoder. The Encoder extracts the features from the images by Base Module (BM) and Multiple Scale Module (MSM), and the Decoder restores the features to images by BM, Dense Path Module (DPM), and Attention Fusion Module (AFM).

Fig. 1.

The architecture of F3-Net, which adopts encoder–decoder architecture and mainly includes four types of modules, namely Base Module (BM), Multiple Scale Module (MSM), Dense Path Module (DPM), Attention Fusion Module (AFM). The input data is the original CT images, and the output results are the lesion areas.

The Encoder input is the preprocessed image, and the input images serially are passed through the BM and MSM modules to obtain $M_1$. $M_i$ performs maximum pooling, reduces the size of the feature maps to half of the original size, and then serially passes them through the BM and MSM modules to obtain the feature maps $M_{i+1}$. This process is repeated four times in total, and the feature map generated by the MSM module is {$M_i\mid i=2,3,4,5$}. The process is described as for in Eq. (1):

$$M_{i+1}=MSM\left(Base\left({\mathcal{P}}_m\left(M_i\right)\right)\right),i=1,2,3,4$$ (1)

In Eq. (1), $Base\left(\cdot \right)$ represents the BM in Fig. 1. $P_M\left(\cdot \right)$ is the maximum pooling with a 2 × 2 pooling window. $MSM\left(\cdot \right)$ is the MSM module.

The input of the decoder is $B_5$, $B_5=M_5$, $M_i$ output $D_i$ after DPM, $B_i$ increases the size of the feature map to twice of the original after upsampling, and then enters AFM together with $M_i$, and then passes BM to obtain $B_{i-1}$. This process is repeated four times in total, among which the features are generated by BM {$B_i\mid i=4,3,2,1$}. After the last output $B_1$ passes through Conv 1 × 1, the output feature representing the segmentation result map has the same resolution as the input images. The computing process for the above process is expressed as Eq. (2):

$$B_{i-1}=Base\left(AFM\left(\mathcal{U}(B_i,DPM\left(M_{i-1}\right))\right)\right),i=5,4,3,2$$ (2)

In Eq. (2), $Base\left(\cdot \right)$ represents the BM. $\mathcal{U}\left(\cdot \right)$ is the Bilinear interpolation algorithm. $AFM\left(\cdot \right)$ is the AFM.

3.1.1. Base module

BM is the basic unit in the model as usual, its main function is to extract features and modify the number of feature channels, which is composed of serial Conv3 × 3, BN (Batch Norm), Relu, Conv3 × 3, BN, and Relu. Its main structure is shown in Fig. 2.

Fig. 2.

Base Module is utilized to extract features mainly through serial convolution, batch normalization and relu operations. Its input and output are features.

3.1.2. Multiple scale module

The COVID-19 lesions in CT images have the characteristics of different sizes and shapes. Therefore, we designed a MSM as shown in Fig. 3, and added it after each Base Module in the encoder. MSM is inspired by ASPP [17] and PPM [40]. It is characterized using convolution modules with different receptive fields, so it can capture features of different scales, so as to increase the ability to perceive multi-scale lesions of F3-Net.

Fig. 3.

Multiple Scale Module is utilized to better deal with lesions at multiple scales. It mainly implements the extraction of features of different scales through two methods, dilated convolution and serial pooling, convolution, and up sampling. Different colored blocks $S_i$ in the figure represent features of different scales.

MSM is divided into two parts, $DU$ and $DC$. $DU$ consists of $U_N$ parallel branches. In each branch, $U$ first changes the size of the feature map to expand the receptive field through the average pooling operation, then changes the number of channels of the feature map to $U_R$ through the convolutional layer, and finally uses bilinear interpolation restores the feature map to its original size. After these branches, $U_N$ feature submaps {$S_1,S_2,\dots ,S_{U_N}$} can be obtained. $DC$ consists of $U_M$ parallel branches. In each branch, $U$ passes through dilated convolution with different dilation rates and changes the number of channels of the feature map to $U_R$, and then $U_M$ feature submaps {$S_{U_N+1},S_{U_N+2},\dots ,S_{U_N+U_M}$} are obtained. The difference between the dilated convolution and the classic convolution is that it enlarges the receptive field to ${(2r-1)}^2$ by setting the dilation rate $r$ without changing the number of parameters and the calculation cost.

Finally, $U$ and the feature submaps $\left\{S_1,S_2,\dots ,S_{U_N},\right)\left(S_{U_{N+1}},\dots ,S_{U_N+U_M}\right\}$ are joined together to obtain $S$, and the number of channels has increased $(U_N+U_M)U_R$. $S$ gets $M$ after Conv 3 × 3, and the dimension and size of $M$ and $U$ are the same, and then enter the next module.

3.1.3. Dense path module

To solve the information loss phenomenon caused by up sampling in the decoder stage, Ronneberger et al. [19] chooses to directly connect the features of the encoder to the corresponding decoder features, which enables the network to propagate directly from the encoder to the decoder. Ibtehaz et al. [24] indicates that the feature in the encoder is close to the original image and is a low-level feature, while the feature in the decoder is a high-level feature. There is a big semantic gap, and if the fusion is directly carried out, it may cause some influences that are not conducive to segmentation. Inspired by MultipleResUNet [24] and Densely Connected Convolutional [41], we designed a Dense Path Modul (DPM) as shown in Fig. 4 to narrow the feature gap between encoder and decoder. Therefore, the low-level feature $M_i$ generates $D_i$ after passing through the DPM in F3-Net, and then $D_i$ is connected with the high-level feature instead of direct fusion, which can reduce the semantic gap.

Fig. 4.

Dense Path Module is utilized to eliminate the semantic gap between encoder and decoder features. Its input is $M_i$ and its output is $D_i$. It contains several serial Dense Modules.

The DPM is composed of $N_D$ Dense Module (DM). Define $T_0=M_i$, each DM concats $T_j$ with $T_j$ after passing through DM to obtain $T_{j+1}$. $T_{j+1}$ has more $N_R$ than $T_j$ in the number of channels, where $0\le j\le N_D-1$ as shown in Eq. (3).

$$T_{j+1}=Concat(T_j,Base\left(T_j\right)),0\le j\le N_D-1$$ (3)

Finally, $T_{N_D}$ has more $N_D{N}_R$ than $M_i$ in the number of channels, so the dimension of $T_{N_D}$ is reduced to be consistent with $M_i$ through the Base Module to get $D_i$, and then they are imported into the next module, where $1\le i\le 4$ as shown in Eq. (4).

$$D_i=Base\left(T_{N_D}\right)$$ (4)

3.1.4. Attention fusion module

The feature fusion strategy used in [19] is concat function. Concat does not distinguish between the two features and directly connects the two features according to a certain dimension. It is like getting two strangers who have never met to work together. Inspired by [42], [43], we designed the Attention Fusion Module (AFM) to replace the concat operation to better integrate $D_i$ and $B_{i-1}$. AFM can make the feature map adaptively adjust the relationship between each channel and adaptively adjust the relationship between multiple feature maps.

Grouping convolution is used in various lightweight models to reduce the number of operations and parameters by dividing the input feature map equally into several groups by channels and then performing regular convolution on each group, while isolating the information exchange between different groups [44].

As shown in Fig. 5(a). There are two fusions in the AFM. In the first fusion, $B_{i-1}$ is bilinearly interpolated and then subjected to the same grouping convolution operation as $D_i$ to obtain feature maps {$A_1,A_2$} respectively. Then we get $A$ from $A_1$ and $A_2$ via an element-wise addition as shown in Eq. (5).

$$A=A_1\oplus A_2$$ (5)

where $\oplus$ is element-wise addition. In the second fusion, $A$ is pooled by the average pooling to obtain a vector $R$ of length $C$. Each element in $R$ represents the comprehensive statistical information of the corresponding channel in $A$. To further enable a certain channel in the feature map to focus on the information of other channels, $R$ is passed through a fully connected layer of size $C\times V$ to obtain a vector $T$ of length $V$ as shown as Eq. (6).

$$T={\mathcal{F}}_{CV}\left({\mathcal{P}}_A\left(A\right)\right)$$ (6)

${\mathcal{P}}_A\left(\cdot \right)$ represents average pooling, and ${\mathcal{F}}_{CV}\left(\cdot \right)$ represents a fully connected layer with length from $C$ to $V$. Then let $T$ goes through two fully connected layers of size $V\times C$, and the we get two vectors $t_1$ and $t_2$ of length $C$ as Eq. (7). $t_1$ and $t_2$ are passed into the $Softmax$ layer to generate two vectors $a_1,a_2$ of length $C$. {$a_1,a_2$} and feature map {$A_1,A_2$} are correspondingly element-wise prodcuted and then element-wise addition to get the $S$.

$$t_i=F_{VC}\left(T\right),i=1,2$$ (7)

$$\{a_i\mid i=1,2\}=Softmax\left(\{t_i\mid i=1,2\}\right)$$ (8)

$$S=a_1\otimes A_1\oplus a_2\otimes A_2$$ (9)

In Eq. (8) $Softmax$ is softmax function. In Eq. (9), $\otimes$ is element-wise product. The first fusion is preliminary and naive, and the two features pay attention to each other’s weights in the second fusion, which are more detailed and have an inter-channel attention mechanism. It is worth mentioning that the AFM is not limited to the fusion between the two features in this work. It can also be used to replace the concat operation of multiple features and can be embedded as a separate component in an existing network model, as shown in Fig. 5(b).

Fig. 5.

Attention Fusion Module contains an attention mechanism, which can make features more harmoniously merged. (a) is the application of AFM in F3-Net. (b) is an expanded form that can fuse multiple features.

3.2. Loss function with lesion location knowledge

Due to factors such as blurred borders and low contrast, the COVID-19 lesions in CT are often similar to normal human tissues, and even some lesions are connected with blood vessels and lung parenchyma. This makes it hard for the deep learning model to identify areas with similar pixels, shapes with COVID-19 lesions as normal tissue areas. Especially, these detected areas are often not even within the lung regions, so the segmentation accuracy is quite poor.

After collecting the clinical experiences from multiple doctors and observing the CT images of the cases, this work obtained the area $(256\times 256)$ by scaling all the labeled lesions in the training set and then element-wisely summarize all pixels. The statistical matrix $Z(256\times 256)$ of pixels of all lesions in the training set. Summary of $Z$ distribution characteristics is computed for later usage. As shown in Fig. 6(c), It is easy to see that the lesion area is concentrated in the left and right lung lobes, and the area other than the lung lobes does not contain the lesion. As shown in Fig. 6, (a) is the histogram after $Z$ is projected on the first dimension ($X$ axis of image). (b) is the histogram after $Z$ is projected on the second dimension ($Y$ axis of image).

Fig. 6.

Statistics of our dataset. (a)(b) The distribution map of lesion pixels in different dimensions. (c) The heat map of the pixels of the lesion. (d) The heat map of $Los{s}_{Covid-BCE}$.

According to the facts in Fig. 6(a), (b), and (c), we can verify the conclusion that the ROI (Region of Interest) in the training set is concentrated in the left and right lung lobes, and the rest are basically free of focus. Assuming that the distribution of lesions in the test set is the same as that in the training set, this paper designs a loss function $Los{s}_{Covid-BCE}$ according to the characteristics of lesions distribution with Eq. (10).

$$Los{s}_{Covid-BCE}=\frac{-1}{W\times H}\sum _{i=1}^W\sum _{j=1}^H(g_{i,j}\times ln\left(Sig\left(p_{i,j}\right)\right)+(1-g_{i,j})\times ln(1-Sig\left(p_{i,j}\right)))$$ (10)

The details of $g_{i,j},Sig\left(p_{i,j}\right)$ and $dist(i,j)$ in Eq. (10) is computed as Eq. (11).

$$g_{i,j}=\left\{\begin{array}{cc}\hfill \frac{dist(i,j)-e}{dist(W,H)-l}+1\hfill & \hfill dist(i,j)>e\hfill \\ \hfill 1\hfill & \hfill s<dist(i,j)<e\hfill \\ \hfill \frac{s-dist(i,j)}{s}+1\hfill & \hfill dist(i,j)<s\hfill \end{array}\right)$$

$$Sig\left(p_{i,j}\right)=min(1,q_{i,j}\times \frac{1}{1+e^{-p_{i,j}}})$$ (11)

$$dist(i,j)=\sqrt{{(i-\frac{W}{2})}^2+{(j-\frac{H}{2})}^2}$$

In Eq. (10), $p_{i,j}$ represents the result predicted by the F3-Net for the input CT image at the position $(i,j)$, $g_{i,j}$ represents the value at the position $(i,j)$ of the lesion image corresponding to the input CT image. $q_{i,j}$ is a piecewise function, representing the attention weight in $Los{s}_{Covid-BCE}$, $s,e$ are the start$\left(s\right)$ and end$\left(e\right)$ positions of the lesion area distribution in the training set, which must be obtained according to the distribution characteristics of the pixels in $Z$. Fig. 6(d) is the heat map of the weight of each position of $q_{i,j}$ in a 256 × 256 area.

The points in the non-lesion area in $p_{i,j}$ can be given a coefficient value greater than $1$ by $q_{i,j}$, and the label value of the points in the non-lesion area is $0$, so the larger the prediction value is, the greater the loss value will be. During the training process, the points in the lesion area can obtain more attention than points the non-lesions area, which can improve the segmentation accuracy.

4. Experiments and discussion

This section describes the dataset created in this paper and the implemented details of the method firstly. The performance evaluation is conducted, and general metrics are analyzed.

4.1. Experimental setting

4.1.1. Setting of neural network

In the experimental environment, the F3-Net is implemented on the pytorch platform. The type of GPUs in server is 8*GTX 1080Ti.

The specific parameters of the method are configured as follows. In the encoder stage, the size of the feature is changed through max pooling, and the number of channels in the feature is changed through Base Module. The size of $M_i$ is $M_1(64\times 256\times 256),M_2(128\times 128\times 128),M_3(256\times 64\times 64),M_4(512\times 32\times 32),M_5(512\times 16\times 16)$. The reason why the number of channels of $M_5$ is not doubled compared to $M_4$ but is consistent with it is that the up sampling method is a bilinear interpolation, which does not contain parameters and cannot change the number of channels compared to transposed convolution. It can save computing resources in the decoder stage. In the decoder stage, the scale of the feature is changed by bilinear interpolation. The sizes of $B_i$ are $B_5(512\times 16\times 16),B_4(256\times 32\times 32),B_3(128\times 64\times 64),B_2(64\times 128\times 128),B_1(64\times 256\times 256)$.

In the MSM module, we set $U_N=U_M=4$. In the four branches of the $DU$ part, the multiple mean pooling scales are $1,2,4,6$ and the method for up sampling is also bilinear interpolation. The sizes of the convolution kernels in the four branches of the $DC$ part are $1\times 1,3\times 3,3\times 3,3\times 3$ respectively. The dilation rates are $1,2,3,6$. We set $U_R$ to $1/16$ of the number of $U$ channels. MSM module can be easily extended to more branches, and it can be embedded as a separate component in the existing network model.

When $i$ in $M_i$ is smaller, $M_i$ is closer to the original image, and $B_i$ is closer to the segmentation result. Therefore, it can be seen that the semantic gap between $M_i$ and $B_{i+1}$ after up sampling is different, and as i becomes larger, the gap will gradually become smaller. So in the DPM, we set the values of $N_D$ in the DPM after $M_i$ are $4,3,2,1$. At the same time, the growth rate $N_R$ respectively are set as $8,16,32,64$.

In the AFM, because the dimensions of $D_i$ are different, the number of groups in the corresponding group convolution should not be static. Therefore, the size of the convolution kernel for the group convolution is 3 × 3, and the number of groups in group convolution is $1/8$ of the number of $D_i$ channels. The length $V$ of the $T$ vector is $1/2$ of the length of the $R$ vector.

According to the Summary of $Z$ distribution characteristics, the $Los{s}_{Covid-BCE}$ function parameter $e$ is set to 107, and parameter $s$ is set to 8.

4.1.2. Evaluation metrics

In order to quantitatively analyze the experimental results just as segmentation tasks, we use the following indicators to measure the results to evaluate the performance of the method, that are Sensitivity ($Sen$), Specificity ($Spe$), Precision ($Pre$), Dice score. We also introduce two golden metrics, Enhanced-Alignment Measure ($E_{\varphi }$), Mean Absolute Error (MAE), which are defined as Eqs. (12) to (13).

$$E_{\varphi }=\frac{1}{W\times H}\sum _{i=1}^W\sum _{j=1}^H\varphi (p_{i,j},g_{i,j})$$ (12)

$$\text{MAE}=\frac{1}{W\times H}\sum _{i=1}^W\sum _{j=1}^H|p_{i,j}-g_{i,j}|$$ (13)

where $p_{i,j}$ and $g_{i,j}$ have the same meaning as in Eq. (10).

4.2. Materials

4.2.1. COVID-19 segmentation dataset

Since the number of samples in the existing public dataset is small and fuzzy, it is very easy to cause the phenomenon that the model cannot be fitted. Therefore, this paper constructs a dataset for F3-Net to perform image segmentation experiments. The details are as follows: 358 cases of different new coronary patients from the same hospital were used.

Each case file format is ‘.nii’. In each case, personal information was hidden, and RT-PCR tests were done, and the results were positive. Each case contains more than 300 CT images, and each CT image has been manually marked with the lesion area by two well experienced doctors using ITK-SNAP tool to obtain the lesion image corresponding to each CT series. A group of 4472 images with relatively good quality images are selected, and one pairwise of image lesion contains a CT source image and a corresponding lesion annotated image. (See Table 1.)

Table 1.

Some parameters of COVID-19 patients undergoing CT testing.

Item	Value
Slice thickness	DS: “2.5”
KVP	DS: “120.0”
Data collection diameter	DS: “500.0”
Reconstruction diameter	DS: “320.0”
Distance source to patient	DS: “541.0”
Gantry/detector tilt	DS: “0.0”
Table height	DS: “124.0”
Rotation direction	CS: ‘CW’
Exposure time	IS: “1160”
Generator power	IS: “31800”
Focal spot(s)	DS: “1.2”

4.2.2. Preprocessing

In data preprocessing, scale of all CT images and lesion images are set as 256 × 256, and then the method normalizes the pixel values of CT image to $0\sim 255$ by setting the window width (WW) and window level (WL) as 1500HU and −450HU respectively. Finally, the value of all point will be normalized from $0\sim 255$ to $0\sim 1$.

The training strategies of the method is preset as following. Firstly, the data set is divided into training set, validation set, and test set according to the ratio of $14:3:3$, and then the training set is rotated by $180°$ for data enhancement. Finally, the data of the training set is used to train F3-Net model. The batch size is set to 6, the learning rate is set to $1e^{-4}$, and the optimization algorithm is $Adam$ [45], where ${\beta }_1=0.9$, ${\beta }_2=0.999$, $eps=1e^{-8}$ and $weigh{t}_{decay}=1e^{-8}$, loss function is $Los{s}_{Covid-BCE}$, and the value of iterations is 1000. The validation and test sets are used to evaluate the method in train stage and test stage.

To prove the superior performance of F3-Net for comparison with models used in COVID-19 image segmentation task which are the U-Net [19], the FCN [14], the Attention-UNet [46] and COPLENet [37].

4.3. Results

4.3.1. Quantitative results

We compare our F3-Net with four state-of-the-art networks for semantic segmentation. FCN [14] only consists of multiple convolutional layers. U-Net [19] uses the architecture of down-sampling and up-sampling and uses skip-connection to reduce semantic gaps between features. Attention-Unet [46] involves an attention mechanism based on U-Net. COPLE-Net [37] replaced the skip connection of U-Net with a bridge layer and integrates a combination of maximum pooling and average pooling to reduce information loss in the pooling process and achieves top performance in COVID-19 lesion segmentation. Table 2 shows the results of quantitative analysis.

Table 2.

Quantitative results of lesion infection regions on our dataset.

Methods	Dice	Sen	Spe	Pre	MAE	$E_{\varphi }$
FCN [14]	0.6329	0.5951	0.9849	0.5616	0.0255	0.9728
U-Net [19]	0.6489	0.6904	0.9913	0.7375	0.0172	0.9782
Attention-Unet [46]	0.6699	0.7142	0.9937	0.7649	0.0159	0.9787
COPLE-Net [37]	0.7313	0.7230	0.9911	0.7148	0.0166	0.9768
F3-Net(Ours)	$\mathit{0.7758}$	$\mathit{0.8079}$	$\mathit{0.9916}$	$\mathit{0.7461}$	$\mathit{0.0138}$	$\mathit{0.9857}$

As we can see from the Table 2, F3-Net achieved scores of 77.58%, 80.79%, 99.16%, 74.61%, 1.38% and 98.57% in terms of Dice, Sen, Spe, Pre, MAE and $E_{\varphi }$ respectively [47], [48]. Compared with other networks, F3-Net has achieved the best results on the key indexes, $Dice$, $MAE$ and $E_{\varphi }$. The DPM can reduce the semantic gap between the encoder and decoder features. Moreover, AFM can provide attention mechanism to make the features more harmonious fusion. Furthermore, the performance of F3-Net can be even improved by using $Los{s}_{Covid-BCE}$, which is reflected as the increment of 0.81%, 0.09%, 2.01%, 0.07% and 0.25% on the $Dice$, $Spe$, $Pre$, $MAE$ and $E_{\varphi }$ respectively. It can be concluded from the addition of prior knowledge of lesion distribution in $Los{s}_{Covid-BCE}$, which limits the prediction results of the model and makes the model more biased towards the lesion areas.

4.3.2. Qualitative results

The segmentation results, shown in Fig. 7, indicate that the segmentation result of F3-Net is more close to the original lesion image and contains few errors and magazines than others. However, compared with F3-Net, the experimental results of other networks are worse. For example, the experimental results of FCN basically do not contain the lesion area, which is almost completely inconsistent with ground truth. The result of U-Net is better than FCN, which is basically the lesion included, but there are many impurities and errors in addition to the lesion. Attention-UNet and COPLE-Net improve the experimental results, but also have many errors. This accuracy of F3-Net is in line with the design idea of F3-Net. Because of the different sizes and special shapes of COVID-19 lesions, the MSM module is designed so that the network can adapt to multi-scale features to capture lesions of different sizes better.

Fig. 7.

Visual comparison of COVID-19 lesions segmentation results of different methods on our test set.

4.3.3. Ablation study

We propose three key improved points of MSM, DPM and AFM modules. They are important components of F3-Net. MSM is applied to the encoder stage, which can sense features of different sizes to capture ROI of different sizes and shapes, DPM can eliminate the semantic gap between the features in the encoder and the decoder, AFM adds an attention mechanism is applied to the decoder stage, which can enable a more harmonious fusion between multiple features. The ablation study for every module is conducted on the dataset and the evaluation results are shown in Table 3.

Table 3.

Ablation study for the proposed DPM, MSM and AFM in the our model. The best top two results are shown in red and blue fonts, where * and blank represent switches. The F3-Net will use the title’s module of the column when the value is *, otherwise it is blank.

No	DPM	MSM	AFM	Dice	Sen	Spe	Pre	MAE	$E_{\varphi }$
1				0.6904	0.6489	0.9913	0.7375	0.0172	0.9782
2	*			0.7185	0.6933	0.9928	0.7457	0.0161	0.9843
3		*		0.7419	0.7651	0.9909	0.7201	0.0158	0.9854
4			*	0.7387	0.7583	0.9910	0.7200	0.0159	0.9857
5	*		*	0.7331	0.7637	0.9902	0.7048	0.0165	0.9836
6		*	*	0.7652	0.7662	0.9928	0.7643	0.0139	0.9886
7	*	*		0.7658	0.8144	0.9905	0.7227	0.0147	0.9812
8	*	*	*	0.7758	0.8079	0.9916	0.7461	0.0138	0.9857

The ablation experiments of these modules are shown in the Table 3, where No. 1 result is the baseline without MSM, DPM and AFM. We first apply MSM, DPM and AFM to the baseline separately and perform experiments on the test set, and get results No. 3, No. 2 and No. 4. Their improvements in terms of Dice score compared to No. 1 are $5.15\text{\%},2.81\text{\%}$ and 4.83% respectively. In addition, it can be clearly observed that by applying any two of these modules to baseline, various indicators can also be improved. Through No. 7 vs No. 3, No. 6 vs No. 3 and No. 6 vs No. 5, the effectiveness of DPM, AFM and MSM can be seen separately. Finally, it can be observed that No. 8 (baseline ＋ DPM ＋ MSM ＋ AFM) increases the baseline performance in terms of major metrics, such as Dice, MAE, etc. Therefore, we can conclude that the MSM, DPM and AFMs have significant contributions to the performance improvement of F3-Net.

4.4. Discussion

To address the difficulties of the scales of lesions are not uniform, the area of the lesion is quite different and the morphology of lesion is variable, we proposes a COVID-19 Lesion Segmentation network, where the multiscale perception module enables the model to adaptively select different size perceptual fields to better capture COVID-19 lesions of different sizes in CT images, the dense path module eliminates the semantic gap between feature maps, the attention fusion module, the attention fusion module gives specific weights to different channels of different feature maps so that they can mutually suppress the unimportant information when fusing. Although our proposed model achieves good results in automatic segmentation of COVID-19 lesions, the performance and generalization ability of the model can be further improved in the future.

Although our proposed model achieves good results in automatic segmentation of COVID-19 lesions, the performance and generalization ability of the model can be further improved in the future. First, we only considered the specificity of CT images for the COVID-19 lesion segmentation task, ignoring other medical tests of patients, such as blood routine, which are not in the same modality as CT images, but contain some important information as well. Then, the 3D texture features were ignored in the feature extraction process. Finally, we did not further classify the segmented lesion regions. Therefore, our future work has the following points. First, we will extend our proposed framework to other modalities besides CT images, such as ultrasound, to achieve automatic segmentation of COVID lesions. Then, we will try to extract image features from multiple spatial dimensions. Finally, we plan to develop multi-task learning models to design COVID-19 lesion segmentation and classification algorithms.

5. Conclusions

In this paper, we propose a Fully Feature Fusion Based Neural Network for COVID-19 Lesion Segmentation in CT images, which is mainly composed of three core modules MSM, DPM, and AFM, where MSM module focuses on the extraction of multi-scale features, DPM is used to eliminate semantics between features gap, and AFM focuses on the fusion between features. We have constructed a dataset for COVID-19 lesion segmentation to facilitate the neural network’s training, which contains 4472 sets of CT and corresponding COVID-19 lesion images. Extensive experiments on the dataset have demonstrated that the proposed F3-Net advance the state-of-the-art performances. Note that the proposed model can help clinicians more accurately segment COVID-19 lesions and conduct further research.

CRediT authorship contribution statement

Wei Li: Experimental ideas, Evaluated experimental data, Drafted manuscripts. Yangyong Cao: Designs experimental procedures collects data, Assists in manuscript writing. Shanshan Wang: Designs experimental procedures collects data, Assists in manuscript writing. Bolun Wan: Revised the manuscript and evaluates the data.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

The data that has been used is confidential.