High-resolution CT Image Retrieval Using Sparse Convolutional Neural Network

Abstract

We propose a high-resolution CT image retrieval method based on sparse convolutional neural network. The proposed framework is used to train the end-to-end mapping from low-resolution to high-resolution images. The patch-wise feature of low-resolution CT is extracted and sparsely represented by a convolutional layer and a learned iterative shrinkage threshold framework, respectively. Restricted linear unit is utilized to non-linearly map the low-resolution sparse coefficients to the high-resolution ones. An adaptive high-resolution dictionary is applied to construct the informative signature which is highly connected to a high-resolution patch. Finally, we feed the signature to a convolutional layer to reconstruct the predicted high-resolution patches and average these overlapping patches to generate high-resolution CT. The loss function between reconstructed images and the corresponding ground truth high-resolution images is applied to optimize the parameters of end-to-end neural network. The well-trained map is used to generate the high-resolution CT from a new low-resolution input. This technique was tested with brain and lung CT images and the image quality was assessed using the corresponding CT images. Peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) and mean absolute error (MAE) indexes were used to quantify the differences between the generated high-resolution and corresponding ground truth CT images. The experimental results showed the proposed method could enhance images resolution from low-resolution images. The proposed method has great potential in improving radiation dose calculation and delivery accuracy and decreasing CT radiation exposure of patients.

1. INTRODUCTION

Due to CT x-ray exposure and respiratory motion, a routine planning CT (lung or abdomen) is usually captured with a large slice thickness (e.g. 3-4mm) (1-5). Since the out-of-slice resolution is distinctly lower than the in-slice resolution in these CT images, such CT images will affect both contouring (tumor and organ) and dose calculation in treatment planning (6-10). Thus, CT images with super-resolution or high-resolution are needed for the radiotherapy (3, 9, 11-16). The purpose of this work is to develop a deep-learning-based method to retrieve/reconstruct high-resolution CT images from routine low-resolution CT for radiotherapy treatment planning.

2. METHOD

Suppose we have a set of pairs of low-resolution and high-resolution training CT images. For each pair, the high-resolution image is used as the mapping target of the low-resolution image. The mapping is represented as a deep sparse convolutional neural network (17-20) that takes the low-resolution image as the input and outputs the high-resolution one. In the training stage, we first upscale low-resolution to the high-resolution size by the bicubic interpolation, then extract features of up-scaled image. A classical feature extraction strategy in image restoration is to densely extract patches and then represent them by a set of pre-trained bases such as PCA (21), DCT (22), Haar (23), etc. This is equivalent to convolving the image by a set of filters, each of which is a basis. Thus, the input up-scaled low-resolution image I_LR first goes through a convolutional layer H₁(I_LR) = max(0,F₁*I_LR +B₁) to extract features for each patch, where F₁ denotes the filters and B₁ denotes the biases, * represents the convolution operation. Secondly, the learned iterative shrinkage threshold algorithm (LISTA) (24-26) layer H₂ is applied to adaptively sparse represent the extracted feature by iteratively optimize the low-resolution dictionary D_LR and high-dimensional sparse code H₂(H₁(I_LR)). Thirdly, in order to construct the accurate mapping from low-resolution to high-resolution, the obtained sparse vector is mapped to the sparse coefficient of high-resolution by a non-linear mapping H₃ based on restricted linear unit (ReLU) method (27). Then, we use a high-resolution dictionary D_HR to reconstruct the dense features of high-resolution from sparse coefficients and then feed the features into the final layer H₄ to reconstruct the high-resolution patches. To obtain the adaptive D_HR, we first randomly set D_HR with Gaussian noise, and then optimize it by iterative shrinkage threshold algorithm (ISTA) (28, 29). The predicted overlap patches are averaged to produce the final full image, which can be regarded as a pre-defined filter on a set of feature maps (where each position is the “flattened” vector form of a high-resolution patch). Motivated by this, we define a convolutional layer to produce the final high-resolution image H₄ (H̄(I_LR)) = max(0,F₄*H̄(I_LR)+B₄), where H̄ denotes the previous operations, F₄ is composed of several filters and B₄ is a bias vector. The framework of the proposed algorithm is shown in Fig. 1.

Figure 1.

Schematic flow chart of the proposed algorithm for high-resolution CT image restoration.

2.1. Learned Iterative Shrinkage Threshold Algorithm

Our proposed sparse operator H₂ is based on the intimate connection between sparse coding and neural network studied by (24). After feature extraction, for a given input vector x=H₁(I_LR)∈Rⁿ with fixed low-resolution dictionary D_LR, the goal is to find the optimal sparse code vector z∈R^m which minimizes an energy function that combines the square reconstruction error and a l₁ sparsity penalty on the code:

$$\underset{z}{\arg \min }\frac{1}{2}{\Vert x-D_{\mathit{LR}}z\Vert }_2^2+\alpha {\Vert z\Vert }_1$$ (1)

where D_LR is an n ×m dictionary matrix whose columns are the normalized basis vectors, α is a coefficient controlling the sparsity penalty. In order to adaptively optimize the sparse coefficient, we first initiate the low-resolution dictionary by randomly set D_LR(0) with Gaussian noise. Then we iteratively optimize the dictionary and sparse code by LISTA:

$$z_{k+1}=h_{\theta }\left(\frac{1}{L}{D}_{\mathit{LR}}(k)x+(1-\frac{1}{L}{D_{\mathit{LR}}}^T(k)D_{\mathit{LR}}(k))z_k\right),D_{\mathit{LR}}(k+1)=\underset{D_{\mathit{LR}}}{\arg \min }\frac{1}{2}{\Vert x-D_{\mathit{LR}}(k)z\Vert }_2^2+\alpha {\Vert z_{k+1}\Vert }_1$$ (2)

where hθ denotes the shrinkage function ${\left[h_{\theta }(v)\right]}_i=\text{sign}(v_i)(\mid v_i\mid -{\theta }_i),{\theta }_i=\alpha ∕L,$, L is the upper bound on the largest eigenvalue Of D_LR^T(k)D_LR(k).

2.2. Restricted Linear Unit

Previous layers extract a n₁ – dimensional sparse representation for each low-resolution patch. In order to enhance the performance of neural network, it is demanded to learn the connection of low-resolution sparse representations with high-resolution sparse codes. In other word, it is needed to obtain the informative and relevant representation which can be highly connected to the patch of high-resolution from previous low-resolution representation. We perform a non-linear mapping to build the n₂ – dimensional high-resolution vectors from n₁ – dimensional. This is equivalent to applying filters which have a trivial spatial support 1×1:

$$H_3(z)=\max (0,F_3\ast x+B_3)$$ (3)

Here F₂ contains n₂ filters of size n₁×1×1, B₂ is n₂ – dimensional. Each of the output n₂ – dimensional vectors are conceptually a sparse representation of a high-resolution patch that will be used for reconstruction. Rectified Linear Unit (ReLU, max(0, x)) (27) is used on the filter responses. It is possible to add more convolutional layers to increase the non-linearity. But this can increase the complexity of the model (n₂×1×1×n₂ parameters for one layer), and thus demands more training time.

2.3. Loss Function

Learning the end-to-end mapping function F_SCNN requires the estimation of all the network parameters W ={F₁,B₁,D_LR,α,F₃,B₃,D_HR,F₄,B₄}. This is achieved through minimizing the loss between the reconstructed images F_SCNN (I_LR, W) and the corresponding ground truth high-resolution images I_HR. Given a set of high-resolution images {I ⁱ_HR} and their corresponding low-resolution images {Iⁱ _LR}, we use mean squared error (MSE) as the loss function:

$$L(W)=\frac{1}{n}\sum _{i=1}^n{\Vert F_{\text{SCNN}}(I_{\mathit{LR}}^i,W)-I_{\mathit{HR}}^i\Vert }_2^2$$ (4)

where n is the number of training samples. The parameters of each convolutional layer are initialized by drawing randomly from a Gaussian distribution with zero mean and standard deviation 0.001 and 0 for biases. The MSE of loss function is evaluated by the difference between the {Iⁱ _HR} and the network output. We use the Caffe package (30) to enhance the training performance.

3. RESULTS

The proposed high-resolution CT retrieve method was tested with a dataset with 50 brain and lung CT images with the pixel size was 1.00×1.00×2.00 turn³ and 0.98×0.98×3.00 min³. We performed leave-one-out cross-validation method to evaluate the proposed high-resolution reconstruction algorithm. We repeated the training and testing for our method twice using two different resolution CT images down-sampled by 2 and 3 from original brain and lung CT images. Our retrieved CT images (output) were compared with the original planning CT images. In order to get a quantitative evaluation, we used the mean absolute error (MAE), peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) to evaluate the difference between original CT (gold standard) and our reconstructed CT.

Fig. 2 shows a comparison of CT images retrieved from different low-resolution CT images (down-sampled by 2 and 3) using Bicubic interpolation, Dong el al’s method (31) and our proposed method. The retrieved CT images by our method are much closer to original CT images. Table 1 and Table 2 shows average MAE, PSNR and SSIM of three methods for all brain and lung patient’s data. The smaller MAE, as well as higher PSNR and SSIM demonstrated the restoration accuracy of the proposed method.

Figure 2.

Comparison of high-resolution CT images retrieved from different low-resolution CT images using three different methods. From left to right (a) is the ground truth CT, (b) is the high-resolution CT obtained by bicubic interpolation, (c) is the difference between the bicubic-based high-resolution and ground truth CT, (d) is the high-resolution CT obtained from a published state-of-art method [11], (e) is the difference between the high-resolution CT obtained by this method and ground truth CT, (f) is the high-resolution CT obtained by the proposed method, (g) is the difference between the high-resolution CT obtained by our proposed method and ground truth CT. The first and third rows show the results of CT images down-sampled by 2. The second and fourth rows show the results of CT images down-sampled by 3.

Table 1.

MAE, PSNR and SSIM for the lung CT images using the three methods.

	Down-sampling factor = 2			Down-sampling factor = 3
	MAE	PSNR (dB)	SSIM	MAE	PSNR (dB)	SSIM
Bicubic	18.61	34.49	0.94	28.05	30.91	0.89
Dongei et al. (31)	15.42	36.71	0.96	22.82	33.03	0.91
The proposed	13.64	37.93	0.96	21.16	34.40	0.92

Table 2.

MAE, PSNR and SSIM for the brain CT images using the three methods.

	Down-sampling factor = 2			Down-sampling factor = 3
	MAE	PSNR (dB)	SSIM	MAE	PSNR (dB)	SSIM
Bicubic	19.75	33.04	0.97	35.11	28.62	0.91
Dongei et al. (31)	14.90	35.74	0.98	27.10	30.77	0.94
The proposed	9.51	38.71	0.99	20.08	32.32	0.96

4. DISCUSSION AND CONCLUSION

In this paper, we propose a high-resolution CT retrieve method based on sparse convolutional neural network. The novelty of our approach is the integration of the sparse representation and deep convolution neural network into the high-resolution CT reconstruction framework. This approach has 2 distinctive strengths: 1) Instead of using conventional sparse representation, learnable network-based dictionary learning and sparse representation is used to sparse represent the low-resolution image. 2) Contrary to the classical deep networks for image super-resolution with sparse prior, which uses sparse coefficients as the representation of both low-resolution and high-resolution, we used a ReLU-based non-linear mapping to build the informative and relevant representation that is highly connected to the patch of high-resolution. The parameters of each layer include the non-linear mapping are optimized by loss function iteratively. Thus, we build an adaptive connection between low- and high-resolution CT images. We compared the proposed method with the state-art-of methods and demonstrated its feasibility and reliability. The proposed method has great potential in improving radiation dose calculation and delivery accuracy, and decreasing CT radiation exposure of patients.