Evaluation of motion artifacts in brain magnetic resonance images using convolutional neural network-based prediction of full-reference image quality assessment metrics

Abstract.

Purpose

Motion artifacts in magnetic resonance (MR) images mostly undergo subjective evaluation, which is poorly reproducible, time consuming, and costly. Recently, full-reference image quality assessment (FR–IQA) metrics, such as structural similarity (SSIM), have been used, but they require a reference image and hence cannot be used to evaluate clinical images. We developed a convolutional neural network (CNN) model to quantify motion artifacts without using reference images.

Approach

The brain MR images were obtained from an open dataset. The motion-corrupted images were generated retrospectively, and the peak signal-to-noise ratio, cross-correlation coefficient, and SSIM were calculated. The CNN was trained using these images and their FR–IQA metrics to predict the FR–IQA metrics without reference images. Receiver operating characteristic (ROC) curves were created for binary classification, with artifact scores $<4$ indicating the need for rescanning. ROC curve analysis was performed on the binary classification of the real motion images.

Results

The predicted FR–IQA metric having the highest correlation with the subjective evaluation was SSIM, which was able to classify images requiring rescanning with a sensitivity of 89.5%, specificity of 78.2%, and area under the ROC curve (AUC) of 0.930. The real motion artifacts were classified with the AUC of 0.928.

Conclusions

Our CNN model predicts FR–IQA metrics with high accuracy, which enables quantitative assessment of motion artifacts in MR images without reference images. It enables classification of images requiring rescanning with a high AUC, which can improve the workflow of MR imaging examinations.

1. Introduction

Compared with other imaging modalities, such as x-ray and computed tomography, the scan time of magnetic resonance imaging (MRI) is relatively long, so motion artifacts arising from patient movement are often unavoidable. Motion artifacts are a major problem affecting MR images¹ and typically appear as blurring or ghosting along the phase-encoding direction. Their appearance depends on a complicated interaction between the image structure, type of motion, specific MR pulse sequence, and k-space acquisition strategy.² Motion artifacts degrade the quality of an image such that a rescanning is sometimes required. However, motion artifacts may go unnoticed by radiological technologists and thus only become apparent during the radiologist’s reading, which may affect the diagnostic quality. Furthermore, especially for inexperienced technologists, it is sometimes difficult to decide if rescanning should be performed.

To mitigate motion artifacts, various strategies have been proposed; these include using faster imaging, a non-Cartesian k-space trajectory,³ navigators,⁴ and retrospective⁵ or prospective⁶ motion correction. Recently, postprocessing artifact reduction techniques using deep learning approaches have also been introduced.^1,7,8 In these reports, motion-corrupted images are generated by retrospective simulation, and the level of similarity to reference images is evaluated using metrics, such as the mean square error, peak signal-to-noise ratio (PSNR), and structural similarity (SSIM).⁹ Calculation of these so-called full-reference image quality assessment (FR–IQA)¹⁰ metrics requires a reference image; hence, clinical images without reference images cannot be used to assess motion artifacts. For this reason, subjective visual evaluation is usually used to assess motion artifacts in clinical images. However, subjective evaluation is poorly reproducible, expensive, and time consuming.

Recent research on motion artifacts in MRI has reported their detection and classification using a convolutional neural network (CNN);^11–13 however, this machine learning-based approach is focused on the binary classification of image motion, which is insufficient for distinguishing between sequences that can be corrected for motion, those that require rescanning, and those that have so little motion that they require no correction. Therefore, Mohebbian et al.¹⁴ developed a stacked ensemble classifier that sorts motion severity into five levels and demonstrated that it can distinguish between each of them irrespective of the MRI acquisition parameters.

To quantitatively evaluate motion artifacts, we develop a model to predict FR–IQA metrics of images without a corresponding reference using a CNN trained with FR–IQA of motion-corrupted images that have references. The purpose of this study is to assess the utility of the model by comparing the predicted FR–IQA with the subjective visual assessment of the severity of motion artifacts in MRI.

2. Materials and Methods

2.1. Datasets

Image data were obtained from the New York University fastMRI Dataset, the contents of which have previously been described in detail by Zbontar et al.¹⁵ The brain images in this database are stored in Digital Imaging and Communication in Medicine format and were obtained at 1.5 T and 3 T field strengths using 11 MRI scanners across five clinical locations. Scanners included the 3 T Prisma, Skyra, Biograph, Tim Trio, and the 1.5 T Avanto and Aera (Siemens Healthcare, Erlangen, Germany). The dataset includes axial T1-weighted (pre-T1W), T2-weighted (T2W), fluid-attenuated inversion recovery (FLAIR), and some post-T1-weighted (post-T1W) images that were acquired following administration of a contrast agent. Among the brain images in this dataset, 2000 (1151 for 3T and 849 for 1.5T) images of 45 patients, each of whom underwent T2W, FLAIR, and post-T1W scans, were selected for training and validation. If precontrast T1W images were included, half of the image data would be T1W-contrast images, so we only use post-T1W images. A total of 240 (144 for 3T and 96 for 1.5T) images (60 images each of T2W, FLAIR, pre-T1W, and post-T1W) from five patients not used for training and validation data were used as test data.

2.2. Motion-Corrupted Image Generation

Images were resized to a resolution of $256\times 256\mathrm{pixels}$ and then rigidly transformed by rotation and translation (motion image). The original and motion images were transformed into k-space ($\mathrm{k}\text{-}{\mathrm{space}}_{\text{Original}}$ and $\mathrm{k}\text{-}{\mathrm{space}}_{\text{Motion}}$) using a two-dimensional fast Fourier transformation (2D FFT). The $\mathrm{k}\text{-}{\mathrm{space}}_{\text{Original}}$ and $\mathrm{k}\text{-}{\mathrm{space}}_{\text{Motion}}$ data were merged by exchanging specific data points from $\mathrm{k}\text{-}{\mathrm{space}}_{\text{Original}}$ with the corresponding points in $\mathrm{k}\text{-}{\mathrm{space}}_{\text{Motion}}$, producing $\mathrm{k}\text{-}{\mathrm{space}}_{\mathrm{Motion}\text{-}\text{corrupted}}$. Finally, motion-corrupted images were obtained with inverse 2D FFT of $\mathrm{k}\text{-}{\mathrm{space}}_{\mathrm{Motion}\text{-}\text{corrupted}}$ (Fig. 1). Values of the rotation angle (1 deg to 20 deg), translation (1 to 20 pixels), number of motion lines (1 to 100), and location of motion line were generated randomly for each image using a uniform random number generator. All processing was implemented in MATLAB 2018a (The MathWorks, Inc., Natick, Massachusetts).

Fig. 1.

Strategy to generate a motion-corrupted image.

2.3. Objective Quality Assessment

To quantitatively evaluate the severity of the motion artifacts in the motion-corrupted images, the true FR–IQA metrics [PSNR, normalized cross-correlation coefficient (CCC), and SSIM] were calculated using the original image as the reference. The PSNR measures pixelwise differences between the original and motion-corrupted images. The CCC has been extensively used as a metric to evaluate the degree of similarity or patterns between images.¹⁶ The CCC of the original image and the motion-corrupted image was calculated as follows:

$$\mathrm{CCC}=\frac{1}{N}\sum _x\sum _y\frac{\left(A\right(x,y)-\overline{A})\left(B\right(x,y)-\overline{B})}{{\sigma }_A{\sigma }_B}$$ (1)

where $A$ is the original image, $B$ is the motion-corrupted image, and are the mean values, and ${\sigma }_A$ and ${\sigma }_B$ are the standard deviations of $A$ and $B$, respectively. The SSIM shows the similarity of signal intensity, contrast, and structure for each local region, but not at a pixelwise level,⁹ and was calculated as follows:

$$\begin{array}{c}\mathrm{SSIM}=\frac{(2{\mu }_A{\mu }_B+C_1)(2{\sigma }_{AB}+C_2)}{({\mu }_A^2+{\mu }_B^2+C_1)({\sigma }_A^2+{\sigma }_B^2+C_2)},\end{array}$$ (2)

where ${\mu }_A,{\mu }_B$, and ${\sigma }_{AB}$ are the local mean and cross-covariance for images $A$ and $B.C_1=(k_{1}L{)}^2$ and $C_2=(k_{2}L){,}^2$ where $L$ is the dynamic range of the pixel values and $k_1$ and $k_2$ were set to 0.01 and 0.03, respectively.

2.4. CNN Architecture

The architecture of the CNN is shown in Fig. 2. It comprises a grayscale input image with a resolution of $256\times 256\mathrm{pixels}$, six convolutional layers with a $3\times 3$ kernel size, two max-pooling layers, and the rectified linear unit activation function. The number of filters was increased after the max-pooling layer, being 32, 64, and 128 for the first, second, and third blocks, respectively. The last two layers were fully connected to combine the feature vectors. The output contained the predicted FR–IQA metrics (i.e., scalar values of PSNR, CCC, and SSIM). For a single input image, three CNN models were designed with outputs as predicted, PSNR, CCC, and SSIM. The total number of parameters was 268,722,913.

Fig. 2.

Architecture of the CNN.

2.5. Model Training

The motion-corrupted images were randomly assigned to separate training ($N=1600$) and validation ($N=400$) sets. Only data in the training and validation sets were used for training and hyperparameter selection. The model was trained using the training set for 50 epochs, and its performance was evaluated using the validation set after each epoch. A single NVIDIA GeForce RTX 2080Ti GPU (NVIDIA Corporation, Santa Clara, California) was used to train the CNN. The batch size was 8, and a learning rate of 0.01 was used. Network training was performed using Keras with a TensorFlow backend (Google, Mountain View, California), and the network was optimized using the Adam algorithm to minimize the mean squared error (MSE).

2.6. Subjective Image Evaluation

The severity of motion artifacts in the test dataset of 240 motion-corrupted images was subjectively evaluated by three radiological technologists and compared with the predicted FR–IQA metrics. The technologists (readers A, B, and C) had 12, 8, and 12 years of experience as MRI operators, respectively. The readers evaluated the images using a five-point scale based on the severity of the artifacts, ranging from 1 (extensive) to 5 (negligible), as shown in Fig. 3. They were not provided with the motion-free original images or FR–IQAs at the time of evaluation.

Fig. 3.

Example images demonstrating the five-point scale used to grade the severity of motion artifacts (1 = extensive; 2 = severe; 3 = moderate; 4 = minimal; 5 = negligible).

2.7. Binary Classification of Images with Real Motion Artifacts in Detecting the Need for Rescanning

From the remaining images in the dataset that were not used in the training, validation, and test data, a total of 100 images (25 images each of T2W, FLAIR, pre-T1W, and post-T1W) with real motion artifacts were extracted such that images with different artifact levels (artifacts score, 1 to 5) were uniformly included, and the predicted SSIM were calculated using the CNN model.

2.8. Statistical Analysis

Model performance was characterized using Pearson’s correlation coefficient ($r$) with 95% confidence intervals (CI) and coefficient of determination ($R_2$) between the true and predicted FR–IQA metrics for each type of image contrast (FLAIR, T2W, pre-T1W, and post-T1W). The artifact scores generated by the three readers during the subjective image evaluation were averaged, and the intraobserver agreements were calculated using Cohen’s kappa statistics with a 95% CI. The Spearman rank-order correlation coefficient ($\rho$) was used to calculate the association between the artifacts scores and each of the predicted FR–IQA metrics. Receiver operating characteristic (ROC) curves were created for binary classification, with an artifact score $\ge 4$ indicating no need for rescanning, and area under the ROC curves (AUC) for the predicted PSNR, CCC, and SSIM were compared using DeLong’s test. The Youden index was used to determine the optimal sensitivity and specificity. All statistical analyses were performed using JMP14 software (SAS Institute, Cary, North Carolina). All $P$ values were two-sided, and $P<0.05$ was considered statistically significant.

3. Results

Our CNN model was found to predict the values of FR–IQA metrics with high accuracy (Fig. 4). The Pearson $r$ and $R^2$ values for each FR–IQA metric are shown in Table 1, which shows that there was a significant correlation between the true and predicted FR–IQA metrics for all types of metrics and image contrast ($P<0.001$).

Fig. 4.

Scatterplots of the true and predicted FR–IQA metrics [(a) PSNR; (b) CCC; (c) SSIM] for the test dataset ($N=240$). FR–IQA, full-reference image quality assessment; PSNR, peak signal-to-noise ratio; CCC, normalized cross-correlation coefficient; and SSIM, structural similarity.

Table 1.

Correlation between the true and predicted FR–IQA metrics.

	PSNR		CCC		SSIM
	Pearson $r$ [95%CI]	$R^2$	Pearson $r$ [95%CI]	$R^2$	Pearson $r$ [95%CI]	$R^2$
T2WI	0.978 [0.963, 0.987]	0.956	0.787 [0.666, 0.867]	0.619	0.866 [0.830, 0.894]	0.750
FLAIR	0.810 [0.701, 0.883]	0.657	0.413 [0.178, 0.604]	0.171	0.876 [0.800, 0.924]	0.598
Pre-T1WI	0.984 [0.973, 0.990]	0.968	0.808 [0.697, 0.881]	0.652	0.881 [0.808, 0.927]	0.776
Post-T1WI	0.982 [0.971, 0.989]	0.965	0.869 [0.788 0.920]	0.754	0.908 [0.851 0.945]	0.825
Over all	0.969 [0.960, 0.976]	0.939	0.753 [0.693, 0.803]	0.568	0.866 [0.830, 0.894]	0.750

The Spearman $\rho$ values, measuring the association between the artifact scores and each of the predicted FR–IQA metrics, were 0.715 ($P<0.001$), 0.674 ($P<0.0001$), and 0.853 ($P<0.0001$) for PSNR, CCC, and SSIM, respectively (Fig. 5). Intraobserver reliability, measured using Cohen’s kappa coefficient, was good: readers A−B = 0.824 (95% CI: [0.703, 0.945]); readers A−C = 0.752 (95% CI: [0.612, 0.893]), and readers B−C = 0.805 (95% CI: [0.672, 0.939]).

Fig. 5.

Scatterplots of the artifact score and the predicted FR–IQA metrics: (a) PSNR; (b) CCC; (c) SSIM for the test dataset ($N=240$).

Figure 6 shows the ROC curve describing the diagnostic performance of the predicted FR–IQA metrics in detecting the need for rescanning. The AUCs (95% CI) of the predicted PSNR, CCC, and SSIM values were 0.864 (0.809 to 0.905), 0.828 (0.767 to 0.876), and 0.930 (0.891 to 0.956), respectively, and there was a significant difference between SSIM and CCC ($P<0.0001$) and between SSIM and PSNR ($P=0.0002$). The sensitivity and specificity were as follows: PSNR, sensitivity = 75.9%, specificity = 82.1%; CCC, sensitivity = 87.0%, specificity = 70.5%; and SSIM, sensitivity = 89.5%, specificity = 78.2%.

Fig. 6.

ROC curves showing the diagnostic performance of predicted FR–IQA metrics in detecting the need for rescanning.

For the binary classification of images with real motion artifacts using predicted SSIM, the average predicted SSIM (95% CI) was 0.646 (0.618 to 0.674) with need for rescanning and 0.804 (0.788 to 0.821) without need for rescanning. The AUC (95% CI) was 0.928 (0.858 to 0.964), the sensitivity was 90.0%, and the specificity was 83.3%.

4. Discussion

In this study, we developed a CNN model to predict FR–IQA metrics quantifying the severity of motion artifacts in MRI without the need for motion-free reference images. A simple small-layer CNN yielded highly correlated predictions of FR–IQA metrics when results from all types of image contrast were pooled, with Pearson $r$ correlation coefficients of 0.969, 0.753, and 0.866 for PSNR, CCC, and SSIM, respectively. Separate analysis of each type of image contrast showed that high correlation coefficients were obtained for all contrasts, suggesting that training with a mixture of image contrasts is not a problem. In addition, although only post-T1WI was used for training, the predictive performance of the pre-T1WI test data was high, indicating that it was sufficient to have training data that included images of T1W contrast with or without contrast media. The predicted FR–IQA metric that correlated most highly with the subjective evaluation was SSIM ($\rho =0.853$). The MSE and the related PSNR are widely used as FR–IQA metrics because they are simple to calculate, have clear physical meanings, and are mathematically convenient in the context of optimization. However, they do not correlate well with perceived visual quality.¹⁷ In contrast, the SSIM index measures three visually relevant parameters, namely luminance, contrast, and geometric structure.⁹ Kumar et al. showed that SSIM is a more accurate measure of the quality of compressed MR images than PSNR because it measures the structural properties of an image.¹⁸ Using a CNN to predict SSIM, which had the highest correlation with the subjective evaluation in this study, it is possible to quantify motion artifacts without using reference images.

Ma et al. used a CNN to detect artifacts and classify MR image quality as either nondiagnostic or diagnostic, achieving a sensitivity of 84% and an AUC–ROC score of 0.79 in a binary classification task with a simple four-layer CNN architecture.¹³ However, in this type of model, subjective evaluation is necessary for training, and the number of training images is limited. In addition, as is the case for visual evaluation, we believe that the model is unreliable in terms of reproducibility and objectivity. In contrast, the subjective evaluations are not necessary when using CNNs to predict objective measures of image quality, as demonstrated herein, and a large number of images can be used for training.

Mason et al. used retrospectively degraded images (simulated by introducing white Gaussian noise, Gaussian blur, Rician noise, k-space undersampling, wavelet compression, and motion artifacts) to investigate the correlation between 10 FR–IQA metrics and five radiologists’ subjective image quality scores;¹⁹ more complex FR–IQA metrics, such as visual information fidelity, feature similarity index, and noise quality metric (NQM), had a higher correlation with subjective assessment, while those with the lower computational cost, such as SSIM and MSE, had a lower correlation. However, even though the difference in FR-IQA performance depended on the type of image degradation, there was a correlation between all image quality metrics and subjective evaluation when grouped into the same image quality degradation. This is consistent with the high correlation between FR–IQA and subjective evaluation observed in this study, which also focused on one type of image distortion (motion artifacts). Therefore, our method of predicting objective image quality metrics using a CNN trained specifically for a particular type of image distortion is useful. However, evaluations that account for all types of image degradation require the use of more complex objective image quality indices, such as VIM, FISM, and NQM rather than SSIM, PSNR, and MSE. This task will be the objective of future studies. Similar to the reports of Mason et al.,¹⁹ variability existed between the FR–IQA and subjective assessment scores in our study. This indicates the difficulty in predicting subjective scores with objective image quality metrics because there is always variability in subjective scores, and each observer may have a different sensitivity and tolerance for different forms of distortion.

In validating the effectiveness of artifact and noise removal techniques using deep learning, our findings show that it may be possible to use the predicted FR–IQA as a substitute for visual assessment without a reference image at the clinical stage by including an objective evaluation during the training of the reconstruction model. In addition, if objective evaluation metrics can be provided from the scanner at the time of image reconstruction, the operator can use them to decide if rescanning is necessary. This will improve the efficiency of the scan and the uniformity of image quality. In this study, the predicted SSIM was able to identify images requiring rescanning with an AUC–ROC score of 0.930 in motion-corrupted images generated with simulation and 0.928 in real motion images. Informing the operator if a rescan is necessary may prevent unidentified artifacts from adversely affecting the quality of the examination.

This study has several limitations. The motion artifacts simulated in this study were generated from the rotation and translation of the in-plane direction and did not include the motion of the through-plane direction. Because the actual artifacts are affected by the motion of the through-plane direction as well as the in-plane direction, further improvement in model performance is expected if the motion artifacts of the through-plane direction can be simulated. In addition, because the motion artifacts in this study were generated assuming Cartesian sampling, motion artifacts in non-Cartesian sampling such as PROPELLER or radial acquisition will have different characteristics from those in Cartesian sampling, and there is a high possibility that they cannot be evaluated correctly. Therefore, it is necessary to simulate the motion artifacts corresponding to the sampling strategy of the image to be evaluated. MR images of postsurgery patients show brain deformation and susceptibility artifacts due to postoperative implants. Because the data used in this study did not include postsurgery patients, the effect of surgery on the assessment of motion artifacts is unknown. Moreover, postoperative patients are at higher risk for motion, so further study is needed. The datasets used for training in this study included those with static field strengths of 1.5 T and 3 T. The accuracy of the CNN model can be higher if it is trained with images obtained under scan conditions (e.g., image contrast and static magnetic field strength) comparable with those used during training;¹⁴ however, it is straightforward to improve the accuracy of our method by fine-tuning and to improve the generalizability by increasing the number of training images. Although some of the original images in our dataset already contained motion artifacts, we were able to obtain good accuracy. The prediction accuracy may be further improved by selecting clean images for training, but it is also expected to be improved by increasing the number of training images, the latter method being simpler. Finally, we note that the subjective evaluation of the need for rescanning was performed in our study by radiological technologists, whose assessments may differ from those of radiologists.

5. Conclusion

In this study, we generated motion-corrupted images from motion-free reference images and built a CNN model trained with FR–IQA metrics that was able to accurately predict these metrics using motion-corrupted images without a reference. The predicted FR–IQA and the subjective evaluation were highly correlated, and the predicted FR–IQA could be used to determine if rescanning is necessary with a high AUC. Predicted FR–IQAs could be used to help the operator decide if rescanning is necessary or to replace a subjective assessment.

Biographies

Hajime Sagawa is a PhD student in the Division of Health Sciences, Graduate School of Medical Sciences, Kanazawa University and a radiological technologist at Kyoto University Hospital. His main research interests include the development of MR physics and imaging techniques, medical data analysis using deep learning methods, and image processing.

Koji Itagaki is a radiological technologist at Kyoto University Hospital. His research interests focus on the physics and technology of magnetic resonance and nuclear medicine, image processing, and applications of deep learning methods to medical data analysis.

Tatsuhiko Matsushita is a radiological technologist at Kyoto University Hospital and a research assistant at the Faculty of Health Sciences, Institute of Medical, Pharmaceutical and Health Sciences, Kanazawa University. He received his PhD from the Division of Health Sciences, Graduate School of Medical Sciences, Kanazawa University. His research interests include MR physics and imaging techniques.

Tosiaki Miyati is a faculty dean at Kanazawa University. He received his Doctor of Engineering degree from Gifu University and Doctor of Medical Science degree from Nagoya City University. He was appointed as an assistant professor of health sciences in 2000 and was promoted to full professor in 2007 at Kanazawa University. During this period, he has been developing noninvasive functional imaging methods in MRI and published 30 books and more than 300 journal articles.

Disclosures

There are no conflicts of interest to declare.