Survey on Neural Networks Used for Medical Image Processing

Abstract

This paper aims to present a review of neural networks used in medical image processing. We classify neural networks by its processing goals and the nature of medical images. Main contributions, advantages, and drawbacks of the methods are mentioned in the paper. Problematic issues of neural network application for medical image processing and an outlook for the future research are also discussed. By this survey, we try to answer the following two important questions: (1) What are the major applications of neural networks in medical image processing now and in the nearby future? (2) What are the major strengths and weakness of applying neural networks for solving medical image processing tasks? We believe that this would be very helpful researchers who are involved in medical image processing with neural network techniques.

1. Introduction

With medical imaging playing an increasingly prominent role in the diagnosis of disease, interests in medical image processing have increased significantly over the past decades [1]–[3]. Especially methods based on artificial neural networks (ANNs) have attracted more attention. In 1992, a comprehensive survey of neural networks in image processing has been published by Miller et al. [4]. In their review article, Miller etc. predicted that neural networks would become widely used in medical image processing. This predication turned out to be right. According to our searching results with Google Scholar, more than 33000 items were found on the topic of medical image processing with ANNs during the past 16 years. The intention of this article is to cover those approaches introduced and to make a map for ANN techniques used for medical image processing. Instead of trying to cover all the issues and research aspects of ANNs in medical image processing, we focus our discussion on three major topics: medical image preprocessing (i.e. de-noising and enhancement), medical image segmentation, and medical image object detection and recognition. We include methods published before 1992 as well, because those methods provide key ideas and are still useful to learn. In this way, we retain the continuity and provide a complete view of ANNs in medical image processing research. We do not contemplate to go into details of particular algorithms or describe results of comparative experiments, rather we want to summarize main approaches and point out interesting parts of the neural networks in medical image processing, further more in this survey, we want to give some ideas to answer: 1) what the major applications of neural networks in medical image processing now and in the nearby future are; 2) what the major strengths and weaknesses of ANNs in applications of medical image processing would be. We think this should be the main contributions of our work in this paper.

The remainder of this paper is organized as follows: In Section 2 image processing operations used in medical imaging are reviewed briefly with an emphasis on the techniques that have been employed and on the tasks which these techniques aim to solve. Artificial neural network approaches used in medical image processing are reviewed in Section 3. Section 4 surveys main issues for medical image processing with neural networks. Section 5 concludes main trends in the research

2. Image processing in medical imaging

In recent years, image processing techniques pose a far important role in the analysis of medical images in a wide range of modalities because of the reasons below.

First, most medical images have a poor noise-to-signal ratio than scenes taken with a digital camera, which often leads a frustratingly lower spatial resolution and makes the contrast between anatomically distinct structures too low to be computed reliably. For example, in the case of ultrasonic images, speckle noise, which is caused by the scattering of the ultrasonic beam from microscopic tissue in homogeneties, tends to mask the presence of low contrast lesions and reduces the ability of a human observer to resolve final detail [5]. A similar case also appears in nuclear medicine (NM) images [6]–[7]. Because of these reasons, image preprocessing techniques used for reducing noises and blurs of medical image are indispensable.

Second, changes to image content must be done in a highly controlled and reliable way that does not compromise clinical decision-making. For example, whereas it is generally acceptable to filter out local bright patches of noise, care must be taken in the case of mammography not to remove microcalcifications. In order to obtain this goal, some sophisticated operations have to be done. An important step in this process is image segmentation. In image segmentation, we expect the pixels in the same class to have similar pixel values independent of their locations. However, in magnetic resonance imaging (MRI), in-homogeneity in the magnetic field usually gives rises to intensity non-uniformity (INU) artifact. This common artifact exhibits itself as a smooth, slowly varying change in image pixel values and could have adverse effect on the performance of intensity based automatic segmentation methods. In addition to intensity variation due to field in-homogeneity, there may be a lack of tissue specific meanings for MRI intensities within scans, even for the same patient obtained on the same scanner using the same protocol.

Thirdly, information gained from two images acquired in the clinical track of events is usually of a complementary nature, and a proper integration of useful data obtained from the separate images may be more desired. There are, therefore, potential benefits in improving the way in which these images are compared and combined [8].

Table 1 shows the main operations of image processing used in different medical image modalities.

Table 1.

Main operations of image-processing used in different medical image modalities.

	Preprocessing	Segmentation	Registration	Recognition
X-rays	√	√	-	√
Ultrasonic	√	√	√	√
CT	√	√	√	√
MRI	√	√	√	√
NM	√	√	-	√

3. Neural networks used in medial image processing

Neural networks are well known for their good performance in classification and function approximation, and have been used with success in medical image processing over the past years, particularly in the case of preprocessing (e.g. construction and restoration), segmentation, registration and recognition. Table 2 gives an overview of the main types of neural network used in these fields; detailed description of most important applications is included in the remainder of this section.

Table 2.

Main neural networks used in medical image processing

	Preprocessing	Segmentation	Registration	Recognition
Feed forward NN	√	√	-	√
Radial basis function NN	-	-	√	√
Hopfield NN	√	√	-	√
Self organizing feature NN	√	√	√
Adaptive resonance theory NN	-	-	-	√
Cellular NN	√	-	-	-
Convolution NN	-	-	-	√
Probabilistic NN	-	√	-	√
Fuzzy NN	√	√	-	√
Neural ensemble		√	√	√
Massive training NN	√	-	-	√

3.1 Preprocessing

Image preprocessing with neural networks generally falls into one of the following two categories: image reconstruction and image restoration (Including de-noise and enhancement). Neural networks used for these two medical image processing operations are summarized in table 3.

Table 3.

neural networks for medical image preprocessing

Neural networks	Image reconstruction	Image restoration
Hopfield NN	[9]–[13]	[25]
Feed forward NN	[14] [15]	[18]–[24]
Self organizing map NN	[16] [17]	-
Fuzzy NN	-	[26]–[27]
Cellular NN	-	[28]–[30]

^*[] reference index

From table 3, it can be seen that the Hopfield neural network is one of the most used neural works for image reconstruction [9]–[13], and it poses 55 percent of our reviewed literatures related to this areas. The feed forward neural network [14]–[15] and the self-organizing Kohonen neural network [16]–[17] are also found to be used for medical image reconstruction, which pose 2 of 9 papers among our reviewed literatures, respectively.

The majority of applications of neural networks in medical image preprocessing are found in medical image restoration, 13 of 24 papers among our reviewed literatures focused their interests here [18]-[30]. Among which, one paper for Hopfield neural network, seven papers for the feed forward neural network, and two papers for fuzzy neural network and cellular neural network respectively.

3.2. Medical image segmentation

Table 4 gives an overview of neural networks used in medical image segmentation.

Table 4.

an overview of neural networks used in medical image segmentation

	Remarks
Du Yih TSAI [31]	Using a three layers BP NN for segmenting liver structure from abdominal CT images
A. Hasegawa [32]	Employing a shift invariant NN for the segmentation of lung fields in chest radiography
M. Ozkan [33]	Developing a BP NN approach to the automatic characterization of brain tissues from MR images
Yan Li [34]	Describes a LSB NN based method
Ian Middleton et al. [35][36]	Combining a MLP NN and active contour model for structures segmenting in MR images
John. E. Koss [37]	Employing Hopfield NN for segmenting liver from CT abdominal images
Lin et al. [38] [39]	Proposing a fuzzy Hopfield NN based segmentation approach
K.S. Cheng [40]	Developing a competitive Hopfield NN for medical image segmentation
Wei Sun [41]	Proposing an MRI segmentation method based on fuzzy Gaussian basis NN
Lee [42]	Describing a contextual NN based segmentation
Wang [43]	Presenting a probabilistic NN based segmentation technique
Reddick et al. [44]–[47]	Hybrid NNs are developed for MR/MRI/CT image segmentation

From Table 4, it can be seen that the feed forward neural network is the most used neural network for medical image segmentation. Among our reviewed papers, 6 of 17 papers employed the feed forward network for medical image segmentation [31]–[36]. Compared with the traditional Maximum Likelihood Classifier (MLC) based image segmentation method, it has been observed that the feed forward neural networks-based segmented images appear less noisy, and the feed forward neural networks classifier is also less sensitive to the selection of the training sets than the MLC. However, most feed forward neural network based methods have a very slow convergence rate and requires a priori learning parameters. These drawbacks limited the application of feed forward neural networks in medical image segmentation, especially since the medical images training sets required are very large.

Hopfield neural network is another most used neural network for medical image segmentation. Among our reviewed literatures, 4 of 17 paper used Hopfield Neural Network to segment some organs from a medical image [37]–[40]. Hopfield neural networks used here were introduced as a tool for finding satisfactory solutions to complex optimization problems. This makes them an interesting alternative to traditional optimization algorithms for medical image reconstruction which can be formulated as optimization problems. A further discussion of Hopfield networks for solving optimization problems see section 4

Other neural networks, such as fuzzy Gaussian basis neural networks [41], contextual neural network [42], probabilistic neural network [43], a hybrid neural network method [44]–[47], are also found to be employed for medical image segmentation.

3.3. Object detection and recognition

Detection and recognition of organs and tumors in medical images are prerequisite in medical applications. It is also the final step in the medical image processing, where the goal is to interpret the image content. Therefore, it couples techniques from segmentation or object recognition with the use of prior knowledge of the expected image content. As a consequence, it is potentially the most fruitful application area of neural networks, as using a neural network approach makes it possible to roll several of the preceding stages (preprocessing, segmentation) into one and train it as a single system. Neural network methods used for medical image detection and recognition are summarized in table 5.

Table 5.

Main neural network methods used for medical image detection and recognition

	NN model	Application
Wu [48]	Feed-forward NN	Interpretation of mammograms
Tourassi [49]	Feed forward NN	Localization of cold lesion in SPECT images
Maclin [50]	Feed forward NN	Diagnosing classes of liver diseases based on ultrasonographic
Wolberg [51]	Feed forward NN	Breast cancer diagnosis
Fikret [52]	BP NN	Melanoma detection
Ashizawa [53]	BP NN	Lung diseases distinguish
Suzuki [54]–[58]	Modified BP NN	Reduction of false positives in computerized detection of lung nodules
Zhu [59]	Hopfield NN	Detection of brain tumor boundaries
Innocent [60]	ART NN	Radiographic image classification
Yasser [61]	RBF NN	Diagnosis of liver diseases using ultrasonic images
Chen [62]	Probabilistic NN	Liver tumors recognition
Pavlopouls [63]	Fuzzy NN	Liver ultrasound image recognition
B. Verma [68]	Fuzzy NN	Detecting microcalcifications in digital mammograms
H.P. Chan [64]–[67]	Convolution NN	Detection of clustered microcalcifications on mammograms
Zhou et. al. [69][70]	Neural ensemble	Lung cancer cell identify

From table 5, it can be seen that for the use of neural networks in medical image detection and recognition, the back propagation neural networks poses most places, 11 of 23 papers among our reviewed papers employed it [48]–[58]. Compared with conventional image recognition methods, no matter used for the interpretation of mammograms [48], or used for cold lesion detection in SPECT image [49], or used for diagnosing classes of liver diseases based on ultrasonographic [50], or used for separation of melanoma from three benign categories of tumors [52], or distinguish interstitial lung diseases [53], or used for reduction of false positives in computerized detection of lung nodules in LDCT [54]–[58]and chest radiography [61], all these neural network based methods show their preference in recognition accuracy and computing time.

Other neural networks, i.e. Hopfield neural network [59], ART neural network [60], radial basis function neural network [61], Probabilistic Neural Network [62], convolution neural network [64]–[67], and fuzzy neural network [63] [68], have also found their position in medical image detection and recognition, and pose 1 of 23, 1 of 23, 1 of 23, 1 of 23, 2 of 23 and 2 of 23 papers, respectively.

Different from what mentioned above, in [69] and [70], artificial neural network ensembles are employed to cancer detection. The ensemble is built on two-level ensemble architecture. The first-level ensemble is used to judge whether a cell is normal with high confidence where each individual network has only two outputs respectively normal cell or cancer cell. The predictions of those individual networks are combined by some a method. The second-level ensemble is used to deal with the cells that are judged as cancer cells by the first-level ensemble, where each individual network has several outputs respectively, each of which represents a different type of lung cancer cells. The predictions of those individual networks are combined by a prevailing method, i.e. plurality voting. Experiments show that the neural network ensemble can achieve not only a high rate of overall identification but also a low rate of false negative identification, i.e. a low rate of judging cancer cells to be normal ones, which is important in saving lives due to reducing missing diagnoses of cancer patients.

4. Discussion

4.1 Major strengths and contributions of applying neural networks for medical image processing

From the reviewed literatures, we found that the Hopfield neural network, the feed-forward neural network and the self-organizing feature map are the most and widely neural network model in medical image processing. The main advantage of using Hopfield neural network for medical image processing is that the problem of medical image reconstruction can be taken as an optimization problem, and thus be solved by letting the network converge to a stable state while minimizing the energy function. Compare with other conventional technique, the idea of just taking medical image processing as the optimization of Hopfield neural network makes the problem of medical image processing more easer to solve.

The feed forward neural network is a supervised neural network. From our reviewed literatures, we find that when a gold standards is available, this kind of neural network is good choose for medical image processing, Compared with Hopfield neural network based method or other conventional techniques, the advantages of the this neural networks based image reconstruction methods are their ability to control the compromise between the noise performance and resolution of the image reconstruction and their conceptual simplicity and ease of implementation.

When no gold standard is available, the self-organizion feature map (SOM) is an interesting alternative to supervised techniques. It can learn to discriminate different medical image information, e.g., textures when provided with powerful features.

From the reviewed literatures, we also find that no matter what neural network model employed for medical image processing, compared with conventional image processing methods, the time for applying a trained neural network to solve a medical image processing problem was negligibly small, though the training of a neural network is a time cost work and also medical image processing tasks often require quite complex computation [18]–[24].

4.2 Main weakness of applying neural networks for medical image processing

Despite their success story in medical image processing, artificial neural networks have several major disadvantages compared to other techniques.

The first one is that how to choose the best neural network model and its corresponding architecture. Although there is some work on model selection [71], there are no appropriate data in literature describing what type of network to construct for a given task and no general guidelines exist which guarantee the best trade-off between model bias and variance for a particular size of the training set [72]. Networks must be designed by trial and error: this empirical approach to network design is difficult to surmount. Furthermore, there is always a danger of overtraining a neural network because that minimizing the error measure occasionally does not correspond to finding a well-generalizing neural network.

The second problem is their black-box character. A trained neural network might perform well but offers no explanation on how it works. That is, given any input a corresponding output is produced, but it is usually hard to explain why this decision was reached, how reliable it is, etc. In medical image processing, this is certainly problematic, so the use of neural networks in such applications will remain limited.

The third problem is that a neural network is hard to express human expert’s knowledge and experience, and the construction of its topological structure lacks of theoretical methods [73]. Moreover the physical meaning of its joint weight is not clear. All these can make the image processing method by neural networks unstable. A solution to these problems may be to combines fuzzy technique with neural networks together by using neural networks to process fuzzy information. It provides neural networks the ability to express qualitative knowledge, and network topological structure and joint weight have clear physical meaning. Also, it can make the initialization of network easier, avoid the local optimization of network training, and ensure the stability of networks [74].

The fourth problem in medical image processing relates to the amount of input data. For achieving a high and reliable performance for non-training cases, a large number of training cases are commonly required [75] [76]. If an ANN is trained with only a small number of cases, the generalization ability (performance for non-training cases) will be lower (e.g. the ANN may fit only the training cases). Because medical images are progressing rapidly as technology advances, the timely development of CAD schemes is important. However, it is very difficult to collect a large number of abnormal cases for training, particularly for a CAD scheme with a new modality, such as lung cancer screening with multi-detector-row CT. This significantly degraded the results obtained.

Finally, there is a clear need for a thorough validation of the developed image processing algorithms. Such validation will necessarily involve a comparison of network classification with that of experts in the field. Unfortunately, in our reviewed literature, validation on a large set of test images had only occasionally been performed, and most of the publications about neural networks applications did not ask the question whether a neural network really is the best way of solving the problem, comparison with traditional methods is often neglected.

In addition, for the using of Hopfield neural network to medical image processing, it also should be noticed that it is not so easy to cast an actual medical image processing problem to the architecture of the Hopfield network. Occasionally, the original problem had to be modified before it could be solved by the Hopfield network architecture. Also, convergence to a global optimum can not be ensured.

5. Conclusion and future perspectives

After a general review of neural networks in medical image processing, what the major strengths and weakness of applying neural networks for solving medical image processing tasks are has been answered. Some interesting possibilities for artificial neural network in medical image processing are also noticed.

The first one is what called on-job training, which makes the use of neural networks in changing environments possible [77]. In the use of neural networks in medical image processing, an especially important problem is how to ensure the transferability of a classifier. When a neural network is trained to classify patterns obtained from one setting with a specific class, the neural network may have a poorer and possibly unacceptably low performance when transferred to a novel setting with another class distribution. How to cope with varying prior class distributions is a hard subject for current research. The on-job training may be a valuable improvement over current systems and facilitate transferability solution between different medical image patterns.

The second one is that the using of the emergent novel neural network methods in medical image processing, e.g. artificial neural network ensembles [69], the combination of neural networks and intelligent agents [78], the combination of genetic algorithms with fuzzy fitness and neural networks [79] etc. represent a promising alternative to increase further the effectiveness of medical image processing.

Perhaps, efficient image compressing techniques, Internet compatibility and medical image data base management should be included within the challenge of medical image processing in the near future.