Deep Learning in Nuclear Medicine and Molecular Imaging: Current Perspectives and Future Directions

Abstract

Recent advances in deep learning have impacted various scientific and industrial fields. Due to the rapid application of deep learning in biomedical data, molecular imaging has also started to adopt this technique. In this regard, it is expected that deep learning will potentially affect the roles of molecular imaging experts as well as clinical decision making. This review firstly offers a basic overview of deep learning particularly for image data analysis to give knowledge to nuclear medicine physicians and researchers. Because of the unique characteristics and distinctive aims of various types of molecular imaging, deep learning applications can be different from other fields. In this context, the review deals with current perspectives of deep learning in molecular imaging particularly in terms of development of biomarkers. Finally, future challenges of deep learning application for molecular imaging and future roles of experts in molecular imaging will be discussed.

Introduction

The amount of biomedical data including images and medical records as well as omics data is rapidly increasing [1]. However, the heterogeneous, complex, and multidimensional nature of biomedical data make it difficult to decipher clinical meaning [2]. Thus, machine learning (ML), a method set of artificial intelligence in which a computer captures patterns underlying data and utilizes them to help decision making, have been extensively and increasingly applied for handling biomedical data. Recently, deep learning (DL), a special type of machine learning methods, has emerged as a new area of machine learning research [3]. DL is a class of machine learning that automatically learns hierarchical features of data by multiple layers composed of simple and nonlinear modules. It transforms the data to representations which are important for discriminating the data [4]. Since this method was successfully applied and overwhelmingly beat other previous ML methods for visual recognition tasks at a competitive challenge in 2012, ImageNet [5, 6], it has dramatically improved tasks of various scientific and industrial fields including not only computer vision but speech recognition, drug discovery, and bioinformatics [4, 7, 8]. Subsequently, DL techniques have become the method of choice in computer vision and several image processing tasks. This success came from development of new modules enabling learning of deep structures and reducing overfitting as well as efficient use of hardware including general-purpose graphics processing units. Most of all, structured large datasets such as ImageNet data contribute to the success of this technique.

DL has been gradually applied to medical image data as medical image analyses are considerably similar to computer vision techniques. As well-structured and labeled medical data are relatively limited, applications of DL were relatively delayed. However, the number of reports regarding DL applications in medical data grew rapidly in 2016 (Fig. 1). The application areas are widening and range from anatomical segmentation to disease classification [9]. Although the many studies initially started from relatively small datasets by using a pretrained DL model as a feasibility study, a robust validation of the medical application has been required [10, 11]. In this regard, medical image big data have started to be collected to validate the feasibility of medical applications. For example, recently, Google researchers collected large data sets consisting of more than 120,000 retinal fundus images for diagnosing diabetic retinopathy and showed high sensitivity and specificity for the detection [12]. An automated skin lesion classification system comparable to dermatologists was recently developed using more than 120,000 clinical images [13]. These systems based on large datasets will further improve DL-based diagnosis and eventually make considerable changes in medical fields by supporting clinical decisions in the clinic in the near future.

Fig. 1.

Number of papers regarding deep learning applications in biomedical fields. Number of papers dealing with deep learning in biomedical fields are recently rapidly increasing. Bar graphs represent results of the number of PubMed paper searched by words: [Deep learning AND medical image] and [Deep learning AND (medical OR biomedical)], respectively

Due to the rapid application of DL in biomedical data, molecular imaging can also be influenced. The application of DL for molecular imaging can be ranged from disease diagnosis and classification to image processing. The potential impact of DL throughout the entire process of molecular imaging acquisition and interpretation requires nuclear medicine physicians and molecular imaging researchers to have knowledge about this technique. To this end, this review will give an overview of deep learning techniques particularly focusing on image recognition. In addition, the next section will introduce recent progress and current perspectives of medical application of DL focusing on molecular imaging. As the adaptation of DL in molecular imaging may inevitably happen, the direction of DL application in molecular imaging and the possible changes in the role of physicians will be discussed.

Brief Overview of Deep Learning for Images

Basics of Neural Networks: Single Layer Perceptron to Deep Neural Networks

In this section, DL concepts particularly for image analysis are briefly introduced. The detailed basic concepts of DL are reviewed by LeCun et al. [4] and general applications of DL in biomedical fields are reviewed by Mamoshina et al. [14].

DL is a type of ML defined by models with many hierarchical layers of information processing compared with conventional ‘shallow’ learning. ML including both shallow and deep learning is a field of artificial intelligence in which learning occurs without explicit programming. Both methods capture patterns underlying complicated data and utilize them as discriminative features of data. DL has advantages in learning intricate patterns from high dimensional raw data while conventional shallow learning usually requires handcrafted features extracted from raw data. Regardless of depth of layers, ML is generally divided into two methods, supervised and unsupervised learning. Supervised learning typically predicts a target variable representing a specific class (classification) of data or specific continuous values (regression). The inputs are raw data or features paired with the target label or target variable. Thus, a supervised ML model can be simply summarized by a prediction model that predicts target values from the data. Parameters of the model are adjusted to accurately predict the target by training process. On the other hand, unsupervised learning is a type of ML that finds patterns of the data without target labels or variables. Data clustering is an example of unsupervised learning. In terms of applications of unsupervised learning, representations extracted by unsupervised learning can be used for the other supervised tasks as well as data clustering.

DL typically means deep neural networks. Thus, neural networks, a type of machine learning, are the basis of recent deep learning techniques. The perceptron is the earliest neural network model which consists of a single layer (Fig. 2a). A simple single layer model which aims at discriminating malignancy from benign lesion is exhibited in Fig. 2a. Inputs of the model are features of the lesion such as tumor size and FDG uptake. The model parameters, W = (w₁, w₂,…,w_n), are multiplied to the inputs and then the activation function is applied. An example of simple activation function is that the output is 1 if the value is more than 0 and otherwise, the output is 0. The parameters of this simple model are optimized for the accurate discrimination. This process is defined as training. The training can be formulated as the minimization of the error between predicted outputs and real values. Practically, the gradient descent method which iteratively updates the model parameters according to gradient of the error function is used for the training [15]. This single layer perceptron has limitations in complicated nonlinear data patterns (Fig. 2b). As shown in the left plot of Fig. 2b, a linear discriminative line can differentiate malignancy from benign lesions while it cannot differentiate it for some nonlinear data distribution patterns such as the right plot. To overcome these certain issues, hidden layers between inputs and outputs are added. This type of neural network is a well-known traditional neural network, multi-layer perceptron (Fig. 2c). Although the application of hidden layer has improved performance of neural networks, it is difficult to train deep architectures. Deep neural networks have received attention after utilization of a new training method, pretraining [16, 17]. Efficient training was achieved by unsupervised feature extraction from input data followed by supervised learning.

Fig. 2.

Brief overview of neural networks. a An example of a single-layer perceptron is built for discriminating malignancy from benign lesion. Image features such as tumor size and FDG uptake are inputs of the model. Model parameters, weights, are multiplied to input variables and the output is predicted by a simple activation function. b A simple perceptron model has limitations in nonlinear problems. Data exhibited in the left plot are simply divided by a discriminative line while those in the right plot require a nonlinear and complex curve for the discrimination. c Multilayer perceptron models can be built by hierarchical composition of perceptron. Hidden layers hierarchically capture representations of data. d While conventional multilayer perceptron models use manual features as inputs, current convolutional neural network models directly use pixels/voxels. e An explanation of convolutional filter is exhibited. Convolutional layers of the network architectures have a certain number of convolutional filters as network model parameters. It produces feature maps as outputs by multiplying input pixels and filter values. f An example of a convolutional neural network represents multiple convolutional layers followed by pooling layers. Hierarchical feature maps are eventually connected to feature vectors and then connected to outputs

While the pretraining initiated the popularity of deep learning, current models particularly for image analysis fields can be summarized by simplified direct supervised training of specialized architecture such as convolutional neural networks (CNN). While previous network architecture used data with vector form, current models using CNN utilize structural information of neighboring pixels/voxels of the image. Besides, another currently popular architecture, recurrent neural network (RNN), considers information of the time-varying pattern of the data. In addition to these specialized network architectures, various techniques have improved performance by reducing overfitting and efficient training. They include special layers, such as dropout [18], batch normalization [19], and modified activation function such as rectified-linear unit (ReLU) [20]. Furthermore, network models trained by large datasets including ImageNet are used to extract general image features, so that they can transfer to another domain such as medical image classification as aforementioned pretraining [21]. Thus, current concepts of deep learning mean not only neural networks with multiple layers but also special architectures, such as CNN and RNN, combined with the various techniques improving the performance and training process. As CNNs are the most widely used for medical image analysis, the next section will additionally describe brief concepts of CNNs.

Convolutional Neural Networks

Traditional neural networks and general machine learning models use inputs as features with vector form. As aforementioned example, to build a machine learning model to differentiate malignancy from benign lesion, we select features such as tumor size and FDG uptake from images. Instead of these features, CNN generally directly uses pixels/voxels as inputs (Fig. 2d). To use structural information of neighboring pixels, convolutional layers are used instead of densely connected perceptron layers. A convolutional layer consists of a specific number of convolutional filters. The example of a 3 × 3 convolutional filter application to an image matrix is exhibited in Fig. 2e. For convolutional layers, values of convolutional filters correspond to weight parameters of the fully-connected neural network layers. Thus, convolutional filter values are adjusted during the training process. As a convolutional filter generated an output feature image, the number of convolutional filters are the number of feature images. In general, after the convolutional layer, consisting of a certain set of convolutional filters, an activation layer follows. Pooling layers are usually added to convolutional layers to subsample feature maps to reduce the amount of learning parameters of the model. These types of layers are hierarchically connected to the output. An example of a simple CNN model is represented in Fig. 2f [22]. The recent model is similar with this figure, however, it has deeper layers and a larger number of convolutional kernels. The dramatic improvement of image recognition in 2012 was achieved by AlexNet, which consists of five convolutional layers and combines modified activation function, ReLU [6]. After 2012, novel architectures improve the performance by using deeper architectures. In 2014, a 22-layer network named Inception [23] was the winner of ImageNet and in 2015, 152-layer network named ResNet further improved performance of image recognition [24]. These architectures and trained models were applied to other domains including medical images. For example, the diabetic retinopathy detection model [12] was based on fine-tuning of Inception model. The application of DL has widened becuase of its flexibility. For example, combining CNN with RNN produced interesting results such as automatic image captioning [25] and various CNN models were also used for object detection [26, 27] and object segmentation [28, 29] from images. These techniques have also been applied to medical images for diagnosis, lesion detection, and segmentation.

Perspectives of Deep Learning in Nuclear Medicine Molecular Imaging

The Distinct Characteristics of Molecular Imaging in DL Application

Even though DL applications in medical images rapidly grow, molecular imaging has distinct characteristics particularly in terms of its purposes and application. Firstly, the aim of the application of deep learning to molecular imaging is substantially different from computer vision fields. While DL models in image recognition aimed at identifying the ground-truth class, the ground-truth of DL is ambiguous in medical fields. Although recent diagnostic applications of DL to medical images generally used clinical or pathologic diagnosis as a ground-truth label, diagnosis can be varied according to pathologists as well as clinicians [30]. Furthermore, in many cases, various disorders are defined as a deviation from the normal population spectrum rather than clear-cut discriminative states. Thus, the application of DL to molecular imaging needs to aim at developing a good biomarker which can define the spectrum of disorders, monitor the status and reflect patients’ outcome instead of simple classification. Besides, molecular imaging aims at representing and visualizing the biological process with regard to physiologic and pathologic changes. The biological process which molecular imaging aims at already reflects the subjects’ outcome as well as surrogate marker of disease pathophysiology. To this end, the application of DL should focus on the discovery of crucial and latent information from imaging data. It should make the best use of molecular imaging information and discover a key pathophysiologic process by summarizing high-dimensional data to a few characteristic and discriminative parameters. The importance of this purpose of DL application in molecular imaging can be inferred from the reason maximum standardized uptake value (SUV) is widely used as the representative parameter for cancer PET imaging in spite of its limitations [31]. Maximum SUV is a single representative value for characterizing a lesion which consists of more than thousands of voxel data. This is a one-dimensional value that quantitatively summarizes an image as a set of high dimensional data to explain tumor glucose metabolism associated with patients’ outcome. Going beyond such a conventional imaging parameter, DL can be a method to capture the most important and discriminative information from the multidimensional and multimodal data of molecular imaging.

Secondly, in terms of data structure and size, molecular imaging is different from natural images. Even more, compared with other medical images such as simple X-ray, it is more difficult to collect large datasets for molecular imaging. As the spectrum of biological targets is very broad, there are numerous types of molecular imaging. In addition, several modalities, e.g., PET, SPECT, optical imaging, and MRI, increase their variety. As it substantially requires case-by-case applications of DL, training with small-sized samples is inevitable. Thus, a challenging issue is to apply deep learning to small medical image datasets. To overcome this issue, transfer learning has been widely used for the application of DL. The models trained by ImageNet, such as AlexNet and ResNet, were used for medical image feature extraction which can be further used for classification, segmentation, and lesion detection. For example, Wang H et al. used AlexNet as a feature extractor for differentiating mediastinal lymph node metastasis of lung cancer from FDG PET/CT [32]. Another method to overcome the limited number of images was to use image patches instead of full-size images. It was commonly used for anatomical segmentation in medical images [33–36]. For segmentation tasks, because classes are designated for each voxel, it is effective to use patch-based learning. Though data augmentation is generally used for natural images, medical images are cautious because manually transformed images could have different clinical information. Nonetheless, to overcome the data size issue, some studies have tried image rotation and flipping for data augmentation [32, 37, 38]. Furthermore, molecular imaging is sometimes used for detecting rare conditions, which require appropriate training methods to solve class imbalance. While the commonly used DL models were developed by data with uniform classes, the training for molecular imaging is usually performed by uneven diagnostic classes. Segmentation and detection also requires identifying small sized lesions in large image matrices. These issues of unique characteristics of molecular imaging require an appropriately adjusted neural networks model.

Briefly, application of DL for molecular imaging needs to focus on the development of good biomarkers rather than simple image classification. As molecular imaging itself provides biologically important information, DL has roles in discovering a crucial biomarker from high-dimensional data and maximum utilization of image information instead of simply replacing human’s visual cognition. To achieve this goal, DL should be properly applied to molecular imaging considering its unique characteristics compared to natural images. Thus, the next sections will discuss recent DL application results and directions in the light of biomarker development. It will be discussed by dividing the applications of DL into two categories: biomarker acquisition supported by DL and discovery of DL-based biomarker (Table 1).

Table 1.

Directions of deep learning applications for molecular imaging

	Indirect	Direct
Explanation	Imaging biomarker acquisition supported by DL	DL-based imaging biomarker
Examples	✓ Segmentation helps measurement of tracer uptake in target regions	✓DL output scores for diagnostic classification
	✓ Low dose acquisition and image reconstruction	✓DL-based image features
	✓Predicting CT images for attenuation correction	✓Combining multimodality data

Acquisition of Imaging Biomarker Supported by Deep Learning

DL application of medical images could indirectly improve well-known biomarkers provided by molecular imaging. Because recent clinically applicable molecular imaging studies, e.g., PET/CT and SPECT/CT, are combined with anatomical imaging, the application of DL to anatomical imaging for lesion delineation and organ segmentation can support quantification of coregistered molecular imaging. Furthermore, DL can be used for improvement of image processing and acquisition which support robust imaging biomarkers by producing better images.

The most common subject of DL application in medical image is segmentation using anatomical CT and MR images. It is ranged from normal anatomical structure segmentation such as whole brain extraction [34, 39], brain substructure, heart and airway segmentation [36, 40–43] to pathological lesion segmentation [44–46]. Before the DL applications, several methods have been reported for the segmentation. However, DL-based medical image segmentation shows faster results as well as better performance. For example, CNN-based striatum segmentation took only a few seconds and showed better accuracy than conventional methods, while FreeSurfer, a widely used brain image processing tool, required several hours [36]. Because most of DL-based segmentation methods are fully automated, quantitative analysis of combined functional images can be accurately and easily performed by utilizing them. For example, using tumor segmentation techniques based on DL, we can simply obtain well-known parameters which require tumor volumes such as mean SUV or total lesion glycolysis. Utilization of this technique may offer a good solution for the issue of variability in tumor volume parameters associated with tumor delineation [47, 48]. Using automated DL-based brain structure segmentation, brain molecular imaging such as amyloid PET and dopamine transporter SPECT/PET can be accurately and easily quantified without complicated processing such as spatial normalization (Fig. 3).

Fig. 3.

An example of imaging biomarker acquisition supported by deep learning. Using automated deep learning-based brain structure segmentation, a well-known brain imaging biomarker such as dopamine transporter SPECT/PET can be easily and accurately quantitated

Another application area for improving imaging biomarker is related to image acquisition and processing. DL has markedly improved image processing fields as well as image recognition. An example is super-resolution, which predicts a high resolution image from low resolution information [49, 50]. This kind of concept could be extended to the PET images estimating standard images from low-dose PET images [51]. Recent studies suggest DL-based reconstruction of sinograms which can outperform computational time and show comparability with conventional iterative algorithms [52, 53]. Even though this kind of technique requires evidence of clinical equivalence, it can be expected that they support reducing radiation exposure as well as scan and reconstruction time.

As one of various application fields of DL, specific neural network models can be used for generating images of different modality [54]. Some neural network models were developed for predicting pseudo-CT images from MR images [55, 56]. These models could produce realistic CT images corresponding to MR images though synthesized CT images could hardly apply to clinical decision making such as lesion detection and characterization. Nonetheless, this method can be used for solving CT-less attenuation correction particularly for PET/MR [57]. Moreover, as this method is bidirectional, various ways of translation are feasible. As a future work, pseudo-image generation can be combined with conventional image processing and quantification methods such as partial volume correction for SPECT and PET images.

Discovering of Deep Learning-Based Biomarker

DL can be directly applied to molecular imaging to obtain reliable biomarkers. Deep CNN models for classifying each subject’s image is a representative direct application of DL as a supervised learning. A study of deep CNN application to mediastinal metastatic lymphadenopathy detection using FDG PET was built by supervised training [32]. Multimodal images including PET and MRI were used as inputs of DL models for the disease classification problems. A report used manual image features of PET and MRI to discriminate Alzheimer’s disease from normal controls [58]. Another recent study suggested a deep 3-dimensional CNN model that used a pair of FDG and florbetapir PET images as input and discriminated Alzheimer’s disease from normal controls [59]. In terms of biomarker development, a quantitative marker representing patients’ outcome based on image data has been required instead of simple diagnostic classification of images. The CNN model using FDG and florbetapir PET images was directly transferred to mild cognitive impairment patients to identify who would rapidly convert into full-blown dementia [59]. In result, the final output of the model could be used for future cognitive score decline. Thus, the final output of deep neural network could be used for a biomarker to predict future cognitive decline (Fig. 4a). Another directly extracted biomarker using DL was reported in dopamine transporter imaging [60]. The model trained for discriminating FP-CIT SPECT images of Parkinson’s disease from normal was applied to scans without evidence of dopaminergic deficit (SWEDD) patients determined by human reading. The DL-based model revealed that a number of SWEDD patients already had abnormal imaging patterns at baseline, which suggested the output of DL could be used for refinement of diagnosis for Parkinson’s disease and SWEDD. Briefly, the role of DL in such functional imaging is not limited to image classification but extended to provide a quantitative output value summarizing multidimensional and multimodal image data, which can be used for the predictive marker.

Fig. 4.

Two types of deep learning-based imaging biomarkers. a Outputs of deep learning model can be used as a biomarker. A deep convolutional neural network model for discriminating Alzheimer’s disease could be directly transferred into identification of mild cognitive impairment patients who would rapidly convert to dementia. Additionally, the output score of the model was well correlated with future decline of cognitive score. b As deep learning is an automated discriminative feature learning, hierarchical features of neural networks can be used as multidimensional biomarkers. Image features related to prognosis or treatment response can be selected after the training of deep learning models

Since DL is a representation learning, DL-based features automatically extracted from images can be used for imaging biomarkers. Some recent studies define these features as ‘DL-based radiomics features’ [61, 62]. ‘Radiomics’ is originally an emerging concept that represents a series of quantitative analysis for identifying high-dimensional image features which can provide predictive and prognostic information [63, 64]. So far, the features have been manually extracted, which included texture features as well as simple parameters such as mean and maximum SUV. On the other hand, as DL is a feature learning method, the hierarchical features of hidden layers can be used as radiomics features (Fig. 4b). A study suggested DL-based image features could be related to patients’ prognosis from chest CT images [61]. Another study using MR images showed that DL model trained for tumor segmentation can be used as a feature extractor, which could identify predictive features for IDH1 mutation in low-grade glioma [62]. DL-based features can be obtained by supervised learning as well as unsupervised learning. A study identified autism-related functional connectivity patterns using an autoencoder, a type of unsupervised learning, for brain network analysis [65]. Though DL-based features are optimized for image discrimination and effectively summarize high-dimensional image data, their usage as a clinically reliable biomarker should be validated in large, independent cohorts.

The direct extraction of DL-based image biomarkers may facilitate utilization of molecular imaging in precision medicine. The key to successful precision medicine relies on the development of good biomarkers that define individual variability as the basic concept of precision medicine to plan prevention and treatment strategies taking individual variability into account [66]. DL can play a role in translating high-dimensional and complex images into quantitative and objective values. Though visual analysis and manual feature extraction such as SUV also provide information on molecular imaging, application of DL enriches the quantitative information by capturing the most discriminative features of images.

Future Directions and Challenges

DL has received great attention in the medical imaging domain because of its success in various scientific and industrial fields. Due to the unique features of molecular imaging, DL should be appropriately modified for molecular imaging analysis. Most of all, the aim of DL application in molecular imaging is different from previous common usage of DL. These images represent physiologic and pathologic processes which require discovering crucial and latent biological information instead of simple diagnostic classification. Thus, DL in molecular imaging should focus on extracting good biomarkers from multimodal and high-dimensional image data.

Although DL has only recently begun to be applied to molecular imaging fields, there are some challenges to be solved in terms of biomarker discovery based on DL. Firstly, as the aim of DL application is to capture good prognostic and predictive information, the target of DL training needs to be diversified. So far, most of the medical application of DL has been aimed at diagnostic classification. However, the DL-based biomarker is aimed at predicting individual subjects’ outcome and treatment response. A good example of another clinically feasible training target is the usage of specialized regression for survival data. Recent studies suggested a prognostic stratification model based on genomic data and DL, which was trained by survival time and events [67, 68]. This DL-based survival analysis can provide a robust prognostic score from the patients’ data. This type of method can be applied to the patients’ molecular imaging data for risk stratification. Secondly, clinical application of DL-based molecular imaging diagnostics requires diagnostic uncertainty and diagnosis of multiple domains. Current DL models are optimized for classification and regression without suggestion of decision reliability. However, the uncertainty measure of the DL-based decision is critical in the clinical setting. Cases where it is difficult to determine diagnosis or produce quantitative value should be identified as they need additional diagnostic tests. Recent studies have attempted to combine Bayesian approximation with DL for uncertainty measurement [69], which will be used in clinical application of DL. Third, application of unsupervised learning will be increased to exploit unlabeled image data and clinical implication. The large dataset of pairs of image data and confirmed diagnosis is limited in spite of the large amount of raw image data. Unsupervised learning is used for image feature extraction, which can be applied to supervised learning with relatively small-sized datasets. Additionally, clustering of data, a result of unsupervised learning, may give additional insights of subgroups of disorders. A clinically permeated example of unsupervised clustering is breast cancer subtyping based on gene expression data [70]. In this context, we expect patients subtyping based on DL-based molecular imaging biomarkers. DL-based molecular imaging biomarker may support individualized risk stratification and personalized treatment by playing an important role in precision medicine.

Roles as an Expert in Nuclear Medicine and Molecular Imaging

Due to the recent development of DL, a myth that the practice of visual tasks in medicine including nuclear medicine image reading will be replaced by machines has emerged. Even though it is difficult to predict the future medical environment, it will be inevitable that some specific tasks are replaced by automated systems. For example, conventional image interpretation tasks such as detecting lesions and simple diagnostic classification may be considerably replaced. Although the clinical value of DL-based conventional nuclear image interpretation has not yet been fully proved, it is evident that DL-based interpretation has the great advantage of the absence of interobserver variability as reported in the DL-based model for FP-CIT SPECT interpretation [60]. Furthermore, the model showed higher accuracy than visual reading for discriminating Parkinson’s disease [60]. For common radiologic images such as mammography, DL-based interpretation showed comparable diagnostic classification to experts’ reading for specific tasks [71]. Nonetheless, the key to molecular imaging is to provide good biomarkers reflecting biological processes. The role of DL in molecular imaging will be capturing maximum information from images by generating predictive biomarkers, which is not related to replacement of human tasks. In this regard, the role of nuclear medicine physicians will be changed to translate and integrate biomarkers automatically extracted from image data for clinical decision making. Various DL applications such as anatomical segmentation and image processing will indirectly help to obtain and augment well-known biomarkers. Image features and biomarkers directly obtained by DL will be used to predict clinical outcomes as well as select individualized treatment options. Thus, for nuclear medicine physicians, knowledge of multimodal data and analytic methods will be needed for the translation of molecular imaging data to clinical implications.

DL applications in molecular imaging will eventually facilitate precision medicine. It will help to make the most of information in molecular imaging. To accelerate proper DL application in molecular imaging, more emphasis should be placed on the intent of each molecular imaging as DL aims at developing good biomarkers. Furthermore, large and open databases are needed to advance the techniques. The innovation of DL in computer vision was initiated by a publicly available large dataset, ImageNet. Large genomic databases such as The Cancer Genome Atlas have accelerated the understanding of cancer and development of novel biomarkers based on big omics data analysis. Recently, data sharing has started to be seriously discussed for raising integrity and accelerating medical research [72]. In medical imaging fields, large datasets in common modalities such as chest X-ray data have started to open [73]. Accordingly, it is urgently required to establish large databases of molecular imaging data as well. It will be the starting point for molecular imaging as an important part of future data-driven precision medicine.

Conclusion

Medical decisions are made by a comprehensive interpretation of all relevant patient data including symptoms, signs, laboratory tests, and imaging. As reviewed in this article, DL has advantages in automatically extracting discriminative features in high-dimensional data. Thus, DL will largely impact medical field tasks particularly related to quantitative analysis. As molecular imaging is a set of image data which provides quantitative information, various DL techniques will be rapidly applied to molecular imaging. Even though some current tasks may be partly replaced by automatic systems, DL may enhance usage of molecular imaging as it can maximally extract information obtained by imaging data to develop biomarkers. As experts in translating molecular imaging information to clinical decision making, we need to focus on exploiting DL for developing biomarkers from molecular imaging to be an important part of the precision medicine era.

Compliance with Ethical Standards

Conflict of Interest

Hongyoon Choi declares no conflict of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

For this study formal consent is not required.