AI for Response Evaluation with PET/CT

Abstract

Positron emission tomography (PET) / computed tomography (CT) are nuclear diagnostic imaging modalities that are routinely deployed for cancer staging and monitoring. They hold the advantage of detecting disease related biochemical and physiologic abnormalities in advance of anatomical changes, thus widely used for staging of disease progression, identification of the treatment gross tumor volume, monitoring of disease, as well as prediction of outcomes and personalization of treatment regimens. Among the arsenal of different functional imaging modalities, nuclear imaging has benefited from early adoption of quantitative image analysis starting from simple standard uptake value (SUV) normalization to more advanced extraction of complex imaging uptake patterns; thanks to application of sophisticated image processing and machine learning algorithms. In this review, we discuss the application of image processing and machine/deep learning techniques to PET/CT imaging with special focus on the oncological radiotherapy domain as a case study and draw examples from our work and others to highlight current status and future potentials.

Introduction

Recent years have witnessed exponential growth in the use of imaging for diagnostic and therapeutic radiological purposes. In particular, positron emission tomography (PET) has been widely used in oncology for the purposes of diagnosis, grading, staging, and assessment of response. For instance, PET imaging with ¹⁸F-FDG (fluoro-2-deoxy-d-glucose), a glucose metabolism analog, has been applied for diagnosis, staging, and treatment planning of lung cancer (1-10), head and neck cancer (11, 12), prostate cancer (13), cervical cancer (14, 15), colorectal cancer (16), lymphoma (17, 18), melanoma (19) and breast cancer (20-22). Moreover, accumulating evidence supports that pre-treatment or post-treatment FDG-PET uptake could be used as a prognostic factor for predicting outcomes (23-27). The combination with computed tomography (CT) further augment the functional/physiological information with anatomical information to better localize disease. A detailed review of recent advances in PET/CT imaging for oncology is presented in our previous work (28).

Quantitative imaging information from hybrid-imaging modalities could be related to biological and clinical endpoints, a new emerging field referred to as “radiomics” (29, 30). We and others have published work to demonstrate the potential of this new field to monitor and predict response to radiotherapy in head and neck (31, 32), cervix (31, 33), sarcoma (34), and lung (35) cancers, allowing for adapting and individualizing treatment regimens. More recently, with the development of deep neural networks (DNNs) as well as the enhanced graphics processing unit (GPU) computing power, deep learning-based image analyses showed promising results in several applications, including segmentation, reconstruction, and outcome modeling, etc. These algorithms allow learning the image representation directly from the raw images in opposite to conventional machine learning approaches that would require manual extraction of these features (36). For instance, Sharif et al. presented an artificial neural networks (ANN) in the wavelet domain for PET volume segmentation (37). Zhao et al. proposed a multi-modality segmentation method based on a 3D fully convolutional neural network (F-CNN), which is capable of taking into account both PET and CT information simultaneously for tumor segmentation (38). Ypsilantis et al. showed that one CNN based PET imaging representation are highly predictive of response to neoadjuvant chemotherapy (39).

In this review, we will discuss the application of advanced machine/deep learning techniques based on PET/CT imaging for response evaluation with specific focus on two major areas of obtaining regions of interest (e.g., cancerous tumors) and image-based prediction of treatment outcomes.

Quantitative image analysis from PET/CT

Hand-crafted radiomics methods, with hundreds or even more radiomics features available, requires feature selection and/or feature extraction as a necessary prior step that aims to obtain the optimal subset or combination of features to achieve better performance and also to reduce the risk of overfitting. Filter, wrapper and embedded methods are commonly used supervised methods for feature selection. Principal component analysis (PCA) (40), clustering and t-Distributed Stochastic Neighbor Embedding (t-SNE) (41) are some examples of unsupervised methods, where no labels are needed. Based on the endpoints, either time dependent (survival) or independent (classification) machine learning algorithms could be applied for the feature selection (42). Typical classification algorithms include logistic regression, support vector machines (SVM), random forests (RF), and neural networks (43-45). Cox regression (46), random survival forests (47) and support vector survival (48) methods, as well as recently proposed deep learning based survival models (49, 50) are available for survival analysis.

Though hand-crafted features introduced above can provide prior knowledge, they also suffer from the tedious designing process and may not faithfully capture the underlying imaging information for the task at hand with potential selection bias. Alternatively, with the development of deep learning technologies, as a branch of machine learning, the extraction of machine learnt features is becoming widely applicable recently. These deep learning methods are generally based on multi-layer neural networks, especially the convolutional neural networks (CNN). In deep learning, the processes of data representation and prediction (e.g., classification or regression) are performed jointly. In such a case, multi-stack neural layers of varying modules (e.g., convolution for filtering or pooling for data reduction) with linear/non-linear activation (mapping) functions perform the task of learning the representations of data with multiple levels of abstraction and subsequent fully connected layers are tasked with classification/regression, for instance (36).

A typical scenario to get such features is to use the data representation CNN layers as feature extractor. Each hidden layer module within the network transforms the representation at one level. For example, the first level may represent edges in an image oriented in a particular direction, the second may detect motifs in the observed edges, the third could recognize objects from ensembles of motifs (51). Patch-/pixel-based machine learning methods use pixel/voxel values in images directly instead of features calculated from segmented objects as in other approaches (52). This removes the need for segmentation, one of the major sources of inter- and intra-variability of radiomic features. Moreover, the data representation removes the feature selection portion eliminating associated statistical bias in this process as noted earlier.

For the CNN network, either self-designed (from scratch) or existing pre-trained structures with natural or medical images (e.g., VGG (53), Resnet (54), two CNN network structure that performed well for ImageNet classification task) can be used. Depending on the data size, we can choose to fix the parameters or fine tune the network using our data, also called transfer learning. Instead of using deep networks as feature extractors, we can use them directly for the whole modeling process. Similar to the conventional machine learning methods, there are also supervised, unsupervised and semi-supervised methods. CNN are similar to regular neural networks, but the architecture is modified to fit to the specific input of large-scale images.

Inspired by the Hubel and Wiesel’s work on the animal visual cortex (55), local filters are used to slide over the input space in CNNs, which not only exploit the strong local correlation in natural images, but also reduce the number of weights significantly by sharing weights for each filter. Recurrent neural networks (RNN) can use their internal memory to process time-dependent sequence inputs (e.g., dynamic PET acquisition) and take the previous output as inputs. There are two popular types of RNN – Long short-term memory (LSTM) and Gated recurrent units (GRU) (56, 57). They were invented to solve the problem of vanishing gradients for long sequences by internal gates that are able to learn which data in the sequence is important to keep or discard.

Deep autoencoders (AE), which are unsupervised learning algorithms (no labels are needed for the input data), have been applied to medical imaging for latent representative feature extraction. There are variations to the AEs, such as variational autoencoders that resemble the original AE and variational Bayesian methods to learn a probability distribution that represents the data (58), convolutional autoencoders that preserve spatial locality (59), etc. Another unsupervised method is the restricted Boltzmann machine (RBM), which consists of visible and hidden layers (60). The forward pass learns the probability of activations given the inputs, while the backward pass tries to estimate the probability of inputs given activations. Thus, the RBMs lead to the joint probability distribution of inputs and activations.

Deep belief networks can be regarded as a stack of RBMs, where each RBM communicates with previous and subsequent layers. RBMs are quite similar with AEs, however, instead of using deterministic units, like RELU, RBMs use stochastic units with certain distribution. As mentioned above, labeled data is limited, especially in the medical field. Neural network based semi-supervised approaches that combine unsupervised and supervised learning by training the supervised network with an additional loss component from the unsupervised generative models (e.g. AEs, RBMs) (61). More details on the machine learning aspect is provided here (62, 63).

Comparing with feature-based methods, deep learning methods are more flexible and can be used with some modifications in various tasks. In addition to classification, segmentation, registration, and lesion detection are widely explored by deep learning techniques. Fully CNN (FCN), trained end-to-end, merge features learnt from different stages in the encoders and then upsampling low resolution feature maps by deconvolutions (64). Unet, built upon FCNN, with the pooling layers being replaced by upsampling layers, resulting in a nearly symmetric U-shaped network (65). Skipping structures combines the context information with the unsampled feature maps to achieve higher resolution. In order to identify Alzheimer’s disease or mild cognitive impairment, Shen et al. exploited deep Boltzmann machine (DBM) to obtain a latent hierarchical feature representation from a 3D patch, and then devised a systematic method for a joint feature representation, as shown in Figure 1 (a), from the paired patches of MRI and PET with a multi-modal DBM by fusing neuroimaging and biological features. Figure 1 (b) and Figure 1 (c) visualize, respectively, the learned connection weights from the MRI pathway and the PET pathway (66).

Figure 1.

(a) The shared feature learning from patches of the heterogeneous modalities, e.g., MRI and PET, with discriminative multi-modal DBM and (b, c) visualization of the learned weights in Gaussian RBMs (bottom) and those of the first hidden layer (top) from MRI and PET pathways in multi-modal DBM (From Shen et al., 2017).

Once the model is built using the algorithms introduced above, the final step is the model evaluation. Ideally, the model should be tested on a large independent patient cohort. Alternatively, resampling methods were developed to evaluate the model performance in the situation of limited sample sizes. Two commonly applied methods are bootstrapping and cross-validation. The basic idea of bootstrapping is to make inference about a population using the sample data by resampling with replacement methods. It was first introduced by Bradley Efron in 1979 (67), and the more commonly used bias-corrected and accelerated (BCa) bootstrap was developed by Efron as well (68). Cross-validation is used to measure (evaluate) the goodness of fit for a prediction model. Models will be developed on the training folds and tested on the validation fold. Typically used frames are 5-fold or 10-fold cross-validation. Cross-validation is commonly used for model hyper-parameter tuning as well. In terms of the metrics that measure the goodness of fit, the receiver operating characteristic (ROC) curve, or the area under the ROC curve (AUC) quantifies the sensitivity and specificity of the model and represents the probability that a randomly selected case is correctly distinguished with a larger probability than a randomly selected case is incorrectly distinguished for the classification task, and c-index is similar to AUC and is used as a measure for survival models, which is the probability of concordance between predicted and observed survival. The instruction of using these techniques can be found in the TRIPOD guidelines (69).

One thing to point out is that the training data used for feature selection and model construction should not be used for the evaluation of the model, which will lead to overoptimistic results and worse generalization for new data. Careful design of training, validation, and test sets within any resampling technique is vital for reducing the selection or optimization bias (70-72), where a balance between fitting and generalization can be achieved. The size of the training data is an intractable problem that depends on the dataset, the complexity of the problem and the learning algorithm. Ideally, more data is desired to train complex models (e.g., deep learning models) and estimate the generalizability of the developed models. However, in reality, especially in the medical field, data scarcity is a challenge hard to be resolved. To deal with this issue, synthetic data methods were applied, such as generative adversarial networks, or simply using slices or patches of images. Theoretically, power analysis is used to estimate the sample size needed to detect an effect. The definition of power is the probability of rejecting the null hypothesis when it is true.

In a review by Zwanenburg, the radiomics workflow as shown in Figure 2 was summarized. Meanwhile several pitfalls during radiomics analysis was pointed out as well, such as the split of training, validation and testing data, class imbalance and incomplete reporting, which should be payed attention to when conducting radiomics study (73).

Figure 2.

a. Data analysis strategies and typical analysis workflow. Generalizability of a radiomics model is assessed through internal and external validation (a). Internal validation should be reported by repeatedly dividing the data into development and validation datasets instead of reusing the development data for validation to avoid optimistic biases. (b). Many steps are only fully performed on the development dataset, and the resulting parameters and results are applied to the validation dataset.

Application of PET/CT based AI models

In the following, we discuss application of PET/CT to radiotherapy, as a case study for oncological applications, with focus on two cases: PET/CT regions of interest delineation/detection and outcome prediction for response evaluation and clinical decision-making using deep learning for PET/CT.

PET/CT deep learning based tumor delineation detection

PET/CT imaging can simultaneously acquire functional metabolic information and anatomical information of the human body. To fuse the complementary information in PET/CT for accurate tumor segmentation is challenging. Some multimodality tumor segmentation methods have been proposed. EI Naqa et al. proposed a multivalued level set model based method to integrate information from multimodality images to identify the biophysical structure (74). Markel et al. used 3D texture features to train a combined decision tree with K-nearest neighbor classifier for segmentation in PET/CT (75). There are works for PET/CT segmentation based on Markov Random Field optimization, graph cut method as well as a ‘topo-poly’ graph model with the intention to combine the information from multimodality images (76-78).

Recently, deep learning based region of interest (ROI) segmentation/abnormality detection methods showed improved performance for different disease sites. Li et al. proposed to use a 3D FCNN to produce a probability map from the CT image in non-small cell lung cancer dataset (79). A fuzzy variational model was then proposed to incorporate the probability map and the PET intensity image for an accurate multimodality tumor segmentation, where the probability map acted as a membership degree prior. They found that only a few samples were needed for training the network to produce the probability map, which is beneficial in clinic research where the dataset is small. The results showed an average dice similarity indexes (DSI) of 0.86 ± 0.05, sensitivity (SE) of 0.86 ± 0.07, positive predictive value (PPV) of 0.87 ± 0.10, volume error (VE) of 0.16 ± 0.12, and classification error (CE) of 0.30 ± 0.12. Moe et al. described an automatic segmentation algorithm using the U-Net architecture for delineation of the gross tumor volume and pathologic lymph nodes of head and neck cancers in PET/CT images (80). Zhao et al. proposed a network started with a multi-task training module, in which two parallel sub-segmentation architectures constructed using deep CNNs were designed to automatically extract feature maps from PET and CT respectively for lung cancer dataset (81). A feature fusion module was subsequently designed based on cascaded convolutional blocks, which re-extracted features from PET/CT feature maps using a weighted cross-entropy minimization strategy. The tumor mask was obtained as the output at the end of the network using a softmax function. Figure 3 shows the network structure and Figure 4 shows the example tumor segmentation results for CT only, PET only and PET/CT conditions.

Figure 3.

The architecture of the proposed co-segmentation network. Two parallel branches were used for feature extraction from the CT image and the PET image, respectively, followed by the feature fusion module. The segmentation result was the output at the end of the network.

Figure 4.

Visual comparison of the segmentation results (blue contour) of different methods and the ground truth (pink contour) for three patients. (a)–(c) Segmentation results of the CNN-based method using CT only, and (d)–(f) segmentation results of using PET only, and (g)–(i) segmentation results of the proposed co-segmentation method.

Gsaxner et al. proposed a method to generate a training dataset from PET/CT by applying thresholding to the PET data for obtaining a ground truth and by utilizing data augmentation to enlarge the dataset for the urinary bladder segmentation (82). Ben-Cohen et al. combined a fully convolutional network (FCN) with a conditional generative adversarial network (GAN) to generate simulated PET data from given input CT data (83). The synthesized PET can be used for false-positive reduction in lesion detection solutions. Clinically, such solutions may enable lesion detection and drug treatment evaluation in a CT-only environment, thus reducing the need for the more expensive and radioactive PET/CT scan. Quantitative evaluation was conducted using an existing lesion detection software, combining the synthesized PET as a false positive reduction layer for the detection of malignant lesions in the liver. Current results showed a 28% reduction in the average false positive per case from 2.9 to 2.1.

A multimodal spatial attention module (MSAM) that automatically learns to emphasize regions (spatial areas) related to tumors and suppress normal regions with physiologic high-uptake was introduced by Fu et al. (84). The resulting spatial attention maps are subsequently employed to target a convolutional neural network (CNN) for segmentation of areas with higher tumor likelihood. Two clinical PET-CT datasets of non-small cell lung cancer (NSCLC) and soft tissue sarcoma (STS) validated the effectiveness of the MSAM in these different cancer types. This MSAM method surpassed the state-of-the-art lung tumor segmentation approach by a margin of 7.6% in Dice similarity coefficient (DSC).

PET/CT deep learning based modeling

In conventional radiomics, El Naqa et al. investigated intensity-volume histogram metrics and shape and texture features extracted from PET images to predict patient's response to treatment for cervix and head and neck cancers and showed good preliminary discriminant power for utilizing functional imaging in clinical prognosis (85). Cook et al. conducted research on patients with non-small cell lung cancer (NSCLC) treated with chemoradiotherapy underwent pretreatment 18F-FDG PET/CT scans and found that baseline 18F-FDG PET scan uptake showing abnormal texture as measured by coarseness, contrast, and busyness is associated with nonresponse to chemoradiotherapy by RECIST and with poorer prognosis (86). Yip et al. assessed the association of and predictive power of FDG PET-based radiomic features for somatic mutation in NSCLC patients (87). They found that found that eight radiomic features and two conventional features were significantly associated with EGFR mutation status (FDR Wilcoxon = 0.01–0.10). Figure 5 showed the PET images for different mutation groups and the predictive power for each feature. Hsu et al. introduced a FDG-PET radiomics tissue classifier for differentiating FDG avid-normal tissues from tumor (88). This work showed that the radiomics features from PET images are able to correctly capture the characteristics of normal tissues and tumor. Figure 6 showed one of the decision tree, the threshold for the random forests and the t-SNE visualization for better understanding of the role of different features.

Figure 5.

(a) From left to right are patients with EGFR mutation, KRAS mutation, and EGFR– and KRAS– tumors. Stage I and III tumors are shown in top and bottom rows, respectively. Arrows indicate locations of lung tumors; (b) AUC. * indicates that AUC is significantly > 0.50 (random guessing) assessed with Noether test (FDR_Noether <= 0.10). Many of the features significantly predict EGFR+ tumors; however, they are not able to predict KRAS+ tumors.

Figure 6.

Visualization of random forest classifier. (a) Binary decision tree for classifying FDG-avid tissues and tumor. Fifty binary decision trees were trained with the random forest classifier. (TR: Tumor, KL: kidney left, BL: bladder, KR: kidney right, HT: heart, BR: brain). (b) Display of threshold values for volume and maximum SUV in all 50 trees. The color coding (red: bladder, green: brain, blue: heart, purple: kidney left, gold: kidney right, teal: tumor, white: undecided) shows the tissue classification made on two features: volume and centroid Z. (c) t-SNE plot illustrating the classification results for all segmented volumes and features using dimension reduction.

Though hand-crafted features are able to embed experts’ prior knowledge, it suffered from tedious feature selection process and might miss other important information in the images. Deep learning method developed on large dataset that uses the whole image as input avoid this issue and provides a promising option for response evaluation in PET/CT. Ypsilantis et al. compared the performance of the two competing strategies: an approach using state-of-the-art statistical classifiers based on radiomics features; and a CNN trained directly from the PET scans. For radiomics feature-based approach, 103 radiomics features including texture and model-based features were extracted and four statistical classifiers (logistic regression, gradient boosting, RFs and SVMs) were adopted for the discrimination of responders/non-responders for 107 patients treated with neoadjuvant chemotherapy. For the deep learning approach, in order to capture patterns of FDG uptake across multiple adjacent slices that might be important for predicting chemotherapy response, all possible sets of three adjacent slices were constructed as input to a three-channel CNN network (3S-CNN). The input and network structures were shown in Figure 7. They showed that the CNN based PET imaging representation are highly predictive of response to therapy and outperformed the feature-based approach (39). Kawauchi et al. conducted a retrospective study included 6462 patients who underwent whole body FDG PET-CT. The CNN network was able to predict the sex, as well as age and body weight to prevent patient misidentification in clinical settings (89). The same group conducted a retrospective study on 3485 sequential patients with malignant disease, who underwent whole-body FDG PET/CT (90). A residual network (ResNet)-based CNN architecture was built for classifying patients into the 3 categories (benign, malignant or equivocal). In addition, they performed a region-based analysis of CNN (head-and-neck, chest, abdomen, and pelvic region). In the patient-based analysis, CNN predicted benign, malignant and equivocal images with 99.4, 99.4, and 87.5% accuracy, respectively. Schwyzer et al. studied the performance of artificial neural network discriminating lung cancer patients based on clinical standard dose, tenfold and thirtyfold reduced radiation dose FDG-PET images. They found that machine learning algorithms may aid fully automated lung cancer detection even at very low dose (91).

Figure 7.

(a)¹⁸F-FDG PET ROIs of a specific tumor i after segmentation embedded into larger square background of standard size of 100 × 100 pixels. Each enlarged slice is denoted by x_i,j and each set of three spatially adjacent enlarged slides is denoted by x_i,k, where j and k represent the slices and triplets of the specific tumor i. In this example only 3 triplets, from the 5 available slices can be formed, so k = 1,2,3; (b) CNN architecture for fusion of 3 adjacent ¹⁸F-FDG PET intra slices into a vector v. The CNN architecture is composed from 4 convolutional and 4 max-pooling layers denoted by $[U^{(i)}{]}_{i=1}^4$ and $[V^{(i)}{]}_{i=1}^4$ (From Ypsilantis et al., 2015).

Sibille et al. proposed a CNN to detect foci positive for 18F-FDG uptake, predict the anatomic location, and determine the expert classification (suspicious or non-suspicious) in patients with lung cancer and lymphoma (92) . This study included 629 patients. Fig. 8 shows the network structure where there are PET/CT crop inputs, PET–maximum intensity projection (MIP) and atlas position input. The output contains both classification and location predictions.

Figure 8.

Convolutional neural network (CNN) for automated detection of foci positive for fluorine 18 (18F)–fluorodeoxyglucose (FDG) uptake, definition of their anatomic localization, and classification of lesions suspicious for tumor. The position of an 18F-FDG–positive focus, transformed in the atlas space and a PET–maximum intensity projection (MIP) and nine coronal multiplanar reconstructions (MPRs), are extracted from PET and CT and are used by the CNN to predict the localization and classification.

PET volumes of interest were first segmented by using a fixed thresholding algorithm to create candidate regions. Whole-body CT examinations were aligned to an anatomic atlas location; the displacement found was used to translate the manually defined 18F-FDG–positive foci to a standardized position, referred to as atlas position. Nine coronal multiplanar reconstructions were extracted to consider anatomic surroundings of an 18F-FDG–positive focus. A coronal maximum intensity projection (MIP) of the whole-body 18F-FDG PET was reconstructed. They found that both 18F-FDG PET and CT information are needed to obtain the highest accuracy.

Wang et al. compared one deep learning method and four classical machine learning methods (RFs, SVMs, adaptive boosting, and ANN) for classifying mediastinal lymph node metastasis of non-small cell lung cancer (NSCLC) from 18F-FDG PET/CT images (93). The five methods were evaluated using 1397 lymph nodes collected from PET/CT images of 168 patients, with corresponding pathology analysis results as gold standard. It showed that the performance of CNN is not significantly different from the best classical methods and human doctors for classifying mediastinal lymph node metastasis of NSCLC from PET/CT images. However, CNN does not need tumor segmentation or feature calculation, it is more convenient and more objective than the classical methods. Peng et al. constructed radiomics signatures and a nomogram for predicting disease-free survival (DFS) based on the features from PET and CT images for individual induction chemotherapy (IC) in advanced nasopharyngeal carcinoma (NPC) (94). Both deep learning features and handcrafted features were extracted from the lung tumor and lymph nodes (4 ROIs) based on the PET/CT images to quantify the tumor phenotype. For each ROI, a deep CNN was trained to extract deep learning features. A set of handcrafted features was also extracted. The radiomics nomogram achieved a c-index of 0.722 (95% CI, 0.652–0.792) in the test set. Deep learning PET/CT-based radiomics could serve as a reliable and powerful tool for prognosis prediction and may act as a potential indicator for individual IC in advanced NPC.

Current issues and future directions

PET image characteristics

Generally speaking, PET images have lower resolution than CT or MRI in the order of 3-5 mm, which is further worsened under cardiac or respiratory motion conditions due to longer acquisition periods. Moreover, PET images are susceptible to limited photon count noise. Advances in hardware such as crystal detector technologies (95) and software such as image reconstruction techniques (96) are poised to improve PET images quality and their subsequent use. Deep learning reconstruction methods are also promising to obtain better spatial resolution PET images as discussed in previous sections, which cost much less to realize compared with the hardware development approach.

Improved PET-based outcome models

The conventional feature-based machine learning methods embedded experts’ prior knowledge, while the feature selection process is usually blamed to be unstable, especially for high-dimensional feature input. In comparison, deep learning methods are able to bypass the feature designing or selection stages, and have shown great potential in clinical image applications, including segmentation, reconstruction, classification, actuary analysis. However, there are several issues restraining its clinical practice, two major ones are the scarcity of data and the difficulty in interpretability. For limit sample size, data augmentation (e.g., affine transformation of the images) during training is commonly adopted. Transfer learning is also a widely used strategy to reduce the difficulty in training, by transferring deep models trained on other datasets (natural images) and then fine-tuning on the target dataset. The structures of the networks can be modified to reduce overfitting as well, such as, adding dropout and batch normalization layers. Dropout randomly deactivates a fraction of the units during training and can be viewed as a regularization technique that adds noise to the hidden units (97). Batch normalization reduces the internal covariate shift by normalizing for each training minibatch (98).

For the interpretability, the large number of interacting, non-linear parts makes it hard to understand deep neural networks (99, 100). Aiming at improving the interpretability of radiomics for clinicians, graph-based approaches can be utilized (101), and better visualization tools for deep learning are being developed such as activation maps highlighting regions of the tumor that impact the prediction of the deep learning classifier are also being proposed. Several groups have developed methods for identifying and visualizing the features in individual test data points that contribute most towards a classifier’s output. A well-known method is the Local Interpretable Model-Agnostic Explanations (LIME), which provides explanations of decisions for any machine learning model by local approximation with simpler methods (102). Selvaraju et al. proposed a method called Gradient-weighted Class Activation Mapping (Grad-CAM), which uses the gradients of any target concept (say logits for ‘dog’ or even a caption), flowing into the final convolutional layer to produce a coarse localization map highlighting the important regions in the image for predicting the concept at hand (103). Chattopadhyay et al. further generalized this method into Grad-CAM++ that can explain occurrences of multiple object instances in a single image compared to state-of-the-art (104).

Conclusions

Image processing constitutes an indispensable set of tools for analyzing and extracting valuable information from PET/CT images, this is made more powerful with the emergence of artificial intelligence techniques that is reshaping the future of radiological sciences including nuclear medicine (105). We presented in this review an overview of different traditional features that could be extracted from PET/CT images and various modern machine/deep learning methods that could use these images as direct input for different applications including contouring and response prediction. We have shown that incorporation of different anatomical information from CT into PET is feasible and could yield better results. However, there are challenges still in the use of PET with some issues related to inherited image quality, others related to standardization of image acquisition protocols and reconstruction algorithms, and also some general issues like shortage of data and inadequate interpretability of the models for deep learning approaches that appeared not only in PET/CT applications but across the spectrum of medical imaging applications. Nevertheless, advances in hardware and software technologies will further facilitate wider application of advanced image processing and machine learning techniques to PET/CT and hybrid imaging to achieve better clinical results and personalized treatment regimens. In particular, the synergy between image analysis and machine learning could provide powerful tools to strengthen and further the utilization of PET/CT in clinical practice and realize its potentials for a better personalized health care and improved patient’s quality of life.