Classification of Myocardial ¹⁸F-FDG PET Uptake Patterns Using Deep Learning

Abstract

Purpose

To perform automated myocardial segmentation and uptake classification from whole-body fluorine 18 fluorodeoxyglucose (FDG) PET.

Materials and Methods

In this retrospective study, consecutive patients who underwent FDG PET imaging for oncologic indications were included (July–August 2018). The left ventricle (LV) on whole-body FDG PET images was manually segmented and classified as showing no myocardial uptake, diffuse uptake, or partial uptake. A total of 609 patients (mean age, 64 years ± 14 [standard deviation]; 309 women) were included and split between training (60%, 365 patients), validation (20%, 122 patients), and testing (20%, 122 patients) datasets. Two sequential neural networks were developed to automatically segment the LV and classify the myocardial uptake pattern using segmentation and classification training data provided by human experts. Linear regression was performed to correlate findings from human experts and deep learning. Classification performance was evaluated using receiver operating characteristic (ROC) analysis.

Results

There was moderate agreement of uptake pattern between experts and deep learning (as a fraction of correctly categorized images) with 78% (36 of 46) for no uptake, 71% (34 of 48) for diffuse uptake, and 71% (20 of 28) for partial uptake. There was no bias in LV volume for partial or diffuse uptake categories (P = .56); however, deep learning underestimated LV volumes in the no uptake category. There was good correlation for LV volume (R² = 0.35, b = .71). ROC analysis showed the area under the curve for classifying no uptake and diffuse uptake was high (> 0.90) but lower for partial uptake (0.77). The feasibility of a myocardial uptake index (MUI) for quantifying the degree of myocardial activity patterns was shown, and there was excellent visual agreement between MUI and uptake patterns.

Conclusion

Deep learning was able to segment and classify myocardial uptake patterns on FDG PET images.

Keywords: PET, Heart, Computer Aided Diagnosis, Computer Application-Detection/Diagnosis

Supplemental material is available for this article.

©RSNA, 2021

Summary

A deep learning approach was developed to automatically segment the left ventricle and classify patterns of myocardial fluorine 18 fluorodeoxyglucose PET uptake in patients referred for oncologic imaging.

Key Points

• ■ A quantitative myocardial uptake index was developed, which maps patterns of uptake to a quantitative scale from no uptake (0) to diffuse uptake (1).

• ■ There was excellent agreement between left ventricular volume in partial and diffuse uptake categories, with lower agreement in the no uptake category, in patients referred for oncologic imaging.

• ■ There was moderate agreement between ground truth myocardial fluorine 18 fluorodeoxyglucose PET uptake patterns and classifications obtained using deep learning.

Introduction

Fluorine 18 (¹⁸F) fluorodeoxyglucose (FDG) PET is used to evaluate cardiac tumors, inflammation, and myocardial viability. As a glucose analog, FDG is sensitive to carbohydrate metabolism, and increased FDG uptake implies a shift from lipid to carbohydrate metabolism. During fasting, the myocardium predominantly consumes free fatty acids and triglycerides with minimal carbohydrate metabolism (1). This metabolic pattern is reversed following a meal. Heart disease shows similar alterations in the metabolic shift from lipid to glucose metabolism (2), which has been attributed to the susceptibility of free fatty acid b-oxidation to ischemia (3–5). Nonischemic heart disease also shows increased carbohydrate usage (6,7).

The shifting metabolism in normal myocardium and in patients with cardiac disease can result in different spatiotemporal uptake patterns at FDG PET (8). Investigators have spatially classified these patterns using several classification schemes from oncologic studies (9,10). The classification is qualitative, and there is variability among experts because of uncertainty and overlap in uptake patterns. There is a need for robust and automatic techniques to classify patterns of myocardial FDG PET uptake to improve consistency of diagnosis between patients with similar uptake patterns and among caregivers reviewing the same images. Likewise, fully automated segmentation and classification methods of left ventricle (LV) uptake patterns are needed to develop large-scale investigations of their clinical significance and to distinguish between physiologic and pathologic LV uptake. Such large-scale repositories are increasingly becoming available in health biobanks (11). There is a need for automated techniques to extract quantitative imaging traits such as myocardial uptake and pattern of classification from this imaging data at scale.

To address the challenges of conventional analysis of large numbers of images and to improve the consistency of diagnosis, machine learning can provide precise image labeling using automation (12,13). Recent deep learning techniques have reached a high level of automated performance approaching or, in some cases, exceeding experts (14). An area to be explored is applying classification and segmentation deep learning techniques on myocardial FDG PET images. One hurdle in developing deep learning algorithms in this field is that they require large amounts of expertly curated and labeled images that capture the heterogeneity of cardiac FDG PET uptake patterns on which to train.

The purpose of this work was to develop a deep learning computer algorithm to classify physiologic patterns of myocardial FDG uptake using a curated dataset of whole-body FDG PET images from an unbiased source of consecutive imaging scans. We sought to develop an algorithm that would perform the joint task of LV segmentation from whole-body FDG PET images and classification of the pattern of myocardial uptake within the segmented region. The algorithm was validated by comparing the accuracy of segmentation and uptake labeling patterns to radiologist experts. We hypothesized that deep learning could robustly label myocardial contours on FDG PET images and identify the myocardial uptake pattern. We investigated the probability distribution of classification patterns produced by deep learning and suggested quantitative myocardial uptake index (MUI) to describe the degree of uptake.

Materials and Methods

Study Design

A total of 612 consecutive patients between July and August 2018 referred for whole-body FDG PET/CT imaging for an oncologic clinical indication were retrospectively evaluated in a protocol approved by the institutional review board of the University of Pennsylvania. Informed consent and Health Insurance Portability and Accountability Act authorization from patients was waived given the retrospective study design and de-identification of the imaging data. De-identification was performed by using software (MIM Software). Among the 612 FDG PET scans, three scans were excluded because two were not full-body PET scans, and in one, the LV could not be segmented. The pathologic burden of these patients is shown in Table 1. This study used a sample of convenience, and formal sample size considerations were not conducted.

Table 1:

Characteristics of Study Population

A total of 609 patient scans were randomized between training (60%, 365 patients), validation (20%, 122 patients), and testing (20%, 122 patients) datasets for the image segmentation task.

FDG PET Imaging

Patients were imaged with a Biograph mCT PET/CT scanner (Siemens Healthineers). Patients were injected with approximately 555 MBq (15 mCi) of FDG after an at least 4-hour fast. PET images were obtained 60 minutes after the FDG injection, starting from the midthigh and through the base or vertex of the skull. A low-dose CT scan was acquired for tissue attenuation and anatomic correlation covering the same field of view as PET. Across all patients, blood glucose level was 105.7 mg/dL ± 28.6, uptake time was 61.8 minutes ± 10.9, and the dose was 15.4 mCi ± 1.1.

Image Segmentation

Manual contouring of the LV was performed for all FDG PET scans by a research technician with 2 years of experience and a radiologist with more than 15 years of nuclear medicine experience (B.F. and P.E.B., respectively) by using MIM Software and was provided as a ground truth segmentation for deep learning. Images were not reoriented in the cardiac planes.

Model Development

The two-dimensional U-Net architecture consisted of 14 convolutional layers, three max pooling layers, three layers of deconvolution and concatenation, and one layer of spatial dropout after the final convolutional layer (15). The network was adapted from an architecture described in Ronneberger et al (16), and the main adaptations were to accommodate relatively smaller FDG PET image dimensions (256 × 256) than electron microscopy images (572 × 572). The main changes were to the number of convolutional layers (14 vs 18) and skip layers (three vs four). Each convolutional layer was batch normalized and followed by rectified linear unit activation (Fig E1 [supplement]). Training was performed using a cloud computing machine equipped with four vCPUs (15 GB memory) and one NVIDIA Tesla P100 GPU. The network was implemented in Python 3.0 using TensorFlow and Keras.

Model performance was evaluated in several ways. Dice scores and log loss training metrics were reported for each training epoch. Cardiac volume and activity measurements for each patient within the segmented region of interest were calculated for both the manual segmentation and automatic segmentations.

Image Classification

Ground truth classification of myocardial FDG uptake was visually assessed on the axial view of the PET images. No myocardial uptake was defined as myocardial activity similar to or lower than adjacent blood pool activity. Diffuse myocardial FDG uptake was defined as activity, above blood pool activity, involving all four LV walls (interventricular septum, inferior wall, anterior wall, and lateral wall), whereas incomplete or partial myocardial FDG uptake was defined as myocardial activity involving three or fewer LV walls. Similarly, cardiac segmentation was performed on the axial view of the PET images.

Following segmentation, a three-dimensional (3D) image classification network based on a modified two-dimensional VGG-16 network (17) was trained to classify patients as one of three categories on the basis of their uptake pattern of the radiotracer FDG: no uptake, diffuse uptake, or partial uptake. The network architecture consisted of eight convolutional layers, four max pooling layers, two dense layers, and two levels of dropout (Fig E2 [supplement]). The 3D network architecture was adapted from a variant of the VGG-16 classification network designed by Simonyan et al (17) and a 3D classification network used for Alzheimer disease classification (18). In comparison to Yang et al (18), the input layer dimensions were smaller in the FDG PET network (110 × 110 × 47 vs 110 × 110 × 110), there were two additional convolutional layers in the Alzheimer network, one additional dense layer in the FDG PET layer, and one additional category (three total) in the final output FDG PET layer. Training was done using eight vCPUs (30 GB memory) and four NVIDIA Tesla P100 GPUs. Data augmentations were applied, including random field of view adjustment (zoom), rotation, and vertical and horizontal displacement. Model adjustments and evaluation were performed by monitoring the training and validation loss metrics. Model hyperparameters such as batch size, number of training steps, dropout, and validation step size were adjusted to address overfitting and underfitting in multiple training runs.

Measurement of MUI

We developed a quantitative MUI for myocardial uptake at FDG PET by parameterizing the output of the neural network. In general, the output of the neural network is constrained by

where P_X is the probability that the image is in the partial (X = P), diffuse (X = D), or no uptake (X = N) category. Equation (1) implies that there are two degrees of freedom; if two probabilities are known, then the third can be found by Equation (1). Therefore, we parameterized the relationship between the two unknown probabilities and data using the following empirical relationship:

The unknown coefficients A = 0.72, B = −50.0, and C = 0.002 were estimated from the test dataset using curve fitting (Python 3.7.4, SciPy) (Appendix E1 [supplement]). The arc length S_P of Equation (2) over the domain [0, P] is

The MUI is the normalized arc length

Intuitively, Equation (4) provides a single number that indicates the extent to which a myocardial FDG PET image shows a certain pattern of uptake with MUI = 0 (no uptake) and MUI = 1 (diffuse uptake). Intermediate patterns of uptake such as focal patterns lie between these extremes.

Repeatability

A second expert rater (P.E.B.) reviewed images in 198 patients to investigate the repeatability of the classification task. The second rater was blind to the findings of the first rater (B.F.). Agreement was measured using Cohen κ in R (R Foundation for Statistical Computing). A second expert performed segmentation in 30 patients to investigate the repeatability of the segmentation task. The second rater (P.E.B.) was blind to the findings of the first rater (B.F.). Agreement was measured using the volume (in milliliters) of the segmentation using the intraclass correlation coefficient (ICC).

Statistical Analysis

Statistical analysis was performed in R (4.0.0), RStudio (1.2.5033), and RTools (4.0.0). Ternary plots were made using ggplot2 (3.3.0).

Two-way analysis of variance was used to compare expert and deep learning uptake activity and volume, followed by TukeyHSD pairwise comparisons for Bonferroni correction. Agreement between expert classifications was performed using Cohen κ in R. Agreement between segmentation volumes was assessed using the ICC in R. Agreement was interpreted to be poor, less than 0.4; fair, 0.4–0.59; good, 0.6–0.74; excellent, greater than or equal to 0.75. The level of significance for all tests was set at P = .05.

Receiver operating characteristic (ROC) analysis was performed using software (Matlab r2020). Predictor variables (probability that an image belongs to a class) and the binary response variable for each class (0 = false or 1 = true) were used to fit a logistic regression model. The probability estimates from logistic regression were used to derive the ROC curve using a range of cutoffs of the binary response variable, the likelihood of the pattern of uptake. ROC analysis was performed for each type of uptake.

Sensitivity and specificity analysis was estimated using the ground truth and predicted labels. The response variable was binarized by selecting the most likely category (ie, the one that had maximum probability). For example, if the neural network made the following predictions for a reference diffuse uptake image: no uptake = 0.3, partial uptake = 0.2, and diffuse uptake = 0.5, then the binary response was true, because the diffuse uptake category was the most likely. If instead the predictions were: no uptake = 0.5, partial uptake = 0.2, and diffuse uptake 0.3, then the binary response was false, because the no uptake category was the most likely.

Results

LV Segmentation

Automatic segmentation and classification were feasible on images in 609 of 610 patients. On images in one patient, classification was not possible because of a large source of uptake elsewhere in the body due to metastatic disease. Therefore, images in 609 patients were used and split between training (60%, 365 patients), validation (20%, 122 patients), and testing (20%, 122 patients) datasets. Automatic anatomic contours, volume, and uptake patterns within the region of interest identified by the deep learning algorithm are shown in Figure 1.

Figure 1:

Cardiac segmentation and classification of myocardial fluorine 18 fluorodeoxyglucose PET uptake using deep learning. (A) Whole-body PET images are shown to a deep learning convolutional neural network (CNN) and (B) the CNN segments the left ventricular (LV) cavity. (C) A second CNN is shown the segmented LV cavity and classifies the image as belonging to “none,” “diffuse,” and “partial” categories. In this example, the image is classified as diffuse uptake (probability of 87.2%). (D) Confusion matrix shows the accuracy of artificial intelligence (AI) classification compared with experts. (E) Segmented labels (red) show good agreement between expert and AI LV cavities for patients in each category. Axial, coronal, and sagittal views are shown for each uptake category. None = no uptake, diffuse = diffuse uptake, partial = partial uptake (focal or focal-on-diffuse).

LV cavity volume was automatically computed by deep learning and compared with segmentation by an expert (Fig 2). There was a significant difference in cardiac volume between the expert and deep learning segmentations (P < .01). Post hoc testing revealed that the no uptake category showed underestimation by deep learning (expert, 236.4 mL ± 72.4 vs deep learning, 173.5 mL ± 87.9, P < .001), but there was no detectable bias in the diffuse or partial uptake categories. There was good correlation between expert and deep learning measurements of volume (R² = 0.35, β = .71).

Figure 2:

Left ventricular (LV) volume and performance of deep learning segmentation. (A) LV cavity volume (in milliliters) in each category of uptake: none, diffuse, and partial and (B) correlation between expert- and artificial intelligence (AI)–measured LV cavity volume.

There was good overlap of the expert and deep learning segmentations for each category (Fig 1E). Additional data showing the Dice score on the training and validation data revealed a progressive improvement in segmentation, as determined by the Dice coefficient (Fig E1 [supplement]). The Dice scores of the training and validation data diverged with longer training periods, which suggests that the network began overfitting. The coefficients used for the testing data were selected prior to overfitting and achieved a Dice score of 0.79 ± 0.14. The algorithm incorrectly segmented images in one patient with peritoneal implants (Fig E3 [supplement]).

There was fair agreement between the volume of the two segmentations with an ICC of 0.45 (95% CI: 0.12, 0.70; P < .005), and this was comparable to the volume between the first expert and artificial intelligence segmentations, with an ICC of 0.52 (95% CI: 0.20, 0.74; P < .001).

Uptake Classification

The distribution of reference standard classification data is shown in Table 2. As indicated by the fraction of correctly categorized uptake patterns, agreement between the expert and deep learning algorithm was 78% (36 of 46) for no uptake, 71% (34 of 48) for diffuse uptake, and 71% (20 of 28) for partial uptake. The performance of the classification network for training and validation data are shown in Figure E2 (supplement). A total of 11% (three of 28) of patients with partial uptake were incorrectly labeled as no uptake, 18% (five of 28) of patients with partial uptake were labeled diffuse, 22% (10 of 46) of patients in the no uptake category were labeled as partial, and 25% (12 of 48) of patients with diffuse uptake were labeled as partial. The performance of the segmentation and classification algorithm is shown for three patients with uptake in each category in Figure 1E. ROC curve analysis was reported for each uptake pattern (Fig 3 and Table 3). The area under the curve was high (> 0.90) for no uptake and diffuse uptake classification. For partial uptake patterns, area under the curve was lower (0.77).

Table 2:

Distribution of Reference Standard Classification Data

Figure 3:

Receiver operating characteristic plot. The optimal point is indicated by the corresponding circles on each curve. The area under the receiver operating characteristic curve was 0.96 ± 0.06 for no uptake, 0.91 ± 0.08 for diffuse uptake, and 0.77 ± 0.16 for partial uptake.

Table 3:

ROC Analysis of Myocardial FDG PET Uptake by Machine Learning

There was moderate agreement between the two expert raters’ judgments, κ = 0.68 (95% CI: 0.60, 0.76; P < .001). This agreement was comparable to the agreement between the first rater and artificial intelligence classifications, κ = 0.57 (95% CI: 0.46, 0.69; P < .001).

Myocardial Uptake Index

On the basis of the modest agreement between experts and deep learning in the pattern classification task, we further investigated the probability distribution of FDG PET images among no, partial, and diffuse uptake categories (Fig 4). The probability distribution was highly heterogeneous (Fig 4A), and 64% for no uptake, 0% for partial uptake, and 20% for diffuse uptake images were classified with greater than 70% probability. We measured the MUI for each patient using Equation (4) (Fig 4B). Myocardial FDG PET images in 16 patients are shown and arranged sequentially in order of their MUI (Fig 4B). By sorting images by their MUI in order of low to high, we observed that the MUI accurately arranged the images by their degree of no uptake, partial, and diffuse uptake. Images showing diffuse uptake (Fig 4B, bottom row) all had MUI of greater than 90, while images showing no or limited uptake had MUI lower than 30 (top row). Images with an MUI between 30 and 90 showed varying degrees of partial uptake patterns (middle two rows).

Figure 4:

Machine learning classification of myocardial fluorine 18 fluorodeoxyglucose PET uptake. (A) Ternary plot shows the classification distribution of cases among diffuse, partial, and no uptake categories. The axis indicates the probability that each image belongs to the no uptake, partial, or diffuse category. (B) Images are arranged left to right and then up to down in order of no uptake to partial uptake to diffuse uptake as indicated by machine learning predicted probability. Numbers in A correspond to classification probabilities of images in B.

Discussion

We investigated deep learning as an automated technique to quantify and classify patterns of myocardial uptake on whole-body FDG PET images. Deep learning classification patterns showed good agreement with expert classification, demonstrating the feasibility of deep learning to recognize patterns of myocardial FDG PET uptake. Two important contributions of our work are (a) the successful application of two neural networks for segmentation and pattern classification in myocardial FDG PET and (b) the development of a quantitative pattern classification variable that was derived from neural network probabilities (Eq [4]). On the basis of the heterogeneity and subjective nature of categories of myocardial uptake patterns, we developed an alternative, quantitative measure of the degree of uptake, the MUI, using probable patterns of uptake informed by deep learning. Using MUI, myocardial FDG PET images were ordered by their degree of uptake.

Cardiac segmentation is challenging on FDG PET images in part because of the variability of uptake patterns. While diffuse uptake patterns show myocardial borders, it is more difficult to identify the myocardium on images with focal uptake or no uptake. Dual CT and PET scanners mitigate the issue of identifying borders by allowing for high-resolution imaging of the myocardium at CT. Additionally, segmentation is a preliminary step to help locate the heart before applying additional imaging analytics. Manual segmentation has been a prerequisite step to subsequent analysis (19). To streamline segmentation, automated methods can be used for segmentation while also reducing variability. Atlas-based segmentation has been used for radiologic images, including PET (20,21), CT, and MRI (22). In comparison to atlas-based segmentation, the application of a trained deep learning network to previously unseen data is computationally inexpensive. While deep learning has been applied generally to PET images for attenuation correction and identifying tumor lesions (23–26), to our knowledge, it has not been used for the segmentation and classification of cardiac uptake.

Interpretation and classification of uptake patterns from FDG PET are challenging because of the overlap between physiologic and pathologic uptake patterns and the influence of diet (27,28). A five-category classification scheme was devised for uptake composed of none, diffuse, focal, over half, and focal-on-diffuse with an additional subclassification for focal as ring, over half, or spot uptake (9). We trained for three uptake categories, none, partial, and diffuse, with the partial category combining focal and focal-on-diffuse categories and the latter reflecting a combination of focal and diffuse uptake. Our patients predominantly showed focal disease, but not focal-on-diffuse disease, so there were limited data available to train a supervised learning algorithm using this fourth category. A larger representation of focal-on-diffuse disease in the training dataset would enable this category to be more accurately determined.

Our study complements and expands upon recent imaging informatics studies of myocardial FDG PET uptake in the heart. Computational techniques such as texture analysis have been applied to myocardial FDG PET images to diagnose patients with cardiac diseases (19,29), and several texture features have shown good repeatability and performance in identifying these patients from those with incidental myocardial findings in tumor patients. Many of these techniques rely on manual curation of patient imaging data for uptake, and cases are excluded if they do not show an uptake pattern. Our study investigated an unbiased set of consecutive patients for nuclear imaging and included those that showed no myocardial uptake.

There were several limitations of this study that should be addressed. While our algorithm showed reliable detection of no uptake patterns, segmentation was more challenging. As the boundary of the heart is mainly defined by its uptake in contrast with that of the surrounding tissues, it is not surprising that this pattern would be the most difficult to segment. As it might be expected, there was variation in the segmentation training data for this category and there was an underestimation of cardiac volume by deep learning. Additional training data for this category could improve the robustness of this segmentation. To reduce the error associated with voxels having both myocardium and blood (partial volume effects), it may be helpful to develop a network to resample the heart along the long axis or develop a network using 3D neural network filters. A second limitation was that our classification scheme used only three categories of uptake (none, diffuse, and partial), and additional data are needed to better identify focal uptake within diffuse categories of disease. Although we did not seek to include patients with identified cardiovascular disease, our approach performed well on an unbiased and large set of data from patients with primarily cancer pathology and should help inform the physiologic patterns of uptake. Because we did not specifically exclude individuals with known cardiovascular history, some patients may have myocardial FDG uptake for reasons other than physiologic glucose use. Additional work is still needed to distinguish pathologic patterns of uptake from physiologic patterns, influenced by diet, and to associate these uptake patterns with patient morbidity and mortality. Another limitation is that the patient population of our study consisted mostly of individuals with cancer history, including patients who previously received therapies (eg, chemotherapy and/or radiation therapy) that could potentially affect or modify FDG uptake in the heart, and consequently, the performance of our algorithm. Future studies will be necessary to further characterize the effect of cancer-related treatment on myocardial FDG uptake and deep learning algorithms. Further studies are also required to investigate the generalizability of our findings to other patient cohorts (eg, myocardial inflammation from sarcoidosis infiltration). Finally, the algorithm was not able to segment images in one patient with peritoneal implants (Fig E3 [supplement]). We expect that being given additional images showing a range of pathologic conditions may further improve the neural network through training.

In summary, this work shows a deep learning strategy for the automatic classification of patterns of myocardial uptake on whole-body FDG PET images. There was a moderate level of agreement for classification accuracy, cardiac activity, and volume between radiologist experts and the automatic method. Deep learning also provided a noncategorical classification strategy to help assess degrees of patterns of uptake. Altogether, these approaches should prove useful for large-scale studies of myocardial FDG PET data, automatic extraction and association of imaging phenotypes with disease, and assisting clinical interpretation of myocardial FDG PET uptake.