Predicting the hypoxic volume of head and neck tumors from fluorodeoxyglucose positron emission tomography images using artificial intelligence

Highlights

• •An AI model was developed to predict hypoxic tumor volumes from FDG-PET images in head and neck cancer.
• •The model performed well for primary tumors and metastatic lymph nodes.
• •The hypoxic volume predictions in test cases strongly correlated with FMISO-PET (Pearson R = 0.96; p < 0.001).

Abstract

Background and purpose

Tumor hypoxia is linked to lower local control rates and increased distant disease progression during head and neck (HN) radiotherapy. ¹⁸F-fluoromisonidazole (¹⁸F-FMISO) positron emission tomography (PET) imaging measured hypoxia can aid dose selection for HN patients, but its availability is limited. Hence, we tested the hypothesis that an artificial intelligence (AI) model could synthesize ¹⁸F-FMISO-like images from routinely acquired ¹⁸F-fluorodeoxyglucose (¹⁸F-FDG) PET images in order to predict primary tumor or metastatic lymph node hypoxic volumes.

Materials and methods

One hundred and thirty-four (training = 84, validation = 13, testing = 21, additional testing = 16) HN carcinoma patients, treated with chemoradiotherapy between 2011 and 2018 and scanned at treatment baseline with ¹⁸F-FDG PET/computed tomography (CT) and ¹⁸F-FMISO dynamic PET/CT, were analyzed. A pix2pix-architecture-based generative adversarial network was trained to yield 2D voxel-wise FMISO hypoxia images of target-to-blood ratios (TBRs) directly from the ¹⁸F-FDG PET/CT image slices. The hypoxic volume was defined consistent with clinical procedure as the malignant volume with TBR values above 1.2. The AI model hypoxia predictions were compared against scaled ¹⁸F-FDG PET values.

Results

The AI model hypoxic volume predictions were well-correlated with ¹⁸F-FMISO hypoxic volumes on the held-out test subjects (Pearson correlation testing R = 0.96, additional testing R = 0.91, p < 0.001). Predictions from globally scaled ¹⁸F-FDG PET images also produced a significantly correlated but worse prediction.

Conclusion

Voxel-wise prediction of hypoxia for HN cancers from a 2D deep learning model using FDG-PET images as inputs was shown to be feasible. Testing on larger institutional and multi-institutional cohorts is required to establish generalizability.

1. Introduction

Tumor hypoxia is a common feature of the tumor microenvironment in many cancer types due to uncontrolled cellular proliferation, the resulting microscopic competition for oxygen and glucose between tumor cells, and abnormal tumor vasculature [1]. Shortly after DNA is damaged, oxygen-based chemical reactions decrease the repairability of DNA breaks. Thus, hypoxia is radioprotective. Measured tumor hypoxia has often been associated with poor local control following radiotherapy. Hypoxia also stimulates a more metastasis-prone phenotype [[2], [3], [4], [5], [6]].

¹⁸F-labelled fluoromisonidazole (¹⁸F-FMISO) PET imaging has demonstrated reproducible results in multiple studies for visualizing and quantifying tumor hypoxia [7,8]. ¹⁸F-FMISO has been used in many cancer sites including: prostate cancer [9], breast cancer [10] and head and neck cancer (HN) [[11], [12], [13]]. However, ¹⁸F-FMISO PET imaging is subject to limited availability as it is currently regulated as an investigational tracer, requires a specialized cyclotron-based production facility, and usually involves time-consuming scan protocols.

Compared to ¹⁸F-FMISO PET, ¹⁸F-fluorodeoxyglucose (¹⁸F-FDG) PET is more commonly available and is a routinely performed procedure in oncology clinics for almost all cancer types. ¹⁸F-FDG PET imaging is a surrogate for the rate of glucose metabolism, which is also influenced by hypoxia. Many clinical studies have explored the relationship between ¹⁸F-FDG and ¹⁸F-FMISO uptake [[14], [15], [16]]. Two studies showed higher ¹⁸F-FDG uptake correlated with hypoxic tumor regions [[14], [15], [16], [17]]. One prior work also showed that the entropy texture computed from ¹⁸F-FDG images was the feature most correlated with tumor hypoxia in 38 patients with head and neck cancer [14]. However, those studies are based on a relatively small number of patients, with inconsistent definitions of tumor hypoxia, and different tumor types. Therefore, a goal of this study was to test whether a tumor hypoxia prediction model could be developed using routinely available ¹⁸F-FDG PET images as input, based on a relatively large and homogenously acquired patient dataset.

Artificial intelligence (AI) models extract a multitude of features that could account for tumor shape and ¹⁸F-FDG distribution shape in a non-local and non-linear fashion. Such models could be trained with a large number of images to learn features that are applicable across underlying imaging variations encountered in the training set, thus increasing robustness of the model. Hence, we hypothesized that some basic hypoxia quantities measured by ¹⁸F-FMISO, such as the hypoxic volume within tumors and lymph nodes, could be estimated using an AI-based deep learning method with only the corresponding 2D slices of ¹⁸F-FDG images as an input. In this study, we developed a generative adversarial network (GAN) [18] approach to estimate ¹⁸F-FMISO PET hypoxia images on a voxel-by-voxel basis from ⁸F-FDG PET images. GAN was selected due to its success in synthesizing accurate medical images [19]. In addition, a simple baseline model that scaled ¹⁸F-FDG values within the primary tumor or metastatic lymph node regions was tested as a predictor of ¹⁸F-FMISO hypoxia because prior work has shown a correlation of ¹⁸F-FDG with ¹⁸F-FMISO for head and neck cancer [20]. The prediction accuracy of the AI-based model and the FDG-scaled model was assessed and compared for head and neck primary tumors and metastatic lymph nodes.

2. Materials and methods

2.1. Patient characteristics

This retrospective analysis study was approved by the institutional review board of Memorial Sloan Kettering Cancer Center. One hundred and eighteen head and neck carcinoma patients treated with chemoradiotherapy and imaged at baseline using both ¹⁸F-FDG PET and ¹⁸F-FMISO PET between January 2011 and October 2016 were analyzed. The median time between the two PET scans was 7 days. The ¹⁸F-FDG PET acquisition commenced at approximately 75 min following ¹⁸F-FDG injection. The ¹⁸F-FMISO PET was acquired using three scans: one 30-minute dynamic acquisition, concurrent with the ¹⁸F-FMISO injection, followed by two static 10-minute scans acquired at approximately 90 min and 160 min after the ¹⁸F-FMISO injection. Most patients underwent testing for HPV and p16 status (see Table 1). Clinical software provided by the manufacturer was used for PET reconstruction with corrections for attenuation, scatter, and random counts. Additional details of patient population and scan characteristics can be found in Grkovski et al [11].

Table 1.

Patient and disease characteristics in the training, validation, and testing groups.

	Ntrain (Fraction)	Nvalidation (Fraction)	Ntest (Fraction)
Subsite
Base of Tongue	40 (0.48)	8 (0.62)	7 (0.33)
Tonsil	32 (0.38)	3 (0.23)	12 (0.57)
Supraglottic larynx	1 (0.01)	2 (0.15)	0 (0)
Hypopharynx	1 (0.01)	0 (0)	1 (0.05)
Unknown	10 (0.12)	0 (0)	1 (0.05)
p16 status
Positive	66 (0.79)	8 (0.61)	17 (0.80)
Negative	6 (0.07)	1 (0.08)	2 (0.10)
Unknown	12 (0.14)	4 (0.31)	2 (0.10)
HPV status
Positive	47 (0.56)	7 (0.54)	15 (0.71)
Negative	11 (0.13)	3 (0.23)	1 (0.24)
Unknown	26 (0.31)	3 (0.23)	5 (0.05)
Disease
Primary tumor	56 (0.39)	9 (0.39)	15 (0.37)
Lymph node	89 (0.61)	14 (0.61)	26 (0.63)
Total	145	23	41

The AI model was trained using 1,521 slices from 84 patients with 56 primary tumors and 89 metastatic lymph nodes, and then validated with 223 slices from 13 other patients with 9 primary tumors and 14 metastatic lymph nodes, and finally tested on 439 slices from 21 other patients with 15 primary tumors and 26 metastatic lymph nodes (see Table 1, Supplementary Fig. 1). Random stratified data splitting was performed to ensure a split of roughly 70%, 10%, 20% of primary tumors and lymph nodes in the training, validation, and testing datasets with a similar prevalence of primary tumors and lymph nodes in all groups.

2.2. Image pre-processing

Image preprocessing steps included: (1) alignment of computed tomography (CT) images, with corresponding ¹⁸F-FDG and ¹⁸F-FMISO scans registered to the same spatial coordinate system using rigid alignment performed with commercial software (Advantage Workstation), (2) computation of hypoxia target-to-blood ratio (TBR) images, defined as the ¹⁸F-FMISO image acquired at approximately 160 min post-injection normalized by mean vascular (internal carotid artery/jugular vein) uptake determined from the same ¹⁸F-FMISO image, and (3) extraction of tumor and lymph node volumes by commercial software using a threshold segmentation method based on the intensity values of ¹⁸F-FDG voxels [11]. The segmented tumor and lymph node volumes were transferred from ¹⁸F-FDG images to ¹⁸F-FMISO images. Voxels outside the segmented treatment target were set to zero. (4) Image resampling (bi-linear interpolation) was conducted to a 256-by-256-by-47 matrix, with voxel dimensions of 1.95-mm-by-1.95-mm-by-3.27-mm. (5) Images were cropped to extract 2D image patches of 32-by-32 pixels enclosing tumors on both ¹⁸F-FDG and TBR images, and (6) scaling and shifting was performed to bring images within the range of [-1, 1], using minimum and maximum intensity values determined from the entire dataset. The synthesized images produced by the AI model also lie within the range of [-1, 1]. These resulting images were finally scaled and shifted back corresponding to the range of FMISO TBR which was [0, 4.7].

2.3. Generative adversarial network (GAN) and training details

The GAN model was created based on the pix2pix approach [21], a conditional GAN method, which uses aligned pairs of ¹⁸F-FDG and ¹⁸F-FMISO images. Briefly, the GAN consists of generator (G) and discriminator (D) networks that are simultaneously optimized using a so-called min–max criterion. G is trained to generate increasingly plausible images that pass an evaluation by D, which is trained to maximize its ability to distinguish generated images from real images. The model was thus trained to predict a pseudo-FMISO-TBR 2D image given an ¹⁸F-FDG 2D image.

Pix2pix was selected due to its capability to achieve accurate image synthesis for medical images [22], [23]. The generator is composed of a U-net [24] with three downsampling (or encoder) layers and three upsampling (or decoder) layers with skip connections (Fig. 1). The discriminator was created using a 2-layer PatchGAN with a 4-by-4 effective receptive field. Both networks were optimized using a combination of image similarity and adversarial discrimination ability losses, as elaborated in the Supplementary Section 1.

Fig. 1.

The deep learning pix2pix prediction model architecture is comprised of a U-net based Generator and a four-by-four PatchGAN Discriminator. The dimension (Height, Width, Channel) of the output at each layer is indicated.

Data augmentation was implemented using random image rotations and 2D slice-wise training. The model was trained for 200 epochs, with an Adam optimizer using a learning rate of 0.0002 and a batch size of 1. The model with the best accuracy on validation was selected for testing. The selected model is available on GitHub: https://github.com/cerr/fdg2fmiso_hn_pix2pix.

2.4. Statistical analysis

Model accuracy was evaluated via several metrics: by comparing the predicted TBR with corresponding measured TBR images regarding the agreement between voxel values by using the mean of the x-percent highest intensities (MOHx) [25], [26], including MOH5 (mean of the hottest 5% voxels) and MOH50 (mean of hottest 50% voxels); using the hypoxic volume (HV) defined as volume of voxels with TBR > 1.2 with threshold based on published works [16], [20], [27], [28], [29], [30]; and by computing the spatial overlap of the predicted and measured hypoxic volumes using the Dice similarity coefficient (DSC). Detailed descriptions of the metrics are provided in Supplementary Section 2.

The scaled FDG model was created as a baseline voxel-wise predictor of hypoxia. This was accomplished by first computing a global normalization factor that corresponded to the ratio of the mean TBR to the mean FDG values computed within all analyzed tumors and lymph nodes. Scaling was performed by multiplying all FDG values with this global normalization factor.

Statistical comparisons of the AI model predictions and the FDG scaled model predictions were conducted using paired, two-sided, Wilcoxon signed rank tests at the 95 % confidence level.

2.5. Network design experiments

Network design experiments were conducted to study the impact of the number of layers, losses used to train the network, and the receptive field size used by the discriminator. The experiments consisted of: (1) varying number of layers (three to five) in encoding/decoding parts of the generator network to study the impact of deeper layers on image generation accuracy; (2) changing the receptive field size (sixteen-by-sixteen, four-by-four and one-by-one) in the discriminator to study the impact of a discriminator with higher/lower receptive field sizes to better differentiate real from synthetic images; (3) applying Wasserstein distance loss [31], [32] to emphasize low frequency image differences occurring as long tails in the image distribution to measure differences between synthesized and real images; and (4) adding squeeze and excitation residual blocks to the encoding/decoding layers of the generator [33] to increase the effective network depth. The results of the experiments described here were used to determine whether there was a potential improvement in model performance under different network designs.

2.6. Additional held-out testing of the selected AI model

An additional testing dataset of 16 head and neck cancer patients consisting of 6 primary tumors and 15 metastatic lymph nodes, acquired between October 2017 and September 2018, and chronologically selected from the patient cohort reported in the study by Lee et al [34], provided additional held-out testing to further evaluate the best AI model selected in section 2.3.

3. Results

The TBR, MOH5, and MOH50 values calculated from the predicted hypoxia images (pix2pix and the FDG-scaled model) and from the measured hypoxia images resulted in moderate correlations (MOH5 Pearson R = 0.46 pix2pix vs. R = 0.37 FDG scaled; MOH50 Pearson R = 0.50 pix2pix vs. R = 0.38 FDG scaled) as shown in Fig. 2. For the prediction of hypoxic volume, the pix2pix model showed an excellent correlation (Pearson R = 0.96, p < 0.001), which improved over the FDG scaled model (Pearson R = 0.91, p < 0.001), particularly for tumors with > 5 cm³ hypoxic volumes (see Fig. 3).

Fig. 2.

Predicted values of the Mean-of-the-Hottest 5% and 50% of hypoxia images: MOH5 (left) and MOH50 (right) calculated from the predicted and measured hypoxia/TBR images for the N = 21 test set.

Fig. 3.

Pix2pix predicted hypoxic volumes for 41 primary tumor/lymph nodes in the N = 21 test dataset showed a stronger correlation to the measured hypoxic volume compared to scaled FDG-derived hypoxic volumes.

The pix2pix model predictions resulted in significantly lower absolute errors of the predicted MOH5, MOH50, and HV and higher DSC than the FDG-scaled model as summarized in Table 2. For the pix2pix model, the absolute errors and predicted hypoxic volume accuracy were similar for primary tumors and lymph nodes as summarized for the testing dataset in Supplementary Table 1.

Table 2.

Comparison of pix2pix and FDG scaled model predictions for the 41 primary tumor/lymph nodes in the test dataset using absolute error of the MOH5, MOH50, and HV predictions, and DSC of the predicted hypoxic volumes. The pix2pix average DSC value (0.77) improves greatly on the FDG scaled DSC average (0.61).

Model	pix2pix n = 41	FDG scaled n = 41	P value
MOH5	0.16 (0.06–0.38)	0.32 (0.14–0.59)	0.001
MOH50	0.11 (0.05–0.31)	0.26 (0.11–0.51)	0.001
HV (cm3)	1.65 (0.79–2.44)	2.85 (0.85–4.80)	0.010
DSC	0.77 (0.50–0.91)	0.61 (0.21–0.76)	<0.001

Median and interquartile range (in brackets) are reported.

A 3D scatter plot, displaying the relationship between predicted MOH5, measured MOH5 and the corresponding primary tumor/lymph node volume illustrates that the predictive performance of the deep learning model on MOH5 does not depend strongly on the volume of primary tumor or metastatic lymph node (see Supplementary Fig. 2). Unsurprisingly, a strong correlation (Pearson R = 0.95, p < 0.001) was observed between measured hypoxia volumes and primary tumor/metastatic lymph node volumes.

Fig. 4 shows images of selected measured TBR values and deep learning model predicted TBR values together with the ¹⁸F-FDG images for six representative examples of varying success.

Fig. 4.

Hypoxia/TBR prediction results obtained from pix2pix for 6 examples from the test dataset: four “good” examples (small prediction errors) consisting of two primary tumors (T1, T2) and two lymph nodes (N1, N2), and two “bad” examples (large prediction errors) consisting of one primary tumor (T3) and one lymph node (N3). For each case, the 2D image slice (¹⁸F-FDG, measured TBR and predicted TBR) containing maximum measured TBR voxel is shown. ¹⁸F-FDG maps are in units of standardized uptake value. The same color bar is used for the measured TBR and predicted TBR images.

The network design experiment results for the testing dataset are summarized in Supplementary Table 2. As shown, increasing the network depth did not improve accuracy and in fact degraded accuracy when 4 layers were used. Decreasing or increasing the patch size used by the discriminator also reduced accuracy. Finally, neither robust distribution matching using a Wasserstein distance loss nor relevant feature extraction using squeeze excitation blocks improved accuracy over the standard approach.

The results from the additional (N = 16) testing dataset (see Supplementary Fig. 3) also showed that the pix2pix model hypoxic volumes were well correlated (Pearson R = 0.91, p < 0.001) with the measured hypoxic volume, especially for tumors with > 5 cm³ hypoxic volumes. These findings are therefore consistent with the testing results shown in Fig. 3.

4. Discussion

A pix2pix GAN-based AI-approach was developed to predict hypoxia/TBR for both primary head and neck tumors and lymph nodes on a voxel-by-voxel basis using input 2D ¹⁸F-FDG images. Our analysis showed that the AI-based prediction was significantly more accurate than an FDG scaled model for all evaluated metrics on a relatively large cohort of head and neck cancer patients. The model’s performance does not depend strongly on tumor volume, shape, or type (primary versus lymph node). The model is straightforward to apply as it does not require image registration or radiomics feature computations as required by other hypoxia prediction studies [35], [28]. Our results thus show the potential of an AI-based approach to predict hypoxia/TBR measurements from routinely available FDG scans.

Our study used a TBR threshold of 1.2 in order to be consistent with multiple prior studies and to provide more hypoxic voxels for training [16], [20], [27], [28], [29], [30]. Two prior studies also used a TBR threshold of 1.6 to define hypoxic volume [36], [37]. We also quantified the impact of TBR threshold on the AI model’s predictions with a threshold of 1.6. This showed a decrease in Pearson correlation from 0.96 to 0.81, perhaps caused by the decrease in the number of hypoxic voxels available for model training.

Our analysis of network design showed that many deep learning variations were similarly accurate. This is possibly due to the fact that ¹⁸F-FMISO TBR images have a smaller range of intensity variations compared to natural and anatomic radiographic images, which in contrast might benefit from the Wasserstein distance loss. Similarly, the advantage of squeeze and excitation blocks [33] to extract relevant features was likely diminished due to a lack of sufficient signal intensity variations within the two different PET images. A more detailed analysis of the utility of these design improvements and losses for the PET images versus other types of imaging data is beyond the scope of the study.

Despite the success of the hypoxia prediction for most primary tumors/metastatic lymph nodes (∼90 %) in the test dataset, large errors were observed for four cases with high measured hypoxia where MOH5 is larger than 2.5. The poor prediction performance on these four cases might result from training data limitations. It should be noted that the prediction of hypoxic volume was still accurate even when there were large MOH errors in high hypoxia treatment targets, which may not be surprising since the determination of the hypoxic volume is solely based on a hypoxia/TBR threshold value of 1.2.

One limitation of this study is that the current AI approach is based on 2D transverse slices. We used this approach to address the data size limitations. A future study with a larger dataset could examine the impact of a 3D network on prediction performance, which should theoretically be superior by accounting for proximity to superior and inferior tumor boundaries. We also focused our approach on the pix2pix GAN model, which possibly could be improved with more advanced methods such as diffusion models, but such an approach was not used due to limited training dataset. In addition, the input data is primarily for HPV-positive tumors, possibly limiting generalizability.

Another limitation of this study is that all the patient data used in model training and validation are from a clinical trial performed at a single institution. Although the model performance was evaluated using unseen holdout test data, only a larger multi-institutional study could fully validate model generalizability.

We are aware that metabolism, as measured by ¹⁸F-FDG images, is only one of the factors that is related to tumor hypoxia. In fact, many previous head and neck cancer studies found there is a moderate correlation between ¹⁸F-FDG and ¹⁸F-FMISO uptake for primary tumors [14], [20], while for metastatic lymph nodes discordant results were reported resulting in weaker correlations [20]. Nonetheless, the AI-based prediction of FMISO hypoxia was more accurate than the FDG scaled model. Viewed from this perspective, the results obtained from this model are promising. It is likely that accuracy could be further improved by using multi-modality inputs including magnetic resonance imaging.

In conclusion, given the potential value of estimating hypoxic volumes in tumors, this study tested the hypothesis that an AI/deep learning model could be trained to predict voxel-wise hypoxia measurements from routinely acquired FDG scans for head and neck cancer. Evaluations on clinically relevant metrics showed that the model achieved more accurate predictions than a naive FDG scaled model, especially for highly hypoxic tumors.

CRediT authorship contribution statement

Wei Zhao: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Visualization, Writing – original draft. Milan Grkovski: Data curation, Methodology, Validation, Writing – review & editing. Heiko Schoder: Data curation, Writing – review & editing. Aditya P. Apte: Software, Validation, Writing – review & editing. John Humm: Conceptualization, Data curation, Writing – review & editing. Nancy Y. Lee: Conceptualization, Data curation, Methodology, Validation, Writing – review & editing. Joseph O. Deasy: Conceptualization, Funding acquisition, Methodology, Project administration, Supervision, Writing – review & editing. Harini Veeraraghavan: Conceptualization, Methodology, Software, Validation, Supervision, Visualization, Writing – review & editing.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following are the Supplementary data to this article: