Deep learning driven predictive treatment planning for adaptive radiotherapy of lung cancer

Abstract

Background and purpose:

To develop a novel deep learning algorithm of sequential analysis, Seq2Seq, for predicting weekly anatomical changes of lung tumor and esophagus during definitive radiotherapy, incorporate the potential tumor shrinkage into a predictive treatment planning paradigm, and improve the therapeutic ratio.

Methods and materials:

Seq2Seq starts with the primary tumor and esophagus observed on the planning CT to predict their geometric evolution during radiotherapy on a weekly basis, and subsequently updates the predictions with new snapshots acquired via weekly CBCTs. Seq2Seq is equipped with convolutional long short term memory to analyze the spatial-temporal changes of longitudinal images, trained and validated using a dataset including sixty patients. Predictive plans were optimized according to each weekly prediction and made ready for weekly deployment to mitigate the clinical burden of online weekly replanning.

Results:

Seq2Seq tracks structural changes well: DICE between predicted and actual weekly tumor and esophagus were (0.83±0.10, 0.79±0.14, 0.78±0.12, 0.77±0.12, 0.75±0.12, 0.71±0.17), and (0.72 ± 0.16, 0.73 ± 0.11, 0.75 ± 0.08,0.74 ± 0.09, 0.72 ± 0.14, 0.71 ± 0.14), respectively, while the average Hausdorff distances were within 2mm. Evaluating dose to the actual weekly tumor and esophagus, a 4.2Gy reduction in esophagus mean dose while maintaining 60Gy tumor coverage was achieved with the predictive weekly plans, compared to the plan optimized using the initial tumor and esophagus alone, primarily due to noticeable tumor shrinkage during radiotherapy.

Conclusion:

It is feasible to predict the longitudinal changes of tumor and esophagus with the Seq2Seq, which could lead to improving the efficiency and effectiveness of lung adaptive radiotherapy.

I. INTRODUCTION

Lung cancer by incidence trends higher globally and leads the cause of cancer death despite the advances of understandings of the disease and treatment options [1,2]. Improving the poor survival of locally advanced non-small cell lung cancer (LA-NSCLC) patients treated with chemoradiotherapy would help in mitigating the problem. To enhance the effectiveness of radiotherapy, clinical trials such as RTOG-0617 [3,4] and RTOG-1106 [5,6] were performed to investigate the potential impact of radiation dose escalation. Unfortunately, dose escalation with uniform dose to the entire tumor (RTOG-0617) was found to decrease survival due to increased toxicities accrued to the surrounding organs at risks (OAR); while the adaptive and selective dose escalation to the PET-avid tumor observed in the middle of treatment (RTOG-1106) slightly increased local-regional progression free time, but fell short in improving overall and progression-free survival. On the future design of dose trials, the authors of RTOG-1106 proposed individualizing radiotherapy dose prescriptions by accounting for a patient’s intrinsic sensitivity determined by the radiation resistant genotype on DNA repair pathway genes [7], and increasing normal tissue sparing to improve survival. Therefore, there are urgent needs to further understand the mechanisms of radiotherapy response by probing with well-integrated longitudinal imaging and biological studies, and to dynamically incorporate the findings into more sophisticated dose optimization and delivery strategies.

Weekly cone beam computed tomography (CBCT) and magnetic resonance imaging (MRI) have been routinely integrated into radiotherapy of LA-NSCLC to monitor tumor and OAR changes in our institution. With the longitudinal imaging data accumulated through IRB-approved protocols, we have built effective deep learning models powered by recurrent neural networks to predict the evolution of tumor [8] and esophagus [9] using the first two or three weekly observations. While these predictions are well-positioned to facilitate the design of a mid-treatment adaptive radiotherapy strategy such as RTOG-1106, it would be more desirable to push the timing of the prediction forward so that beneficial interventions can be introduced as early as possible. In this paper, we attempt to use a cutting-edge deep learning algorithm, convolutional long short term memory (Conv-LSTM) [10,11], to make trustworthy predictions at the initial planning stage, and automatically update the predictions with the weekly CBCT observations when the new streams of data become available. We hypothesize that these predictions of anatomical changes of tumor and OARs can serve as valuable prior knowledge for treatment plan optimization, and eventually improve the therapeutical ratio of the treatment.

Capitalizing on plan re-optimization, adaptive radiotherapy (ART) is able to lower the normal tissue complication when the targeted tumor shrinks or moves away from the OARs. However, the approach of mid-treatment replanning implemented by RTOG-1106 didn’t fully realize the effect of continuous shrinkage because replanning was done only once quite late (after 40Gy) into the treatment course. At the other end of the spectrum, daily on-line replanning enabled on the most advanced linacs [12–15] may capture the fine inter-fractional changes on a much more frequent basis and has the potential to achieve the optimal treatment. The only drawback is that it imposes a tremendous burden on the planning, QA, and delivery workflow, preventing it from being widely implemented in a busy clinic [16]. To mitigate these problems and strike a balance between treatment quality and affordability, we propose a novel predictive treatment planning paradigm that takes advantages of weekly predictions (week 1 to 6) of tumor and esophagus, optimizes a weekly adaptive plan based on each weekly prediction (six weekly predictive plans in total for the entire course), and makes them ready for weekly deployment at the initial planning stage. As radiotherapy progresses, we update the predictions with new snapshots acquired via the weekly surveillance CBCTs, and reoptimize the predictive weekly plans only if predictions deviate significantly from the actual observations. Under this new off-line adaptation paradigm, the geometric dynamics between tumor and OARs are well incorporated in the predictive plans. We hypothesize that meaningful sparing of esophagus and lung can be achieved without compromising tumor coverage in the common clinical scenario of noticeable tumor shrinkage, while mitigating the burden of online adaptation.

II. METHODS AND MATERIALS

2.1. Prediction via Seq2Seq

We developed a new comprehensive model for sequential analysis of radiotherapy CT/CBCT images, named Seq2Seq as illustrated in Figure 1, to predict weekly dynamic changes of both primary GTV and esophagus. The model consists of two stacked Conv-LSTM layers with both vertical and horizontal connections, where the spatial features of the input (longitudinal images and contours) are processed in the vertical direction, and the temporal patterns of the extracted spatial features are parsed in the horizontal pathways. Predictions are automatically updated whenever new data are available in the time series.

Figure 1.

The schema of seq2seq deep learning model. Taking the stream of continuous images and contours as inputs, we applied two stacked Conv-LSTM for prediction of further time points. Weekly adaptive plans based on weekly predictions are produced at the initial planning stage. As radiotherapy progresses, predictions are updated with new snapshots acquired via the weekly CBCTs, and the predictive weekly plans are reoptimized only if predictions deviate significantly from the actual observations.

2.1.1. Design of Seq2Seq

Seq2seq used a stacked Conv-LSTM to analyze the longitudinal features among the sequential dataset. The Conv-LSTM module was modified from a standard LSTM [17] by replacing all multiplication operations with convolution operations to accommodate multidimensional datasets [10]. This modification is particularly useful for medical images because the 2D/3D spatial features can be well preserved within the imaging domain, otherwise lost in the approaches of using LSTM where the extracted spatial features are flattened with a 1D tensor [8, 22]. The technical details of Conv-LSTM can be found in the supplemental document.

2.1.2. Perform weekly predictions

At the initial planning stage, we used the observations X₀, including the original planning CT image and the binary mask of the GTV and esophagus, to make a prediction of week 1, ${\widehat{X}}_1$, including both predicted week 1 CBCT image and binary masks of GTV and esophagus. Because the actual observation at the 1^st week was missing at this point, we utilized a combination of actual observation and prediction, $\left(X_{\text{0}},{\widehat{X}}_1\right)$, to predict the 2^nd week ${\widehat{X}}_2$. This process was repeated until all six weekly predictions were derived. During radiotherapy, as the actual weekly snapshot, X_t, was acquired, we replaced the corresponding prediction in the input series with the actual data and updated the predictions for the rest of the weeks. Note that we only trained one comprehensive network for the predictions of the entire radiotherapy course, rather than training six networks for six individual weekly time points, which makes the algorithm more efficient and robust to potential missing weekly data.

The input at a particular time point consisted of two channels, one is a 2D image slice, and the other is the corresponding binary mask image of GTV/esophagus. 10cm margins around the GTV and the esophagus were included in the region of interest because the background information has proved to be critical in making accurate predictions for the longitudinal lung imaging studies [18,19]. 2D slices instead of 3D whole image were utilized mainly due to the memory limitations in the programming environment. The number of hyperparameters in a 3D Conv-LSTM layer was 20 times of that in a 2D unit, and consumed 120 times of memory, which makes the training time intolerable. After the predictions were produced on a 2D slice basis, predicted images and contours were reassembled as a new 3D RT image/structure set, and imported into Eclipse for treatment planning.

2.1.3. Training and testing

The free breathing planning CT and weekly CBCTs of sixty LA-NSCLC patients were obtained under IRB #16–700 as the training and testing dataset for this retrospective study. The primary GTV observed on the planning CT ranged from 6 to 331 cc, with a mean of 108 cc. All patients received conventional intensity modulated radiation therapy with a prescription dose of 60–66Gy in 2Gy/fraction. Consecutive weekly CBCTs were acquired for each patient to monitor tumor/OAR changes. Relative to the planning CT, GTV shrinkage observed on a weekly basis was 3±12%, 9±17%, 14±23%, 22±17%, 27±17%, and 33±22%, respectively. Growth of GTV due to inflammation (volume larger on weekly CBCT than planning CT) were observed for 16 patients, all before week 3. These bi-directional changes also contribute to the larger standard deviations observed among week 2 and 3. 80% of patients had more than 20% shrinkage, and the tumor disappeared in one patient by the end of radiotherapy.

We applied 10-fold cross-validation to comprehensively train the model and evaluate predictions for all patients. We divided the entire dataset into ten subsets. We first used the first nine subsets for training and the last one subset for testing. Subsequently we repeated this process and rotated the training and testing subsets to overcome the overfitting issue and collect the statistics of the performance metrics across the entire dataset. During the training of Seq2Seq, we used the whole sequence of clinical datasets (planning, week 1→6) for determining each hyperparameter in the Conv-LSTM module so that we can use the trained Conv-LSTM module for predicting the next time point from any length of the inferences. Since both image and segmented contours from the sequential datasets were included in the input, we constructed a hybrid loss which is a combination of the mean square error (MSE) between the predicted and actual weekly images, and the Dice between the predicted and actual weekly contours:

$$Loss=\frac{1}{N}\sum _{i=1}^N\sum _{k=0}^{T-1}\sum _{t=k}^T\frac{1}{T-k}\left\{\alpha \times {‖x_{t+1}^i-{\widehat{x}}_{t+1}^i‖}^2+\left(1-\alpha \right)\times dice\left(m_{t+1}^i,{\widehat{m}}_{t+1}^i\right)\right\}$$

where N is the total number of patients allocated for training, T is the total timepoints (6 weeks in this application), x_t and ${\widehat{x}}_t$ are actual and predicted images, and, m_t and ${\widehat{m}}_t$ are actual and predicted contours at time point t. We introduced a weighting factor α to search for the balance between the two loss terms. The Seq2Seq network was optimized by the adaptive moment estimation (ADAM) optimizer [20]. By comparing different values, α=0.2 was selected for the optimal performance.

Seq2Seq was implemented using the Conv-LSTM module provided in the Keras library (Apress, 2017). It took approximately 5 hours to complete the training of 500 epochs with one Intel® Xeon® Silver 4110 CPU @ 2.10 GHz, 96 GB RAM, and a GeForce GTX 2080Ti GPU (11GB memory). It took less than 1 min to generate the prediction images from week 1 to 6.

2.2. Predictive weekly planning

Preparing for a simulated workflow of weekly adaptation, we produced a weekly predictive plan for each weekly prediction at the initial planning stage. We registered the predicted weekly CBCTs to the original planning CT, and propagated the predicted weekly esophagus and GTV onto the original planning CT. The predicted weekly GTV (GTV_pw) was expanded with a uniform 1.5 cm 3D margin to form the predicted weekly planning target volume (PTV_pw) and served as the basis for optimization along with the weekly predicted esophagus. The planning criteria followed our department’s standard 2Gy×30-fraction protocol of LA-NSCLC radiotherapy, except that 50Gy was required to cover at least 95% of the original PTV_o seen on the planning CT but outside PTV_pw. This additional criterion was borrowed from the protocol of RTOG-1106 to ensure robust dosage to the tumor. All the plans were automatically generated in a batch mode using the expedited constrained hierarchical optimization system [21]. At the initial planning stage before the first delivery, all plans would be reviewed with the aid of an automatic plan checker [22] and made ready for weekly deployment to mitigate the cumbersome clinical burden of online replanning. During radiotherapy, with an update of the input on a weekly basis, new predictions would be generated online, and new predictive weekly plans would be reoptimized offline to improve dosimetric quality only if the weekly prediction of GTV volume changes deviates more than 20% from the actual weekly observation. Threshold of 20% is conservatively selected in this pilot investigation because Moller et al demonstrated that decreases in both locoregional failure and pneumonitis can be achieved via adaptive radiotherapy for a group of patients with a median GTV shrinkage of 30% at mid-course re-planning [23].

2.3. Evaluation

The weekly GTV/OAR were contoured by a radiation oncologist on the weekly CBCT. Subsequently a separate radiation oncologist spot-checked the results. Consensus was reached by the two radiation oncologists if corrections were necessary to improve the quality of the segmentation. For the purpose of evaluating geometric changes of GTV/OAR, we transferred the weekly GTV/OAR to the planning CT (reference of origin) via a rigid registration that aligns the spine and outer lung. Comparing the predicted and actual weekly GTV/OAR, we calculated the correlation of the volume, DICE coefficient, HD50 and HD95 to evaluate the geometric accuracy of Seg2Seq on a weekly basis. Contouring tumor and esophagus on CBCT are challenging given the compromised image quality of CBCTs. We performed a simulated study to estimate the impact of potential interobserver variabilities. We shifted the original ground truth GTV contours on the weekly CBCTs by 1mm in the anterior, posterior, left, right, superior, and inferior directions separately to simulate different observers for the test patients, and calculated the DICE between the predictions made by the model and ground truth for evaluation. Furthermore, simulating the common clinical scenario of missing weekly scans, we intentionally omitted a weekly (2^nd or 3^rd) surveillance scan and used the rest of the weekly scans to perform predictions and test the robustness of the predictor.

We evaluated the weekly predictive plans against two clinical scenarios: (1) using the treatment plan optimized solely based on the planning CT throughout the entire course without adaptation, and (2) weekly online re-optimizing the treatment plan based on the actual weekly tumor and OAR. Because we maintained the same 60 Gy prescription dose to the weekly PTV in this retrospective planning study, the potential benefit from adaptation, using prediction or not, came from ensuring tumor coverage and sparing the OAR such as esophagus and lung. For the purpose of accumulating delivered dose, we mapped the weekly GTV/OAR observed on the weekly CBCT onto the original planning CT via a deformable registration and summed the dose distribution on the original planning CT [24].

IV. RESULTS

The performance of Seq2Seq evaluated on the 60 patients is illustrated in Figure 2. At the initial planning stage, Seq2Seq made six weekly predictions of GTV, with a DICE of 0.83±0.10, 0.79±0.14, 0.78±0.12, 0.77±0.12, 0.75±0.12, 0.71±0.17. As a reference, we also calculated the DICE between the initial planning CT contours and the actual weekly CBCT contours, which were 0.80±0.16, 0.77±0.17, 0.76±0.15, 0.75±0.15, 0.73±0.15, and 0.69±0.20, for week1 to 6, respectively. As confirmed in this quantitative evaluation, the weekly predictions using the planning CT only are consistently more accurate than the initial planning CT contours for the estimates of tumor evolutions (p<0.001, two-tailed t-test), reflected by the larger mean and reduced standard deviation. The averaged HD50 and HD95 of all weekly predictions were less than 0.6 mm and 2.1 mm, respectively. As a result, all the actual weekly GTVs were fully contained by the predictive weekly PTV_pw, which makes predictive weekly planning well-positioned for ensuring tumor coverage. The correlation between the predicted and actual weekly tumor volume were 0.98, 0.98, 0.96, 0.95, 0.95, and 0.90, at week 1 to 6, respectively. Among the 16 patients whose GTV increased from planning CT to week1 CBCT, a correct prediction of such an increase was made with 15 patients (93.8%).

Figure 2.

Performance of Seq2Seq, measured by DICE and Hausdorff distance, improves as weekly CBCT scans gradually become available for predictions over the radiotherapy course.

Among fourteen out of sixty patients (23%), the prediction of tumor shrinkage rate made only with the initial planning CT deviated more than 20% from the actual weekly observation during the treatment course, all because the actual shrinkages were faster than predicted. The underprediction of shrinkage can be gradually corrected by recalibrating the predictions with the new weekly images added to the input of the model. As shown in Figure 2 and the tables in the supplement, such recalibrations did improve the predictive accuracy with statistical significance (p<0.05, two-tailed t-test). Equipped with updated and better predictions, a revision of the weekly predictive plans would improve the dosimetric quality.

A typical example of weekly predictions is illustrated in Figure 3, where the GTV shrank 25% at the end of radiotherapy, and Seq2Seq predicted a 20% shrinkage. In a simulation which replaced the actual week 2 observations with the week 2 predictions as the input of Seq2Seq, the predictions of week 3 to 6 deteriorated only 2% for this case, demonstrating that Seq2Seq is robust to handle missing weekly data.

Figure 3.

An example of ground truth and predicted GTV and Esophagus contours from planning to week 6.

The predictions from Seq2Seq led to weekly predictive plans, which maintained a 60Gy prescription dose to the actual weekly PTVs, and reduced the actual weekly esophagus mean dose to 32.0Gy due to noticeable tumor shrinkage, compared to 36.2Gy in the conventional plan optimized using the initial tumor and esophagus alone, as seen in Figure 4 for the patient case used in Figure 3. As a result, normal tissue complication probability (NTCP) of acute esophagitis would be reduced from 0.70 to 0.64 according to the model of Huang et al [25]. An online weekly replanning strategy would lower the actual weekly esophagus mean dose to 31.6Gy. The benefit brought by the weekly predictive planning is less but comparable to online replanning. Meanwhile, if we choose to isotoxically escalate tumor dose, we could keep the same esophagus mean dose, but add 3 fractions at the end as a conedown to further deliver an extra 6Gy to the actual tumor. Either way, the therapeutic ratio of the offline adaptive predictive plans would increase with the help of Seq2Seq.

Figure 4.

DVHs of weekly (1 to 6) predictive plans show that dose coverages to actual weekly PTVs (in red) and GTVs (orange) are properly maintained, while mean dose to actual weekly esophagus (green) are reduced, and mean dose to actual weekly lung (yellow) are unchanged. The dosimetric benefit results from adaptations to noticeable weekly tumor shrinkages.

In a case where the DICE between the actual and predicted (using planning CT only) 4^th weekly GTV was only 0.52, due to an underprediction of shrinkage, a recalibration at the 4^th week improved the DICE to 0.85 and 0.84, for the following 5^th and 6^th week, respectively. A replanning based on the recalibrated 5^th and 6^th weekly prediction lowered the mean dose of the ipsilateral lung to 24.9Gy, compared to 28.5Gy in the plan solely targeting the original PTV, 27.3Gy in the predictive plans using a series of PTV predictions produced at the simulation stage, and 23.3Gy with an online weekly replanning method. Dose to esophagus in this case didn’t change much since it was far away from GTV. A combination of offline recalibration and replanning did have a potential to further improve the quality of the treatment plans, approaching the limit set by the gold standard of online replanning.

In the experiment where we shifted the original ground truth GTV contours on the weekly CBCTs by 1mm in the six directions to simulate inter-observers variabilities, the DICE between the predictions and the simulated contours were 0.81 ± 0.15, 0.78 ± 0.16, 0.78 ± 0.13, 0.77 ± 0.12, 0.75 ± 0.12, 0.71 ± 0.17, respectively. The difference is not statistically significant (p=0.37, two tailed t-test) compared to the results without the shifts. Therefore, the prediction model was able to tolerate the inter-observer variabilities to some extent. This finding is consistent with what Wong et al demonstrated in their study that the accuracy of a deep learning algorithm trained by a single radiation oncologist is comparable to expert inter-observer variability for the radiotherapy planning contours [26].

V. DISCUSSION

Seq2Seq network is a novel and powerful application for understanding and modeling radiotherapy response. It expands both the spatial and temporal scope and provides a complete suite of predictions covering the entire radiotherapy course. Because the correlations between the predicted and actual weekly volumes are very high, it allows a close track of the evolution of tumors, including patterns such as volume increase at the early stage and eventual shrinkage towards the end of radiotherapy. Although in this retrospective study we mainly used the predicted patterns as input for offline adaptive treatment planning, the predicted patterns could be used to classify patients into various categories of response and modify their treatment with an individualized approach for better therapeutic ratio. The prediction of esophageal anatomical changes can also be used to identify early signs of acute esophagitis, thus preparing the clinical team as well as the patients to alleviate the adverse effect by early interventions. In our ongoing effort, we are also feeding Seq2Seq with pre-radiotherapy diagnostic images that show tumor growth rate [27–29] to further improve the accuracy of the predictive model. All of the patient specific signatures of response produced by Seq2Seq explicitly exist in the form of predicted/actual images and contours inside the treatment planning and management system as standard DICOM data; they could be easily extracted and used for outcome analysis and future design of treatment protocols. Seq2Seq is evolving towards an essential tool of adaptive radiotherapy in our clinic.

Integrating the patient specific dynamic patterns of tumor/OAR response into an offline adaptive workflow by optimizing a series of predictive treatment plans, meaningful reductions of esophagus or lung mean doses can be achieved due to the gradual shrinkage of the overall radiation fields. The weekly predictive offline planning strategy has a unique advantage compared to other popular adaptive strategies. Because its improvement is applied on a finer and more frequent weekly basis, it is more effective in the magnitude of toxicity reduction than the one-time mid-treatment adaptation strategy. Because of its critical capability of predicting the geometrical relationship between GTV and OARs, it more accurately delivers the tumoricidal dose while avoiding the adjacent critical structures than the plan of the day approach [30–33], which only provide a small pool of treatment plans accounting for extreme inter-fractional changes. Because it invests the calculation effort offline with a self-correction ability, and receives helps from our proprietary automated planning and QA tools, it doesn’t require intensive and unrealistic resources as the online adaptive approach, although it recedes slightly in terms of the plan quality. Offline predictive and adaptive planning assisted by Seq2Seq is a viable option as demonstrated in this proof of principle study. We are forming a clinical protocol to implement this novel strategy in our institution, and hopefully it can eventually benefit all patients.

Heart is a critical organ at risk for lung radiotherapy as clearly demonstrated by many clinical investigations [34,35]. We haven’t observed any systemic geometrical changes involving the heart during the 6–7 weeks of radiotherapy course on the weekly CBCTs. Dosimetric heart sparing depends on the proximity of heart to the PTV. In the weekly adaptive paradigm, the benefit brought by adaptation depends on the relative distance between the heart and the portion of tumor that disappears during treatment. We don’t observe significant heart dose reductions with predictive weekly plans in terms of heart V30 and mean heart dose. We feel that there were insufficient number of patients in this study population with tumor targets close enough to the heart to result in high cardiac heart doses. Future studies looking specifically at the subset of patients with high cardiac doses are planned to address this clearly important issue and reduce cardiac dose along with esophageal and lung doses. In addition, we plan to investigate whether weekly MRI is more appropriate for monitoring radiation-induced heart changes in a separate study.

Although the dosimetric benefits of adapting to a shrinking tumor have been demonstrated, this is not without perceived risks, such as underdosing residual tumor cells. The “shrunken” area of the tumor may still and probably does harbor microscopic residual disease. In the process of predictive treatment planning, 50Gy was prescribed to cover at least 95% of the original PTV seen on the planning CT but outside the predicted weekly PTV. This plan objective was established as a “safety guard” against marginal failures, because 50–54 Gy is very widely accepted as a conventionally fractionated dose to address microscopic disease such as might be present within the clinical target volume, or in the postoperative setting. Another related concern is that the prediction model is not perfect. It may miss some actual tumors in the weekly predictions, and cause underdoses to these areas. To mitigate this risk, we propose to use the weekly CBCTs as the basis to accumulate delivered dose and validate the applicability of predictive weekly plans. When clinical objectives are compromised, we will re-calibrate the prediction model and regenerate a new set of predictive weekly treatment plans using the accumulated dose as reference for plan reoptimization. This off-line process is currently being implemented as an automated QA pipeline for adaptive radiotherapy in our institution. It will ensure a proper coverage of the tumor.

In conclusion, it is feasible to predict the longitudinal changes of lung tumor and esophagus with the Seq2Seq network, which could lead to improving the efficiency and effectiveness of lung adaptive radiotherapy.