Advances in Automated Treatment Planning

1). Introduction

Treatment planning in radiation therapy has progressed enormously over the past several decades. Such advancements came in the form of innovative hardware and algorithms, giving rise to modalities such as intensity-modulated radiation therapy (IMRT) ^1–7 and volume modulated arc therapy (VMAT) ^8–14, greatly improving patient outcome and quality of life. While these developments have improved the overall plan quality, they have also given rise to higher treatment planning complexity. This has resulted in increased treatment planning time and higher variability in the final approved plan quality. With the use of non-convex optimizers in IMRT and VMAT, there are also no assurances that the final plan quality is within some delta of the theoretically optimal plan. In addition, with the push towards Online Adaptive Radiation Therapy, there is an even higher demand and pressure for reducing treatment planning time down to a few minutes.

The current clinical treatment planning workflow is performed iteratively and meticulously in a trial-and-error fashion. After the physician directives are set, the treatment planner uses the commercial treatment planning system and slowly fine-tunes hyperparameters—such as the structure weights, tuning structures, beam orientations, etc.—until the plan seems acceptable to the planner. Multiple rounds of feedback between the physician and treatment planner are undergone to keep pushing the plan towards the physician’s true intent until the physician finally approves the plan. The many feedback iterations are mainly due to the fact that medicine, to some degree, is still an art, and the portrayal of the physician’s preference—an accumulation of their knowledge and experiences to approximate patient outcome—is difficult to quantify with current technologies and practices.

In the last decade, since AlexNet took first place in the ImageNet competition in 2012, deep learning (DL) and other artificial intelligence (AI) technologies have made explosive progress, particularly in the areas of computer vision and imageing^15–18, as well as in decision-making^19–24. DL has become a major component in many facets of many different fields, such as from self-driving cars to home goods. One of the major areas that DL is beginning to revolutionize is healthcare. Radiation oncology, as an already technologically advanced field, has much research and implementation involving the use of AI. The field has begun to show the efficacy of using such technologies in many of its sub-areas, such as in diagnosis, imaging, segmentation, treatment planning, quality assurance, treatment delivery, and follow-up. Some AI technologies have already been clinically implemented by commercial systems.

In this article, we will provide an overview to methods involved with treatment planning in radiation therapy. In particular, we will review the recent research and literature related to automation of the treatment planning process, leading to potentially higher efficiency and higher quality plans. We will then present the current and future challenges, as well as some future perspectives.

2). Status

Typically, the clinical radiation therapy pipeline can be viewed as consisting of the following steps: image acquisition, segmentation, physician directive generation, treatment plan generation, quality assurance, treatment delivery, and follow-up. The deep learning models discussed in this article largely are focused on physician directive generation, treatment plan generation, and quality assurance. An accurate clinical dose prediction, by creating an accurate 3D distribution of what is likely to be the final plan dose, can be used for both physian directive guidance and plan generation guidance. Dose calculation is used throughout the fluence map optimization stage as a fast intermedicate dose calculation step, as well as for the quality assurance phase as a secondary dose check. Fluence map prediction can occur during the plan generation step, pontially bypassing the fluence map optimization directly and provide a set of fluence maps directly from a desired dose distribution. Lastly beam orientation optimization is used to automatically determine the ganty and couch angles to be used during the plan generation step, a process that is currently done either by protocol or manually.

2.a). Dose Prediction

Initial efforts focused on developing knowledge-based planning (KBP) ^25–42, which was based on taking historical plan data and then hand-crafting useful features for training a model. The features included things such as spatial information of organs at risk (OAR) and planning target volumes (PTV), distance-to-target histograms (DTH), overlapping volume histograms (OVH), structure shapes, number of delivery fields, etc.^34–43. The early version of KPB utilized machine learning (ML) methods where hand-crafted features from the patient data were fed into the ML model to learn the mapping of these features to planning endpoints and constraints, such as the dose volume histogram (DVH) points. When coupled with an optimization engine, these frameworks can be then semi-automated and are capable of generating a plan for a new patient, given their anatomy.

However, the early versions of KPB were highly limited with the data complexity that could be fed into the model, and well as the type of data that the model was able to predict. The outputs were usually limited to 1D or 2D data, such as single constraint values or the dose volume histogram (DVH), leaving the remainder of how the dose distribution should look entirely up to the physician’s and planner’s intuition as they generate the final deliverable plan. In addition, it was unknown exactly what hand-crafted features are needed to be fed into the model, and therefore features were typically determined through trial-and-error. In addition, manually hand-crafting features may result in the loss of subtle yet crucial information, which can result in the reduced predictive performance of the KBP model. As a result, the plan quality is thus still highly dependent on the physician’s and planner’s skills and experience.

The advancement of DL allowed accurate 3D dose distribution predictions to be performed. One such model was the U-net by Ronneberger et al.⁴⁴. This model was originally introduced for semantic segmentation of biomedical images and was able to incorporate both local and global features for learning a pixel-to-pixel mapping between two data. Many variations of the U-net, such as ones with special connections^45,46, as well as 3D variants⁴⁷, were developed shortly after. Its ability for pixel-to-pixel or voxel-to-voxel mapping made it an ideal candidate for volumetric dose prediction, where the 3D anatomical data is fed into the model in order to predict the 3D dose distribution. Furthermore, DL allowed the raw data to be input instead of relying on hand-crafted features like in classical ML.

Currently, many DL-based volumetric dose prediction models have been developed in the literature and can be largely categorized into two major groups: 1) clinical dose prediction and 2) Pareto optimal dose prediction. Clinical dose prediction models take on more of a KBP flavor, as they still rely on learning from historical plans to make a dose prediction. These have been developed for several cancer sites, such as prostate^48–52, lung ^53,54, and head and neck ^55–58. Figure 1 shows an example set of dose distributions, which include the ground truth dose and mutiple DL dose predictions for a head and neck cancer patient. The best peforming DL model, HD U-net had the closest prediction to the ground truth, while the worst performing model, DenseNet, due to its incapability to calculate global features, had predicted a low dose far away from the target. Pareto optimal dose prediction models try to instead learn the feasible trade-off space, specific to a given patient, which is more similar in flavor to multi-criteria optimization (MCO) ^59,60. These models typically can be constructed in order to make predictions from the user tuning the structure weights ^61,62, or by tuning the DVH directly ^63,64. These models can typically make a prediction after the user places their input within a sub-second timescale, making real-time feedback feasible. In addition, by using methods such as ensemble techniques or Monte Carlo Dropout⁶⁵, any of these DL-based dose prediction models are capable of generating an uncertainty estimate—a way for the model to say “I don’t know” when it makes a prediction⁶⁶.

Figure 1.

Dose washes of an example test patient from the study by Nguyen et al.⁵⁵. The colorbar is shown in units of Gy. The clinical ground truth dose is shown on the top row, followed by the dose predictions of the proposed model, HD U-net, and two other comparative models, Standard U-net and DenseNet. Low dose cutoff for viewing was chosen to be 5% of the highest prescription dose (3.5 Gy).

2.b). Dose Calculation

Dose calculation is used in several aspects of treatment planning, from dose influence calculation, intermediate dose calculation, final dose calculation, and secondary dose checks for plan quality assurance. These calculations themselves have primarily existed as full physics calculation engines. The gold standard in dose calculation is considered to be Monte Carlo (MC) dose calculation⁶⁷, and is often considered to be the ground truth when comparing against other dose calculation engines. However, there is a trade-off between the dose calculation accuracy and the computational workload. The challenge of adopting MC in the clinic is due to its low efficiency and long calculation times. MC needs to stochastically and individually simulate particles, their interactions with the medium, and follow child particles that are generated. Dose calculation research has historically been designing physics engines that are trading off between efficiency (reducing dose calculation time) and accuracy (reducing dose calculation error). One primary example was the development of the Analytical Anisotropic Algorithm (AAA)⁶⁸, a pencil beam convolution/superposition algorithm, which is clinically used in commercial software today. AAA is much faster and can calculate doses within a clinically acceptable timeframe. However, it has been found that, compared to MC, AAA dose calculation accuracy can fall short in certain areas, having as high as 12% error near isolated leaf edges and overestimation of some critical structures as high as 9%⁶⁹. Another algorithm, Acuros XB⁷⁰, similarly uses the Boltzmann transport equation like MC, but solves it in a grid-based manner instead. Acuros XB, using some approximations to bring its calculation speed to a practical level, was found to agree with MC calculations within 3% of low-density lung, and within 4.5% error of phantom tissue just above and below an air cavity. While Acuros XB does have accuracy-levels similar to MC, it is still much slower by about 14 fold compared to AAA, taking 110 s for Acuros XB versus 8 s for AAA to calculated for a static field at reference conditions⁷⁰.

Deep learning has made it possible to boost the accuracy of dose calculation without sacrificing the calculation speed. A common approach is to feed in a computationally cheap, first-order approximation as a prior into the model, which then uses it, alongside other anatomical, geometrical, and/or radiological data, as input to learn a more accurate dose calculation such as Monte Carlo or Acuros XB, for example. These deep learning methods have been applied for different modalities such as external beam photons^71–73, protons⁷⁴, and brachytherapy⁷⁵ for a variety of cancer sites. Deep learning-based dose calculation has also been applied for accelerating the dose calculation under the influence of a magnetic field for the purpose of MRI-guided radiation therapy⁷⁶. These methods have high potential to improve areas of the treatment planning process, such as the intermediate dose calculation step during optimization, which currently uses a fast but inaccurate calculation, or the secondary dose check that tries to confirm that the final plan dose calculation is reasonable.

2.c). Fluence Map Prediction

In the typical treatment planning workflow, the treatment planner often has to enter an inverse optimization phase multiple times as the fine-tune hyperparameters and get feedback from the physicians in order to solve the optimal dose distribution and its corresponding fluence maps/apertures. Each optimization is a lengthy process, and can take as long as 20 minutes for one round of optimization for certain modalities like VMAT. Fluence map prediction attempts to accelerate the treatment planning process by predicting the fluence maps directly instead of the dose distribution, potentially bypassing some or the entire optimization step that is current done in the clinical setting. A common DL-based dose prediction method uses an existing dose distribution projected to each field’s beam’s eye view as input to predict the fluence maps for a given beam configuration^77–79, with an example workflow shown in Figure 2. These methods require that a dose distribution already exists, which could be obtained from a dose calculation or a DL-based dose prediction, for example. Other methods may use the target and organs-at-risk segmentations, projected onto beam’s eye view, directly instead of a dose distribution⁸⁰, which offers a direct route from anatomy to a plan, but in turn, has less flexibility in making adjustments to the plan. These map prediction methods are still in the earlier stages of development and implementation, but their great performance and results offer high promise for fast, automated treatment planning by bypassing the time-consuming optimization phase altogether.

Figure 2:

Workflow of the proposed fluence map prediction method. The leftmost figure represents the projections of dose in phantom geometry⁷⁷.

2.d). Beam Orientation Selection

In external beam radiation therapy, the radiation enters the patient’s body from an external source. The orientation that the fields are set up is critical in order to best minimize the toxicity to OARs while maintaining the prescription dose on the PTV. Clinically, the beams are selected either via protocol, leading up a sub-optimal set, or the beams are manually and tediously fine-tuned by the treatment planner. Beam orientation optimization (BOO) is a method that tries to select a suitable set of beam angles for use, and has been widely researched over the past decade. Typically, these methods are trying to solve the problem in the large non-coplanar space, and employ heuristics, to solve the otherwise unwieldy optimization problem. Some major applications have been the development and implementation of 4π Radiotherapy^39,81–91, and station parameter optimized radiation therapy (SPORT) ^92–97. Most of these optimization methods try to solve the problem within the dosimetric space, instead of pure geometrical space, which results in a set of field orientations that best achieves the dosimetric goals that clinicians care about. However, the major downside to dosimetric-based objectives is they require the dose calculation of all beamlets for every candidate beam, which could be easily over 1000 beams in the non-coplanar space with 6 degree separation. This could take from several hours to several days depending on several factors such as dose calculation engine, number of candidate fields, cancer site, tumor size, etc., meaning that clinical implementation of such a framework would require the clinical workflow to account for the additional overhead and calculation that is needed for the calculation.

Using deep learning for BOO can be used to accelerate the process at the deployment stage. One of the main ideas is to offload the major computational expense of BOO, the dose calculation of every candidate, onto the data preparation and training phase of the DL model and have the model internalize the concept and relationships of beam orientations and dose distributions. This way, the model can suggest these beam orientations without needing to perform a full dose calculation on candidate beams. The studies by Sadeghnejad-Barkousaraie et al. found that DL was capable of learning a beam selection policy of a method called column generation (CG) and providing a beam angle set that was non-inferior to that of CG. The DL method could solve the problem on sub-second timescales, as opposed to the many minutes that CG needs to solve the same problem⁹⁸. They further improved on the prediction by adding on a Monte Carlo Tree Search and shows that they were able to beat the CG solution in less time⁹⁹. Monte Carlo Tree Search has also been used in other applications, such as optimal non-coplanar arc trajectories for VMAT⁹². DL-based BOO methods are still an early area of research and development but have high promise for improving the current methods and for clinical implementation.

2.e). Reinforcement Learning Methods for Treatment Planning

Reinforcement Learning (RL)¹⁰⁰ is a broad class of machine learning techniques that allows for a model to learn a policy for a system with respect to maximizing a defined reward. The model (agent) is allowed to interact and explore with the system (environment). Each action of the agent will affect the environment, causing it to be in a different state than before. A reward or penalty can then be computed, which is provided back to the agent to let it know whether its particular action was good or bad. Over time, as the agent continues to interact with the environment, it learns how to take actions that maximize the reward. The reward/penalty itself is a user-defined metric, typically as a function of the state, and reflects whether we see the current state of the environment as desirable or not. Classical reinforcement learning techniques required for the framework maintain a record of the state, possible actions, and the current respective rewards for those actions (which are updated as the agents continue to explore), for every time point of the state. However, as systems became more complicated, maintaining this record became infeasible, leading to the development of other methods. One such development came with the advancement of deep learning, leading to deep reinforcement learning (DRL)¹⁹, which uses a deep neural network to model agents. DRL has the same training strategy as RL, except that, since deep neural networks have high function approximation capabilities, DRL agents can directly internalize and correlate states, actions, and rewards instead of a tabulated record of these values. This allowed for much-improved decision-making by the agent for complex environments and tasks required by many modern applications. For example, a Deep Q-learning²² method was used to achieve super-human level performance on Atari games^20,21. Another example is the creation of AlphaGo by DeepMind^23,24, where this DRL-based model was able to defeat the top professional human players in the game of Go.

Overall, DRL holds a very strong potential for solving sequential-type clinical problems, such as dose fractionation schemes and hyperparameter tuning during optimization. Some early DRL studies looked into using an RL agent to adapt dose fractionation protocols and found that their agent found policies similar to those in clinical protocols^101,102. For hyperparameter tuning during treatment plan optimization, it was found that DRL agents are able to automatically fine-tune structure weights and threshold dose values are used during fluence map optimization, in order to maximize a defined dosimetric reward^103,104, with an example shown in Figure 3. This can help automate the treatment planning process that is typically time-intensive and tedious for human treatment planners.

Figure 3:

Evolution of: (a) dose volume histograms and dose distributions, (b) Treatment planning parameter (TPP) weight values(λ and τ), and (c) and original and modified ProKnow plan scores in the planning process of a test patient case using our DRL-based virtual treatment planner network (VTPN)¹⁰⁴. ProKnow plan score is a scoring system for prostate cancer IMRT plan, using scores for different clinical criteria (ProKnow Systems,Sanford, FL, USA).

3). Current Challenges and Future Perspectives

Despite their great promise, AI methods for automated treatment planning in radiation therapy face several challenges in both theoretical development and practical applications. The first issue is related to the data used to develop these models, namely limited dataset size, data quality, and data distribution diversity. DL models are data-hungry, with the first successful instances of these models in the new DL era, such as AlexNet¹⁶, requiring millions of annotated images in order to show promising results. Unfortunately, healthcare data is typically much more limited in quantity and is oftentimes much messier (e.g., missing data points, high noise, etc.). In addition, model development often happens at a single institution or area, where there is a specific demographic, which can introduce bias into the model. Many advancements have been made, and researchers are focused on improving results on limited and noisy datasets, such as in meta-learning¹⁰⁵, or learning how to adapt to datasets in different distributions than originally trained on, such as transfer learning¹⁰⁶. However, these methods and their applications to healthcare data are still being explored by the research community.

As these models begin to enter the phase of clinical implementation, many practical challenges arise. Oftentimes, the major areas that are needed for clinical implementation are overlooked entirely by the research team, such as exactly what kind of data and how the data is formatted to be presented to the clinician. A study by McIntosh et al. had found that, while ML models hold great promise in the clinic, most methods are currently evaluated in a simulated environment that does not reflect the clinical workflow as realistically as is should. As such, the radiation therapy plans, while having a a high rate of 83% for being selected for treatment in the simulation phase, had a sigficiantly lower rate of 61% of being selected for treatment in the deployment phase¹⁰⁷.This indicates a large gap in model development and simulated evaluation, where the data and simulation environment are not representative of the true clinical setting. In addition, the clinical implementation needs to be broken down into several phases to ensure the model operability within the clinical workflow and the patient’s safety. This can include several stages such as tool readiness evaluation, workflow, and user interface design, initial implementation, debugging and refinement, departmental approval, clinical implementation, and quality assurance (QA) & renewal procedures. Furthermore, model interpretability and explainability are still large challenges for AI, but are necessary for its success to be used in healthcare. Currently, there are ways to make models provide uncertainty estimation^65,66 (a way for the model to say “I don’t know”), as well as letting the model pinpoint where it is looking at in the data using attention modules^108,109. While this is a large step forward, the interpretation of these is still left largely left to the clinician, making assumptions about what the model may be thinking in order to make its prediction. An open discussion needs to be started prior to beginning the research project and model development to ensure that the direction of the model perfectly meets the clinical needs, instead of researchers trying to solve a clinical problem that may not even exist. As AI begins to integrate itself more and more into healthcare, more pressure for open collaboration between researchers and clinicians will likely accelerate AI’s impact into healthcare.

As radiation therapy becomes more-and-more automated, the current tedious and manual tasks will likely require less-and-less human involvement, greatly improving the overall efficiency of the clinical pipeline. However, in line with the automation paradox¹¹⁰, automation tends to introduce new, higher-level roles, and the areas where people are required will likely become more critical to prevent catastrophic failure where the AI models simply fail (e.g. trying to predict on an out-of-distribution patient or data point). Radiation therapy treatment planning will continue to evolve and strive for higher efficacy and higher efficiency. With the current boom of AI, deep learning has begun to integrate itself into several aspects of treatment planning, which the current major areas have been in dose prediction, dose calculation, fluence map prediction, beam orientation selection, and other reinforcement learning methods. As these AI frameworks become more and more integrated, and begin to revolutionize treatment planning towards more intelligent and automated methods, patient outcomes and quality of life will continue to improve.