Automating proton treatment planning with beam angle selection using Bayesian optimization

Abstract

Purpose

To present a fully automated treatment planning process for proton therapy including beam angle selection using a novel Bayesian optimization approach and previously-developed constrained hierarchical fluence optimization method.

Methods

We adapted our in-house automated intensity modulated radiation therapy (IMRT) treatment planning system, which is based on constrained hierarchical optimization and referred to as ECHO (Expedited Constrained Hierarchical Optimization), for proton therapy. To couple this to beam angle selection, we propose using a novel Bayesian approach. By integrating ECHO with this Bayesian beam selection approach, we obtain a fully automated treatment planning framework including beam angle selection. Bayesian optimization is a global optimization technique which only needs to search a small fraction of the search space for slowly varying objective functions (i.e., smooth functions). ECHO is run for some initial beam angle candidates and the resultant treatment plan for each beam configuration is rated using a clinically relevant treatment score function. Bayesian optimization iteratively predicts the treatment score for not-yet-evaluated candidates to find the best candidate to be optimized next with ECHO. We tested this technique on five head-and-neck (HN) patients with two coplanar beams. In addition, tests were performed with two non-coplanar and three coplanar beams for two patients.

Results

For the two coplanar configurations, the Bayesian optimization found the optimal beam configuration after running ECHO for, at most, 4% of all potential configurations (23 iterations) for all patients (range: 2%−4%). Compared with the beam configurations chosen by the planner, the optimal configurations reduced the mandible maximum dose by 6.6 Gy and high dose to the unspecified normal tissues by 3.8 Gy, on average. For the two non-coplanar and three coplanar beam configurations, the algorithm converged after 45 iterations (examining less than 1% of all potential configurations).

Conclusions

A fully automated and efficient treatment planning process for proton therapy, including beam angle optimization was developed. The algorithm automatically generates high-quality plans with optimal beam angle configuration by combining Bayesian optimization and ECHO. As the Bayesian optimization is capable of handling complex non-convex functions, the treatment score function which is used in the algorithm to evaluate the dose distribution corresponding to each beam configuration can contain any clinically relevant metric.

1. INTRODUCTION

In clinical routine, beam angle configurations are often selected using predefined templates and/or a trial-and-error process based on the planner’s experience. Different institutions usually have widely varying templates and it is not always straightforward which beam configurations will provide adequate target coverage without overdosing the surrounding organs at risk (OARs). Additionally, compared to photon-based therapy, fewer beam angles are typically used for proton therapy¹, making beam angle optimization of particular importance for proton therapy.

Previously, different optimization schemes have been suggested for beam angle selection for photon therapy, including exhaustive search², and mixed integer programming^3–5. Only a few studies have investigated beam angle optimization for proton therapy, including Gu et al. who added a group sparsity term into their optimization framework for intensity modulated proton therapy (IMPT), whereby only a few beams would be selected by the algorithm⁶. They later updated their method to also include robustness to setup and range uncertainties⁷. Cao et al. proposed a local neighborhood search algorithm where each beam was exchanged with one of its neighbors to find the optimal configuration⁸.

All the aforementioned studies employ some version of the conventional fluence optimization technique where the weighted-sum of the objectives corresponding to tumor and OARs is optimized. In order to perform a beam angle optimization, the weights for the objective function for each OAR need to be adjusted in advance, for instance based on one specific beam configuration. This approach has two main drawbacks: 1) The objective function weight adjustment is usually a manual trial-and-error and time-consuming process, and 2) using the same set of objective function weights for all beam configurations could result in sub-optimal plans. Given the increase in computational time that comes from adding beam angle optimization, it is critical to integrate beam angle optimization with an automated treatment planning framework to avoid a very long, iterative treatment planning cycle. One approach along these lines for photon therapy is that of Breedveld et al., who demonstrated a greedy-based technique to address this issue, where the most beneficial beams were sequentially added⁹. In the current study, we propose a new approach that integrates Bayesian optimization with our recently developed automated planning technique based on hierarchical constrained optimization¹⁰. This approach can simultaneously optimize the fluence and beam angles and automatically generate a plan with the optimal beam configuration.

The beam angle optimization problem is typically formulated as a two-level optimization problem, with the upper-level problem optimizing the beam configuration and the lower-level problem optimizing the fluence for a given beam configuration. The main challenge with this formulation is that there is then no explicit objective function (closed-form expression) for the beam angle optimization linking the decision variables (the beams) to a treatment plan quality metric, as a fluence optimization problem needs to be solved to obtain the objective function value for each given beam configuration. Therefore, the objective function evaluation for the beam angle optimization is computationally expensive, especially if constrained fluence optimization is used^10–12. Bayesian optimization is a global optimization technique designed to optimize complex objective functions and it has been widely used in Machine Learning and Deep Learning problems for hyperparameter optimization and tuning¹³.

We propose that Bayesian optimization is well-suited for beam angle optimization. Bayesian optimization sequentially evaluates the most probable optimal points and can find the optimal or near-optimal solution after a few function evaluations. We have integrated Bayesian optimization with our in-house automated treatment planning system called ECHO (Expedited Constrained Hierarchical Optimization) which has been recently introduced into clinical use for photon therapy¹⁰. ECHO solves a series of constrained fluence map optimization problems in hierarchical order of the objective function priority. We conduct the beam angle optimization in a nested-loop algorithm where the outer loop applies Bayesian optimization to find the next beam configuration to be evaluated in the inner loop, which runs ECHO for the given beam configuration. While the objective functions used in ECHO optimization could be used in Bayesian optimization for plan evaluation, Bayesian optimization can handle more sophisticated non-convex plan quality metrics. In this study, we define a plan quality metric (referred to as the treatment score) based on the established DVH criteria used at our institution.

Beam angle optimization alone is a difficult optimization problem and combining that with automated planning makes it even more challenging problem. This study will focus on a proof-of-concept using Bayesian optimization and ECHO for proton therapy. While Bayesian optimization has received increasing attention in other fields¹³, this work, to the best of our knowledge, is the first one using Bayesian optimization for beam angle optimization, and the first study to combine automated treatment planning and beam angle optimization for proton therapy.

2. MATERIALS AND METHODS

2.1. Bayesian optimization

Bayesian optimization^a, which belongs to a group of optimization techniques known as black-box optimization¹³, can be used for optimization of a function $f$ that takes a long time to evaluate and may not have an explicit expression or gradient. The function does not even need to be differentiable; however, a certain level of smoothness is assumed. Bayesian optimization sequentially evaluates the most probable optimal points by leveraging an estimated function, $\widehat{f}$, whereby only a few function evaluations of $f$ are necessary, reducing the computational cost. After a few initial evaluations of the function, $\mathcal{D}=\left\{\left(x_1,f\left(x_1\right)\right),\dots ,(x_{n-1},f(x_{n-1}\left)\right)\right\}$, a Gaussian process¹⁴ is established as prior on $f$ and the estimated function $\widehat{f}$ is built by calculating the posterior probability distribution. The posterior distribution also provides the uncertainty of the estimated function quantified by the variance $\sigma$¹⁵. The MATLAB (The MathWorks, Inc., Natick, MA) functions fitrgp and predict provides $\widehat{f}$ and $\sigma$. An acquisition function,$A$, is defined by combining the estimated function $\widehat{f}$ and the uncertainty $\sigma$, and then the point with the highest $A$-value ($x_{\mathrm{next}}={\mathrm{argmax}}_{x}A\left(x\right))$) is used as the next sample point to be evaluated. The acquisition function represents the expected improvement by balancing the trade-off between exploitation (choosing the next evaluation point as the one minimizing $\widehat{f}$) and exploration (choosing the point with the highest uncertainty to encourage searching the unexplored regions of the search space)¹⁵. Here the Expected Improvement equation¹⁵ was used as the acquisition function and was calculated based on the $\widehat{f}$ and $\sigma$ obtained from the function predict.

Figure 1 provides a schematic illustration of the Bayesian optimization process on a simple one-dimensional example. The dashed black line represents the true objective function $f$, which is only known at the two evaluated sample points, $x_1$ and $x_2$ (Figure 1A). The Gaussian process is used to obtain the estimated function $\widehat{f}$ (blue full line) as well as its uncertainty $\sigma$ which gives the confidence interval of the estimated function (red shaded area). The acquisition function $A$ (green solid line) is defined by combining $\widehat{f}$ and $\sigma$, and the $x$-value with the highest acquisition function (green cross) is then used as the next sampling point, $x_{\mathrm{n}\mathrm{e}\mathrm{x}\mathrm{t}}$. After evaluating $f$ at $x_{\mathrm{n}\mathrm{e}\mathrm{x}\mathrm{t}}$ (denoted $x_3$ in Figure 1B), $\widehat{f},\mathrm{ }\sigma$ and $A$ are updated by using all three sample points, and the maximum for the new acquisition function is found. This iterative process terminates after a user-defined number of iterations. In this study, the optimization variable $x$ for the Bayesian optimization will be the beam configuration.

Figure 1:

Schematic illustration of Bayesian optimization technique in 1-dimension. A) The true unknown objective function $f$ (dashed black line) has been evaluated at only two points, $x_1$ and $x_2$. Based on these, the estimated function $\widehat{f}$ (full blue line) and its uncertainty, $\sigma$, is established. The red shaded area represents the confidence interval of $\widehat{f}$ given by $\widehat{f}\pm \sigma$. At the evaluated points, $x_1$ and $x_2,\mathrm{ }f$ and $\widehat{f}$ coincide and $\sigma =0$. Combining $\widehat{f}$ and the width of its confidence interval, the acquisition function $A$ (full green line) is obtained. The next point to evaluate is chosen as $x_{\mathrm{n}\mathrm{e}\mathrm{x}\mathrm{t}}=\underset{x}{\mathrm{argmax}}A\left(x\right)$. B) After evaluating $x_{\mathrm{n}\mathrm{e}\mathrm{x}\mathrm{t}}$, denoted $x_3$ in the next loop, Bayesian optimization continues by updating the estimates, $\widehat{f}$ and $\sigma$.

2.2. Expedited Constrained Hierarchical Optimization (ECHO)

For the fluence optimizations we use our in-house developed automated planning system, ECHO, which is based on constrained hierarchical optimization¹⁰. ECHO solves a series of constrained fluence map optimization in a hierarchical order; first it maximizes the target coverage while respecting all the maximum/mean dose hard constraints on the OARs and the planning target volume (PTV). The results of the first step are preserved in the following steps by turning the obtained objective function value into a constraint for the next step with a slight relaxation, based on a so-called “slip” parameter, to enlarge the search space for further improvements. In the second step, ECHO minimizes the OAR doses while again respecting all the dose constraints. Although ECHO has been initially introduced for photon-based intensity modulated radiotherapy (IMRT), it can be adopted for proton pencil beam scanning by simply replacing the beamlet intensities (photon decision variables) with the proton spot intensities. The original photon version of ECHO also has a third step to improve the plan delivery efficiency with multi-leaf collimators (MLCs)¹⁰ which is skipped for proton therapy, where (typically) no MLCs are used. ECHO uses the generalized equivalent uniform dose (gEUD)¹⁶ for the objective functions and the maximum/mean dose hard constraints are defined based on our institution clinical criteria for HN cases. For proton fluence optimizations, a multi-field optimization (MFO) approach is used. The reader is referred to Zarepisheh et al.¹⁰ for more details about the ECHO approach.

2.3. The integrated beam angle optimization technique

We have integrated ECHO with Bayesian optimization to devise a fully automated treatment planning technique including beam angle selection. ECHO and Bayesian optimization are run independently and communicate with each other in a nested-loop fashion (Figure 2). We first initialize the algorithm by running ECHO for some initial beam configurations. For each optimized plan, a treatment score is calculated. Bayesian optimization estimates the treatment score for all unevaluated beam configurations and finds the next beam configuration to be evaluated, and ECHO is run for the given configuration. The treatment score is extracted from the resultant ECHO plan and is fed back into the Bayesian optimization for updating the estimate function and finding the next configuration for evaluation. For the computational efficiency purposes, ECHO is using gEUDs as convex surrogate functions for optimization; however, the treatment score can include non-convex DVH metrics. We use our institutional clinical guidelines to establish a treatment score function based on some clinically relevant DVH criteria as well as maximum/mean dose criteria of OARs and PTV. All ECHO plans meet the maximum/mean dose clinical criteria since they have been embedded as hard constraints in both steps of ECHO optimization, but different beam configurations could result in different OAR dose reduction, beyond the clinical requirements, and also different PTV coverage.

Figure 2:

Sketch of the iterative beam angle optimization framework. Bayesian optimization estimates the treatment score for all the unevaluated beam configurations and finds the best configuration, $x_{next}$, to be evaluated next by ECHO. ECHO is run for this configuration and then a treatment score is calculated for the resultant plan. The treatment score is fed back into Bayesian optimization to update the estimated function in the next loop. The loop is run for a fixed number of iterations.

The algorithm can in principle handle as many coplanar and non-coplanar beams as desired; however, the computational time will increase with the size of the problem. In this study, we consider optimizing two coplanar, two non-coplanar, and three coplanar beams. The algorithm selects beams out of 36 equally spaced coplanar beams (gantry angle from $0^{\circ }$ to ${350}^{\circ }$ degrees in steps of ${10}^{\circ }$, at least ${30}^{\circ }$ apart to reduce beam overlap and thereby dose to normal tissue). For the non-coplanar, the algorithm can choose the couch angles among $-10°,\mathrm{ }0°$, and $10°$. To filter out some dispensable beam configurations from the search space, we exclude double counting (i.e., (beam-1, beam-2) is the same as (beam-2, beam-1)). This result in total possible configurations of 558 for two coplanar (2-coplanar), 5022 for two non-coplanar (2-non-coplanar), and 4960 for three coplanar (3-coplanar) beams. Our experience shows that the algorithm only needs to evaluate few configurations to find the optimal or near-optimal solution (~20–50), but to conform this and study the convergence behavior of the algorithm, and to measure the optimality gap, we need to obtain the ground-truth optimal configuration (the algorithm’s ground-truth based upon the given treatment score) by optimizing all possible beam configurations in a brute-force fashion. This is computationally challenging for 2-non-coplanar and 3-coplanar beams with so many possible configurations and therefore we only study the patient with the smallest PTV (patient 4) for these scenarios. For one other patient (patient 3), the algorithm was also tested, but without optimizing all possible beam configurations. For 2-coplanar beams, we also compare the automated selected beam configurations with the manually selected beam configurations. We initialize the algorithm by running ECHO for four initial beam configurations which are somewhat evenly distributed in the corresponding search space (for 2-coplanar/2-non-coplanar beams: $x_{\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{t}}=\{\left(20°,90°\right), \left(120°,200°\right), \left(250°,330°\right), (100°,300°\left)\right\}$, for 3-coplanar beams: $x_{\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{t}}=\{\left(20°,60°,120°\right), \left(70°,120°,310\right), \left(60°,200°, 240°\right), \left(220°,310°,340°\right)\})$.

2.3. Patient population

We test our integrated algorithm on five HN patients selected from the public database, The Cancer Imaging Archive (TCIA)^17,18. The five patients all come from the same data set, and the CT images and organ delineations are retrieved for optimization. The five patients all have large tumors consisting of a collection of separate or overlapping target volumes, but we only focus on the primary clinical target volume (CTV) to have a proof-of-principle. All CTVs are located in or in close vicinity to the oral cavity. The delineated structures include the parotid glands, the mandible, the oral cavity, the CTV, the brainstem, and/or the spinal cord. The other delineated structures are not within the treatment fields for any of the beams used in this study and are not included in the optimization. The PTV is created by adding a 0.5 cm margin to the delineated CTV contour, and a dose prescription of 70 Gy to the PTV is used for all five patients.

2.4. Computational framework

We have implemented our algorithm using MATLAB (R2018a, The MathWorks, Inc., Natick, MA) and the calculations are conducted on a workstation with Intel CPU 2.4 GHz and 64 GB RAM. For modeling purposes, the patient body is discretized into small voxels based on the CT grid (voxel size [0.9766,0.9766,2] mm). The dose delivered to each voxel from equal-intensity spots is pre-calculated for each beam, using the pencil beam dose calculation algorithm available in open source software matRad^19,20, applying a proton spot spacing of 5 mm, and is stored as a matrix called the influence matrix. All the plans are generated using ECHO dose optimization algorithm. Table 1 shows the number of voxels and proton beam spots used in the optimization for each patient. Since the number of beam spots differs between the beam configurations, a range is provided in the table.

Table 1:

The number of voxels and beam spots included in the fluence optimization for each patient, along with the volume of the PTV. For the number of beam spots, the range (minimum to maximum), corresponding to different beam configurations, is provided.

Patient number	Number of voxels	PTV volume (cm3)	Number of beam spots (range)
1	90460	162.7	4931 – 5058
2	99576	169.7	5370 – 5491
3	106570	129.9	3887 – 4001
4	54568	12.4	503 – 540
5	88991	107.5	3049 – 3139

2.5. Beam configuration selection criteria

One of our experienced proton planners (J.S.S.) selected the manual beams for the 2-coplanar configurations by looking at the relevant patient data (e.g., CT scan, structure set, beam-eye-view). Parotid and brainstem/cord dose was of top priority to reduce, but other clinical routine considerations for HN treatment planning was also considered. The planner was given the freedom to plan the patients as found best with the only restriction being that only two coplanar beams could be used. The treatment score used in the Bayesian optimization was constructed as a summation of our established institutional clinical criteria (e.g., PTV V_100%, PTV D_99%, Parotid mean dose, Mandible/Brainstem/Cord maximum dose) to reflect our clinical practice. Not to completely disregard the less important clinical criteria (e.g., Parotid max dose, Mandible/Brainstem/Cord mean dose), they were also included in the treatment score but with a smaller weight (0.8/0.2 for important/less-important criteria). ECHO is employed to generate plans using the planner’s selected beam configurations, and then ECHO is used in conjunction with Bayesian optimization to automatically generate plans with optimal beam configurations.

3. RESULTS

3.1. Two coplanar beam configurations

Figure 3A illustrates the progress of the Bayesian optimization for all five patients for the 2-coplanar configurations. The x-axis represents the number of iterations which corresponds to the number of evaluated beam configurations. Both the absolute and relative number of configurations are indicated in the top and bottom of the figure, respectively. The y-axis represents the relative optimality gap which is the relative difference to the treatment score of the ground truth optimal solution (known as we evaluated all possible beam angle configurations). For all five patients, the Bayesian optimization identifies the optimal solution after evaluating at most 4% of all possible configurations (23 out of 558 configurations). The Bayesian optimization does not necessarily improve the solution as it progresses, and therefore, the best-so-far solution needs to be identified at each iteration. For patient 5, the dashed line in Figure 3B shows the treatment score at each iteration of the Bayesian optimization and the solid line shows the treatment score of the best-so-far plan.

Figure 3:

The progress of the Bayesian optimization. The x-axis represents the number of evaluated beam configurations (above: the absolute number; below: the number of configurations relative to all 558 possible configurations) and the y-axis presents the relative optimality gap defined as the relative deviation of the treatment score to the ground truth optimal solution. A) The progress behavior for all five patients when the best-so-far solution is selected at each iteration. B) For patient 5, the dashed (solid) line is the treatment score of the last (best-so-far) plan.

Table 2 lists the beam configurations selected manually by the planner and the beams selected automatically using the Bayesian optimization algorithm. Figure 4 compares the DVHs for the treatment plans generated using the manual and automated beam configurations. The normal tissue structure includes all tissues outside the PTV. For all patients, only one parotid receives any noticeable dose; for patient 4 it is the right parotid, and for the rest it is the left parotid, therefore only one parotid structure is shown. The treatment plans with the algorithm selected beams are comparable to the plans with the planner selected beams. Figure 5 provides a side-by-side dose comparison of the plans with respect to some clinically relevant metrics used at our clinical practice. For each metric, the sign of the dose differences has been chosen so a positive difference means the plan with the automated beam selection is favorable, i.e. delivers less dose to the OAR or reduces the PTV under- and over-dosage. The general trend in Figure 5 is that the algorithm scores better on the mandible maximum dose, the normal tissue hot spot (D_1%) and the parotid mean dose, while it sacrifices the mandible mean dose and the PTV hot spot (D_1%). These differences, which can also be seen in Figure 4, are attributed to the treatment score function and its higher weight (i.e., 0.8) for important clinical criteria like parotid mean dose and mandible max dose and smaller weight (i.e., 0.2) for parotid max dose and mandible mean dose.

Table 2:

The beam angles selected automatically using the algorithm and the beams selected by the planner.

	Automatically selected beams	Planner’s beams
Patient 1	(40°, 320°)	(40°, 90°)
Patient 2	(40°, 320°)	(74°, 285°)
Patient 3	(30°, 60°)	(75°, 130°)
Patient 4	(190°, 280°)	(230°, 310°)
Patient 5	(30°, 300°)	(70°, 280°)

Figure 4:

DVH comparison of the ECHO plans generated using the beam configuration chosen automatically by the algorithm (solid line) and by the planner (dashed line). For patient 4, a sub-plot is added to zoom in on the low dose region and demonstrates the differences. The other delineated organs (e.g. brainstem and cord) received minimal or no dose for these configurations and are therefore excluded from the DVHs.

Figure 5:

Side-by-side dose comparison of the treatment plans generated using the automated beams and the planner’s beams. Positive differences indicate that the results are in favor of the automated beams.

Figure 6 compares the dose distribution for patient 2 of the plan with the algorithm selected beams (left) and the planner selected beams (right). While the two plans are comparable (as also shown in Figure 4B), the automated selected beam configuration results in a lower mandible maximum dose and slightly more conformal dose distribution around PTV (lower normal tissue hot spots) at the cost of larger low dose spillage in normal tissues.

Figure 6:

Dose distribution comparison of the plans generated using the automated beams (left figure) and the planner’s beams (right figure) for patient 2.

Figure 7 reports the timing of the different components of the proposed algorithm. For the influence matrix calculation (Figure 7A), the box plots show the range of the calculation time for the 36 different beams. One can calculate the influence matrix for all the beams in advance or calculate them as they are needed during the optimization, since Bayesian optimization only evaluates some of the beam configurations. The ECHO optimization is currently the most time-consuming part of the algorithm (Figure 7B). A single ECHO optimization for a given beam configuration takes 2–36 minutes (median times for the five patients) and it must be multiplied by the required number of iterations for Bayesian optimization. Figure 7C shows the total time for all 25 iterations of Bayesian optimization and it is obviously very computationally efficient.

Figure 7:

A) The influence matrix calculation time for a single beam (in seconds). The box plot shows the range for the 36 different beams. B) The ECHO optimization time for each beam configuration (2 beams, in minutes). The box plot shows the time range for the different beam configurations tested. C) The total Bayesian optimization time for 25 all iterations (in seconds).

3.2. Two non-coplanar and three coplanar beam configurations

For patient 4, Figures 8A and 8C illustrate the convergence behavior of the algorithm for the 2-non-coplanar and 3-coplanar beams, respectively. In both scenarios, the algorithm reaches the optimal solution after examining less than 1% of all possible configurations which amounts to about 45 iterations. Figures 8B and 8D compare the DVH of the plans generated using 2-coplanar configuration (dashed line), 2-non-coplanar configuration ({210$°$,10$°$; 240$°$,−10$°$}; solid line, 8B), and 3-coplanar configuration ({210$°$; 250$°$; 290$°$}; solid line, 8C). The plans are very comparable and only offer different trade-offs between organs (2-non-coplanar exchanges the parotid mean dose in favor of the mandible max dose and 3-coplanar exchanges the mandible max dose in favor of the PTV max dose). The treatment scores for these plans are also very similar, meaning this patient does not benefit from the non-coplanar beams or extra beam due to its tumor size and location.

Figure 8:

Patient 4: A, C) Progress of the Bayesian optimizer using 2-non-coplanar (A) and 3-coplanar beams (C). B, D) the DVH comparisons of 2-coplanar (dashed line in B and D) against 2-non-coplanar (solid line in B) and 3-coplanar (solid line in D) plans.

For patient 4, the ground-truth optimal configuration was found by optimizing all possible beam configurations in a brute-force fashion. In a real situation though, we do not have the ground-truth and have to stop the algorithm after a certain number of iterations. For patient 3, we do not have the ground-truth and we run the algorithm for 60 iterations. Among the evaluated 2-non-coplanar configurations the best was {40$°$, 10$°$; 300$°$, −10$°$}, and {30$°$; 60$°$; 90$°$} for 3-coplanar. Figures 9A and 9C show the convergence of the algorithm for the 2-non-coplanar and the 3-coplanar beams, respectively. Note that y-axis is the treatment score value and not the optimality gap since it is not available. For both the 2-non-coplanar and the 3-coplanar scenario, the algorithm does not make any progress for many consecutive iterations (Figures 9A and 9C), suggesting the solutions are likely optimal or at least near-optimal. The optimal treatment score for the 2-coplanar configuration is also shown (dashed line, Figures 9A and 9C) for the comparison purposes. Patient 3, as opposed to patient 4, benefits especially from non-coplanar configuration with 6% improvement in the treatment score. Figure 9B also shows that using non-coplanar beams improve both PTV homogeneity and OAR sparing. Less improvement is achieved by adding an extra beam for this patient (Figures 9C and 9D).

Figure 9:

Patient 3: A, C) Progress of the Bayesian optimizer using 2-non-coplanar (A) and 3-coplanar beams (C). B, D) the DVH comparisons of 2-coplanar (dashed line in B and D) against 2-non-coplanar (solid line in B) and 3-coplanar (solid line in D) plans.

4. DISCUSSION

This study presents a proof-of-concept to a solution for a challenging optimization problem of fully automated proton therapy with simultaneous optimization of fluence and beam angle configurations. Breedveld et al.⁹ have previously put forth an interesting greedy-type approach to address this problem for photon therapy. A greedy technique is usually computationally efficient, and it sequentially adds the best beams to the pool of already-selected-beams. However, since the performance of the greedy technique relies on the previously selected beams, it is more appropriate and reliable for photon therapy when more beams are needed. In contrast, the Bayesian optimization is more suitable for proton therapy when few beams are needed, as the computational complexity grows with the number of beams required. This study shows that the automatically selected beams are comparable to or better than the planner’s beam configurations.

The use of Bayesian optimization in this study was inspired by its recent success in parameter tuning for machine learning problems^13,21, where the hyperparameters are analogues to the beams in this study. Bayesian optimization is quite flexible and can be integrated with any fluence optimization technique. For instance, instead of ECHO, one can use a knowledge-based automated treatment planning system^22,23, such as Varian Rapid Plan System²⁴, to optimize a plan for a given beam configuration which would be less computationally expensive. Furthermore, there is a large freedom to choose the objective function (referred to as the treatment score function in this study) in the Bayesian optimization, allowing even non-convex objective functions. While the objective functions used in Bayesian optimization and for fluence optimization need to be related, they do not need to be the same. Therefore, a complicated objective function reflecting broad clinical criteria and preferences (based on DVH indices, TCP, NTCP, or robustness criteria) can be used for Bayesian optimization, even if a simpler convex function is used for fluence optimization.

Bayesian optimization is a non-monotone optimization algorithm meaning that the last objective function value obtained by the algorithm is not necessarily better than the previous one and the so-called best-so-far point needs to be identified after each iteration. As such, the algorithm may progress without any improvement for several iterations but that does not necessarily indicate the achievement of the optimal solution, and therefore it is difficult to establish a convergence criterion. In general, the algorithm is stopped after a pre-defined maximum number of iterations or if no significant improvement is observed in the last few consecutive iterations. In our limited experience, we observed that the algorithm reaches the optimal solution after ~20–50 iterations. Future study is needed to find out if this observation holds true when more beams are used. The implications of early stopping of the algorithm also needs to be investigated.

The Bayesian optimization itself is a fast optimization technique as only 7 s is required to run 25 iterations; hence, the fluence optimization algorithm (ECHO) is the main time-consuming component of the integrated approach (Figure 7). Although ECHO eliminates iterative objective weight tuning, solving constrained optimization problems for every beam configuration is still time-consuming. In our study, each 2-coplanar optimization took between 2 min (smallest tumor, shortest time) and 58 min (largest tumor, longest time). Run time becomes more important if more beams are needed in each configuration. In this study we did not optimize any part of the framework for timing, and there are potential approaches to speed up the algorithm, including: 1) solving ECHO for one beam configuration and then constructing the un-constrained counterpart of the problem using Lagrangian multipliers¹⁰ which can then be used for the optimization of other beam configurations in the following iterations; 2) down-sampling the number of voxels and spots initially for beam angle optimization and solving the full-resolution fluence optimization problem subsequently after identifying the optimal beam configuration; 3) eliminating unnecessary beams upfront based on machine limitations²⁵, couch collisions⁹, or based on the prior clinical experiences; 4) warm-starting Bayesian optimization with some good initial beam angles provided based on prior clinical experiences or based on dosimetric information from beams-eye-views²⁶; or 5) running the Bayesian optimization in parallel where more configurations are optimized simultaneously¹³.

In this study, we used a pencil beam algorithm for dose calculation^19,20; however, for some anatomical sites (e.g., lung²⁷) or due to beam line specifications (e.g., the use of multi-leaf collimators²⁵), Monte Carlo (MC) dose calculation is needed to ensure adequate accuracy. MC dose calculation can easily be used in the proposed framework, as the individual parts of the optimization scheme (dose calculation, fluence optimization, and beam angle optimization) are fully decoupled. While the MC dose engine could potentially increase the overall time, the optimization would still probably remain to be a bottleneck.

The constrained hierarchical optimization approach (ECHO) involves some so-called ‘slip’ parameters that control constraint loosening at different levels of priority, and potentially need to be tuned for each clinical site. However, previous studies on photon optimization demonstrated that these parameters can be easily tuned based on a few training patients and then be used for all the patients^10,12. Using a very similar algorithm, Falkinger et al.²⁹ found that the slip parameters have even less impact for proton therapy due to the higher degree of freedom offered by protons as a result of their limited range in tissues. Here we simply borrowed the slip parameters used in photon planning, leading to good results (Figure 4). We also eliminated the last step of the original photon-based ECHO approach which promotes the photon plan delivery efficiency by smoothing out the fluence map profile¹⁰. Before using this approach in a real clinical setting, an equivalent step has to be added to improve the proton plan delivery efficiency, for example similar to the one suggested by Müller and Wilkens¹¹ which minimizes the total spot weight sum and eliminates the low-weighted spots.

Robust optimization has recently been implemented in ECHO³⁰ and it can be easily integrated with the proposed Bayesian optimization framework. The treatment score in Bayesian optimization then needs to represent all the scenarios and the combination of these should reflect the way the scenarios are combined in the robust optimization approach (e.g., using the $p$-norm formulation as proposed in Ref. 30). Including robustness increases the problem size and the timing scales almost linearly with the number of error scenarios considered. This will further increase the overall time of the framework, but as stated above, there are different ways to tackle this issue.

5. CONCLUSION

A framework for automated treatment planning workflow for proton therapy including beam angle selection using a Bayesian optimization approach combined with the ECHO optimizer was presented. We have shown that Bayesian optimization can automate the beam angle selection and its integration with ECHO is a promising approach for fully automated proton therapy treatment planning.