Development of a Monte Carlo based robustness calculation and evaluation tool

Abstract

Background

Evaluating plan robustness is a key step in radiotherapy.

Purpose

To develop a flexible Monte Carlo (MC)‐based robustness calculation and evaluation tool to assess and quantify dosimetric robustness of intensity‐modulated radiotherapy (IMRT) treatment plans by exploring the impact of systematic and random uncertainties resulting from patient setup, patient anatomy changes, and mechanical limitations of machine components.

Methods

The robustness tool consists of two parts: the first part includes automated MC dose calculation of multiple user‐defined uncertainty scenarios to populate a robustness space. An uncertainty scenario is defined by a certain combination of uncertainties in patient setup, rigid intrafraction motion and in mechanical steering of the following machine components: angles of gantry, collimator, table‐yaw, table‐pitch, table‐roll, translational positions of jaws, multileaf‐collimator (MLC) banks, and single MLC leaves. The Swiss Monte Carlo Plan (SMCP) is integrated in this tool to serve as the backbone for the MC dose calculations incorporating the uncertainties. The calculated dose distributions serve as input for the second part of the tool, handling the quantitative evaluation of the dosimetric impact of the uncertainties. A graphical user interface (GUI) is developed to simultaneously evaluate the uncertainty scenarios according to user‐specified conditions based on dose‐volume histogram (DVH) parameters, fast and exact gamma analysis, and dose differences. Additionally, a robustness index (RI) is introduced with the aim to simultaneously evaluate and condense dosimetric robustness against multiple uncertainties into one number. The RI is defined as the ratio of scenarios passing the conditions on the dose distributions. Weighting of the scenarios in the robustness space is possible to consider their likelihood of occurrence. The robustness tool is applied on IMRT, a volumetric modulated arc therapy (VMAT), a dynamic trajectory radiotherapy (DTRT), and a dynamic mixed beam radiotherapy (DYMBER) plan for a brain case to evaluate the robustness to uncertainties of gantry‐, table‐, collimator angle, MLC, and intrafraction motion. Additionally, the robustness of the IMRT, VMAT, and DTRT plan against patient setup uncertainties are compared. The robustness tool is validated by Delta4 measurements for scenarios including all uncertainty types available.

Results

The robustness tool performs simultaneous calculation of uncertainty scenarios, and the GUI enables their fast evaluation. For all evaluated plans and uncertainties, the planning target volume (PTV) margin prevented major clinical target volume (CTV) coverage deterioration (maximum observed standard deviation of $D98{\%}_{\mathrm{CTV}}$ was 1.3 Gy). OARs close to the PTV experienced larger dosimetric deviations (maximum observed standard deviation of $D2{\%}_{\mathrm{chiasma}}$ was 14.5 Gy). Robustness comparison by RI evaluation against patient setup uncertainties revealed better dosimetric robustness of the VMAT and DTRT plans as compared to the IMRT plan. Delta4 validation measurements agreed with calculations by >96% gamma‐passing rate (3% global/2 mm).

Conclusions

The robustness tool was successfully implemented. Calculation and evaluation of uncertainty scenarios with the robustness tool were demonstrated on a brain case. Effects of patient and machine‐specific uncertainties and the combination thereof on the dose distribution are evaluated in a user‐friendly GUI to quantitatively assess and compare treatment plans and their robustness.

1. INTRODUCTION

A key step in the radiotherapy treatment workflow is the treatment plan evaluation both in terms of dose parameters and in terms of robustness. ¹ In recent years, evaluating robustness to uncertainties on the patient side (e.g., setup uncertainty ² ) or in the mechanical accuracy of the treatment machine (e.g., multileaf‐collimator (MLC) leaf positioning accuracy ³ ) has become an essential part of plan quality assessment. ⁴

To quantify the impact of an uncertainty on the plan robustness, its influence on the dosimetric quality of the plan has to be determined. Due to interpatient anatomical variations and the large variety of treatment techniques available, it is difficult to make a general statement relating a specific type of uncertainty to a dosimetric consequence.

Treatment plan robustness depends on the applied technique. Particularly, the complexity of the techniques impedes the straightforward understanding of the dosimetric impact of a type of uncertainty. Developments in intensity‐modulated treatment techniques for C‐arm treatment units aim to improve plan quality by using increased degrees of freedom (DoF) and consequently increasing the plan complexity for the novel techniques. Volumetric modulated arc therapy (VMAT) increases the DoF by including dynamic gantry rotation as compared to intensity modulated radiotherapy (IMRT), and has become standard of care of radiotherapy. ⁵ , ⁶ , ⁷ Dynamic trajectory radiotherapy (DTRT), which extends VMAT by dynamic table and collimator rotations during delivery, adds two additional DoF, while ensuring acceptable delivery times. ⁸ , ⁹ An additional DoF, and additional complexity, is introduced in mixed beam radiotherapy (MBRT ¹⁰ ) and dynamic MBRT (DYMBER ¹¹ ) by combining photon and electron beams to use the radiation type specific advantages of sharp beam penumbra (photons) and distal dose fall‐off (electrons). Treatment plan robustness therefore depends on the treatment technique. For instance, uncertainties in gantry position have different effects on the dose distribution for the dynamic gantry rotation in VMAT as compared to the static gantry in IMRT, and patient setup uncertainties have different effects on photon treatment compared to electron treatments.

Usually, a differentiation between systematic and random uncertainties is made. ¹² Uncertainties in patient setup have both systematic and random components. On the machine side, systematic uncertainties such as calibration errors, and intrafraction uncertainties of each machine component, influence the delivery accuracy and thus the delivered dose distribution. In recent years, the study of machine logfiles has been a focus topic in research. Logfiles contain time‐resolved information of the machine status during delivery and are used to calculate the fraction‐specific uncertainty of the machine. In routine clinical practice, logfiles are used to monitor the performance of treatment machines, improve efficiency in quality assurance (QA) workflows, ¹³ , ¹⁴ , ¹⁵ , ¹⁶ , ¹⁷ and examine the robustness of treatment plans. ¹⁸ , ¹⁹

With this increasing variety and complexity in treatment techniques, it is not sufficient anymore to assess, for example, MLC positioning uncertainty ²⁰ , ²¹ or uncertainties in patient setup ² alone. These investigated uncertainty scenarios have to be extended to account for the added DoF of new techniques and the uncertainties must be assessed in combination to achieve a more comprehensive robustness assessment of a treatment plan. There is a need for simple and comprehensive calculation and evaluation tools of the dosimetric impact of all patient and machine‐related uncertainties, including their interplay.

Therefore, this work aims to develop a robustness tool to evaluate the dosimetric robustness of treatment plans. The robustness tool will be applicable to a wide range of treatment techniques and used to investigate the impact of patient and machine component‐related uncertainties on the dose distribution, individually and in combination. To ensure the accurate calculation of the dose distributions, including the aforementioned uncertainties, Monte Carlo (MC) dose calculation ²² using the Swiss Monte Carlo Plan (SMCP) ²³ , ²⁴ will serve as the backbone of this tool.

Finally, following the idea of a plan quality index, ²⁵ we introduce a robustness index (RI), which condenses dosimetric robustness of multiple uncertainties into one number. This streamlines the robustness evaluation and robustness comparison of different treatment plans, for the considered uncertainties.

2. MATERIALS AND METHODS

The robustness tool consists of two components: the calculation and the evaluation part. In the calculation part, the dose distribution including the desired uncertainties is calculated. In the evaluation part, the dosimetric impact of the uncertainties on the dose distribution is assessed. Dose calculation uses SMCP on a high‐performance computing cluster for efficient calculation. ²⁴ To evaluate the dosimetric impact of the uncertainties, a GUI is developed. The workflow is illustrated in Figure 1.

FIGURE 1.

Workflow for the robustness tool. Preparation: The treatment plan is created in a treatment planning system (TPS) based on the CT and structure set. To include logfile and time‐resolved information, the created treatment plan must be delivered first and the logfile recorded. Part 1, Calculation: user defines the robustness space in terms of uncertainty scenarios. Subsequently, the respective dose distributions incorporating the desired uncertainties are calculated (see Section 2.4). Part 2, Evaluation: evaluation of the dose distributions of the robustness space (see Section 2.5)

2.1. Terminology

To assess robustness of a treatment plan, a so‐called robustness space is evaluated. The robustness space has the dimensionality N, related to the N uncertainty types considered for the evaluation. The robustness space is spanned up by N uncertainty axes $a_i,i\in \{1\text{…}N\}$, with $z_i$ uncertainty scenarios at $\{n_{a_i}^1,n_{a_i}^2,\text{…},n_{a_i}^{z_i}\}$ along the axis $a_i$. The location of a single scenario in the robustness space is given by $n_{a_1,\text{…},a_N}^{x\in \{1\text{…}{z}_1\},\text{…},x\in \{1\text{…}{z}_N\}}$. Thus, each scenario represents a combination of uncertainties. An uncertainty axis is characterized by the origin of the considered uncertainty: it can be patient (e.g., setup, motion) or machine (e.g., gantry uncertainty) related. The uncertainty axes and scenarios for calculation and evaluation are specified by the user of the robustness tool. A scenario is associated with the dose distribution including the corresponding uncertainties. In the center of the robustness space, the reference scenario represents the nominal plan with the planned dose distribution involving no uncertainty, the reference CT, and the reference structure set. For example, the robustness space considering patient setup uncertainties with respective scenarios in longitudinal, lateral, and vertical patient setup direction and their combination, has scenarios in three dimensions.

The following uncertainty axes are implemented in the tool: patient setup uncertainties (longitudinal, lateral, vertical, table rotation angle, table pitch angle, table roll angle) are modeled by table translations and rotations. Further uncertainty axes consider the mechanical accuracy of machine components (gantry angle, collimator angle, MLC leaf positions, translational jaws position), monitor units, and rigid three DoF intrafraction patient motions in longitudinal, lateral, and vertical direction. Additionally, the tool supports the re‐calculation of the plan on different CTs to account for anatomical changes. If requested, each scenario has its own CT and structure set. Further not yet considered uncertainty axes are easily integrated in the tool.

2.2. Data structure

Scenarios in the robustness space are structured according to two options: in the gridded data (GD) option, where the scenarios are ordered in a regular grid. Thus, the dose distributions of all possible combinations of the considered uncertainty axes are calculated: Together, the N uncertainty axes define a set of $z_1\times z_2\times \text{…}\times z_N$ scenarios in the robustness space. The spotted data (SD) option requires the specification of each individual uncertainty scenario at $n_{a_1,\text{…},a_N}^{x\in \{1\text{…}{z}_1\},\text{…},x\in \{1\text{…}{z}_N\}}$, for calculation of the respective dose distribution. The SD option potentially limits the number of calculations and subsequent evaluations, which becomes particularly useful to probe the robustness space, for example, for the most sensitive uncertainty axes.

2.3. Uncertainty type

The robustness tool enables to calculate and evaluate patient and machine‐related uncertainties. For the purpose of patient‐related setup uncertainties, a Gaussian distribution is used. ¹² Thereby the mean and the standard deviation (sigma) represent the systematic and the random component of the uncertainty. During the Monte Carlo dose calculation, a shift is sampled from the Gaussian (mean, sigma) and applied to the particle exiting the linac head. Thus, for each particle exiting the linac head, such a shift is applied following the Gaussian distribution. As this distribution is applied on a particle‐by‐particle basis, there is no need to simulate several fractions for a specific setup uncertainty. The obtained dose distribution directly accounts for the setup uncertainty distribution. This approach saves substantial computational resources but assumes a sufficient number of fractions (>10), as shown by van Herk et al. ²⁶ In the context of robust optimization, the effect of number of fractions on the simulation of random setup uncertainties and robustness has been investigated previously by Fredriksson. ²⁷

For machine‐related uncertainties, the robustness tool models global systematic uncertainties by constant offsets in the respective machine components. To simulate treatment plan and machine‐specific uncertainty combinations, information of the machine logfile is used. To this end, the plan has to be delivered at least once prior to application to record the logfile, which, for example, could be done during pretreatment quality assurance. In the logfile of a TrueBeam (Varian Medical Systems, Palo Alto, CA, USA), the status of all machine components is logged as expected and actual value during delivery at a rate of 50 Hz with absolute time stamp. The difference between expected and actual for each control point can be assigned as control point‐specific local systematic uncertainty. The robustness tool enables to consider and to scale these uncertainties for the subsequent respective dose calculation.

The time‐resolved machine logfile information can be further used to synchronize a time series of intrafraction patient‐related motion (e.g., breathing) with machine parameters.

2.4. Calculation

The reference treatment plan is generated with an Eclipse research version (Varian Medical Systems, Palo Alto, CA, USA). An input file, defining the treatment machine axis at all control points of the treatment for the nominal scenario, is created. The user specifies the desired robustness space, and the input file is automatically adapted accordingly using a python framework to serve as an input for the subsequent calculation of the dose distributions of the scenarios in the robustness space, using SMCP. ²³ , ²⁴ , ²⁸ Calculation voxel size is $0.25\times 0.25\times 0.25{\mathrm{cm}}^3$. The statistical uncertainty of the Monte Carlo calculated dose distributions presented in this work is <1.2% (one standard deviation). The number of simulated primary particles per calculation is in the order of 10⁸.

2.5. Evaluation

Evaluation of the uncertainty scenarios dose distributions is performed according to DVHs, gamma passing rate and dose differences. To provide a flexible and user‐friendly robustness evaluation of treatment plans, a GUI is developed. The read‐in of the input of the GUI, loading of the robustness space data (dose distributions, CTs, structures), and the calculation of DVHs and gamma passing rate is parallelized on the number of available CPU cores. The computer memory required for the evaluation is roughly proportional to the size of the dose distribution times the number of scenarios. The GUI is implemented in C++ utilizing the Qt toolkit ²⁹ and has a dynamic structure.

The user of the robustness tool interactively defines the evaluation conditions, including thresholds for acceptance to evaluate the robustness of a scenario in the robustness space. Possible conditions are based on $Dx$%, $Vx$%, D _mean or gamma passing rate, evaluated for selected structures. To assess the “distance” of the individual quantity to the specified acceptable threshold, a conditions meter c is introduced as follows:

$$c=\frac{Q_{\mathrm{current}}-Q_{\mathrm{ref}}}{Q_{\mathrm{acceptable}}-Q_{\mathrm{ref}}}$$ (1)

where c is the ordinate for a given quantity Q (e.g., D _mean). Q _current and Q _ref are the parameter values for the current and reference scenario, respectively. Q _acceptable is the threshold for the parameter value used to determine if Q is robust. For $c>1$, the scenario is not considered robust and for $c<1$ it is robust. For $c=0,$ the scenario fulfils the condition equally well as the reference scenario and for $c<0$ the scenario achieves the condition better than the reference scenario.

The calculation of the gamma passing rate is based on the algorithm of Ju et al., ³⁰ which calculates the gamma without using a linear interpolator and discretization of the search space (Supporting Material S1). This has the advantage that the time needed for calculating the gamma passing rate is independent of the distance‐ and dose difference criteria and an exact gamma value is calculated: the common limitation of lacking an infinite resolution grid is overcome by calculating the gamma passing rate values of each reference dose point as the closest geometric distance between this point to the hypersurface defined by the evaluation dose distribution. Additionally, a maximal gamma threshold value, representing the maximal difference in gamma value to the nominal scenario is set, after which the calculation is automatically terminated to save calculation time. The robustness tool employs a global gamma analysis, with the reference scenario set as the reference dose in each evaluation. The dose criterion is given relative to a user‐defined value. The default is the prescribed dose. Additionally, a low‐dose threshold relative to the beforementioned value is used.

Visual evaluation of the difference between dose distributions of any scenarios in the robustness space is conducted in three dimensions (lateral, sagittal, coronal), with an adjustable dose difference threshold.

2.6. Interpolation between scenarios

Evaluation of the treatment plan robustness is not limited to the initially defined uncertainty scenarios. The scenarios populating the robustness space serve as sampling points, between which interpolation is applied. To this end, a metric defining the distance d between two scenarios at b and c in the robustness space is specified as

$$d^2=\sum _{l=1}^N{\left(\frac{b_l-c_l}{m_l}\right)}^2$$ (2)

with the uncertainty normalization $M=(m_1,m_2,\text{…},m_N)$. The uncertainty normalization is defined by default as

$$m_l=\frac{n_{a_l}^{z_l}-n_{a_l}^1}{z_l-1}$$ (3)

but is adjustable. The resulting distance is unitless. The evaluation quantity Q can then be interpolated based on the nearest neighbors.

2.7. Robustness index

To summarize the information of treatment plan robustness considering multiple uncertainties, the robustness index (RI) is introduced. When all scenarios in the robustness space are considered equally important, the RI describes the fraction of scenarios passing the conditions: $\mathrm{RI}=\frac{N_{\mathrm{p}}}{N_{\mathrm{all}}}$, where N _p is the number of scenarios passing all robustness conditions and N _all is the number of all scenarios in the present robustness space.

However, a greater distance to the location of the reference scenario usually represents a decrease in the likelihood of occurrence of such a scenario. Therefore, a second option to calculate RI is introduced, given by $\mathrm{RI}=\frac{{\sum }_{t=1}^{N_{\mathrm{p}}}{w}_t}{{\sum }_{t=1}^{N_{\mathrm{all}}}{w}_t}$ with $w_t=e^{\frac{-d_t^2}{2R_{\mathrm{E}}^2}}$. $d_t$ is the distance between the location of scenario t and the reference scenario in the robustness space and R _E is the evaluation range, the standard deviation of this scenario distribution. The maximal value of the RI is 100% (in the weighted and unweighted case) and states that all scenarios pass the conditions. In the unweighted case, an RI of 10% means that 10% of the scenarios in the robustness space pass the conditions. In the weighted case, an RI of 10% describes that the fraction of the summed weights of the passing scenarios is 10%. Thereby the scenarios are weighted according to a Gaussian distribution $G(\mu =0,\sigma =R_{\mathrm{E}}$) centered around the reference scenario (Supporting Material S2). The mean and standard deviation of a quantity Q evaluated on the whole robustness space change to a weighted $\mathrm{mean},{\mu }_Q=\frac{{\sum }_t{w}_t{Q}_t}{{\sum }_t{w}_t}$ and weighted $standarddeviation,{\mathrm{\Sigma }}_Q$ with ${\mathrm{\Sigma }}_Q^2=\frac{{\sum }_t{w}_t{Q}_t^2{\sum }_t{w}_t-{({\sum }_t{w}_t{Q}_t)}^2}{{({\sum }_t{w}_t)}^2-{\sum }_t{w}_t^2}$, where $w_t$ is the Gaussian weighted distance factor from the reference as defined above and $Q_t$ is the value of the quantity Q at scenario t.

2.8. Application and demonstration of the robustness tool

Application of the robustness tool and the GUI is demonstrated on different treatment plans for a right‐sided brain case (Figure 2), namely, an IMRT, a VMAT, a DTRT, and a DYMBER plan (Table 1). Prescribed dose is 60 Gy to 50% of the planning target volume (PTV). The clinical target volume CTV–PTV margin is 0.5 cm. The respective plans are optimized according to clinical goals applied in our institute. Application of the main functionalities of the GUI is demonstrated on the DTRT plan. To conduct the robustness comparison against uncertainties in patient setup of different treatment techniques, applications 1–6 (Table 1) are assessed. To demonstrate the flexibility of the robustness tool in terms of different treatment techniques and uncertainty scenarios, applications 7–12 are investigated.

FIGURE 2.

Right‐sided glioblastoma brain case. The PTV is shown in red, the OARs in color

TABLE 1.

Application of the robustness tool, demonstrated for different treatment techniques and uncertainties

#Application	Treatment technique	Description of uncertainty axes and scenarios	#Scenarios
1	IMRT (3 fields)	Systematic patient setup uncertainty: longitudinal, lateral, and vertical directions modeled with Gaussian distribution $G(\mu ,0),\{\mu =-0.5,-0.2,0.0,0.2,0.5\text{cm}\}$ ∀ axes, and combinations of them	125
2	IMRT (3 fields)	Random patient setup uncertainty: longitudinal, lateral, and vertical directions modeled with Gaussian distribution $G(0,\sigma ),\{\sigma =0.0,0.1,0.2,0.3,0.5\mathrm{cm}\}$ ∀ axes, and combinations of them	125
3	VMAT (2 arcs)	Systematic patient setup uncertainty: longitudinal, lateral, and vertical directions modeled with Gaussian distribution $G(\mu ,0),\{\mu =-0.5,-0.2,0.0,0.2,0.5\text{cm}\}$ ∀ axes, and combinations of them	125
4	VMAT (2 arcs)	Random patient setup uncertainty: longitudinal, lateral, and vertical directions modeled with Gaussian distribution $G(0,\sigma ),\{\sigma =0.0,0.1,0.2,0.3,0.5\mathrm{cm}\}$ ∀ axes, and combinations of them	125
5	DTRT (3 trajectories)	Systematic patient setup uncertainty: longitudinal, lateral, and vertical directions modeled with Gaussian distribution $G(\mu ,0),\{\mu =-0.5,-0.2,0.0,0.2,0.5\text{cm}\}$ ∀ axes, and combinations of them	125
6	DTRT (3 trajectories)	Random patient setup uncertainty: longitudinal, lateral, and vertical directions modeled with Gaussian distribution $G(0,\sigma ),\{\sigma =0.0,0.1,0.2,0.3,0.5\mathrm{cm}\}$ ∀ axes, and combinations of them	125
7	IMRT (3 fields)	Global systematic uncertainty counter‐moving jaws X1 and X2 and Y1 and Y2, $\{-1,-0.5,0.0,0.5,1.0\mathrm{cm}\}$; Global systematic uncertainty leaf bank A and B, $\{-0.1,-0.05,0.0,0.05,0.1\mathrm{cm}\}$; and combinations of them	625
8	DTRT (3 trajectories)	Global systematic uncertainty in gantry‐, table‐, collimator angle $\{-4.0^{\circ },-2.0^{\circ },-1.0^{\circ },0.0^{\circ },1.0^{\circ },2.0^{\circ },4.0^{\circ }\},$ and combinations of them	343
9	DYMBER (3 photon trajectories and 1 electron field [6, 9, 12, 15, 18, 22 MeV])	Systematic patient setup uncertainty for electron fields: longitudinal, lateral, and vertical directions modeled with Gaussian distribution $G(\mu ,0),\{\mu =-0.5,0.0,0.5\text{cm}\}$ ∀ axes, and combinations of them	27
10	VMAT (2 arcs)	Rigid intrafraction motion in transversal and sagittal planes, Rotation amplitude of isocenter around dens axis in transversal and sagittal planes, $\{0.0^{\circ },0.5^{\circ },1.0^{\circ },2.0^{\circ },3.0^{\circ }\}$, and combinations of them	25
11	VMAT (2 arcs)	Global systematic collimator and table angle uncertainty, $\{-4.0^{\circ },-2.0^{\circ },-1.0^{\circ },0.0^{\circ },1.0^{\circ },2.0^{\circ },4.0^{\circ }\};$ Local systematic gantry uncertainty: mean absolute gantry logfile difference between expected and actual scaled to $\{0.0^{\circ },1.0^{\circ },2.0^{\circ },3.0^{\circ },4.0^{\circ }\}$, and combinations of them	245
12	VMAT (2 arcs)	Global systematic uncertainty in table angle and gantry angle, $\{-1.0^{\circ },0.0^{\circ },1.0^{\circ }\}$; Local systematic gantry uncertainty: mean absolute gantry logfile difference between expected and actual scaled to $\{0.0^{\circ },1.0^{\circ }\}$; Systematic and random lateral patient setup uncertainty: modeled with Gaussian distribution $G(\mu ,\sigma ),\{\mu =-1.0,0.0,1.0\text{cm;}\sigma =0.0,0.5\text{cm}\}$ and combinations of them	108

Note: Applications 1–6 evaluate systematic and random patient setup uncertainty, and applications 7–12 evaluate different robustness spaces for different treatment techniques.

The robustness conditions are set to: D98% and D2% of the CTV are not allowed to be reduced or increased by more than 1 Gy, respectively. Additionally, D2% and mean dose are not allowed to be increased by more than 1 Gy for serial and parallel OARs, respectively. The plan robustness is then quantified by the RI: once for all scenarios weighted equally and once with $R_{\mathrm{E}}=1.5$ to increase the weight of the more likely scenarios.

2.9. Validation of the robustness tool

Extensive validation of the SMCP dose calculation framework has been conducted in the past. ⁹ , ¹⁰ , ¹¹ , ²⁴ , ²⁸ , ³¹ This included validation of photon‐based treatment techniques such as IMRT, VMAT, DTRT, as well as mixed beam treatment techniques, such as DYMBER. Measurements are conducted with the Delta4 device (ScandiDos, Uppsala, Sweden). First, 10 × 10 cm² fields and 5 × 5 cm² fields at different gantry angles (0°, 90°, 180°) are delivered and the measured dose is compared to the calculation. Second, to validate the accuracy of the dose calculation of selected scenarios (Table 2) in the robustness space of the VMAT plan for the brain case (Figure 2), the scenarios are calculated with the robustness tool and the corresponding XML files needed for delivery on a TrueBeam (Varian Medical Systems, Palo Alto, CA, USA) in developer mode are created, the plans are delivered, and the delivered dose is measured and compared against the calculation.

TABLE 2.

Validation scenarios

Validation scenario	Description
1	10 × 10 cm2 field at gantry angles 0°, 90°, 180°
2	5 × 5 cm2 field at gantry angles 0°, 90°, 180°
3	Reference scenario (no uncertainty)
4	Systematic patient setup uncertainty of +1 cm in lateral, longitudinal, and vertical directions
5	Systematic uncertainty of 1° in gantry
6	Systematic uncertainty of 1° in gantry and collimator
7	Systematic uncertainty of 1° in gantry, collimator, and table rotation
8	Systematic uncertainty of 1 mm for MLC leaf 40 in leafbank B
9	Systematic uncertainty of 3 cm in Jaw X1
10	Intrafraction motion of 1 cm in longitudinal direction during treatment
11	Systematic 10% more MU
12	Gantry logfile uncertainty scaled up by factor 10
13	Systematic gantry uncertainty of 1° combined with systematic patient setup uncertainty of +1 cm in lateral, longitudinal, and vertical directions

Note: Validation scenarios 1–2: validation of 10 × 10 and 5 × 5 cm² open fields at different gantry angles. Validation scenarios 3–13: specific uncertainty scenarios for validation of the robustness space of the VMAT plan.

To validate the calculation of quantities Q, for example, DVH parameters such as volume, $Dx$% and D _mean, the robustness tool calculations are compared to the results of the corresponding Eclipse implementation. The dose distribution of the reference VMAT plan is loaded into Eclipse, and the DVH parameters for several organs are compared to the evaluation returned by the robustness tool. The gamma passing rate calculations of the robustness tool are validated against the open‐source gamma passing rate calculation of pymedphys 0.37.1, an implementation based on the work of Wendling et al. ³² : gamma passing rates of the reference scenario of the VMAT plan and the scenarios in the robustness space incorporating systematic setup uncertainties in lateral, longitudinal, and vertical directions between −0.5 and 0.5 cm are calculated and compared.

3. RESULTS

3.1. GUI for evaluation

Robustness evaluation with the GUI is shown in Figure 3 for the DTRT plan Table 1, application 8. The DVH viewer (Figure 3, number 1) represents the DVH distributions of the selected structures over all scenarios in the robustness space as DVH bands. The boundaries of the DVH bands correspond to the scenarios, which induce the maximal deviations in terms of dose distribution for this structure. The reference DVH is indicated by a solid line and the DVH of the current scenario is shown with a dashed line. In the axes selection window (Figure 3, number 3), the user switches the uncertainty axes of the robustness space to select a two‐dimensional plane in the robustness space. The selected plane is displayed in the robustness map (Figure 3, number 2) in real time. Additionally, the evaluation range is superimposed on the robustness map: in Figure 3, an R _E of 1.5 and standard metric is chosen to emphasize the smaller uncertainties. The evaluation range is visualized by the shaded area. The conditions meter for the selected conditions set in the conditions list is shown on the bottom right of Figure 3.

FIGURE 3.

Main GUI of the robustness evaluation tool. Top left, 1: DVH viewer with DVH bands of all scenarios of the robustness space of the DTRT plan (Table 2, application 8). Top right, 2: Robustness map displaying 2D plane of the robustness space and pop‐up metric window to change the metric if needed, Bottom left, 3: Axes selection window to select a plane in the robustness space for closer inspection. Bottom left, 4: Specification of parameters gamma passing rate calculation. Bottom right, 5: Conditions meter for evaluating structure‐specific conditions; 6 and 7: Open pop‐up windows: conditions list (definition of robustness conditions), dose window (shows dose distribution and dose difference), and statistics window (summarizes key quantities of the robustness space)

3.1.1. Conditions list

The conditions from Section 2.8 are specified in the conditions list (Figure 3, number 5 opens the conditions list seen in Figure 4) and are adjustable on the fly according to the interest of the user. The following structures are part of the evaluation: brain, brainstem, left and right eye (Eye_l./r.), chiasma, left and right lens (Lense_l./r.), left and right optic nerve (N.opticus_l./r.), left and right lacrimal glands (Lacrimal_l./r.), CTV, and PTV.

FIGURE 4.

Conditions list opened by Figure 3, number 6. 1: Select structure, 2: Select parameter, 3: Add to evaluation

3.1.2. Conditions meter

The conditions meter in Figure 3, number 5, displays how well the current scenario is fulfilling the user‐defined conditions. If all conditions are passed, the respective rectangle in the robustness map (Figure 3, number 2) is marked green, and red, if one condition is not fulfilled. To evaluate changes in the distance or dose difference criteria in the calculation of gamma passing rate, the gamma passing rate calculation needs to be restarted by ticking the red circle in the main GUI window (Figure 3, number 4).

3.1.3. Dose window

The dose window (Figure 5) displays the dose distributions of the selected scenario (systematic uncertainty of 1° in table and gantry angle: Figure 5, number 1), the reference scenario (Figure 5, number 2), and the dose difference (Figure 5, number 3) superimposed on the respective CT. The dose difference is always superimposed on the CT of the reference scenario and the structures of the underlying anatomy are visualized by their contours. The user explores the dose distributions in the transversal, coronal, and sagittal planes by scrolling through them, and switches from one plane to the other with the help of the crosshairs. Additionally, an adjustable dose threshold is implemented to visualize dose differences in a desired range.

FIGURE 5.

The dose window (opened by Figure 3, number 6) displays dose distributions superimposed on the CT. Structures are indicated by the fine lines (PTV and CTV in red and orange here). 1: Dose of current scenario. 2: Reference dose. 3: Dose difference between reference and current doses, includes a user‐defined threshold to visualize relevant dose ranges. 4: View control to switch through transversal, coronal, and sagittal planes. The red cross serves as a guideline when switching planes from transversal (shown here) to coronal or sagittal

3.1.4. Statistics window

The statistics window, displayed in Figure 6, summarizes dosimetric key quantities for all OARs. Additionally, the RI for the selected evaluation range is displayed at the bottom.

FIGURE 6.

Statistics window (opened by Figure 3, number 7) displays key quantities and robustness index (RI) for all selected structures in the current evaluation range

3.1.5. Interpolation

A robustness map of SD and corresponding GD is shown in Figure 7. Here only some scenarios are available (shown as white dots). Green and red areas indicate whether a scenario is considered robust and acceptable or not, respectively. Between the available scenarios, the robustness map displays gray rectangles (Figure 7, number 1). So far, no information on the robustness of these areas in the robustness space is available. These empty spaces are filled by interpolation. Besides interpolation, the zoom functionality is implemented in the GUI. It inspects a single two‐dimensional slice through the robustness space in greater detail. The zoom function increases the number of scenarios along the uncertainty axes of this slice by a factor of two, thus leading to additional combinations of these uncertainty axes in the selected plane and performs an interpolation on all inserted scenarios. Interpolation and zooming are implemented on a multithreaded basis.

FIGURE 7.

1: Robustness map of robustness space with SD, with scenarios at every other combination (white dots). 2: Robustness map of robustness space with GD. 3: Zoom functionality: doubles resolution of scenarios in this slice and interpolates on all inserted scenarios

3.2. Robustness evaluation of different treatment plans

The results of the applications 1–12 from Table 1 are shown in Table 3. The default uncertainty normalization is chosen. Furthermore, the standard deviation Σ of the $D98{\%}_{\mathrm{CTV}}$ and $D2{\%}_{\mathrm{CTV}}$ as well as the structure with the greatest standard deviation in mean dose and in D2% are displayed. RI is higher for a weighted analysis ($R_{\mathrm{E}}=1.5$), compared to equal weighting of the scenarios. Due to the Gaussian weighted evaluation method, RI increases, as scenarios located closer to the reference have more weight compared to scenarios located further away from the reference scenario and they usually pass the conditions better.

TABLE 3.

Evaluation of applications 1–12

#Application description	RI (all scenarios equally weighted)	RI $({\mathbf{R}}_{\mathbf{E}}\mathbf{1}.\mathbf{5})$	$\mathbf{\Sigma }$ of $\mathit{D}98{\%}_{\mathbf{CTV}},$ $\mathbf{\Sigma }$ [Gy]	$\mathbf{\Sigma }$ of $\mathit{D}2{\%}_{\mathbf{CTV}},$ $\mathbf{\Sigma }$ [Gy]	Structure with greatest $\mathbf{\Sigma }$ for mean dose, $\mathbf{\Sigma }$ [Gy]	Structure with greatest $\mathbf{\Sigma }$ for D 2%, $\mathbf{\Sigma }$ Gy]
1. IMRT setup (systematic)	0.8%	2.2%	1.3	0.5	Optic nerve r.: 8.8	Optic nerve r.: 15.1
2. IMRT setup (random)	9.6%	31.3%	0.1	0.1	Chiasma: 1.8	Eye_r.: 2.4
3. VMAT setup (systematic)	8.0%	12.5%	0.5	0.3	Optic nerve r.: 7.3	Optic nerve r.: 14.4
4. VMAT setup (random)	14.4%	49.5%	0.1	0.2	Chiasma: 2.2	Chiasma: 1.8
5. DTRT setup (systematic)	16.8%	23.2%	0.5	0.4	Optic nerve r.: 6.2	Chiasma: 14.5
6. DTRT setup (random)	5.6%	23.4%	0.1	0.1	Optic nerve r.: 0.8	Chiasma: 1.5
7. IMRT MLC/Jaws	21.0%	29.9%	1.7	1.7	PTV: 1.9	Lacrimal gland r.: 1.9
8. DTRT gantry, table, collimator	12.0%	33.5%	0.4	0.3	Lacrimal gland r.: 0.9	Optic nerve r.: 1.5
9. DYMBER electron setup	25.9%	31.2%	0.9	0.7	Lacrimal gland r.: 0.4	PTV: 0.7
10. VMAT rigid intrafraction motion	16.0%	43.2%	0.2	0.8	Chiasma: 0.6	Optic nerve r.: 1.2
11. VMAT, logfile	5.1%	15.3%	0.2	0.2	Chiasma: 1.6	Lacrimal gland l.: 3.5
12. VMAT, systematic and random patient and machine uncertainties	13.9%	16.9	0.1	0.2	Optic nerve r.: 1.6	Chiasma: 3.6

Note: Robustness index (RI) for all scenarios weighted equally and $R_{\mathrm{E}}=1.5$. Standard deviations for $D98{\%}_{\mathrm{CTV}}$ and $D2{\%}_{\mathrm{CTV}}$ and structure with greatest standard deviation in mean dose and D2%. The calculation of standard deviations is based on the default metric and $R_{\mathrm{E}}=1.5$.

Evaluating applications 1–6 compares the robustness of different treatment techniques to patient setup uncertainties. The RI indicates better dosimeteric robustness to setup uncertainties for the VMAT and the DTRT plans as compared to the IMRT plan. Assessing RI and the standard deviation of $D98{\%}_{\mathrm{CTV}}$ and $D2{\%}_{\mathrm{CTV}}$ show that the CTV–PTV margin compensates for the uncertainties to maintain CTV coverage, but some of the OARs are strongly affected by setup uncertainties, potentially leading to reject the plan completely when clinical constraints are no longer met.

In Figure 8, the DVH bands of the robustness space including random setup uncertainties of the IMRT, VMAT, and DTRT plans are shown. Especially the OARs near the PTV experience a great variation in their dose, as seen by the width of the DVH bands compared to the ones of PTV and CTV.

FIGURE 8.

DVH bands for the IMRT, VMAT, and DTRT plans, including random uncertainties $G(0,{\sigma }_{x,y,z}=0,0.1,0.2,0.3,0.5$ cm), in patient setup

3.3. Validation

In Table 4, the dose measurements of the selected scenarios (Table 2) are compared against the calculation. For the gamma analysis, dose difference/distance criteria of 2% (global)/1 mm for validation scenarios 1–2 and 3% (global)/2 mm for validation scenarios 3–13, including a 20% low‐dose threshold, are applied in the evaluation for all measurements. All measurements agree with the calculation of the robustness tool with a gamma passing rate of >96.4%.

TABLE 4.

Gamma passing rate evaluation of scenarios specified in Table 2

Validation scenario	Gamma passing rate (2% global/1 mm)
1	Average 98.5%
2	Average 99.0%
	Gamma passing rate (3% global/2 mm), Arc 1	Gamma passing rate (3% global/2 mm), Arc 2
3	97.8%	99.6%
4	98.7%	97.8%
5	97.0%	99.8%
6	96.4%	99.6%
7	97.0%	99.6%
8	97.2%	99.8%
9	98.7%	99.6%
10	99.8%	100%
11	97.4%	99.4%
12	99.0%	99.4%
13	99.4%	100%

Note: 2% (global)/1 mm dose difference and distance criteria, including a 20% dose threshold was applied to evaluate scenarios 1–2. The clinical dose difference and distance criteria, 3% (global)/2 mm, including a 20% dose threshold was applied for the evaluation of scenarios 3–13.

In Table 5, the results of the Eclipse implementation to calculate volume, Dx% and D _mean agree with the calculation of the robustness tool (e.g., PTV volume differentiates by 0.2%). For small structures greater differences (e.g., chiasma volume) occur. To validate the accuracy of the gamma passing rate calculation, a total of 64 scenarios are evaluated (2% of prescribed dose/2 mm, 20% threshold). The results cover a range of 91%–100% passing rate. The robustness tool and pymedphys deviate by a maximum of 0.8%. However, pymedphys needs substantially longer calculation time for the same number of CPU cores (on average: 5 min as compared to 2 s for the robustness tool).

TABLE 5.

Comparison of the DVH parameter evaluation: robustness tool (bold, left) and Eclipse (right)

	Planning target volume		Clinical target volume		Brain		Brainstem		Chiasma		Body
Volume [cm3]	272.9	272.4	166.4	165.5	1479.5	1479.1	39.8	39.3	0.5	0.1	7599.6	7508.2
D2% [Gy]	63.3	62.5	62.2	62.1	61.8	61.3	28.7	27.7	27.3	24.1	59.8	59.9
D98% [Gy]	55.4	56.6	57.4	58.2	0.0	1.3	0.0	0.4	13.7	14.6	0.0	0.1
D mean [Gy]	59.5	59.9	60.5	60.1	25.7	26.2	7.4	6.4	18.1	16.8	7.6	8.0

4. DISCUSSION

In this work, a novel robustness tool is developed and implemented to evaluate the robustness of treatment plans of different techniques to patient and machine‐related uncertainties. In contrast to usually applied robustness evaluation procedures (e.g., evaluation limited to consider only setup uncertainties ² ), this tool has a great flexibility in terms of accurately calculating and evaluating different uncertainty scenarios and types (individually and in combination), in terms of simultaneously evaluating multiple uncertainty scenarios according to various criteria for the target and the OARs, and in terms of its applicability to different treatment techniques.

Accurate dose calculation of uncertainty scenarios is ensured by employing validated MC dose calculation with SMCP. It is necessary to simulate a sufficient number of primary particles for each MC dose calculation to achieve a reasonable statistical uncertainty ³³ and hence reliable robustness results within the statistical uncertainty of the Monte Carlo calculated dose distributions. As MC dose calculations are usually computationally expensive, the tool is constructed in a modular way, and the dose calculation algorithm is interchangeable.

The robustness tool provides high flexibility to the user to enter specific scenarios for the robustness space. Owing to this flexibility in the calculation and evaluation of the robustness space, the developed robustness tool efficiently overcomes the restriction of standardized uncertainty scenarios: for example, limited robustness spaces, such as only considering MLC ³⁴ or only patient setup ² , ³⁵ uncertainties. The user of the robustness tool defines the robustness space incorporating a variety of uncertainty types and their combinations. With the help of this tool, the dosimetric impact of systematic and random uncertainties in patient setup and uncertainties in the mechanical accuracy of machine components on the reference scenario are quantified. The robustness tool offers the possibility to simulate systematic and random setup uncertainties of Gaussian distributions. This reflects that, for example, the user can determine up to what level of uncertainty the given treatment plan is robust for the conditions considered, which then trigger, for example, an appropriate setup strategy. Additionally, it incorporates machine logfile information for realistic fraction‐specific modeling of the uncertainties in each machine component, which has been determined to be a useful tool to assess machine‐specific delivery uncertainties. ¹⁹ , ³⁶ , ³⁷ , ³⁸ The robustness tool therefore permits comprehensive robustness evaluation adaptable to different use cases.

The flexibility to evaluate different treatment techniques, as for example described by Quian et al., ³⁶ is fundamental to the design of the robustness tool. Owing to the availability of validated dose calculation for various treatment techniques, robustness comparison of different treatment techniques, including different radiation sources, is possible with the robustness tool. This has the potential to redefine and restructure the treatment planning process to already include robustness evaluation at the stage of appropriate treatment technique and treatment plan selection.

The robustness tool assesses robustness according to multiple criteria. The dosimetric impact of uncertainties on target and OARs is investigated visually and quantitatively on a scenario‐by‐scenario base as well as in summary. The DVH viewer with the DVH bands including all investigated scenarios, the robustness maps and the RI contextualize the detailed evaluation according to dose‐volume parameters, dose difference, and gamma passing rate of the individual scenarios. The gamma passing rate calculation is successfully validated against pymedphys and permits efficient assessment of the dosimetric impact of an uncertainty type. Differences in DVH parameter calculation, especially for small structures, arise due to the different volume calculations and the dose value: the triangular meshes are not constructed in the exact same way in the robustness tool and in Eclipse. Furthermore, Eclipse interpolates the dose between the voxels for DVH calculation and the robustness tool does not. In literature more complex interpolation methods, for example, the polynomial chaos expansion ³⁹ or (quasi‐) Monte Carlo methods, ⁴⁰ as compared to our interpolation method are described, usually operating on the level of dose distributions. However, these methods assume that the probability distribution function of the uncertainties is known beforehand, and that the uncertainties are independent of each other. In general, the independence of the uncertainties cannot be guaranteed and there is limited knowledge, especially about the cumulative probability distribution of several uncertainties. The presented applications of the robustness tool confirm that restricting robustness investigations on the target volume ¹² gives a misleading picture of the plan robustness, and simultaneous evaluation of dosimetric robustness on OARs is needed for a comprehensive robustness analysis of a treatment plan.

The application of the robustness tool fits into existing evaluation strategies: the flexibility in incorporating numerous uncertainty scenarios in the dose calculation and their evaluation makes it compatible with existing robustness approaches such as the “good practice scenario selection” and the “statistically sound scenario selection” evaluated by Sterpin et al. ⁴¹ However, the tool extends these strategies by considering new potential scenarios, various uncertainties, different treatment techniques, and flexible evaluation including the RI. The RI facilitates the robustness evaluation, summarizes, and efficiently compares the robustness of competing treatment plans and techniques for a given robustness space. Robustness evaluation is directly depending on the selected robustness space along with the conditions as robustness acceptance criteria. The RI is consequently also depending on those settings. Robustness evaluation by means of RI should always be reported with information about robustness space and the robustness conditions. Additionally, if the user of the robustness tool has knowledge about the probability distribution of the scenarios, a confidence interval for the RI can be reported.

We understand that the full potential of the RI only unfolds when standardized robustness spaces in terms of uncertainty axes, scenarios, and evaluation range, as well as relevant evaluation conditions such as D98%, D2%, or D _mean for the structures are defined. This standardizes facilitates and streamlines the comparison of RI of different robustness studies in various radiotherapy centers. Furthermore, the RI has the potential to serve as a threshold action level for replanning, discarding the plan, or recommending specific QA tests.

The robustness tool is expected to play a key role in our group in the field of robust optimization, the margin concept, and the implementation of new treatment techniques. With the help of this tool, we aim to gain an understanding of meaningful robust plan optimization by investigating treatment type‐specific correlations between different uncertainties and by studying the sensitivity of a plan to different uncertainty types. Additionally, with the tool there is the potential to investigate the margin concept of target volumes and OARs toward flexible margins. ⁴² Finally, the robustness of new treatment techniques is explorable in order to facilitate their safe clinical implementation.

5. CONCLUSION

In conclusion, by combining dose calculation of treatment plans using different treatment techniques, including patient and machine‐related uncertainties, and flexible evaluation according to user‐defined criteria, this robustness tool provides accurate comprehensive robustness evaluation, and fills the need for an overarching robustness evaluation tool to determine plan quality.

CONFLICT OF INTEREST

The authors have no relevant conflicts of interest to disclose.

Supporting information

Supporting Information

Loebner HA, Volken W, Mueller S, et al. Development of a Monte Carlo based robustness calculation and evaluation tool. Med Phys. 2022;49:4780–4793. 10.1002/mp.15683