Benchmarking a commercial proton therapy solution: The Paul Scherrer Institut experience

Abstract

Objective:

For the past 20 years, Paul Scherrer Institut (PSI) has treated more than 1500 patients with deep-seated tumors using PSI-Plan, an in-house developed treatment planning system (TPS) used for proton beam scanning proton therapy, in combination with its home-built gantries. The goal of the present work is to benchmark the performance of a new TPS/Gantry system for proton therapy centers which have established already a baseline standard of care.

Methods and materials:

A total of 31 cases (=52 plans) distributed around 7 anatomical sites and 12 indications were randomly selected and re-planned using Eclipse™. The resulting plans were compared with plans formerly optimized in PSI-Plan, in terms of target coverage, plan quality, organ-at-risk (OAR) sparing and number of delivered pencil beams.

Results:

Our results show an improvement on target coverage and homogeneity when using Eclipse™ while PSI-Plan showed superior plan conformity. As for OAR sparing, both TPS achieved the clinical constraints. The number of pencil beams required per plan was on average 3.4 times higher for PSI-Plan.

Conclusion:

Both systems showed a good capacity to produce satisfactory plans, with Eclipse™ being able to achieve better target coverage and plan homogeneity without compromising OARs.

Advances in knowledge:

A benchmark between a clinically tested and validated system with a commercial solution is of interest for emerging proton therapy, equipped with commercial systems and no previous experience with proton beam scanning.

Introduction

Proton therapy is a radiation treatment modality that uses protons for cancer treatment as opposed to high-energy X-rays used in conventional radiation therapy. Due to the dose deposition pattern of protons, characterized by low entrance dose, followed by a peak (i.e. Bragg peak) and a sharp falloff, it is possible to achieve highly conformal dose distributions, thus improving organ-at-risk (OAR) and normal tissue sparing.¹

In 1996, Paul Scherrer Institut (PSI) inaugurated world’s first proton gantry for treatment of deep-seated tumors, using the proton beam scanning (PBS). With this technique, pencil beams are modulated in energy and magnetically steered within the patient, thus covering the tumor volume in three dimensions.^1–3 Our current clinical practice is a result of two decades of both PBS expertise and continuous improvement and our patient cohort currently represents the most mature set of clinical results for PBS, with published outcomes consisting of large patient numbers with substantial follow-ups.^1,4–8

Last year, PSI introduced into clinics a commercial system, consisting of a proton therapy gantry and the associated treatment planning system (TPS). As a part of commissioning, we have performed a comparative study using a representative set of clinical cases in order to ensure the baseline standard of care as we know from our in-house solutions, with which over 1500 patients were successfully treated. With the increasing number of proton therapy facilities around the world, most equipped with commercial solutions and with no previous experience in the field, we believe results of the present study are of interest for broader audience.

Methods and materials

Patient selection

A total of 31 clinical cases (n = 52 plans) were randomly selected from PSI patient database treated using our in-house solutions (gantries 1 and 2 together with PSI-Plan) between January 2015 andDecember 2017. Although a higher proportion of cases for which our institution was experienced with PBS was included in our sample, the complexity of the cases was not taken into consideration for the selection. The distribution of anatomical sites and indications is shown in Table 1.

Table 1.

Patient sample and characterization (31 clinical cases)

Anatomical site	# of cases
Brain	7 (23%)
Spine	6 (19%)
Head and neck	4 (13%)
Eye	4 (13%)
Craniospinal axis	3 (10%)
Nasal cavity	2 (6%)
Others	5 (16%)
TOTAL	31
Indication	# of cases
Chordoma/Chondrosarcoma	7 (23%)
Astrocytoma	4 (13%)
Meningioma	4 (13%)
Rhabdomyosarcoma	3 (10%)
Sarcoma	3 (10%)
Carcinoma	3 (10%)
Medulloblastoma	2 (6%)
Craniopharyngioma	1 (3%)
Melanoma	1 (3%)
Ependymoma	1 (3%)
Neuroblastoma	1 (3%)
Adenoma	1 (3%)
TOTAL	31

The proportion of selected indications was representative of our institution’s experience with PBS, i.e., the most frequent indications were chordomas and chondrosarcomas of the skull base and spine (cervical, thoracic and sacrum)

From the selection of 31 clinical cases resulted 52 plans, i.e. only 13 (24%) of the cases were prescribed with a single plan for the entire course of treatment. Of the total 52 plans, 14 (27%) were Single Field Uniform Dose (SFUD) plans and 38 (73%) were Intensity Modulated Proton Therapy (IMPT) plans. For SFUD plans, each field is independently optimized resulting in a uniform dose distribution per field to the target, whereas in IMPT the final dose distribution is a combination of several fields which are simultaneously optimized resulting in an inhomogeneous dose per field.^9–11

TPS configuration and calculation algorithms

The original CT’s, structure sets and dose distributions of all 31 cases were exported from the in-house system in DICOM format. Subsequently, these were imported into the new commercial TPS (Eclipse™, from Varian Medical Systems, Palo Alto, CA), where they were re-planned. The major technical differences between the two TPS are summarized in Table 2.

Table 2.

Specifications for combination G1/G2-PSI-Plan and ProBeam^®-Eclipse™

Systems parameters	In-house			Commercial
Version	PSI-Plan (G1): v. 7.7.0 PSI-Plan (G2): v. 2.10.1			Eclipse™ (ProBeam®): v. 13.7.15
Dose calculation algorithm	Raycasting5			PCS (v. 13.6.15)6
Dose calculation grid	G1: 0.5 × 0.5×0.4 cm G2: 0.25 × 0.4×0.4 cm			0.2 × 0.2×0.2 cm
Dose optimization algorithm	Least-squares optimization7			NUPO (v. 13.6.15)6
Layer spacing	Fixed G1: 0.46 cm G2: 0.25 cm <130 MeV 0.50 cm >130 MeV			Energy-dependent 0.37 cm at 70 MeV 0.59 cm at 130 MeV 0.85 cm at 230 MeV
Spot spacing	Rectangular grid G1: 0.5 × 0.5 cm G2: 0.4 × 0.4 cm			Hexagonal grid 0.425 x FWHM in air
					No RS	5 cm RS
				70 MeV	0.58 cm	X
				130 MeV	0.43 cm	1.15 cm
				230 MeV	0.31 cm	0.65 cm
Spot size		σx	σy		σx	σy
	70 MeV	0.58 cm	0.59 cm	70 MeV	0.58 cm	0.56 cm
	130 MeV	0.36 cm	0.35 cm	130 MeV	0.43 cm	0.41 cm
	230 MeV	0.25 cm	0.24 cm	230 MeV	0.31 cm	0.33 cm
Minimum number of deliverable protons per pencil beam	70 MeV - 1.61e5 230 MeV - 2.29e5			70 MeV - 2e6 230 MeV - 4.6e6
RS	G1: 0.46 cm WET 37 RS plates  G2: 4.5 cm WET automated RS			5.73 cm WET RS plate

FWHM, full-width half maximum; NUPO, Nonlinear Uniform Proton Optimizer; PCS, proton convolution superposition; PSI, Paul Scherrer Institut; RS, range shifter; WET, water equivalent thickness.

Treatment planning and evaluation

To avoid any bias due to different calculation algorithms, the structure sets and dose distributions of the original plans, were imported and evaluated in Eclipse™. A total of 52 plans, with planning target volumes (PTV’s) ranging from 11.2 to 2410.6 cc, were re-planned in Eclipse™ by three experienced dosimetrists with the support of two medical physicists, using the same beam geometry (i.e. number and orientation of fields) as the original plans and preliminary beam data derived from the first commissioning measurements of the Varian ProBeam^® gantry.

To perform dose calculations in Eclipse™, a body contour was created for all cases, including the patient’s outline as well as the treatment couch and immobilization devices. Dose calculations were performed using the PCS algorithm and dose optimization using the NUPO approach. All plans were normalized to the mean dose in the PTV or respective technical structure, i.e. PTV excluding overlapping OARs, plus a margin for fall-off dose compensation.

The minimum, maximum and mean PTV doses, as well as quality of target coverage (V_95%) were evaluated for each plan. At our institution, PTVs are a result of an isometric expansion of the CTV with a CTV–PTV margin depending on the anatomic site of the tumor. The CTV–PTV margin used for both plans were identical as these are derived from patient positioning error calculations and no significant differences are expected. Furthermore, as CTV-based optimization is not supported by PSI-Plan, PTV-based optimization was also used for the plans performed in Eclipse™ to ensure a fair comparison.

In Eclipse™, for superficial cases in which the PTV extend outside the body contour, a technical structure was created by cropping the PTV to be inside the body, with an inside margin of 2 mm. For simultaneous integrated boost (SIB) cases, the maximum dose to the lower dose level targets was accessed by excluding the boost target, plus an additional margin of 5 mm to account for the dose gradient. This allowed for more accurate minimum and maximum dose evaluation, respectively, for these particular cases. To assess overall plan quality, the homogeneity index (HI) (equation 1), as defined by ICRU 83¹² and the conformity index as (CI), as defined by van’t Riet¹³ (equation 2) were calculated for each individual plan as follows:

$${\displaystyle Homogeneityinde{x}_{}=\frac{D_{2\mathrm{\%}}-D_{98\mathrm{\%}}}{D_p}}$$ (1)

$${\displaystyle Conformityinde{x}_{}=\frac{T{V}_{{RI}^2}}{TV\times V_{RI}}}$$ (2)

where $D_{2\%}$ , $D_{98\%}$ and $D_p$ correspond to the dose covering the most exposed 2 and 98% of the volume, defined by ICRU Report 83 as near-maximum and near-minimum, respectively, and the prescribed dose to the target volume; $RI$ is the reference isodose (i.e. 95% of the prescribed dose), $V_{RI}$ is the volume of the reference isodose, $TV$ is the volume of the target volume (i.e. PTV) and, hence, ${TV}_{RI}$ is the volume of the PTV encompassed by the 95% isodose. To evaluate the statistical significance of the differences between the plans performed by the two TPS, a paired-sample t-test was performed for all the parameters mentioned above.

Regarding OAR sparing, the mean dose and D_2% achieved with Eclipse™ were compared against both dose constraint prescribed by the responsible medical doctor and the dose achieved with PSI-Plan. Finally, the number of pencil beams per plan was compared between the two TPS. All data analysis was performed with Matlab^® R2018b (v. 9.5) (The MathWorks Inc., MA).

Results

A total of 54 PTVs were used to evaluate dosimetric differences between the two different TPS. Table 3 shows the results for PTV coverage and overall plan quality. Differences in homogeneity and conformity to the PTV between PSI-Plan and Eclipse™ are shown in Figure 1 for an astrocytoma (A) and sacral chordoma case (B).

Table 3.

Comparison between PSI-Plan and Eclipse™

Parameter	PSI-Plan	Eclipse™	p-value
Min (%)	58.4 ± 21.2	63.4 ± 27.4	0.11
Max (%)	110.6 ± 4.6	108.2 ± 4.3	<0.005
Mean (%)	99.6 ± 3.6	99.4 ± 3.5	0.56
V95% (%)	93.9 ± 6.2	97.4 ± 5.3	<0.005
HI (%)	0.23 ± 0.16	0.14 ± 0.18	<0.005
CI (%)	0.78 ± 0.12	0.74 ± 0.10	<0.005

CI, conformity index; HI, homogeneity index; PSI, Paul Scherrer Institut; PTV, planning target volume.

Average values and respective standard deviations of the evaluated parameters for PTV as well as results from the t-test performed to assess statistical significance (paired-sample t-test).

Figure 1.

Difference in homogeneity and conformity between PSI-Plan (left) and Eclipse^TM (right). For both plans, the minimum displayed isodose represents 95% of the prescribed dose. The DVH shows the differences in the dose distribution homogeneity between Eclipse™ (squares) and PSI-Plan (triangles). DVH, dose–volume histogram; PSI, Paul Scherrer Institut.

Although differences for minimum and mean doses are not statistically significant, Eclipse™ plans appear to systematically provide a higher minimum dose, lower maximum dose, comparable mean dose, higher PTV coverage and homogeneity indices closer to unity. On the other hand, plans from PSI-Plan appear to be more conformal.

As shown in Figure 1, the 95% isodose extends outside the PTV, for Eclipse^TM plans. On average, the V_95% isodose volume in Eclipse^TM plans is 1.4 times larger than the respective PTV whereas for PSI-Plan, it is 1.1 times larger.

Regarding OAR sparing, for each prescribed dose constraint (D_mean and D_2%), the dose to the respective OAR was evaluated and compared between TPS’s. Figure 2 shows the comparison between the sparing achieved with PSI-Plan and Eclipse™, with respect to the prescribed dose constraint, for all evaluated OARs.

Figure 2.

Comparison between OAR sparing achieved with PSI-Plan and Eclipse™. Each point represents the difference between the TPS (PSI-Plan or Eclipse) ^TMand D_mean and D_2% with respect to the constraint from the clinical prescription. Points in the third quadrant represent cases in which the constraint was achieved by both TPS, whereas points in the first quadrant represent the cases in which the dose constraint was not met by any of the TPS. Points over the equality line show the cases in which an OAR was better spared with Eclipse^TM than with PSI-Plan. OAR, organ at risk; PSI, Paul Scherrer Institut; TPS, treatment planning system.

D_mean was evaluated for 136 OARs and requested D_mean constraints were not achieved for 18, for both PSI-Plan and Eclipse™ plans. The differences were not significant (p = 0.7). Regarding D_2% analysis, a total of 253 OARs were evaluated and both TPS spared 92.9% of the evaluated OARs. Without a compromise for OARs according to the clinical prescription, D_2% was significantly higher for Eclipse™ (p < 0.005) for 64.4% of the evaluated OARs, i.e. with Eclipse™ it was possible to achieve D_2% closer to the constraint, which seemed to have contributed for a better target coverage.

Finally, for the evaluated number of plans (n = 52), our results show that PSI-Plan requires, on average, 3.4 times more pencil beams than Eclipse™ to cover the same target volume. The largest difference was found for an Ewing’s sarcoma case, for which Eclipse™ required 3222 pencil beams and PSI-Plan 29,429, i.e. 9.1 times more. No correlation was found between the difference in the number of pencil beams per plan and the size of the target volume (r = −0.05).

Discussion

In proton therapy, any TPS and delivery system used in clinical practice should be able to make the most out of the physical properties of protons (spread-out Bragg peak, sharp fall-off etc.) and, at the same time, mitigate its disadvantages (e.g. range uncertainty) in order to effectively treat patients.

While several benchmarking papers have been published on commissioning data and calculation algorithms,^14–16 literature on comparing proton therapy TPS’s is limited.^11,14 Our work aimed to compare a commercially available system against our in-house solutions, with which there is extensive clinical experience and proven clinical outcomes.^1,4–8 As Eclipse™’s module for protons was only recently released, not much literature exists comparing it to other TPSs. As for PSI-Plan, it was used before to benchmark XiO^® photon plans,¹⁷ but never against another TPS for proton planning.

Our results show that both systems are able to provide acceptable plans. While not compromising OARs sparing, Eclipse™ was able to achieve significantly better target coverage and overall plan quality (higher minimum dose, lower maximum dose, comparable mean dose, better HI). On the other hand, PSI-Plan showed better results in terms of dose conformation.

Additionally, our results show that PSI-Plan requires a substantially higher number of spots per plan. This difference is due to, firstly, the conservative approach taken in PSI-Plan regarding spot and layer spacing and the optimization of such parameters in Eclipse™ and, secondly, the relatively high minimum spot weight limit imposed by the ProBeam^® in comparison with Gantries 1 and 2 at PSI. That is, more pencil beams are removed from the plan in Eclipse™ due to undeliverable weights than in PSI-Plan plans. Although a high minimum spot weight has advantages as it reduces the overall spot number and possibly the delivery speed, it can be problematic in some plans, particularly for multiple-field plans to large volumes and with low fraction doses. The impact of layer/spot spacing as well as minimum spot weight on clinical planning has been under investigation at our institution.¹⁸

Lastly, differences in dose calculation and optimization algorithms ought to be considered as limitations to a more accurate comparison. Although both PSI-Plan and Eclipse™ analytical calculations have been validated against Monte Carlo,¹⁹ there are nevertheless substantial differences between the two algorithms, with PSI-Plan using the raycasting approach and Eclipse™ a pencil beam deconvolution approach.²⁰ Doolan et al,¹¹ reported the same limitations when comparing three different proton TPS.

To derive conclusions about each TPS performance, an analogous study using beam data which mimics the same treatment unit as input for both TPS would have to be conducted. However, this exercise was out of the scope of the present study as the main goal was to get insights on the quality of treatment planning from two complete and different PBS solutions (i.e. combination ProBeam^®-Eclipse™ vs G1/G2-PSI-Plan) in order to insure that the commercial solution would be able to provide the standard of care as our in-house systems.

Conclusion

The present study aimed to benchmark a commercial solution against our clinically tested and validated in-house systems, which have been used for the past 20 years. Our goal was to ensure that the new commercial system is able to provide the same standard of care for future patients as well as to reveal systematic differences for two systems in the obtained dose distributions. Both systems proved to be capable of producing satisfactory plans that respect the dose constraints for both target and OARs. While some systematic differences between the two systems were observed, these do not seem significant to be translated into differences in clinical outcomes and, therefore, we conclude that the standard of clinical care in our institution will be maintained with the implementation of the new commercial solution.

Nevertheless, due to differences between systems and some technical limitations, special care should be taken while drawing conclusions about each TPS performance.