Auditing local methods for quality assurance in radiotherapy using the same set of predefined treatment plans

Abstract

Background and purpose

Local implementation of plan-specific quality assurance (QA) methods for intensity-modulated radiotherapy (IMRT) and volumetric modulated arc therapy (VMAT) treatment plans may vary because of dissimilarities in procedures, equipment and software. The purpose of this work is detecting possible differences between local QA findings and those of an audit, using the same set of treatment plans.

Methods

A pre-defined set of clinical plans was devised and imported in the participating institute’s treatment planning system for dose computation. The dose distribution was measured using an ionisation chamber, radiochromic film and an ionisation chamber array. The centres performed their own QA, which was compared to the audit findings. The agreement/disagreement between the audit and the institute QA results were assessed along with the differences between the dose distributions measured by the audit team and computed by the institute.

Results

For the majority of the cases the results of the audit were in agreement with the institute QA findings: ionisation chamber: 92%, array: 88%, film: 76% of the total measurements. In only a few of these cases the evaluated measurements failed for both: ionisation chamber: 2%, array: 4%, film: 0% of the total measurements.

Conclusion

Using predefined treatment plans, we found that in approximately 80% of the evaluated measurements the results of local QA of IMRT and VMAT plans were in line with the findings of the audit. However, the percentage of agreement/disagreement depended on the characteristics of the measurement equipment used and on the analysis metric.

1. Introduction

Intensity-modulated radiotherapy (IMRT) and volumetric modulated arc therapy (VMAT) techniques have become standard practice in radiotherapy. Given the complexity of these delivery methods, the dose delivery needs to be verified against calculation by the treatment planning system (TPS) [1].

Several reports have been written regarding recommendations on quality assurance (QA) for IMRT and VMAT plans [1], [2], [3], [4], [5]. Still, local implementations of plan-specific QA methods may vary because of differences in hardware, software and evaluation metric. To ensure independent verification of plan-specific QA, many dosimetry audits have been conducted using locally devised treatment plans [4], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15]. Since the local QA equipment is also used to devise the local class solution, such audits may not give insight in its ability to detect nonconformities for plans not belonging to the original class solution. Since over time, treatment plans may deviate unnoticed from the intended class solution it is important to determine whether local QA systems can detect errors for plans not belonging to the class solution. To achieve this aim we distributed a limited set of pre-defined treatment plans using the radiotherapy (RT) extension to the Digital Imaging and Communications in Medicine (DICOM) standard among the participating institutes. Using the same set of treatment plans allows the comparison between the local QA and the audit findings of different centres.

2. Materials and methods

The audit was performed at all 21 Dutch radiotherapy centres extended with one satellite location. Due to the lack of plan import options in the treatment planning software, in 3 of 22 sites plans had to be generated by the institute itself. The results of these measurements were not included in the analysis.

2.1. Treatment plans

Treatment plans of different complexity were generated: simple (cervix) and complex (head and neck) IMRT and VMAT, and a stereotactic (brain) VMAT plan. The audit plans reflect typical clinical IMRT and VMAT delivery, selectively chosen such that all delivery parameters were valid for the various combinations of TPS and linac delivery system used by the participating centres, provided that particular treatment technique was used clinically. For comparative and logistical reasons, the audit was performed for 6 MV beams only. Based on the occurrence of linac and TPS (Table 1), two plan sets were created: one for Elekta (Elekta Instrument AB, Stockholm, Sweden) linacs, devised in Pinnacle (Philips Medical Systems International B.V., Best, the Netherlands) and one for Varian linacs, devised in Eclipse (Varian Medical Systems, Palo Alto, California). We strived to keep the planning parameters as similar as possible (Table 2). The IMRT plans designed for the Elekta linac used a step-and-shoot technique, whereas all other plans used a sliding-window technique.

Table 1.

Available linac vendor and TPS combinations in the Netherlands at the time of the audit measurements (October 2014 - August 2015).

Linac vendor	TPS system	Number of institutes	RTP import
Elekta	Monaco (Elekta Instrument AB, Stockholm, Sweden)	2	Not possible*
Elekta	Oncentra (Elekta Instrument AB, Stockholm, Sweden)	2	DICOM
Elekta	Pinnacle (Philips Medical Systems International B.V., Best, the Netherlands)	10	Pinnacle file format
Elekta	Raystation (RaySearch, Stockholm, Sweden)	1	DICOM
Varian	Eclipse (Varian Medical Systems, Palo Alto, California)	5	DICOM
Varian	iPlan (BrainLab AB, Munich, Germany)	1	Not possible

^*Limited DICOM import is possible from version 5.1 but not for externally generated phantom plans.

Table 2.

Audit treatment plan characteristics.

Parameter	Simple plans		Complex plans		Stereotactic plan
Technique	IMRT*** 5 beams	VMAT 1 arc	IMRT*** 7 beams	VMAT 2 arcs	VMAT 1 arc
Preferable linac	Standard	Standard	Advanced	Advanced	Advanced
Elekta	MLCi(2)	MLCi(2)	Agility	Agility	Agility
Varian	Clinac	Clinac	TrueBeam	TrueBeam	TrueBeam
Energy (MV)	6	6	6	6	6
Gantry angles (o)	−144, −72, 0, 72, 144	178–182	−150, −110, −50, 0, 50, 110, 150	178–182, 182–178	178–182
Collimator angle (o)	20	20	20	20	20
Dose (cGy) at isoc
Plans for Elekta linac	184.5	193.2	144.8	146.3	353.8
Plans for Varian linac	188.5	181.5	146.5	143.6	357.5
Total # MU*
Plans for Elekta linac	418.6	391.4	511.7	388.8	675.0
Plans for Varian linac	878.3	565.0	1313.1	461.8	710.5
CPs/segments**
Plans for Elekta linac	35	90	60	180	90
Plans for Varian linac	643	178	1632	356	178

^*1 MU (Monitor Unit) = 1 cGy @ SSD = 100 cm @ dmax.

^**Number of segments for Step-and-Shoot (Elekta) IMRT, number of control points for dynamic delivery (Varian) IMRT and VMAT (Elekta, Varian).

^***Delivery technique: step-and-shoot for Elekta linacs and sliding-window for Varian linacs.

To assess the complexity of the plans, the segment shape, leaf motion and dose distribution were evaluated visually. To ensure accurate dose measurements in the audit phantom, the isocentre was located in a homogeneous high dose region. The linacs were grouped into two types:

• •Standard: Elekta MLCi(2) or Varian Clinac
• •Advanced: Elekta Agility or Varian TrueBeam

Whenever possible, the simple plans were delivered on a standard linac, the complex and stereotactic plans on an advanced one. For the Varian linacs, the same treatment plans could be delivered on both linac types, whereas the standard and advanced Elekta linacs are not interchangeable due to differences in head design (e.g. multi-leaf collimator (MLC) and block design). Besides the differences in head design, there was also a variation in availability of linac options (i.e. not all institutes purchased the VMAT license on Elekta MLCi(2) linacs).

2.2. Audit preparation

The treatment plans, audit phantom Computer Tomography (CT) scan, structure data set and the audit preparation manual were distributed to all institutes. The institutes calculated the dose on a 2 × 2 × 2 mm³ grid, using for the phantom a relative electron density of 1.016 g/cm³ or mass density of 1.04 g/cm³ [16]. All other calculation settings, such as dose algorithm, correction for treatment table were according to the clinical protocol of the institute.

The institutes performed their own QA measurements in advance using their local equipment (Table 3) and analysed the measurements according to the audit criteria (Section 2.3).

Table 3.

QA equipment used by the institutes to measure the audit treatment plans.

ArcCHECK	Sun Nuclear Corporation, Melbourne, USA	2
Delta4	Scandidos AB, Uppsala, Sweden	5
EPIDdosimetry	Elekta	1
EPIDdosimetry	Varian	1
Film (EBT3) and point dosimetry	Gafchromic EBT3, Ashland Specialty Group, Wayne USA	2*
MatriXX	IBA, Louvain-La-Neuve, Belgium	5
Octavius® II	PTW, Freiburg, Germany	4**
Octavius® 4D	PTW, Freiburg, Germany	2

^*Film and ionisation chamber are used in combination with a slab phantom or the OCTAVIUS® II phantom.

^**In one institute, the PTW 729 array is used in combination with a slab phantom.

2.2.1. Measurement equipment

All measurements and irradiation of calibration films for the audit were performed using the OCTAVIUS® II (PTW Freiburg GmbH, Freiburg, Germany) phantom and its associated inserts for the three different dosimeters: ionisation chamber for an absolute dose measurement, ionisation chamber array for a 2D measurement with high reproducibility [11], [17], and radiochromic film for a 2D measurement with high resolution. The ionisation chamber was calibrated by the Dutch Metrology laboratory, VSL; the 2D array was calibrated by its manufacturer (PTW Freiburg GmbH, Freiburg, Germany) and checked for constancy at the Netherlands Cancer Institute – Antoni van Leeuwenhoek.

2.2.1.1. Audit ionisation chamber

The point dose was measured using a 0.016 cm³ PinPoint ionisation chamber (TN31016 PTW Freiburg GmbH, Freiburg, Germany) in combination with an electrometer (Unidos^Webline, PTW Freiburg GmbH, Freiburg, Germany). The readings were converted to absolute dose according to the k_Q formalism [18].

2.2.1.2. Audit array

The OCTAVIUS® II with 729 plane-parallel ionisation chambers was used for the array measurements and the readings were recorded by the VeriSoft software (VeriSoft®, version 6.1, PTW Freiburg GmbH, Freiburg, Germany [19]). To compensate for daily output variations, the dose measured for a 10 × 10 cm² field by the central ionisation chamber of the array was used for normalisation.

2.2.1.3. Audit film

Film measurements were performed using radiochromic films (Gafchromic EBT3, Ashland Specialty Group, Wayne USA) from a single batch. For absolute dose calibration using three colour channels [20], quarters of a film were irradiated in the audit array with 0, 200, 400 and 600 MU (∼0–3.8 Gy) with a 10 × 10 cm² field. The films were converted to dose according to the well-established local protocol of the VU University Medical Centre (Amsterdam, the Netherlands) [21], [22].

2.3. Analysis

2.3.1. Ionisation chamber

The relative difference between the dose as calculated by the local TPS and the audit measurement corrected for daily accelerator output variation was defined as Δ_N. Δ_N was calculated by multiplying for each plan the relative difference between the local TPS calculated dose and the audit measurement, with the ratio of the local TPS calculated dose and the audit measured dose for the 10 × 10 cm² field in the OCTAVIUS® II phantom. Δ_N was reported with a pass tolerance level of ±3%.

2.3.2. Array and film

The comparison between measured and computed dose distributions was performed in VeriSoft. The 2D measurements from the array and film were compared with the 3D dose distribution from the TPS using a 3D global gamma analysis [23], normalised to the maximum computed dose in the phantom. For the simple and complex IMRT and VMAT plans, the gamma pass rates were calculated using gamma parameters 3 mm and 3% [2], [3], [23], while for the stereotactic plan these settings were 1 mm and 5% [24]. A dose threshold of 10% of the measured maximum dose and a pass criterion of 95% or higher was used.

2.3.3. Evaluations

The agreement/disagreement between audit and the institute QA results was evaluated per dosimeter, per treatment plan and per linac type. For the latter, we combined the results of the Varian Clinac and TrueBeam linacs since these could deliver the same plans and were considered interchangeable, in contrast to the standard and advanced Elekta linacs. The median, interquartile range and full range (min/max) of the results for the three audit dosimeters and for institute QA results were calculated per plan type.

3. Results

In total 82 plans were measured with each audit dosimeter: 18 simple IMRT; 14 simple VMAT; 18 complex IMRT; 16 complex VMAT and 16 stereotactic VMAT. The plans measured on Elekta linacs were 52: 23 for the standard and 29 for the advanced type; on the Varian linacs 28 plans were measured.

Most of the audit results agreed with the institute QA results (ionisation chamber: 74/82; array: 69/82; film: 62/82, see Fig. 1). When considering only the plans measured on Elekta linacs, most ionisation chamber (48) and array (44) audit measurements concurred with the institute QA. Similar results were found for Varian linacs: 24 ionisation chamber and 23 array measurements coincided with the institute QA.

Fig. 1.

Pie-charts showing the agreement between results of the audit and the institute QA for all plans (upper panel), plans measured on Elekta linacs (lower left) and plans measured on Varian linacs (lower right). Due to the large number of outliers, with extremely low pass rate, found for the film measurements, the film results were not taken into account in the analysis per linac type. The four categories are: Agreement (pass), indicating that the plan passed both audit and institute QA; Agreement (fail), indicating that the plan failed both audit and institute QA; Disagreement (audit pass/institute QA fail), indicating that the plan passed the audit but failed the institute QA; Disagreement (audit fail/institute QA pass), indicating that the plan failed the audit but passed the institute QA. The acceptance criteria were: a gamma pass rate ≥ 95% (5%/1mm for the stereotactic plan and 3%/3mm for the other plans) for the 2D measurements (array and film); a relative difference within ±3% for the point measurements (ionisation chamber).

For both ionisation chamber and array measurements, the agreement/disagreement rates were not affected by plan complexity (Fig. 2), whereas for film the disagreement rate seems to be slightly higher. Moreover, the film results showed a greater number of fails and fewer passes for increasing plan complexity.

Fig. 2.

Pass rate for the audit results (array: left column and film: right column) as a function of the institute QA for all measurements subdivided according to the plans. The pass rates were calculated using gamma parameters 5%/1 mm for the stereotactic plan and 3%/3 mm for the other plans. Please note the difference in scale for the y-axis between the Array and Film results.

The average of the ratio between calculated and measured dose for the 10 × 10 cm² reference field used in the calculation of Δ_N was 0.7% ± 1.4% (1 SD). These values were largely influenced by the larger spread in this ratio from the institutes using a dose-to-medium, D_m, based TPS for which the ratio was 1.1% ± 1.5% (1 SD) compared to 0.2% ± 1.2% (1 SD) for institutes with a dose-to-water, D_w, based TPS.

For the array measurements, the range of the pass rates observed for the simple (75.8%–100%) and complex VMAT plans (83.3%–100%) was larger than the institute QA result range (80.3% –100% vs 93.2%–100%, respectively). For the film measurements, an unexpected high number of outliers with extremely low (<50%) pass rate was found. Since this was primarily seen for one linac type, the film results were excluded in the analysis per linac type.

The distribution of the measurements performed per linac type and per plan type are summarized in Table 4. The average median Δ_N value per linac type for all plan types was −1.2% (standard Elekta), −2.2% (advanced Elekta) and 0.8% (both Varian types). For the Elekta linacs, the median pass rates per plan type were above 95% for the array measurement except for the simple VMAT plan. The larger range for this plan was not observed in the institute QA results, except for one outlier (second to lowest was 92.1%). For the advanced Elekta linacs, a larger range in pass rates was observed for the complex plans, which was also observed in the institute QA. For the Varian linacs, a wider range was observed for both the IMRT plans. However, this range was in agreement with the institute QA results.

Table 4.

Median values, range (min/max) and interquartile range (IQR) of the audit and institute QA results among all measurements for the Elekta linacs (MLCi(2) or Agility) and the Varian linacs (Clinac and TrueBeam).

Plan type	Linac type	Number of measure-ments	Ionisation chamber	Array	Institute QA
			ΔN (%)	Pass rate (%)
			Median (min−max) IQR	Median (min−max) IQR	Median (min−max) IQR
Simple IMRT	MLCi(2)	12	−1.5 (−2.8−+0.4) 1.3	100.0 (98.1–100.0) 0.0	99.5 (92.4–100.0) 1.2
Simple VMAT	MLCi(2)	8	−0.6 (−2.5−+1.2) 1.0	94.0 (75.8–99.6) 10.6	99.9 (80.3–100.0) 1.3
Complex IMRT	MLCi(2)	3	−1.4 (−1.4−+0.1) 0.7	100.0 (99.8–100.0) 0.1	99.5 (99.3–100.0) 0.3
Complex IMRT	Agility	10	−1.7 (−2.4−+1.3) 1.0	99.6 (79.0–100.0) 0.9	99.6 (95.6–100.0) 2.1
Complex VMAT	Agility	9	−2.7 (−3.7−−1.4) 1.1	96.3 (83.3–100.0) 5.7	98.8 (93.2–100.0) 4.2
Stereotactic VMAT	Agility	10	−2.2 (−2.9−−0.9) 0.6	99.6 (96.9–100.0) 1.9	100.0 (97.7–100.0) 0.1
Simple IMRT	Clinac/TrueBeam	5	−0.2 (−2.1–+0.2) 0.4	99.1 (93.5–100.0) 1.7	100.0 (93.8–100.0) 1.2
Simple VMAT	Clinac/TrueBeam	6	+1.4 (−0.1–+1.8) 0.9	100.0 (99.2–100.0) 0.3	100.0 (92.1–100.0) 0.0
Complex IMRT	Clinac/TrueBeam	5	+0.8 (−3.3–+1.9) 0.8	99.5 (65.8–100.0) 0.8	97.4 (79.0–100.0) 5.5
Complex VMAT	Clinac/TrueBeam	6	+1.2 (−0.9–+1.8) 1.9	100.0 (99.5–100.0) 0.0	99.7 (95.0–100.0) 0.6
Stereotactic VMAT	Clinac/TrueBeam	6	+0.6 (−0.8–+1.3) 1.2	100.0 (98.1–100.0) 0.0	99.7 (94.1–100.0) 0.2

For all plans combined, the average median Δ_N was −1.3 ± 0.6% (1 standard deviation, SD); the average median gamma pass rate was close to 100%, for both the film (99.1%) and array (99.7%) measurements as well as the institute QA result (99.7%).

4. Discussion

A national audit on local methods for treatment plan verification using one set of pre-defined plans has proven to be feasible. Overall, the audit revealed a good agreement between calculated and measured dose. The results of the audit were in line with other multi-institution audits [5], [7], [8], [13].

Overall, the local QA findings were in agreement with those of the audit, possibly due to the observed high pass rate. The observed amount of agreement between local QA and audit depends on the characteristics of both audit and local QA equipment, plan complexity and the evaluation parameters.

Kry et al. [25] found that local QA did not predict their audit results. In our work, the QA methods, treatment plans and evaluation criteria were more homogeneous, partly explaining this difference. In addition, the audit was set-up prospectively and performed by the audit team, thereby reducing the chance of handling errors. The measurements failing both audit and institute QA were mostly related to the complex plan types as presented by Lafond et al. showing lower gamma pass rates for head and neck plans compared to prostate [26]. One institute with a low pass rate for both audit array measurements and local QA did not use the technique clinically since it was still developing the corresponding class solution. For the others, the audit QA was poor due to directional dependence of the audit phantom (see below). Unfortunately, not for all low pass rates a satisfactory explanation could be found. All institutes have been informed of their own results with respect to the average ones.

The level of agreement/disagreement between the audit and institute QA results was found to be dependent on the employed measurement method. This variation is caused by the characteristics and limitations of the employed device, in combination with characteristics of the treatment plan.

In addition, the results are highly sensitive to the chosen pass/fail limit and acceptance gamma criterion. Evaluating the array results with a limit of 90% would increase the level of agreement (both pass) to 91%, the level of agreement (both fail) to 1% and reduce the level of disagreement to 8%. The degree of agreement/disagreement between the audit results and the institute QA is comparable for both Elekta and Varian machines, despite the different set of plans used. Analysing our array results shows an average pass rate of 97.9% (3%/3mm) and 96.2% (3%/2mm) for all Varian plans and an average pass rate of 96.6% (3%/3mm) and 93.4% (3%/2mm) for all Elekta plans. This larger variation for the Elekta plans, with more variation in the combination of linac/TPS vendor, is in accordance with the findings of Clark et al. [11], although they found a bigger difference.

For the audit array, a relatively large number of measurements disagreed for the simple and complex VMAT plans (audit failed, institute QA passed), which was most likely due to the measurement device itself. The response of the audit array has a directional dependence influencing the outcome of the measurements on Elekta linacs [27], [28], [29], in particular because the phantom was defined as homogeneous in the TPS [29] and a substantial number of MU’s was delivered at the affected angles (90 and 270°). For the plans for Varian linacs, the MU’s were more evenly distributed over all angles. The use of for example the OCTAVIUS® 4D array could have mitigated this [16].

A wider range in results was observed for both simple and complex IMRT plans on Varian machines than for Elekta linacs, which was in agreement with the institute QA results. This could be explained by the more complex sliding-window delivery technique used by Varian linacs compared to the step-and-shoot delivery of Elekta machines [30], [31]. For the Varian linac we observed a higher disagreement rate (audit pass, institute failed), which could be explained by the higher spatial resolution of the local QA equipment used by these institutes.

A larger number of outliers with a very low gamma pass rate was observed for the film results compared to the other measurements methods. In these measurements, on average 10 cGy (i.e. 5% on 200 cGy) less was recorded than expected, corresponding to a pass rate below 80% using 3% and 3 mm. This is explained by the 5% variation in the local audit film calibration curves, being much larger than the ∼1.5% variation expected from the standardised procedure for film analysis [22], [32], [33], [34], [35]. Absolute film dosimetry is less accurate than relative film dosimetry [34] but no clear reason was found for the differences in the local calibration curves. Further investigations to clarify this are on-going, but beyond the scope of this work.

The QA methods used in this audit were designed to be insensitive to differences in dose calculation properties, for example type and version of TPS, dose algorithm (D_w based or D_m based), Hounsfield Unit to electron density conversion and inclusion of the treatment couch. This is confirmed by the small variation in Δ_N over all the institutes for the different plan types, and by the differences between D_m and D_w based TPS for the ratio between calculated and measured 10 × 10 cm² field dose. Other tests should elucidate such aspects.

The main challenges in the audit were the workload and circumventing RT-Plans DICOM import problems. Separate plans for the different linac types had to be created causing unavoidable differences among these plans. For Elekta linacs, certain machine settings and limits can be chosen by the user, regardless of the TPS being used. To ensure that the audit treatment plans could be delivered and the dose could be calculated by each institute, the most conservative settings (e.g. maximum leaf speed, minimum and maximum dose rate) were chosen. For Varian users, such freedom in machine settings and limits is not available.

Another limitation is that, for comparison purposes, the audit team defined the evaluation criteria according to national guidelines which may not be exactly the same as the ones used locally. Despite national guidelines [2], [3], the evaluation criteria varied slightly among institutes but the impact on the results of this work is negligible. Finally, small differences exist in the implementation of the gamma analysis between QA vendors.

Since the performance of the local QA devices is to a large extent energy independent, we believe that this evaluation of local QA methods can be translated to other energies as well.

Concluding, this work described the implementation and results of a national audit on validation of clinically used QA methods for plan-specific verification. For this, the same set of treatment plans was used for all institutes enabling a multi-centre comparison of the local QA results not influenced by other steps of the treatment chain such as planning protocol and optimization algorithms. The results showed that overall local QA of IMRT and VMAT plans in the Netherlands gave similar results as those of an external audit. However, the disagreement rates between local and audit QA results indicate that independent audits of QA systems, combined with other types of dosimetry audits, are indispensable tools in the continuous improvement of radiotherapy as part of the treatment for patients with cancer.

5. Conflict of interest notification

Actual or potential conflicts of interest for this study do not exist.