Study on the established customized limits for the daily quality assurance procedure

Abstract

Since there are no corresponding specification limits for some new daily quality assurance (QA) items in the TG-142, it is a compromise that the specification limits used in the monthly or annual QA procedures are used for the daily QA procedure in work. But there is no basis for whether this is feasible. The purpose of this article is to analyze QA results using SPC to determine the tolerance limits at our institution, and to present the usefulness of the analysis method using SPC. The data of three groups daily QA processes performed with Daily QA3 in three years were analyzed using statistical process control (SPC). For calculating capability indices (C_p, C_pk, C_pm and C_pmk) of processes, the appropriate number of calculation points was analyzed firstly. Then, in calculating the capability indices for output, limits ±3% of the daily QA in the TG-142 were used as the specification limit, while for flatness and symmetry, an annual QA limits of ±1% was used. For putting forward measures to solve the problem, customized tolerance and action limits were established for each process. And the process control charts calculated using data measured by the five therapists and a medical physicist were compared. At least six to eight weeks of control daily check data points (i.e. 30–40 points) should be used for calculating the individuals and moving range (I-MR) control chart to ensure the stability of control lines. Process capability indices of output were all ≥1, some were up to 3–4. While for symmetry, some processes failed to meet the requirements that capability indices were < 1. For different processes of the same daily QA items, the calculated customized limits were quite different. The range of upper control line (UCL) and lower control line (LCL) was smaller for output and the CL was closer to the target value of 0 for flatness and symmetry in the I-MR control chart calculated using data measured by one staff. For different quality control processes without management by the SPC method at our institution, calculated tolerance and action limits of the same measurement item were quite different. And in most measurement items, the specification limits used in the monthly or annual QA procedures in the TG-142 are not suitable to the daily QA procedure. So the analysis method using SPC is useful and necessary.

INTRODUCTION

If the performance of linear accelerators deviates from normal, the effectiveness of radiotherapy may be affected [1]. In American Association of Physicists in Medicine task group 142 (AAPM TG-142), it is recommended that the geometry or dose parameters that may influence the radiotherapy dose should be checked as part of daily quality assurance (QA) procedures, and the limits are specific to the type of treatments delivered with the treatment unit [2]. However, these limits in AAPM TG-142 are the minimum requirements for quality control, which is mainly based on clinical efficacy [3]. It is not impossible to list specific limits for individual products in the report, let alone actual quality control processes. Because these thresholds evolve from the QA data [2].

In addition, with the rapid development of equipment, such as morning check equipment and smart radiotherapy platforms [4–6], some monthly QA procedures such as flatness and symmetry have gradually become check items for daily quality control. However, there are no corresponding limits for these new daily QA items currently in TG-142, it is a compromise that the specification limits used in the monthly or annual QA procedures are used for the daily QA procedure in work. But whether it is feasible to do that, there is little literature to report at present.

Furthermore, when there are not proper limits in a professional society guidance document, limits that are based on the process are more needed. The method of statistical process control (SPC) can be used when establishing customized limits. SPC was first applied to industrial manufacturing [7–9], and was gradually utilized in aviation and health care [10, 11]. In 2005, Pawlicki et al. [12] analyzed the quality control of radiotherapy through SPC. Subsequently, this method was also applied to the QA of linear accelerators [13, 14], tomotherapy [15], proton accelerators [16] and intensity-modulated radiotherapy plans [17–19]. In 2013, Sanghangthum et al. [20] proposed to use SPC for establishing customized tolerance and action limits.

The purpose of this study is to discuss the feasibility of using the specification limits in the monthly or annual QA procedures to the daily QA procedure and put forward measures to solve the problem.

BACKGROUND

SPC is a process control tool that uses mathematical statistics to find signs of systematic error in time and take measures to eliminate their effects. It has two main aspects of application: one is to use the control chart to analyze the stability of the process, distinguish the random error and systematic error in the process by the threshold of the upper (UCL) and lower control lines (LCL), and provide early warning for abnormal factors in the process; the other is to evaluate the process quality using process capability indices and provide numerical measures of whether or not a QA process is capable of meeting the predetermined specification limits.

The Shewhart chart is one of the most commonly used control charts for continuous data. It includes an UCL, central line (CL), LCL and some data points arranged by sequence of time. If all the data points fall between the UCL and LCL, it indicates that the process is stable, i.e. the process is only affected by random errors. The UCL and LCL are usually set at a distance of three expected standard deviations (σ). This implies that 99.7% of the data points would fall within the control limits when the data are normally distributed. Then, when the process is in control, there is only a 0.3% chance that a point will be outside the control limits.

The individuals and moving range (I-MR) control chart is one kind of the Shewhart chart. If the subgroup size of the quality control item is 1, the I-MR control chart is used. The respective calculation formulae for UCL, CL and LCL in I-MR control charts are as follows:

where and are mean values of I-MR and d₂ is a bias correction constant that depends on the subgroup size n. It is customary to use the constant value d₂ = 1.128 for subgroup size n = 1.

The is obtained as follows:

Capability indices include C_p, C_pk, C_pm and C_pmk. The C_p characterizes the state of fluctuation inherent in the process. The C_pk considers both the fluctuation inherent in the process and the bias of mean values. The C_pm considers both deviations of target values and mean values. The C_pmk is a comprehensive result of the deviation of process capability and mean values from target values. These indices are defined explicitly as shown:

where is the average, σ is the standard deviation, the process average and variance are calculated from the data only under a specific process operation criterion. USL and LSL are the upper and lower specification limits, limits in the TG-142 report are used as specification limits in this study. T is the target value, the value of T is 0 in this study.

METHODS OF SETTING TOLERANCE LIMITS AND ACTION LIMITS

Tolerance limits should serve as warning limits and it serves as an indication that a process is changing and in need of attention. Control chart limits are ideal to use as tolerance limits because control chart limits are designed to characterize process performance. In this study, the tolerance limit was established using the larger one of the UCL and LCL in the I-MR control chart.

There are two types of action limits: one type is that defined by professional societies, guidance documents or best practice documents, such as the recommended limits in the TG-142 report. This type is taken as universal or absolute recommendation that is not varied from institution to institution, and exceeding these limits is likely to affect clinical treatment effect. The other kind is unspecified action limits that are determined empirically by an analysis of available data or sometimes simply by using clinical experience, and exceeding these limits does not necessarily affect clinical treatment effectiveness, but staying within these limits can improve the treatment process [20]. In this study, the action limits belong to the second type and are process-based action limits. The action limit is calculated by the formula of Sanghangthum [20]:

where UAL–LAL is the width of the action limits, A is used to adjust the width of the limit to balance the occurrence of type I errors (errors rejecting the true null hypothesis) and type II errors (errors accepting the false null hypothesis) in the statistical process. The average and variance are calculated from the data only under control in the I-MR control chart. The calculation is performed with A being 6, 5, 4 and 3, respectively, and the calculated value of, which covers the tolerance limit with the minimum A value, was used as the action limit.

MATERIALS AND METHODS

Data of daily QA were obtained from a clinical photon beam (Elekta Synergy linear accelerator, UK) measured using Daily QA3 (Sun Nuclear, US). Daily QA3 has a total of 25 detectors, five ionization chamber detectors of 0.6 cm³ effective measurement volume used for the measurement of output dose, flatness and symmetry. Annually, the temperature and pressure within Daily QA3 are calibrated with a calibrated barometer and mercury thermometer. The detector calibration of Daily QA3 will be performed according to the operating procedure if the consistency of Daily QA3 detectors becomes worse.

In this study, after performing absolute calibrations on photon energy 6 MV or adjusting the flatness/symmetry or autocalibrating the multi-leaf collimator (MLC) position of the accelerator that all may influence the dosimetric parameter, dose calibration of Daily QA3 was performed at a field size of 20 × 20 cm² at 100 SSD for 100 MU and the baseline was recalibrated.

Usually, daily QA is performed by therapists in rotation before patient treatment. Daily QA3 is aligned on the field light. In this study, a hundred sets of continuous daily check data every year (five groups per week for 24 weeks) were analyzed for three years and the first data of every set was selected after dose and detector calibration of Daily QA3 were performed. The reason of doing these is to reduce the impact of Daily QA3 performance on data measurement. Meanwhile, a finger-type ionization chamber (0.6 cc, PTW30013) and dosimeter (Sun Nuclear, PC Electrometer) were used to measure the accelerator output and a parallel plate detector (Sun Nuclear, MapCHECK2) was used to measure the flatness and symmetry every week for the verification of Daily QA3 measurement results.

In addition, another 30 sets of daily check data were collected by a medical physicist after therapists performed daily QA every morning with the same setup and measurement conditions over six weeks.

Analysis of the control line stability

Control chart limits as a function of the number of data points was investigated. The data of first 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 75 and 100 points, representing to 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15 and 20 weeks, were varied to calculate the control limits in each group for each item. However, if there were any points in the calculation limit that were out of control and the source of the error was known, then those out-of-control points were removed and the control limits were recalculated [21].

Capability assessment of quality control process

Based on the results of control line stability above, an appropriate number of data points were selected to calculate the capability indices (C_p, C_pk, C_pm and C_pmk) of each quality control item. Before the calculation, it was tested whether the distribution of the data is the normal distribution, if not, the data was transformed according to Johnson transformation and then was calculated the capability indices [22]. Next, limits ±3% of the daily QA in the TG-142 was used as the specification limits in calculating the capability indices for output, while an annual QA limits of ±1% was used as the specification limits for flatness and symmetry.

Calculation of tolerances and action limits

The tolerance and action limits were also calculated from an appropriate number of data points for three groups daily QA processes. To demonstrate the ability to detect process changes by establishing customized tolerance and action limits, customized tolerance and action limits were used to monitor and analyze the rest of data points of daily QA process. And for verification, the measurement results of the ionization chamber and MapCHECK2 were used to compare.

The effect of individual operation constancy for the location of the UCL, CL and LCL

Two sets of process control charts were calculated separately using 30 sets of daily check data points collected from the five therapists and one physicist. And the effect of individual operation constancy for the location of the UCL, CL and LCL was analyzed.

RESULTS

The results of control line stability

As shown in Table 1, mean , standard deviation σ and the number of run points before out-of-control limits on the X-chart for all measurement items in 2019 were calculated. The value of (/σ)/(#pts) varied significantly in the first six weeks and tended to be stable after six to eight weeks (Fig. 1).

Table 1.

The mean, standard deviation and the number of run points before out-of-control limits on the X-chart for 2019. Each week is equal to 5 data points. N is the number of data points used to calculate the limits

	Output Constancy			Axial Symmetry			Transverse Symmetry			Flatness
N (weeks)	Mean	Standard deviation σ	The number of run points	Mean	Standard deviation σ	The number of run points	Mean	Standard deviation σ	The number of run points	Mean	Standard deviation σ	The number of run points
1	-0.42	0.15	5	-0.68	0.33	5	-0.43	0.10	5	-0.22	0.05	5
2	-0.41	0.28	10	-0.67	0.22	10	-0.42	0.15	10	-0.22	0.08	10
3	-0.44	0.26	15	-0.63	0.24	15	-0.44	0.14	15	-0.23	0.07	15
4	-0.39	0.31	20	-0.60	0.25	20	-0.41	0.15	19	-0.20	0.09	20
5	-0.38	0.30	25	-0.60	0.24	25	-0.37	0.19	25	-0.19	0.10	25
6	-0.35	0.30	30	-0.58	0.24	30	-0.36	0.19	29	-0.18	0.10	29
7	-0.35	0.28	35	-0.55	0.24	35	-0.34	0.19	34	-0.17	0.10	34
8	-0.36	0.27	40	-0.52	-1.04	40	-0.32	0.20	39	-0.17	0.09	38
9	-0.36	0.26	45	-0.49	0.25	45	-0.33	0.17	43	-0.17	0.09	43
10	-0.37	0.21	48	-0.48	0.24	50	-0.34	0.17	48	-0.17	0.09	48
15	-0.33	0.31	74	-0.45	0.25	74	-0.39	0.23	64	-0.20	0.12	65
20	-0.30	0.32	96	-0.48	0.26	96	-0.44	0.18	66	-0.22	0.08	71

Fig. 1.

Signal to noise ratio (/σ) normalized by the number of in-control data points for the data of: (i) Output constancy, (ii) Flatness, (iii) Axial symmetry and (iv) Transverse symmetry for 2017–2019.

Capability assessment of quality control process

For three groups processes in 2017–2019, process capability indices of output were all ≥1, some were up to 3–4. However, most process capability indices of symmetry failed to meet the requirements, especially of transverse symmetry (see Table 2).

Table 2.

Process capability indices of each quality control items in 2017–2019

	Output Constancy						Axial Symmetry						Transverse Symmetry						Flatness
	Mean	Standard deviation σ	C p	C pk	C pm	C pmk	Mean	Standard deviation σ	C p	C pk	C pm	C pmk	Mean	Standard deviation σ	C p	C pk	C pm	C pmk	Mean	Standard deviation σ	C p	C pk	C pm	C pmk
2017	0.41	0.53	1.89	1.63	1.49	1.29	-0.44	0.38	0.88	0.49	0.57	0.32	-0.03*	1.11*	0.70*	0.48*	0.69*	0.48*	-0.07	0.1	3.33	3.10	2.73	2.54
2018	-0.05	0.28	3.57	3.51	3.52	3.46	0.01	0.29	1.15	1.14	1.15	1.14	-0.17	0.43	0.78	0.64	0.72	0.60	-0.06	0.09	3.70	3.48	3.08	2.90
2019	-0.36	0.29	3.45	3.03	2.16	1.90	-0.59	0.26	1.28	0.53	0.52	0.21	0.11*	1.02*	1.04*	0.90*	0.59*	0.51*	0.08*	1.03*	1.45*	1.11*	0.82*	0.63*
Specification limits	±3%						±1%						±1%						±1%

Note: Data marked with * represents that the value is the result of the original data after Johnson transformation. The Cp characterizes the state of fluctuation inherent in the process. The Cpk considers both the fluctuation inherent in the process and the bias of mean values. The Cpm considers both deviations of target values and mean values. The Cpmk is a comprehensive result of the deviation of process capability and mean values from target values.

Calculation of tolerances and action limits

As shown in Table 3, the calculated tolerance and action limits of the same measurement item during different quality control processes were quite different. The tolerance limits of output in 2017 and 2018 were ± 2.01% and ± 0.89%, respectively, and the action limits were ± 2.24% and ± 0.94%, respectively. In 2017 and 2018, the chosen value of A for the action limits of all measurement items was 5, while in 2019, it was 4.

Table 3.

Process-based tolerance limits and action limits of each quality control items in 2017–2019

QA type	Date	The control chart limits			±Tolerance limits	±Action limits
		UCL	CL	LCL		A = 3	A = 4	A = 5	A = 6
Output Constancy	2017	2.01	0.41	−1.19	2.01	1.34	1.79	2.24	2.69
	2018	0.78	−0.05	−0.89	0.89	0.57	0.75	0.94	1.13
	2019	0.51	−0.36	−1.24	1.24	0.93	1.24	1.55	1.86
Axial Symmetry	2017	0.71	−0.44	−1.59	1.59	1.16	1.54	1.93	2.32
	2018	0.88	0.01	−0.86	0.88	0.58	0.77	0.97	1.16
	2019	0.20	−0.59	−1.36	1.36	1.28	1.71	2.14	2.56
Transverse Symmetry	2017	1.70	0.04	−1.61	1.70	1.10	1.47	1.84	2.21
	2018	1.11	−0.17	−1.45	1.45	0.92	1.23	1.53	1.84
	2019	0.18	−0.38	−0.93	0.93	0.84	1.12	1.40	1.68
Flatness	2017	0.22	−0.07	−0.36	0.36	0.24	0.32	0.40	0.48
	2018	0.22	−0.06	−0.33	0.33	0.22	0.29	0.36	0.43
	2019	0.10	−0.19	−0.48	0.48	0.43	0.57	0.72	0.86

Note: UCL: upper control line, CL: central line, LCL: lower control line.

As shown in Fig. 2, the monitoring charts based on process tolerance and action limits were analyzed. The rest of the 60 data points of each quality control item were monitored and identified. Most points of flatness and transverse symmetry exceeded the lower action limit after the 45^th data point.

Fig. 2.

Monitoring charts of using the process-based tolerance and action limit calculated with the data in 2019. ‘Unfilled triangles’ represents the first 40 data points used for calculating the tolerance and action limit; ‘unfilled squares’ represents the 60 data points requiring monitoring analysis. The dashed lines represent the UCLs and LCLs and CLs of the control chart calculated from the first 40 data points; the solid lines represent the range of action limits calculated from the first 40 data points. The red-filled squares represent the data points that beyond the tolerance and action limit. UCL: upper control line, CL: central line, LCL: lower control line

The effect of individual operation constancy

A comparison of process control charts calculated by five therapists and one physicist shows, the range of UCL and LCL was smaller for output and the CL was closer to the target value of 0 for flatness and symmetry in the I-MR control chart in which data points were collected from the physicist (see Fig. 3).

Fig. 3.

The process control charts calculated using 30 sets data points collected by five therapists in turns and only one physicist.

DISCUSSIONS

The study of Sanghangthum et al. [13] found that the results become more consistent when the number of data points to calculate limits was increased. If the control line is too wide some errors will be omitted, however, if it is too narrow the error may be a false positive. Therefore, they recommend at least eight to 12 weekly QA data points (i.e. over two to three months) should be used for calculating limits, and they speculate that two to three weeks of daily QA data points may be needed. The same method was used in the current study, and it was found that at least six to eight weeks of control daily check data points (i.e. 30–40 points) should be used for calculating the I-MR control chart to ensure the stability of control lines. Due to the increase of uncontrollable risks during long-term operation of the quality control process, more data points may not be better [13]. Therefore, when using SPC to study the quality control process, the minimum number of calculation points required for the control line to be stable should be calculated first by analyzing the rule of changes of /σ.

When calculating the process capability index, the size of sampled data should be greater than 25 to be representative, while in this study, 40 data points were used based on the study result of control line stability. On analysis of the process capability indices, it was found that part of the output indices were up to 3–4, which were much higher than the A ++ (C_pk ≥ 2.0) level. Usually, if the value of C_pk is more than 2.0, it means the quality control process has reached an excellent level and cost reduction can be considered. But the result of this study may not mean that. Because the specification limit of this study was ±3% (as per TG-142), which is the minimum level that meets the clinical requirements. In order to improve the quality control level in this study, tolerance specifications needed to be stricter. In addition, it is not always true that the capability indices of transverse symmetry failed to meet the requirements in this study. Because an annual tolerance of ±1% (as per TG-142) is used for a three-dimensional water tank or a large parallel plate detector and the performance of this equipment are better than daily check equipment, so the requirement of annual tolerance may be stricter than daily tolerance [2]. Binny et al. [4] pointed out that transverse and axial symmetry measurement variations from baselines using the water tank were within ±1.5% and flatness variations from baselines were within ±0.5%. These findings explain why, for the same quality control process, in this study most process capability indices of symmetry did not meet the requirements while those of flatness did. Therefore, only by establishing a more suitable specification limit can the capability index be used to better evaluate the quality control process.

The tolerance limits were used in this study for Level 1 and 2 actions, as per the TG-142 report. Exceeding the tolerance limits indicates that the process may change, but observation can be continued or actions can be taken as appropriate. The action limits were used for Level 3 actions from the TG-142 report, and such immediate action should be taken if the action limits are exceeded. It can be seen that when the quality control process is not managed by the SPC method, the process may be significantly different even if the tested accelerator, quality control equipment and testing personnel remain basically unchanged (see Table 2). Furthermore, the calculation of the action limits is set considering the Taguchi-type process capability index C_pm = 1.33, the maximum value of A for the action limit shall not be greater than 6, which means that the process meets the 6σ management requirements; the smaller the value of A, the smaller the change in data and the better the process capability [20]. In the work of Sanghangthum et al. [20], for the weekly check process of linear accelerator output，where the measuring equipment is the morning check equipment RBA-3 (GAMMEX RMI Inc., Middleton, WI) and the monthly check process (where the measuring equipment is the finger-type ionization chamber), the chosen value of A for all the action limits was 3. The same method was used in this study, and the chosen value of A was 4–5. This indicates that the daily quality control capability of the accelerator needs to be improved.

The chosen value of A (A = 4) in 2019 was smaller than that in 2018 (A = 5) (see Table 3), so it indicated less variability in the data, and thus a better QA program in 2019. However, the tolerance and action limits in 2019 were larger than those in 2018. The reason is that the deviation of the mean values of data from the target values was relatively high in 2019, resulting in the value of the LCL being significantly higher than that of the UCL, and the tolerance and action limit are determined by the control line with a larger value. So, it is necessary to manage and adjust the quality control process using the SPC method before setting customized tolerance and action limits.

When using tolerance and action limits to monitor the quality control process, the system deviations in the process should be detected in time to ensure effective monitoring of changes in accelerator dose parameters. As seen from Fig. 2, it appeared there was a system error in the process from the 45^th data point. However, during the same period, axial flatness, transverse flatness, axial symmetry and transverse symmetry measurement results of MapCHECK2 are respectively 2.68 ± 0.13 (average ± standard deviation), 2.43 ± 0.07, 1.39 ± 0.27 and 0.60 ± 0.20, and it did not show systematic deviations when using SPC method to analyze.

It can be proved that the performance of linear accelerator is normal, so the system error may be caused by the worsening constancy of the detector of Daily QA3. After recalibrating the detector constancy, the system error was eliminated. When the morning check equipment has been used for many years, and the detector is severely aged, it is necessary to increase the frequency of calibration or replace it with new test equipment. It was also found that the range of tolerance limits of axial symmetry was greater than that in the transverse direction in 2019, which was consistent with the measurements of MapCHECK2. It can be concluded that this phenomenon is indeed the true state of the accelerator and can be effectively detected using customized tolerance and action limits.

Accuracy and precision were used to evaluate the daily quality control level. The smaller the deviation between the average value and the target value in the Shewhart chart, the higher the accuracy of the process; the smaller the range between the UCLs and LCLs, the higher the precision of the process. The results in Fig. 3 indicate that when the operation constancy is good, the daily quality control level is higher.

Limitations of this study: none of three groups in the quality control process were managed by the SPC method. Having a comparison could better explain the positive effect of the SPC method in the improvement of the quality control process. The authors intend to compare the improved quality control data with existing data in the next work and further study the positive effect of the SPC method.

CONCLUSION

In this study, for different quality control processes without management by the SPC method at our institution, calculated tolerance and action limits of the same measurement item were quite different. And in most measurement items, the specification limits used in the monthly or annual QA procedures in the TG-142 are not suitable to the daily QA procedure. So the analysis method using SPC is useful and necessary.

CONFLICT OF INTEREST

All the authors declare they have no personal, financial, commercial or academic conflicts of interest.

FUNDING

The work was supported by Shanxi Key Research and Development Program (201803D31172).