Application of an incident taxonomy for radiation therapy: Analysis of five years of data from three integrated cancer centres

Abstract

Aim

To develop and apply a clinical incident taxonomy for radiation therapy.

Background

Capturing clinical incident information that focuses on near-miss events is critical for achieving higher levels of safety and reliability.

Methods and materials

A clinical incident taxonomy for radiation therapy was established; coding categories were prescription, consent, simulation, voluming, dosimetry, treatment, bolus, shielding, imaging, quality assurance and coordination of care. The taxonomy was applied to all clinical incidents occurring at three integrated cancer centres for the years 2011–2015. Incidents were managed locally, audited and feedback disseminated to all centres.

Results

Across the five years the total incident rate (per 100 courses) was 8.54; the radiotherapy-specific coded rate was 6.71. The rate of true adverse events (unintended treatment and potential patient harm) was 1.06. Adverse events, where no harm was identified, occurred at a rate of 2.76 per 100 courses. Despite workload increases, overall and actual rates both exhibited downward trends over the 5-year period. The taxonomy captured previously unidentified quality assurance failures; centre-specific issues that contributed to variations in incident trends were also identified.

Conclusions

The application of a taxonomy developed for radiation therapy enhances incident investigation and facilitates strategic interventions. The practice appears to be effective in our institution and contributes to the safety culture. The ratio of near miss to actual incidents could serve as a possible measure of incident reporting culture and could be incorporated into large scale incident reporting systems.

1. Background

While radiation therapy has made impressive gains as a treatment modality for cancer, radiation therapy is not without risk and effective quality management remains essential. Incident reporting systems are invaluable for improving quality and safety; however, developing a suitable system for radiation therapy has been challenging.1, 2 The International Atomic Energy Agency (IAEA) developed a safety reporting and learning tool — Safety in Radiation Oncology (SAFRON) — that supports the voluntary reporting of radiotherapy incidents and near misses.³ In Europe, the Radiation Oncology Safety Information System (ROSIS) was established in 2001 as a voluntary, web-based reporting tool.⁴ A voluntary, web-based reporting tool that uses a comprehensive incident taxonomy has been developed in Canada.⁵ In the United States, recommendations for the content and structure of incident learning databases in radiation oncology have been made with the support of professional societies.² A national Radiation Oncology Incident Learning System (RO-ILS) was launched in 2014 by the American Society for Radiation Oncology (ASTRO) and the American Association of Physicists in Medicine (AAPM).6, 7 In Australia, the importance of a national incident monitoring system as a risk management and quality improvement tool has been recognised in the Australian Tripartite Radiation Oncology Practice Standards (ROPS)⁸ and the National Strategic Plan for Radiation Oncology 2012–2022.⁹ However, there is no uniform system for classification of clinical incidents although a number of paper-based and electronic incident reporting systems are in use.10, 11 Our efforts to meet the national standards at our regional centres have been reported previously.¹²

In New South Wales (Australia), incidents in cancer treatment centres are reported and managed using the Advanced Incident Management System (AIMS). Incident definitions in AIMS are necessarily broad as they apply to all patient services; the system lacks the native ability to capture radiation therapy-centric incident information such as dose level variation. Moreover, the four-step Severity Assessment Code (SAC), used for grading severity of outcomes from incidents, is incapable of adequately assessing the potential for future changes in toxicity arising from radiation incidents.

2. Aim

In the light of the above, we set out to develop a clinical incident taxonomy for radiation oncology (incident classification system) based in part, on the work of Arnold et al.¹³ and modified to align with the Australian radiation oncology practice standards. As in the US, it was felt that a database structure more closely reflective of local (Australian) practices would be desirable.² The aim was to develop a system, which could provide information about clinical incident trends in radiation therapy, that might support quality improvement.3, 5, 14 In particular, near-miss incidents form a valuable source of quality improvement information1, 3, 7, 15, 16, 17, 18 and this aspect was incorporated into the system outputs/learning tools.¹⁹ Here, we report on the application of the taxonomy and analysis of five years of incident data from three integrated cancer centres.

3. Methods and materials

3.1. Department characteristics

Our radiation oncology service comprises three centres, which opened between 2007 and 2010, in regional Australia. All staff can lodge incident reports electronically; near-miss reports are encouraged to aid learning and quality improvement. The majority of errors are detected, reported and investigated by experienced Radiation Therapists. Quality and safety initiatives have included revisions to planning quality assurance (QA) checks and audits against national radiation oncology practice standards.⁶ Staff have been encouraged to report imaging-related variations as near-miss incidents.

The present study covers the period from January 2011 to December 2015, during which 8432 courses of treatment were provided. The same period saw rapid development and expansion with additional linear accelerator (linac) capacity and staff, along with a range of new technology and practices. This included changes in intensity modulated radiotherapy (IMRT) delivery from step and shoot to dynamic and volumetric modulated arc therapy (VMAT)-based approaches, along with the implementation of a stereotactic body radiation therapy (SBRT) protocol.

3.2. Data collection and definitions

All incidents — including complaints, staff-related occupational health and safety, trips and falls as well as those from clinical causes — were recorded in the AIMS database in compliance with local and state health policy directives. Definitions of incidents and near misses also met this policy. True adverse events were defined as incidents that resulted in unintended treatment and, potentially, patient harm. A ‘near miss’ was defined as an actual incident that could have had adverse consequences but did not; this includes an arrested or interrupted sequence, where the incident was intercepted before causing harm.

Incident management processes did not include formal classification of dose error levels, however, investigations typically incorporated an examination of dose variance.

3.3. Local taxonomy classification system

Radiation therapy incidents were classified according to a locally developed taxonomy, based in part on the approach used by Arnold et al.¹³ and Lam et al.²⁰ The taxonomy consisted of 12 incident categories, each with a varying number of more specific subcategories (for details, see Supplementary Table 1). All investigations and coding activities were completed in AIMS. Incidents affecting a patient over multiple fractions or days were counted as a single incident per course. Quality improvement and feedback was conducted by local radiation therapy site managers and included a six-monthly audit of cross-site incidents, periodic review of codes, collection and analysis of data.

This study was reviewed by the Northern New South Wales Human Research Ethics Committee (reference: NCNSW HREC QA197) and was considered to be a quality improvement project.

3.4. Data analysis

Incident reports and investigation notes were extracted from AIMS, using customised reports (Supplementary Table 1). Overall incident rates were collected over five years and benchmarked against published data.²¹ Incident rates were based on the number of treatment courses each year. Incident data from 2011 was compared with 2015; in addition, the first two years were compared with the last two years. Treatment rates were compared over time, using the two-tailed Fisher's exact test. A P-value of less than 0.05 (two-tailed) was considered statistically significant.

4. Results

Incident rates are summarised in Table 1 and are benchmarked against published studies in Table 2. Treatment activity for the three cancer centres increased by 60%, from 1369 courses in 2011 to 1933 courses in 2015. A total of 720 incidents were recorded (Table 1), of which 566 (78.6%) were coded radiation therapy incidents. None of these incidents resulted in an adverse event. The lowest number of coded incidents recorded occurred in 2012 (75; range for the three sites 16–55), the highest number in 2013 (130). No incident involved a >5% radiotherapy dose deviation.

Table 1.

Incident rates by year for the three cancer centres (includes both total incidents and radiation therapy-coded incidents).

Year	XRT Courses	Workload growth rate	Incidents
			Total	RT codeda	AO	NAO	NM	% AO of total reported	Total	RT coded	AO	NAO
									Rate per 100 courses of RT
2011	1369	19.6%	141	115	10	69	62	7.09%	10.29	8.40	0.73	5.04
2012	1571	14.8%	114	75	16	22	76	14.04%	7.26	4.77	1.02	1.40
2013	1748	11.3%	148	130	23	29	96	15.54%	8.47	7.44	1.32	1.66
2014	1811	3.6%	144	124	25	62	57	17.36%	7.95	6.85	1.38	3.42
2015	1933	6.7%	136	122	15	51	70	11.03%	7.04	6.31	0.78	2.64
Total	8432	11.2%	720	566	89	233	361	12.36%	8.54	6.71	1.06	2.76

AO, adverse outcome; NAO, no harm adverse outcome; NM, near miss incident; RT, radiation therapy.

^aClassification system (taxonomy) is described under Section 2 and summarised in the Supplementary Table 1.

Table 2.

Comparison of the incident rates of the present study and the literature.

Study/Author	Time period	Adverse incidents per 100 courses	Near miss incidents per 100 courses (Calculated)	Incidents (all) per 100 courses	Scope of reporting/comments
Macklis et al.29	1995	1.8	NR	3.6	59 treatment related errors. Shielding block errors most common (30%)
Calandrino et al.32	1991–1996	0.45	NR (3.05)	3.5	Limited to monitor units calculation only
Fraass et al.27	1996–1997	1.2	NR		Treatment processes only. Setup and accessory errors most common
Huang et al.28	1997–2002	1.97	NR		All errors
Yeung et al.38	1992–2002	1.9	(2.74)	4.7	Documentation and set up errors
Arnold et al.11	2004–2007	3.9	13.6		All phases of RT processes
Bissonnette and Medlam25	2001–2007	2.0 (2001) 1.2 (2007)	0.56 (2004) 1.37 (2007)		Over 6-year period
Chang et al.26	2001–2011	2.64			Introduced “e” reporting during 10 year study period
Mutic et al.14	2007–2009	NR	NR	60	Event severity: low: 69.9%, medium: 22.6%, high: 7.7%, severe: 0.1%
Current study (2017)	2011– 2015	1.056 (adverse) 2.76 (no Harm)	4.28	8.54 (all) 6.71(RT)	Incidents cover entire RT experience

NR, not reported; RT, radiation therapy.

Across the five years the total incident reporting rate was 8.54 incidents per 100 courses and 6.71 per 100 courses for radiation therapy specific coded incidents. The rate of true adverse events (unintended treatment and potential patient harm) was 1.06 adverse events per 100 courses. Adverse events, where no harm was identified, occurred at a rate of 2.76 per 100 courses. Overall and actual rates both exhibited a downward trend over the 5 year period.

Total and radiation therapy coded incident rates were relatively stable, averaging 136.6 and 113.2 reports respectively per year. Despite increases in workload of 60% over the study period incident reports grew at an average of 11.19% per year.

Incident numbers by incident category, for the combined centres, are shown in Fig. 1. A high proportion of incidents were noted at the treatment stage. A relatively high number of incidents (103) were imaging related, of which 18% were also associated with a failure in QA processes, i.e. use of two or more codes recorded in the incident investigation (Fig. 2). This is slightly lower than treatment related QA failures, which were recorded at a rate of 25%. Over time the use of the “other” imaging category grew, while other defined imaging codes trended downwards.

Fig. 1.

Incident numbers by year and incident group for the three cancer centres. Incidents relating to “consent” are included under “documentation”; incidents relating to “Positioning” are included under “treatment”.

Fig. 2.

Incident imaging codes for the three cancer centres: 2011–2015.

4.1. Incident rates assessed over time periods

Comparisons of first and last years (2011 and 2015) and also first two (2011, 2012) and last two years (2014, 2015) are shown in Table 3. Incidents per 100 courses were somewhat lower in 2015 (7.04) than in 2011 (10.29) while the difference in rates of adverse incidents was minimal (0.71 in 2011, 0.78 in 2015). The rates of no harm adverse outcomes (NAO) and near misses (NM) tended to be lower in 2015; the rate of near misses was significantly lower in 2014–2015 than in 2011–2012 (3.38 vs 4.68; P = 0.008). The ratio NM/(AO + NAO) was somewhat lower in 2014–2015 than in 2011–2012 (0.86 vs 1.39).

Table 3.

Changes in incident rates between 2011 and 2015 (reflecting changes in technology). The ratio NM/(AO + NAO) is proposed as a measure of a “safety culture”.

Year	All incidents	AO	NAO	NM	Ratio NM/(AO + NAO)
		Rate per 100 courses			All sites	Sites 1 & 2a
2011	10.29	0.71	5.04	4.53	0.79	1.18
2015	7.04	0.78	2.64	3.62	1.06	1.14
2011–2012	8.78	0.87	3.22	4.68	1.39	1.48
2014–2015	7.49	1.08	3.03	3.38	0.86	1.25
Pb	0.077	0.534	0.886	0.008	NA	NA

AO, adverse outcome; NAO, no harm adverse outcome; NA, not applicable; NM, near miss incident.

^aExcludes site 3 due to confounding factors.

^bYears 2011–2012 compared with 2014–2015.

5. Discussion

Incident reporting has been recognised as an important safety tool.1, 6, 7, 14, 16, 17, 18, 22 Typically, reports in relation to radiation therapy have examined detailed causative factors, while assuming the presence of a good safety culture. Analysis of incident reports is based on confidence that sufficient representative reports are captured of events not sought. Incidents are generally undesirable and pressure to achieve low rates may lead to systematic underreporting. When combined with the tendency to “fix and forget”,²³ many hazardous events are addressed but potentially go unreported, leaving the “system” none the wiser. Establishing a quality culture that encourages no-blame incident reporting, on actual and near-miss incidents, is therefore crucial.²⁴ Some authors have taken this further, introducing “no fly” safety practices, supported by checklists.²⁵

Many authors have examined the nature of radiation therapy errors6, 13, 18, 26, 27, 28, 29, 30, 31 and while this is valuable, we are of the opinion that additional value can also potentially be derived from considering the organisation's safety culture.19, 24 Therefore, we have looked broadly at incident reporting and have considered the propensity to report near-miss incidents. We suggest that a measure of safety culture can potentially be found in the ratio of near-miss events to actual incidents. Holmberg and McClean¹⁵ suggested that a high ratio of near misses per actual error demonstrates an effective error prevention system, independent of the quality of the initial treatment preparation. In their study, the ratio was approximately 14. In our study, the best safety culture was possibly achieved during 2012–2013, when the number of near-miss reports to actual incidents averaged 1.9 (data not shown). While differences in the state of technology between the two studies may be partly responsible for the differences in the ratio, it is also possible that improvements in safety systems in the intervening years are a factor. Our regional cancer centres have implemented a robust quality assurance system³² inclusive of incident management practices. Given the rates of incident reporting are high and the rate of adverse incidents are low, this would support the hypothesis that centre size is no barrier to safety and that smaller regional practices can be safe.³³

The rates of radiation therapy related incidents reported here compare favourably with previous studies (Table 2). Chang et al.²⁸ reported a rate of 2.64 incidents per 100 courses over a 10-year period from 2001 to 2011. Interestingly, the rate of incidents increased with the implementation of an electronic reporting system, implying perhaps that fewer incidents went unreported. Our experience had no such transition period, with electronic reporting established well before the study period. Ota et al.²¹ reported on 1063 incident reports with an average incident rate per 100 RT courses of 1.7 ± 0.4; when excluding near misses, their rate fell to 1.4 ± 0.3. The relatively small difference between the two rates could indicate a poorer reporting culture around near-miss events.

The work by Bissonnette and Medlam²⁷ is perhaps contextually closest to our work, although their facility is one of the largest clinics at a single site in North America. Their examination of the effect of implementation of IGRT on incident rates appears to have, over time, contributed to reduced incident rates, which declined from 2.0 to 1.2 per 100 courses between 2001 and 2007, a decrease of 60%. The overall average rate of actual incidents was 1.38 ± 0.33 per 100 courses. Our experience is similar, although IGRT has been a constant and evolving component of our practice. We agree with Bissonnette and Medlam that wide-spread introduction of IGRT helped reduce the rate of actual (i.e., non-near miss) incidents. Interestingly, they collected few near-miss reports initially but rates increased from 0.56 per 100 courses in 2001 to 1.37 per 100 courses by 2007. These rates are low compared to our own. When the rate of near-miss reports is greater than the rate of actual incidents, we believe we can infer a strong safety culture, however we acknowledge there may be many confounding factors, which may make this difficult to prove.

The high rates of pre-treatment coded incidents demonstrates the effective use of QA checks prior to treatment. These were exclusively near-miss incidents, where one check failed to detect a problem, which was subsequently detected at later stages of QA. Incidents of most concern are those that passed multiple QA checks and were identified at the treatment stage. Incidents that occur over multiple fractions of treatment could be considered to be the most serious and most likely to result in significant harm. This was rarely seen in our data, and we have not sought to identify or categorise these events separately.

As in earlier studies,7, 13, 27, 28, 29, 30, 31, 34 most incidents were reported in the treatment stages. We observed a trend — perhaps not observed as strongly previously — where, over time, increasing rates of incidents were coded with an imaging-related causative factor. Imaging was also associated with many near-miss reports. We found our imaging incidents taxonomy lacked the granularity to identify new error pathways (Fig. 2). These incidents were predominantly repeated scans, due to software or hardware failures of the imaging system.

The apparent growth in the “other” imaging category, illustrated in Fig. 2, demonstrates both the benefit and shortcomings of a comprehensive and detailed taxonomy. Reporting categories were based on clinical experience and knowledge of probable errors – these categories were therefore directed towards capturing “known” incidents. The reporting process led to safety interventions, which resulted in a decline in the incident rates (as demonstrated in Fig. 2). At the same time, errors arose which were not predicted – the “unknowns”, captured in the “other” category. It may be expected that as patterns in these “new” incidents emerge, the incidents would be categorised and specifically targeted in future safety interventions. Thus, the growth in the “other” category is not a flaw in the taxonomy but rather a sign of an active quality improvement cycle.

The present study comes off an existing base of “high technology” and therefore does not replicate Marks’ study of 2007,³⁵ which examined the deviation rates among patients treated on “high-technology” versus “low technology” machines (defined as those with or without a multi-leaf collimator). The current study examined the general effect of quality improvement efforts inclusive of technology. Major changes in our centres during the study period, which could influence incident rates, included a 60% increase in workload, along with an increase in the scope of complex treatment delivery modalities provided.35, 36, 37, 38, 39 During the five years under review a range of new capabilities was introduced, such as four-dimensional computed tomography, IMRT lung, VMAT and SBRT. The role of image-guidance radiotherapy was also expanded. It could be suggested that as the organisation matured, the ratio of near-miss to actual incidents grew, while the overall incident rate declined, suggesting that a safety culture prevailed and was enhanced, even as adverse incident reports decreased. The reduction in incidents from 2013 may represent the result of sustained, deliberate efforts by management to provide feedback to staff and to focus attention on reporting incidents especially those involving imaging tasks.²⁶ However, there may also be other ‘technical’ reasons for the reduced incident rate, such as reporting fatigue — due to protracted technical issues — and improved procedural protocols and quality improvement activities generally.

6. Conclusion

The taxonomy was useful for capturing data on clinical incident trends in radiotherapy that support quality improvement and we recommend ongoing refinements to any taxonomy, in order to accommodate new error pathways. The measure of a safety culture that we have proposed is the ratio of near-miss to actual incidents. Despite the various limitations of such a metric, we believe that the process of seeking out and identifying a measure of safety culture is, in and of itself, a positive contributor to the creation of that culture. Further research on the value of using the metric as a marker of safety culture is recommended.

Conflict of interest

None declared.

Financial disclosure

None declared.

Appendix A. Supplementary data

The following are the supplementary data to this article: