Determining the impact of pre-radiation treatment verification simulation/dry run by analyzing intradepartmental reported incidents and surveying staff and patients

Abstract

Purpose:

Error identification in radiation therapy is critical to maintain a safe and efficient therapeutic environment. A verification simulation (VS; also called a dry run for patient information) provides a dedicated time prior to treatment to duplicate steps of patient setup, imaging, and treatment process as a final quality assurance step. Through the use of surveys and analysis of reported incidents, we sought to determine the value of a VS before initiating patient treatment.

Methods and materials:

In November 2014, a VS was instituted across our network of 11 radiation oncology clinics. A comparison of the incident rate reported through our departmental incident learning system (ILS) was made between a non-VS group (965 patients who were treated in the 18 months prior to instituting the VS) and a VS group (984 patients who were treated over 18 months with the VS policy in place). From August to December 2016, surveys were completed by 211 patients and 55 physicians, nurses, and therapists detailing their perspectives on the VS.

Results:

There were 28 incidents (2.9%) in the non-VS group compared with 18 incidents (1.8%) in the VS group (P = .03). In the VS group, more incidents were detected before the day of treatment (P = .03) and fewer incidents on the day of treatment (P = .02). In addition, a trend toward fewer incidents after treatment started (P = .09) was observed. Patient surveys indicated that 99.5% of patients were informed of the VS, 83% reported decreased anxiety during treatment, and 5% indicated concerns about delaying treatment. The majority of staff members (67%) were satisfied with the VS.

Conclusions:

A VS helps identify and correct incidents before the administration of radiation therapy and reduces patient anxiety.

Introduction

Systems that deliver therapeutic radiation are complex and therefore can be error-prone. Many staff members are involved in treatment planning and delivery, which requires a workflow that relies heavily on effective communication. There is a technical aspect of radiation delivery that stems from intricate treatment machines and sophisticated planning software. The reliance on this technology and the perils associated with its malfunction were highlighted in a New York Times article entitled “Radiation offers new cures, and ways to do harm.”¹ The authors emphasized that these malfunctions could have been detected by radiation oncology staff members and thereby could have prevented serious accidents. Thus, a system that lends to the detection of both mechanical and human errors is essential.

The system relies on an efficient workflow that has safety barriers in place and adherence to the workflow by the staff is important to mitigate errors. However, since errors are typically propagated via system failures rather than individual negligence, a movement has emerged toward nonpunitive voluntary reporting of errors, which is highlighted by incident learning systems (ILS).^2,3 In this type of environment, incidents are used to improve the quality of care and continually advance these workflows.

The purpose of our study was to assess whether a verification simulation (VS; also called a dry run for patient information) that is integrated into the radiation therapy workflow improves the safety and quality of radiation therapy. We analyzed departmental incidents and performed networkwide surveys that were aimed at eliciting feedback on the VS from radiation oncology staff members and patients.

Methods and materials

In an effort to improve the system and mitigate errors, a VS was established in November 2014 throughout our radiation oncology network, encompassing 11 clinics in western Pennsylvania and eastern Ohio. A VS is a session that typically occurs the day before the first fraction of radiation therapy is delivered and duplicates the steps of patient setup, imaging, and treatment without actually administering the radiation. This session is designed to allow staff time to verify that the parameters of treatment are accurate and troubleshoot problems in an organized team approach. If a patient needs to urgently initiate treatment, similar quality verification occurs shortly before the delivery of treatment; however, it is not defined as a VS.

Incident learning system

After an institutional review board determined the study’s exempt status, we performed a retrospective review of patient-related incidents that were documented throughout our departmental ILS. Incidents are defined per the consensus recommendations for incident learning database structures in radiation oncology as “an unwanted or unexpected change from a normal system behavior, which causes or has the potential to cause an adverse effect to persons or equipment.”⁴ Voluntary reporting is encouraged by our department and incidents are reviewed by the quality improvement committee every 2 weeks. The reviews are nonpunitive, and through feedback and troubleshooting, system improvement is accomplished.

We reviewed incidents that occurred during the 18 months preceding the implementation of the VS and the 18 months after. Incidents were categorized into detection before treatment, on the first treatment day, and after the start of treatment. We grouped incidents by: 1) radiation prescription, 2) patient position and immobilization, 3) bolus and shielding placement, 4) quality assurance (QA) documentation, and 5) localization imaging. Radiation therapists (RTs) represent a final checkpoint in the detection of errors before the administration of radiation; therefore, we classified incidents on the basis of whether they were detected by an RT during this final verification.

Questionnaires

Patient and staff surveys (Figs 1 and 2, respectively) were conducted and accompanied by a cover letter that emphasized that the surveys were voluntary and anonymous. The patient survey started in August 2016 and concluded after 200 patients had completed the survey in December 2016. The patient survey included 8 “yes or no” questions and focused on whether or not the patient felt informed throughout the VS and if their mindset was affected by the VS. The surveys were distributed to patients at the treatment machines either on the VS day or after they had started radiation treatment. Patients who received brachytherapy procedures, urgent starts, and framed stereotactic radiation surgery were excluded from the study because a VS was not performed in these instances. The staff members who were surveyed included the radiation oncologists, nurses, and RTs and were given the option to provide suggestions on how to improve the VS. The authors of this paper did not participate in the surveys.

Figure 1.

Anonymous patient survey.

Figure 2.

Anonymous physician and therapist survey.

Statistical analysis

The non-VS group included 965 patients who were treated in the 18-month period prior to the institution of the VS and the VS group included 984 patients who were treated in the initial 18 months after the VS was implemented. SPSS version 20 was used for the descriptive statistics and comparative analysis, primarily via the χ² test. When analyzing the staff responses on the value of the verification simulation, Bonferroni correction was used given the multiple comparisons. Compliance with the patient VS surveys was calculated as the ratio of returned surveys to VS over a set time period of the study.

Results

Incident learning system

A total of 28 incidents were reported among the 965 patients in the non-VS group for a 2.9% incident rate per patient treated compared with a 1.8% rate (18 incidents for 984 patients) in the VS group (odds ratio: 2.06; P = .03). Therefore, an estimated 93 patients needed to be treated in the absence of a VS to result in 1 additional error. No errors required state or federal reporting; however, 1 error that occurred in the non-VS group resulted in a hospital incident report. In the VS cohort, more incidents were detected before the day of treatment (P = .03) and less incidents on the day of treatment (P = .02). A trend was observed toward fewer errors after the treatment started (P = .09). Notably, there was a nonsignificant difference in the number of incidents on or before the first treatment day in the non-VS group compared with the VS group. RTs detected 44% of the incidents. In the non-VS group, 9 of 28 incidents (32%) were detected by RTs compared with 11 of 18 incidents (61%) after starting the VS (odds ratio: 3.67; P = .04). A complete list of the number of errors, what caused the errors, and the timing of detection is shown in Table 1.

Table 1.

Quantitative computed tomography scans categorized by when they were identified in patients’ treatment course and by error type

Error type	Before day of treatment	Day of treatment	Any other treatment day	Total
Before dry run
Prescription	1	3	5	9
Position	0	4	2	6
Bolus placement	0	1	2	3
Image	0	6	3	9
QA documentation	0	0	1	1
Total	1	14	13	28
After dry run
Prescription	1	0	4	5
Position	4	1	0	5
Bolus placement	0	0	0	0
Image	2	1	0	3
QA documentation	2	1	2	5
Total	9	3	6	18

QA, quality assurance

Questionnaires

The compliance rate of the patient surveys was 53% with a total of 211 patient surveys returned. Patient surveys indicated that 99.5% of patients were informed of the VS and 68% were informed by more than 1 person or source. The majority of patients (68.7%) knew that the VS would occur before entering the department on the VS day. Patients listed the following as informants to the purpose of the VS: physician (n = 150; 71.1%), therapist (n = 146; 69.2%), and nurse (n = 130; 61.6%). Of those who listed only 1 person, the patient’s therapist was most often the source (n = 42; 62.6 % of those reporting a single source). A minority of patients listed the following as sources of information: informed by the front desk staff (n = 23; 10.9%), informed by friends and family (n = 5; 2.4%), informed by the Internet (n = 1; 0.5%), and informed by other patients (n = 1; 0.5%). The staff surveys reflected that the majority of staff members indicated that they always educate patients about the VS (Table 2).

Table 2.

Staff members’ perspectives on education of patients about dry run

	Never	Sometimes	Mostly	Always
Physician (n = 15)
Do you educate patients	1 (7%)	4 (27%)	4 (27%)	6 (40%)
Are patients well informed	0	3 (20%)	8 (53%)	4 (27%)
Radiation therapists (n = 27)
Do you educate patients	0	1 (4%)	3 (11%)	23 (85%)
Are patients well informed	1 (4%)	2 (7%)	9 (33%)	15 (56%)
Nurses
Do you educate patients (n = 13)	1 (8%)	4 (31%)	2 (15%)	6 (46%)
Are patients well informed (n = 12)	0	1 (8%)	7 (58%)	4 (33%)

The majority of patients reported that the VS lessened their anxiety and concerns about radiation treatment (n = 175; 82.9%), and 96% affirmed that they were able to familiarize themselves with the department. When adjustments needed to be made during the VS, 94% responded that staff members explained the adjustments. Only 5% of patients were concerned there would be a delay of treatment (n = 11).

The compliance rates were 94% (n = 15), 93% (n = 13), and 90% (n = 27) for the physician, nurse, and therapist surveys, respectively. Table 3 outlines the physicians’, nurses’, and therapists’ perspectives of the VS in assuring correct treatment delivery, allaying patient’s fears or concerns about treatment, evaluating patient’ s pain or discomfort in the treatment position, and identifying errors or potential errors. RTs reported that they believed the VS was more valuable in assuring correct treatment (P < .01), identifying potential errors (P < .05), and allaying patient fears (P < .01) when compared with physicians.

Table 3.

Staff members’ opinions on aspects of the dry run

	Not at all valuable	Not very valuable	As expected	More valuable	Much more valuable
Physician (n = 15)
Assuring correct treatment	0	1 (7%)	9 (60%)	5 (33%)	0
Allaying patient's fears	2 (13%)	1 (7%)	10 (67%)	2 (13%)	0
Evaluating patient's pain	1 (7%)	0	10 (67%)	4 (26%)	0
Identifying potential errors	0	0	10 (67%)	4 (26%)	1 (7%)
Radiation therapists (n = 27)
Assuring correct treatment	0	2 (7%)	10 (37%)	4 (15%)	11 (41%)
Allaying patient's fears	0	4 (15%)	8 (30%)	6 (22%)	9 (33%)
Evaluating patient's pain	1 (4%)	2 (7%)	11 (41%)	8 (30%)	5 (19%)
Identifying potential errors	0	0	9 (33%)	7 (26%)	11 (41%)
Nurses (n = 13)
Assuring correct treatment	0	0	7 (54%)	2 (15%)	4 (31%)
Allaying patient's fears	0	1 (8%)	4 (31%)	5 (38%)	3 (23%)
Evaluating patient's pain (n = 12)	0	0	3 (25%)	6 (50%)	3 (25%)
Identifying potential errors (n = 12)	0	0	6 (50%)	0	6 (50%)

All staff members believed that the VS performed as expected or better in identifying potential errors. The majority of RTs and nurses believed that the VS always allowed for troubleshooting of treatment delaying issues in contrast to physicians who reported that the VS was only sometimes helpful. The majority of physicians were not sure if their recognition of errors improved with experience with the VS (7% yes vs 40% no, with 53% not sure) while the majority of therapists believed their error recognition improved (63% yes vs 15% no, with 22% not sure). Overall, most staff members (67%) were satisfied with the VS.

Discussion

The VS is a workflow quality assurance (QA) strategy that can help reduce human and mechanical errors. Dedicating time and resources to a VS promotes a culture of safety and has been shown to improve the patient’s experience. ⁵ In addition, a VS allows for interactive patient feedback, which enables autonomy in the treatment course.⁶

QA is essential to the complex delivery of safe and effective radiation therapy. We analyzed a VS by reviewing anonymously reported incidents before and after the implementation of our VS policy. In our study, the overall frequency of reported incidents was low (2.4% of 1949 total patients). After the VS was implemented, contrary to our expectations, the overall incident rate decreased. We suspect this may be because staff members were not reporting incidents because they were being addressed during the improved VS workflow and did not lead to the propagation of an error through the remaining workflow. This is supported by the VS-group trend of identifying incidents earlier and before treatment is under way. In addition, the efficiency of catching and mitigating incidents prior to treatment initiation cannot be understated because interruptions in treatment have been shown to be detrimental to cancer control outcomes in numerous studies.⁷ This error mitigation comes without added complexity or burden because the VS is simple and efficient.

Our study demonstrates the importance of RTs in error detection and providing sufficient time to perform QA. Nearly half of all of incidents were detected by RTs, with a significantly greater percentage of incidents detected by RTs after starting the VS. We attributed this to the allotment of dedicated time for RT-driven QA prior to the implementation of treatment, which has shown to be beneficial.⁸ Seventy-eight percent of RTs reported a favorable perception of the VS compared with 57% of physicians and nurses who reported a favorable perception. Delays that are caused by alterations in patient set up or imaging verification significantly disrupt the work flow and patient convenience. Therefore, allowing time for alterations during the VS mitigates potential disruptions and is one explanation for this discrepancy in valuation between RTs and clinical staff members.

Effective, patient-centered communication enhances trust in the treatment team and predicts successful outcomes. ⁹ It is important to set patient expectations for the day of the VS. During a VS, patients are informed about the steps in radiation therapy and the reasons behind them. In addition, patient comfort in the treatment position can be assessed in a predictable fashion and managed proactively.¹⁰ A VS allows for a controlled environment for patient counseling and troubleshooting patient compliance with treatment setup.

An unforeseen benefit of the VS was that 96% of patients stated that the VS enabled them to become familiar with the department and 82.9% expressed that their anxiety was lessened after having their VS. Anxiety at the start of any radiation therapy treatment course is common and underrecognized.¹¹ The VS reinforces to patients that safety is important and allows them to become accustomed with the daily routine of treatment.

Treatment-related anxiety and psychologic distress have been shown to contribute to fatigue, side effects, and decreased quality of life.^12,13 RTs represent the first line of screening for treatment-related anxiety due to their intricate and persistent relationship with patients. Accordingly, 85% of RTs perceived the VS as effective to alleviate patient fears. Halkett et al. showed that RT-led counseling prior to radiation therapy was effective to alleviate anxiety in a prospective controlled trial.¹⁴ Perhaps effective communication and subtle coaching during a dedicated VS day functioned to decrease anxiety.¹⁵ Mitigating anxiety through the VS was an unintended but welcome by-product.

Criticisms of a dedicated VS include delays in treatment, patient inconvenience, and cost. In our survey, only 5% of patients indicated concerns about the delay of treatment. Currently, no publications that analyze the cost effectiveness of a VS exist and further quantification is warranted. The limitations of this study include the biases of retrospective review and subjective incident reporting. In addition, the non-VS patient group was not surveyed; thus, the impact on decreased anxiety or department familiarity in the VS group may be overestimated given that these may be common feelings after the first treatment regardless of the presence of a VS.

Conclusions

A VS can be an integral part of any radiation oncology QA program and a risk-reduction strategy in the administration of radiation therapy. The VS decreased errors globally and changed the pattern when potential incidents were detected, which resulted in a more efficient and uninterrupted treatment course. Furthermore, the VS is a patient-centric QA activity that has the potential to decrease treatment-related anxiety and improve patient satisfaction. Given our positive findings, we recommend that radiation departments consider adopting a VS or dry run.