Automated Plan Checking Software Demonstrates Continuous and Sustained Improvements in Safety and Quality: A 3-year Longitudinal Analysis

Abstract

Purpose:

Perform a longitudinal analysis of the performance of our automated plan checking software by retrospectively evaluating the number of errors identified in plans delivered to patients in three, month-long, data collection periods taken between 2017 and 2020.

Methods:

11 automated checks were retrospectively run on 1169 external beam radiotherapy treatment plans identified as meeting the following criteria: planning target volume (PTV) based multi-field photon plans receiving a status of treatment approved in either March 2017, March 2018, or March 2020. The number of “passes” (true positives) and “flags” were recorded. “Flags” were sub-categorized into “false negatives”, “false negatives due to naming conventions”, or “true negatives”. 2×2 contingency tables using a two-tailed Fischer’s exact test were utilized to determine whether there were nonrandom associations between the output of the automated plan checking software and whether the check was manual or automated at the original time of treatment approval.

Results:

A statistically significant decrease in flags between the pre- and post-automation datasets was observed for four contour-based checks, namely “adjacent structures overlap”, “empty structures and missing slices”, “overlap between body and couch”, and “laterality”, as well as a check that determined whether the plan’s global maximum dose was within the planning target volume. Review of the origins of false negatives was fed back into the design of the checks to improve the reliability of the system and help avoid warning fatigue.

Conclusions:

Periodic and longitudinal review of the performance of automated software is essential for monitoring and understanding its impact on error rates as well as for optimization of the tool to adapt to regular changes of clinical practice. The automated plan checking software has demonstrated continuous contributions to the safe and effective delivery of external beam radiotherapy to our patient population, an impact that extends beyond its initial implementation and deployment.

INTRODUCTION

The physics pre-treatment plan check is the most effective individual quality control check for detecting high severity radiotherapy (RT) incidents¹. Moving from manual to automated checks is expected to lead to higher rates of error detection². Our automated tool for physics pre-treatment plan checks developed at the Uni-versity of Michigan^3,4 and later co-developed with with Memorial Sloan Kettering Cancer Center⁵, has been shown at both institutions to decrease error and increase plan checking efficiency. Other studies have similarly reported a decrease in error rates when introducing automated plan checks (APC’s)^6–9. The recently released AAPM Task Group 275 Report states that automation holds great promise for both efficiency and effectiveness of physics pre-treatment plan checks. However, they also caution that if such automated tools are not properly tested or implemented, errors could be systematically unidentified¹⁰. Limiting testing to a thorough set of pre-clinical use cases is insufficient when considering that these tools will be used for a variety of treatment techniques on a heterogenous patient population with a user base having varying levels of training and expertise. This is especially true when considering the evolution in treatment complexity, clinical workflow, and departmental policy and procedures over time. Therefore, continual post-deployment surveillance of the clinical utilization and performance of APC’s is of paramount importance to ensure that systematic errors or unintentional behaviors do infiltrate into the RT planning process.

Investigators have measured the impact of their APC’s using data from incident learning systems (ILS’s)^6,9 or by having human plan checkers manually record identified errors^3,5. However, while voluntary reporting ILS’s are essential components of a radiation oncology quality control program, the data is often inconsistently entered^11–13. This can be confounding when evaluating the efficacy of a safety intervention, such as an APC. Therefore, we performed a longitudinal analysis of our APC’s performance by retrospectively evaluating the number of errors that exist in plans actually delivered to patients relative to the date of deployment of a given automated check. We focused on errors that are fundamental to safety, e.g. correct organ-at-risk (OAR) laterality, so as to ensure that they were intended to have been detected regardless of whether the plan was to be checked manually by a human or by an APC.

METHODS:

Our APC is written in C# and utilizes the Eclipse v13.6, and later v15.5, (Varian Medical Systems, Palo Alto, CA) Applications Programming Interface (API), supplemented with database queries in circumstances where data is not provided by the API. It consists of a graphical user interface (GUI) that displays the results of the automated checks and a configuration file that defines which checks to run and the rules and passing criteria as a function of anatomical site and institution³. In our clinical workflow, the planner is expected to run the APC in the time point between the MD review and approval of the plan and submission to the physicist for an initial chart check. Although not required, planners typically run the tool a few times throughout the planning process so that they can identify and resolve flags as they go along in order to minimize the need to undo or repeat any of their work. Planners are expected to evaluate and resolve all flags. If there is a valid reason for which a flag cannot be resolved the planner is instructed to enter a note into the interface which gets saved for subsequent users to review. The physicist performing the initial chart check will also independently run the APC as an essential component of their procedure. They, likewise, are instructed to not approve the plan for treatment with unresolved flags. For this analysis, a batch processing mode was utilized which uses a query to create a list of patients, courses, and plans onto which the APC will be run in the background and the results saved as an XML file.

Table 1 lists the 11-automated checks under investigation, a short description of their behavior, and the date of their initial release relative to the retrospective data mining interval. These specific checks were chosen because they were performed manually prior to the introduction of automation and they are fundamental to our practice, meaning that the rules of the checks would not change over time. Data was collected in June and July of 2020 by identifying all external beam radiotherapy treatment plans in the Eclipse database meeting the following criteria: planning target volume (PTV) based multi-field (>= 3) photon (6X, 15X, 16X) plans that received a status of treatment approved in the record and verify system during one of the respective month-long data collection period time points: March 2017, March 2018, March 2020. Only one plan per patient simulation scan was included in the analysis. As this was a retrospective collection of cases that had already been treatment approved and treated, any identified flags were present in the plan delivered to the patient.

Table 1:

Automated Plan Checks Under Investigation and Timeline of Data Collection

	Name	Description – verifies that:	Date of Initial Release	Output	Pre-Automation	Post-Automation	p-value
1.	Bolus Material Check	Bolus structures have Hounsfield Unit override of 0.	2/19/16 (in initial release)	Flags	0	0	1.000
				Passes	0	1169
2.	Verify Dose Rate	Dose rate per beam is the expected value.	2/19/16 (in initial release)	Flags	0	0	1.000
				Passes	0	1169
Mine data from plans treatment approved between 3/1/2017 – 3/31/2017
3.	Prescribed Percentage	Prescribed percentage is 100%	7/30/2017	Flags	0	0	1.000
				Passes	326	843
4.	Hot Spot in PTV	Global maximum hot spot is contained within a PTV.	7/30/2017	Flags	14	10	0.0019
				Reports	312	833
5.	Adjacent Structures Overlap	Structures pre-defined as known to overlap (e.g. cord/cauda, cord/brainstem, stomach/esophagus) do overlap on at least 1 CT slice.	12/18/2017	Flags	99	51	<0.0001
				Passes	227	792
6.	Empty Structures / Missing Slices	Empty structures are deleted and no segmentations skip slices unless pre-defined as permitted to do so.	12/18/2017	Flags	148	87	<0.0001
				Passes	178	756
7.	Overlap between Body and Couch	Inserted couch models do not overlap with the patient body contour.	12/18/2017	Flags	10	2	<0.0001
				Passes	316	841
8.	180E Used When Appropriate	Gantry approach to 180 degrees (i.e. clockwise vs counter-clockwise) is most efficient. Note: limited to fixed gantry deliveries.	12/18/2017	Flags	2	0	0.0776
				Passes	324	843
Mine data from plans treatment approved between 3/1/2018 – 3/31/2018
9.	CT slice thickness	The slice thickness is less than or equal to the expected value.	5/10/2018	Flags	0	1	0.4055
				Passes	695	473
10.	Verify Primary Collimator Not Exposed	Jaws are not open such that the primary collimator would shadow the intended irradiated field. Note: Varian linac jaw settings can go to 40x40 cm2 but the primary collimator is circular and can shadow the field in the corners for very large field sizes.	5/10/2018	Flags	4	0	0.1513
				Passes	691	474
11.	Laterality of paired organs	Laterality is correct relative to the patient’s orientation for structures identified as a pair or in pre-defined list of structures known to have a laterality (e.g. parotid, eye, lung).	5/10/2018	Flags	114	41	0.0001
				Passes	581	433
Mine data from plans treatment approved between 3/1/2020 – 3/31/2020

A description of each check as well as when it was introduced into clinical use relative to the three retrospective data collection periods. The first two checks were in the initial clinical release, and therefore all of the data related to these checks was labelled as “post-automation”. Checks 3–8 were released after the first data collection period but prior to the second data collection period. Checks 9–11 were released after the first two data collection periods but prior to the final data collection period. P-values in bold indicate that the relationship between the relative number of flags versus passes in the pre-automation and post-automation datasets were statistically significantly different (p < 0.05) for a given check.

The XML result files were imported into Excel (Microsoft corporation, Redmond WA) for analysis and the number of “reports”, “flags”, and “passes” were recorded for each check. “Flag” was further sub-categorized. “False negative” denoted flags related to a fault in the check’s logic or configuration rather than something incorrect with the patient study itself. “False negative due to naming convention” denoted flags thrown because the user hadn’t followed the appropriate pre-defined and agreed upon institutional naming conventions for structures, plans, and reference points. If they had done so, the status would have been “pass”. “True negative” denotes that PCT found a real issue with the patient plan. Reports, flags, and passes were categorized as “pre-automation” if the dataset was from a period before the automated version of that specific check was clinically released. Similarly, “post-automation” refers to data collected from the time period after the automated version of that specific check was released.

2×2 contingency tables using a two-tailed Fischer’s exact test were utilized to determine whether there were nonrandom associations between the output of the APC and whether the check was manual or automated at the time of treatment approval. The null hypothesis was that there was no difference between the two groups. A p-value less than 0.05 was chosen as the level of statistical significance, below which the null hypothesis was rejected.

RESULTS

2×2 contingency tables are shown in Table 1, p-values in bold indicate statistical significance of the given test.

There are no pre-automation checks for the “bolus material” and “verify dose rate” checks since the automated versions were contained within the initial clinical release of our APC. There were zero instances of the incorrect bolus HU value or dose rate being used throughout the three data collection periods, which consisted of 1169 plans reviewed. Similarly, there were zero instances observed, either pre- or post-automation, of the incorrect prescribed percentage being chosen. A plan is always normalized at our institution such that the 100% relative isodose line corresponds to the prescribed absolute dose.

The “CT slice thickness” check had no observed pre-automation flags. There was one patient in the post-automation period where a plan was created on a slice thickness of 3.27 mm rather than 3 mm. This was a large pelvic volumetric modulated arc therapy (VMAT) plan, no sharp dose gradients, and prescription dose of 5040 cGy in 28 fractions. Given that, the clinical decision was made at the time to proceed with treatment, despite the flag.

“Verify primary collimator not exposed” registered 0 observed events in post-automation datasets. In the 4 pre-automation primary collimator events the jaws were open far enough to expose the primary collimator but the irradiated field shape was delivered as intended by the physician since the MLC leaves were blocking that part of the field, indicating that the design of the checker could be refined to better reflect the clinical circumstances.

The addition of automation in the detection of “hot spots outside of the PTV” appears to have reduced the number of observed instances. Further, while 6 of the pre-automation instances were noticeably outside the PTV, the hot spot was restricted to the periphery of the PTV in all cases that were flagged in the post-automation dataset.

A large and statistically significant decrease in flags between the pre- and post-automation datasets was observed for the four contour based checks. 8 of 10 of the pre-automation events for the “overlap between body and couch” check had overlap between those structures in the beam entrance area, 1 of 2 in the post-automation set were in the beam entrance area. The couch model misplacement overwrites the HU value of the CT scan so overlap between the couch model and the body contour results in pixels of the patient being overwritten with the couch HU. In all instances the flags were true negatives, but the overlap was minimal and unlikely to have a major clinical impact. The other three contour-based checks elicited more flags in the post-automation dataset than anticipated. Therefore, as shown in Table 2, those cases were sub-categorized into false negatives/naming convention/true negatives.

Table 2:

Analysis of the reason why flags were thrown for a subset of the automated checks

Check name	Output sub-category	Output sub-category example and explanation	Pre-Automation	Post-Automation
Adjacent Structures Overlap	False Negatives	“Structures C5_Esophagus and Stomach overlap on 0 slices. They should overlap on at least 1 slices. Carefully review the segmentation(s) and either fix or save a note saying why this is okay.” [note: false b/c this patient is being treated to 2 sites on same CT and esophagus in C5 paraspinal target area wouldn’t need to overlap with the stomach]	15 / 326	28 / 843
	False Negatives naming convention	“Structures Esoph RUL and Stomach_NOT_PTV overlap on 0 slices. They should overlap on at least 1 slices. Carefully review the segmentation(s) and either fix or save a note saying why this is okay.” [note: Stomach_NOT_PTV was used as an optimization structure and by convention should have a “z” prefix, which would make the check ignore it]	0 / 326	2 / 843
	True Negatives	“Structures Cord and Brainstem overlap on 0 slices. They should overlap on at least 1 slices. Carefully review the segmentation(s) and either fix or save a note saying why this is okay.”	84 / 326	21 / 843
	True positives	“Automatic Checks Passed”	227 / 326	792 / 843
Empty Structures / Missing Slices	False Negatives	Empty: No examples Missing Slices: “Structure ITV extends from slice z=−9.90 to slice z=6.00 and slices −1.20, 4.20 are missing contours. Carefully review the segmentation(s) and either fix or save a note saying why this is okay.” [Note: our check skips GTV, CTV, PTV b/c could be disparate targets but includes ITV in the evaluation]	6 / 326	21 / 843
	False Negatives naming convention	Empty: “Structure ‘MATCH’ is currently empty. Either complete the segmentation or delete the empty structure.” [Note: MATCH is a planning structure and should have a “z” prefix, which would make the check ignore it] Missing slices: “Structure ‘guide’ extends from slice z=−8.67 to slice z=5.73 and slices −1.47, −1.17 are missing contours. Carefully review the segmentation(s) and either fix or save a note saying why this is okay.” [Note: guide is a planning structure should have “z” prefix, which would make the check ignore it]	18 / 326	32 / 843
	True Negatives	Empty: “Structure Heart is currently empty. Either complete the segmentation or delete the empty structure.” Missing slices: “Structure Artery_L extends from slice z=−2.70 to slice z=1.20 and slices −2.40, −1.80, −1.50, −1.20, −0.90, −0.60 are missing contours. Carefully review the segmentation(s) and either fix or save a note saying why this is okay.” [note: interpolation issue]	124 / 326	34 / 843
	True positives	“Automatic Checks Passed”	178 / 326	756 / 843
Laterality of Paired Organs	False Negatives	“Structures Parotid_L and Carotid_Lt are both labeled as left. Carefully review these contours and ensure that the laterality is correct.” [Note: Check is configured to pair up structures if share 3 characters in a row, here “arotid_L” matches].	18 / 695	20 / 474
	False Negatives naming convention	“Cannot determine the intended laterality of structure Lung_Total. Rename Lung_Total so that intended laterality is clear.” [Note: Our conventional name for combined left and right lung is “Lungs”, not “Lung_total”]	93 / 695	21 / 474
	True Negatives	“Structure Parotid_R is labeled as right but appears to be to the patient’s left. Carefully review these contours and ensure that the laterality is correct.”	3 / 695	0 / 474
	True Positives	“Automatic Checks Passed”	581 / 695	433 / 474

A break-down of the reason why flags were thrown for the adjacent structures overlap, empty and missing structures, and laterality of paired organ automated checks. False negative refers to a situation where the algorithm throws a flag but there should have been a pass. Reasons for this could include a lack of sophistication in the algorithm or poor configuration of the check. False negative due to naming convention is a specific type of false negative related to the fact that the user didn’t follow agreed upon naming conventions, circumventing the ability of the automated logic or configuration to identify those structures properly. True negatives refer to a situation where there was a real problem that should have been detected by the plan check, addressed by the planner, and prevented from propagating to the patient. Ideally, these are the only types of flags that should be thrown by the automated software.

There were three instances in the pre-automation data collection period where incorrectly labelled laterality made it through to treatment. In none of the cases did the mislabeling have an impact on how the plan was constructed, evaluated, or delivered but they did cause misleading and confusing documentation. Zero instances of incorrect laterality were identified in the post-automation dataset. Automation also reduced the frequency of true negatives in the check of empty structures and missing slices and in the adjacent structures overlap check.

DISCUSSION

The benefits of automated plan checking software have been documented at many institutions^3,5–8. Similarly, these results demonstrate that our APC has continuously increased safety at our institution over the course of 3 years. There was a statistically significant decrease in errors reaching the patient for image-based checks where the APC augmented the human review of the relationship of contours and isodose lines with the underlying patient anatomy. This type of manual review has been demonstrably problematic in the field^14–16, so it is encouraging that our automated interventions have resulted in improvement.

The role of automation to facilitate, rather than replace, a human plan checker is reflected in our use of the terminology and symbolism of “flags” rather than “fails”. A “flag” indicates to the human user an area of the plan that needs their scrutiny and, ultimately, a human judgement about whether the state is appropriate or not. While an incorrect grid size or a misplaced couch structure may not have a significant clinical impact for a given patient, our preference is for the user to resolve these flags whenever possible, as increased standardization in practice ultimately increases both quality and efficiency¹⁷.

One pitfall of automation is that algorithmic limitations can lead to false negatives. These potentially waste time, introduce confusion and errors of their own, and lead to user fatigue and loss of buy-in. We would expect the deleterious cognitive effects of the increased complexity arising from false negatives to be similar to checklist fatigue¹⁸. Worse, this could ultimately lead to the disregard of true negatives, as has been noted in human factors research^19,20. Therefore, in addition to initial testing and commissioning of automated algorithms, a periodic analysis of algorithmic performance on clinical cases, especially in the presence of evolving clinical procedures and workflows, as done in this manuscript, is essential. In doing so we’ve identified potential algorithmic or configuration modifications to apply to future versions of the software. For example, the “primary collimator not exposed” check could be modified to analyze the irradiated field as defined by the combination of the jaws, blocks, MLC, or electron cutouts, rather than the jaws alone. This would have eliminated the 4 false negatives identified in the analysis. The “hot spots outside of the PTV” check, based on the observation that all of the post-automation flags occurred directly on the periphery of the PTV, could be modified to consider a hot spot location as acceptable if it is within a one pixel expansion of the PTV. The configuration of the “missing slices” check had already set GTV, CTV, and PTV as structures to exclude from the check as there are often disparate targets, but many of the false negatives in the “missing slices” indicated that ITV should also be included on this list. For the “adjacent structures overlap” check, the false negative example in Table 2 is typical of what was observed. The check could be modified to limit evaluations of segmentations to within some pre-defined distance of the irradiated field.

Non-compliance with established policy and procedures by individuals is also the cause of a large proportion of flags. Many users appear to observe the flag but mentally note, or even note within the software interface, that the structure isn’t one of concern, whereas our preference would be for them to rename it so that it does comply with our naming conventions. Compliance with naming conventions has also been observed to be problematic in the community^21,22 which is why the AAPM and other groups have attempted to introduce universal naming conventions^23,24. We utilize disease site-specific structure templates and continuously work with staff to reinforce the importance of using these templates rather than adding individual structures ad hoc and, more generally, the need to follow naming conventions for the sake of general clarity, automation, outcomes analysis, and data mining.

The user does have the ability to note within the software interface why they believe that proceeding through a particular flag would be okay. In the course of clinical use, we find that users often omit such comments. When they do enter an explanation, that judgement is often misguided. We address this problem with a combination of individual coaching feedback and systematically changing the software messages to make them more descriptive and explanatory. Clarity in error messaging is an issue that the entire field of radiotherapy is often grappling with²⁵. For this analysis, we independently determined whether each flag was a false negative or a true negative without relying on notes left by individual users.

We intend to continue to grow our APC to encompass the subset of checks still performed manually and to increase its capability to check characteristics of the plan in Eclipse against data outside of the Varian ecosystem, for example the institutional electronic health information system, in-house QA software, and MiM (MiM Software Inc, Cleveland, OH), the platform that we use for segmentation and registration. Our APC is dynamic and adapts to changes in the clinical workflow. Requests for additional or modified functionality come from our user base and our departmental Quality Assurance committee based on a review of events in our incident learning database system. We will incorporate checks recommended by AAPM Task Group 275¹⁰ as well as the soon to be released AAPM Medical Physics Practice Guideline 11a on plan and chart review in external beam radiotherapy.

CONCLUSION:

Periodic and longitudinal review of the performance of APC software is essential for understanding its impact on error rates as well as for optimization of the tool to adapt to changes of clinical practice. The APC, a result of an interinstitutional collaboration between two major medical centers, has been proven to continuously contribute to the safe and effective delivery of external beam radiotherapy to our patient population, beyond its initial implementation and deployment.

Data Availability Statement:

The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions