New Ideas and Fresh Perspectives: Improving Processes and Quality Assurance Using Automation and Expertise

Summary

The Vermont Cancer Registry (VCR) previously had a practice of visually reviewing and providing feedback to Vermont reporting facilities for each case received. Limited resources and increasing responsibilities, caseloads, and case complexity made this practice unsustainable. Following a 9-month Systems Thinking exercise in 2019, VCR made significant changes to its processing procedures.1 However, the backlog of unprocessed cases grew (Figure 2), particularly during the pandemic, and VCR contended with an unsustainable level of effort. Staffing transitions along with a return to pre-COVID-19 activities provided an opportunity to reevaluate and adjust thinking and practices.

Figure 2.

Average Number of Days Between Date Case Exported and Date Case Loaded by Year of Diagnosis

Challenges

• Despite a compressed visual review process (comparing coded values to free text), VCR had an ongoing and significant backlog (Figures 1 and 2) in its case processing and consolidation (combining the best information about a tumor from multiple case reports).

• VCR had not met the Advanced NPCR Data Quality Standards in over 5 years (Figure 1), and as a team, felt that completing processing and annual submission tasks for even the 24-month standard “came down to the wire” each year.

• VCR had already undergone a Systems Thinking process that addressed ingrained institutional practices. The process resulted in a significant team mindset change.

• A small core team of 4 (program manager, data quality and education coordinator, health data administrator, and public health analyst) devoted a significant amount of time to processing tasks but did not feel that additional “headway was being made.”

• Staff feared that additional changes to how we handled data would have negative repercussions for data quality of our data. VCR's high-quality data has aways been a source of pride and worth the effort.

• Overall, the team felt demoralized when examining “the situation” we were in.

Figure 1.

NPCR Data Evaluation Reports Percent Completeness, Vermont Submissions, 2013–2022 and Estimated 2023

* Estimate.

Solution

• The prior Systems Thinking process always left an opening to revisit VCR's practices and had instilled open-mindedness to the possibility of change despite ingrained practices. Confronted with VCR's backlog, the team was open to consideration and reexamination of its work. As a team we asked: “What if we didn't do any visual review?” A decision was made to forgo visual review and process data directly into the VCR database and perform quality assurance (QA) after processing.

• The registry chief, analyst, and quality and education coordinator (QEC) meet weekly to formulate and proceed with a plan to review registry data adequately and efficiently.

• Extracts for QA processes were created that included cases and fields that were focused on for review. For fields that require support from text fields, regular expressions (regex) with R were used to search and extract common patterns in the associated text fields to validate what was reported.

  • For example, to validate “Progesterone Receptor Summary,” the lab text fields were scanned for expressions like PR (+), ER/PR: Pos, and Positive for progesterone receptors. Each instance matching the search pattern was extracted into a separate field and flagged based on whether the extracted text matched what was reported in the “Progesterone Receptor Summary.”
• For fields that could be validated without text fields, corresponding fields were used to flag cases where there may be a mismatch.

  • For example, Brain Molecular Marker were validated using ICD-O-3 histology, ICD-O-3 behavior, and the diagnostic confirmation method following the coding instructions found in the Site-Specific Data Item (SSDI) Manual.

• After the data were processed, data were returned to the QEC and health data administrator for final review.

Results

• A larger core set of data items were reviewed with this new process and VCR was able to review more data items than when completing visual review (grade and SSDIs).

• QA was completed for an entire diagnosis year and by groups of sites rather than reviewing cases in a submitted file that included a mix of diagnosis years with different reporting requirements and multiple sites. The QA process became focused and more efficient.

• The benefits of using some automation quickly became apparent. Queries grouped cases and fields, flagged discrepancies, and identified possible errors to be examined.

• VCR's internal timeliness improved, backlogs of data that needed to be processed were eliminated, and VCR was able to provide meaningful feedback to reporting facilities. Quality Indicator Reports that had not been produced since 2014 resumed.

• A total of 8,444 cases were extracted and went through the automation procedure. Of those cases, 4,533 were identified as needing possible additional QA. Cases were reviewed, errors were identified and corrected, and feedback was shared with reporting facilities.

• To address the concern that the new QA process might not capture all errors that would have been identified with visual review, VCR compared prior visual review and the new QA strategy for a limited time frame and a set of facilities and determined that similar types and quantity of errors were identified. We feel confident with our new process. A new State Added Field is used to track cases that have been reviewed.

Sustaining Success

• The VCR feels that this new process is sustainable. However, we will continue to ask, “Is this working?” If not, we will evolve and try different approaches until we are successful.

• For the 2023 Data Submission, VCR set an internal goal of 70% completeness for this year's 12-month data. The education and training coordinator estimates that we are 90% complete (Figure 1).

Story Quote

“While we have not abandoned our original core belief that ‘all data should be reviewed all of the time,’ we have committed to using an agile approach to the mixture of core data items and percentage cases visually reviewed. We will continue to ask, ‘Is the ratio working?’ If not, we will try a different ratio, and keep repeating the process.”

—Sustaining Success (2019)