Skip to main content
NCBI home page
Applied Clinical Informatics logoLink to Applied Clinical Informatics
. 2025 Sep 5;16(4):1005–1013. doi: 10.1055/a-2594-3722

Improving Discrete Documentation of Cancer Staging—An Alert-Free Approach

Renee Potashner 1,2, Adam P Yan 1,2,✉
PMCID: PMC12413274  PMID: 40280549

Abstract

Background

Cancer staging is integral to ensuring cancer patients receive appropriate risk-adapted therapy. Discrete cancer staging using a structured staging form helps ensure accurate staging, provides a single source of truth for staging information, and allows for reporting to regulatory authorities. Our institution created pediatric oncology specific discrete staging forms that have been shared with the broader Epic community. By November 2023, baseline utilization of the staging form for patients with leukemia or lymphoma was 43%, and the override rate for our existing alert was 99.9%.

Objectives

Improve discrete documentation of cancer stage for patients with leukemia or lymphoma within 60 days following initiation of chemotherapy to >80% by July 2024 as measured by signed staging form.

Methods

Model for improving plan-do-study-act (PDSA) cycles was implemented, and statistical process control charts were used to evaluate impact. The first intervention was educational training to oncology providers. The second PDSA cycle involved sharing monthly individual completion data with the primary oncologist regarding their personal patient metrics. The third PDSA cycle involved removing the interruptive alert.

Results

Within 6 months, documentation of primary oncologist improved from 86 to 100%, and initiation of staging form improved from 57 to 90%. Completion of signed cancer staging form reached 80%. Patients marked as not needing staging increased from 5 to 17%.

Conclusion

Completion of a digital cancer staging form is important for continuity of care, and to facilitate reporting to regulatory authorities, though frequent interruptive alerts were an ineffective method for improving documentation. Education and data sharing increased staging completion to near target, with ongoing efforts to reach the goal of 80%.

Keywords: electronic health records and systems; clinical information system, clinical decision support; clinical information systems, oncology domain; clinical care, pediatrics domain; socio-technical aspects of

Background and Significance

Cancer staging is important for maintaining a consistent and clear understanding of a cancer patient's diagnosis, prognosis, and treatment plan. Clear documentation of a patient's cancer stage helps the many clinicians involved in a cancer patient's care have a shared mental model of the patient's disease. In addition, many jurisdictions require institutions to provide data on care for cancer patients, including cancer staging information, to local and federal cancer registries. Often, the data managers who collect this information are required to complete manual chart review, attend tumor board presentations, and contact the primary clinical team to ensure that they have all of the required information to complete regulatory submissions. 1 2 Extracting and validating this staging data often require significant resources, especially if staging is not clearly documented in a consistent and uniform way. 2

The use of a standardized data language is critical to support the shared understanding of cancer staging and helps facilitate the application of guideline-concordant management algorithms, as well as the use of staging information in research. The development of a clinical management algorithm for colorectal cancers using manually generated documentation using the Unified Medical Language System (UMLS) allowed for the correct treatment recommendation for all patients with early stage disease, with reduced accuracy in more advanced stages. 3 Another standardized data model, the Observational Medical Outcomes Partnership Common Data Model (OMOP CDM), was the foundation for standardizing staging information in thyroid cancer. 4 Using natural language processing (NLP) to convert the required staging information into OMOP CDM allowed for diagnostic and staging information to be standardized across multiple centers to support data sharing for research. 4

The use of standardized language for cancer staging is also important in the pediatric population. Our institution developed pediatric specific oncology staging forms, which have been shared with the broader Epic community. There is significant variability among forms depending on the patient's cancer type, and the number of elements varies by disease. For example, for the most common pediatric malignancy B-ALL, our staging form has 14 discrete elements that must be filled out manually by the oncologist ( Fig. 1 ).

Fig. 1.

Fig. 1

Open in a new tab

Pediatric staging form for B-ALL (© 2024 Epic Systems Corporation).

Various strategies have been employed to improve documentation of a patient's cancer stage, including with the use of education by clinical champions, sharing of team staging metrics, and employing NLP. 5 6 7 NLP techniques continue to evolve in their maturity; however, most studies have focused on extracting limited diagnosis codes for a small subset of diseases, limiting the use of such tools for wide scale registry reporting. 8 9 10

A common tool that has been leveraged in optimizing the completion of cancer staging is the Epic Our Practice Advisory (OPA, formerly Best Practice Advisory, BPA). An OPA can be employed in two ways: it may be passive, visible on the screen as a banner or highlighted text box which does not stop a provider from completing the workflow, or it can serve as an active interruptive alert, which requires action before continuing a workflow. Interruptive alerts are a common strategy employed to improve patient safety by requiring a user to complete an action based on the prompt. Interruptive alerts have been employed to guide clinical management, ensure documentation completion, identify concerns regarding duplicate orders, and ensure medication safety. 11 12 13 14 15 However frequent alerts contribute to alert fatigue and provider burnout. 16 17 Repeated overriding of the same alert may also lead to habitual alert overriding where the provider does not actually process the content of the alert and simply clicks through the alert's prompts. 18

Many organizations have initiated efforts to identify and remove interruptive alerts that are not leading to the intended behavior, to limit the risk of reduced efficacy of interruptive alerts overall. At Nationwide Children's Hospital, a quality improvement (QI) initiative to reduce interruptive alerts utilized provider feedback, alert removal, and redesign, to achieve a nearly 40% reduction in alert firing rate after 1 year. 19 The COVID pandemic prompted another institution to implement an interruptive alert to notify staff to order the infection prevention precautions for patients who reported symptoms on initial screening in the Emergency Room. Over time, this alert was more of a burden, and other strategies were used including passive alerts and automated ordering of precautions alongside PCR testing. 20 Similar work has been done at our institution to optimize the utility of interruptive alerts.

Our institution employs the use of an interruptive alert to notify providers upon chart opening that a patient's cancer staging documentation is incomplete. The existing interruptive alert for missing staging information was silenced over 2,600 times in 2022 (99.9% of firings). This prompted our team to explore the functionality of the interruptive alert and consider alternative methods for optimizing completion of cancer staging documentation.

This interruptive alert was a form of clinical decision support (CDS) that was intended to ensure that oncologists provided staging information about their patients in a standardized fashion. Unfortunately, CDS may not lead to the intended effect. Wright et al developed a taxonomy to characterize CDS malfunctions based on the causes, mode of discovery, and outcomes of the malfunction. 21 22 There is emerging literature within pediatric oncology describing the experiences of interruptive alerts aimed to improve patient care management, yet due to design errors, and challenges with incorporating a new CDS tool into an existing workflow, the interruptive alerts were ultimately more problematic than helpful. 23 24

This project was undertaken to improve completion of cancer staging documentation in order to allow for prompt discontinuation of this malfunctioning CDS tool. The primary aim of this project was to increase the proportion of patients with newly diagnosed leukemia or lymphoma who had complete staging documentation by 2 months after initial diagnosis to 80% within 1 year of project initiation.

Methods

Context

This study was conducted at The Hospital for Sick Children, a tertiary academic pediatric center in Canada. Our hematologic malignancies program sees approximately 200 new patients per year. Eight oncology attendings, four nurse practitioners, and 20 oncology fellows provide primary oncology care to these patients. At the initiation of this project, all providers were expected to document their patients' cancer stage using discrete staging forms created locally in our electronic health record (EHR), Epic Systems®. For oncology providers, an interruptive OPA would fire upon opening the chart of a patient whose cancer was not staged ( Fig. 2 ). This alert could be suppressed by filling out the staging form or by marking the cancer diagnosis as “not applicable to cancer type.” In addition, for cancer patients a storyboard grouper “cancer diagnosis” is displayed, and for each cancer diagnosis the patient's staging status is displayed below that diagnosis. The staging form can be initiated by a fellow or nurse practitioner; however, the final form must be signed by an attending physician.

Fig. 2.

Fig. 2

Open in a new tab

Initial Our Practice Advisory alert designed to improve cancer staging (© 2024 Epic Systems Corporation).

Project Team and Framework

The project team consisted of oncology physicians, a clinic nurse, and a quality improvement specialist. Monthly meetings of team members allowed the project team to evaluate factors contributing to success and evaluate opportunities for improvement.

Interventions

At the start of the project a Key Driver Diagram was developed ( Fig. 3 ). Identified drivers include understanding the importance of discrete staging, understanding how to complete the staging workflow, identifying which patients were not staged, and having appropriately timed reminders to stage a patient. Based on these drivers three plan-do-study-act (PDSA) cycles were designed. A SMART aim of staging 80% of newly diagnosed patients by 2 months after initial diagnosis within 1 year of project initiation was chosen. We felt that providers needed a 2-month time window after initial diagnosis for most diagnostic results to be available for staging (e.g., end of induction minimal residual disease [MRD] for acute lymphoblastic leukemia patients is done on day 29) so we opted to give providers a 2-month window to stage patients. The target of 80% was selected as we anticipated that there would always be some patients where the staging forms did not apply (e.g., rare lymphoma patients for whom no staging form exists), or where all results were not available within 2 months (e.g., a patient with acute lymphoblastic leukemia with a positive MRD on day 29 needing a repeat MRD at the end of consolidation).

Fig. 3.

Fig. 3

Open in a new tab

Driver Diagram of factors promoting improved cancer staging documentation.

Baseline Data (August 1, 2023–October 31, 2023): Baseline data regarding discrete cancer staging prior to project initiation was collected over a 3-month period. A report was developed using the reporting workbench tool in Epic to query staging data. The report contained filters to identify all patients with an active chemotherapy plan that had been applied in the last year, and who had been seen in the last 30 days in the hematologic malignancy program. Patients were manually removed from the report if they had transitioned their care into the bone marrow transplant program, had relapsed disease, if their primary cancer care site was not at our institution (e.g., second opinion consult), or if staging was appropriately marked as not needed.

PDSA Cycle #1 (November 1, 2023–November 30, 2023): Two educational sessions were conducted by project leads—one with the oncology attendings and nurse practitioners, and one with the oncology fellows. The educational sessions explained the importance of discrete cancer staging and demonstrated how staging a patient in Epic was performed. A tip sheet for navigating the staging workflow was also prepared and shared with all oncology attendings, nurse practitioners, and fellows. The session also covered how to assign yourself as the primary oncologist or primary oncology fellow in Epic.

PDSA Cycle #2 (December 1, 2023–May 30, 2024): A new mechanism to deliver staging data to oncology providers was devised to improve recognition of patients who are missing a cancer stage. In brief, using the previously validated workbench report, each primary oncologist and oncology fellow received a monthly email that contains a list of their unstaged patients, and their aggregate staging performance. A monthly email was also sent to the entire hematologic malignancies program containing a graph showing monthly staging adherence and a congratulations section that highlighted individual provider's successes (e.g., Dr. X has 100% of their patients staged for 3 months in a row!).

PDSA Cycle #3 (June 1, 2024–July 30, 2024): The previous interruptive OPA was sunset in Epic.

Measures

The outcome measure was the proportion of patients with a completed and signed staging form. The first process measure was proportion of patients with a primary oncologist listed in Epic. In order to track individual attending's staging adherence, we needed to be able to easily identify which attending was responsible for staging that patient. The second process measure was the proportion of patients with an initiated staging form (e.g., started by a fellow but not signed by an attending). Staging form initiation is an intermediary step to staging form completion. The balancing measure was the number of patients for whom the staging form was newly marked as “not applicable” inappropriately. We anticipated that providers might mark a patient as not needing staging to suppress the alert, and have the patient removed from their monthly email list. Appropriateness of marking a staging form as not needed was done manually by AY via chart review. Appropriate indications to mark staging as not applicable would be if a staging form did not exist for that patient's disease. Any patient for whom the staging form was appropriately marked as not needed was removed from the analysis.

Analysis

This interrupted time series evaluated staging adherence over time using statistical process control charts. We applied standard control chart rules to denote special cause variation—a non-random signal of change indicating that the intervention had an impact on the measure. 25 Baseline staging adherence was taken as the mean adherence per month during the baseline period. We also performed an interrupted time series analysis (ITS) by estimating a logistic regression model with number of patients with their cancer staged as the dependent variable and time (measured monthly) and time-by-study period interaction as the independent variables. 26 This model compared the pre- and post-implementation rates while accounting for pre-existing trends in the outcome. All analyses were performed using Python Version 3.10.12 and QI Macros (QI Macros for Excel 2024). 27 28 The SQUIRE 2.0 guidelines were used to develop this manuscript. 29

Ethical Considerations

This project was conducted between August 2023 and July 2024. This project was approved by our institution's Quality Improvement Board and therefore Research Ethics Board review was not required.

Results

Outcome Measure: Changes in the Proportion of Patients with Signed Staging

During the baseline data collection period from August 1 to October 31, 2023, the average monthly proportion of patients who had their cancer stage documented discretely in Epic was 38.2%, and this increased to 43.3% at the of the first PDSA cycle, to 73.2% at the end of the second PDSA cycle, and to 77.5% at the end the third PDSA cycle. Over the course of the project, there was a mean 171.6 patients per month (range: 143–214). Special cause variation was observed starting in November 2023 leading to shift in the central line ( Fig. 4 ). In February 2025, 6 months following completion of the formal project, the proportion of patients who had their cancer stage documented discretely in Epic was maintained at 85%.

Fig. 4.

Fig. 4

Open in a new tab

P statistical process control chart for the proportion of patients with a signed cancer stage.

Process Measure: Changes in the Proportion of Patients with Primary Oncologist in Epic

During the baseline data collection period, the average monthly proportion of patients with an assigned attending in Epic was 74.7%. The proportion of patients with a primary oncologist in Epic was 86.0, 99.4, and 99.4% at the end of the first, second, and third PDSA cycles, respectively. Special cause variation was observed starting in November 2023 with project initiation resulting in a shift in the central line ( Fig. 5 ).

Fig. 5.

Fig. 5

Open in a new tab

P statistical process control chart for the proportion of patients with an assigned attending in Epic.

Process Measure: Changes in the Proportion of Patients with Initiated Staging

The average monthly proportion of patients for whom staging was initiated in the baseline data collection period was 58.3%. The proportion of patients for whom staging was initiated was 56.7, 88.1, and 87.3% at the conclusion of each of the three PDSA cycles, respectively. Special cause variation was observed starting in November 2023 ( Fig. 6 ).

Fig. 6.

Fig. 6

Open in a new tab

P statistical process control chart for the proportion of patients for whom staging has been started.

Balancing Measure: Changes in the Number of New of Patients Whose Staging Was Inappropriately Marked as Not Applicable

During the baseline data collection period, between 0 and 1 patients were newly marked as staging “not applicable” when staging was in fact required. During the first and second PDSA cycles between 0 and 5 patients per month were inappropriately marked as staging “not applicable.” The OPA alert, where a provider could indicate that staging was not applicable, was turned off during the third cycle and the balancing measure was no longer meaningful during that time period.

Discussion

Through this quality improvement initiative, we were able to increase the proportion of patients with a completed cancer staging form while also decreasing the number of interruptive alerts clinicians are exposed to. We were able to accomplish this improvement in staging adherence through several strategies, including providing education, clearly delineating staging responsibilities, and by using audit and feedback methods to share individualized staging metrics with attending physicians and oncology fellows. No changes were made to the storyboard group “cancer diagnosis” display which shows the patient's staging status for each cancer diagnosis.

We were able to maintain a nearly 80% staging completion rate even after the interruptive alert was turned off, with ongoing stability over the following 6 months, reaching 85% completion. This continues to allow for shared understanding of a patient's clinical status for all care team members, facilitates sharing of staging information to regulatory authorities, and improves the quality of patient data our institution maintains for clinical research. Furthermore, this supports that the CDS was not functioning as intended.

Employing Wright's extended taxonomy of CDS malfunctions, we further characterize this CDS malfunction in Table 1 . 21 22 The cause for our CDS malfunction would be classified as a “conceptualization error” as the interruptive alert triggered for all users and at every encounter, rather than firing only for those primary care team members who would be responsible for completing the staging documentation. Furthermore, the alert did not take into consideration the lag time between diagnosis and full data availability for staging assignment.

Table 1. Characterization of the interruptive staging alert based on Wright's clinical decision support taxonomy 21 22 .

Axes Classification
Original axes Cause Conceptualization
Mode of discovery End-user reporting
Initiation From its implementation in production
Effect on rule firing Rule fires in situations where it should not fire
Expanded axes Time to discovery 6 years
Architecture Integrated
Data source Electronic health record
Data format Electronic health record
Clinical impact Unknown
Scale of impact More than 2,600 excess alerts per year
Open in a new tab

The error was discovered via end-user reporting; however, the alert was live for over 5 years before concerns were brought to the attention of our institution's Clinical Decision Support Committee and Oncology Informatics Program. Upon review of the alert by the appropriate committees, it was identified to be one of the top 5 most frequently firing alerts reaching providers, despite its firing for oncologists only. This highlights the importance of frequent review of CDS metrics, and specifically the evaluation of interruptive alerts that are frequently silenced. On additional end-user consultation with oncology providers, it became clear that many users were aware of the alert's limitations and had been frustrated by the alert for many years but were not aware that they were able to give feedback on the alert and were not aware to whom they would give that feedback. This speaks to the importance of end-user engagement to ensure timely reporting of problematic CDS tools, and a clear mechanism for end-users to engage with clinical informatics and development teams not only at initial implementation, but also throughout its use. 30 This ongoing sustained end-user engagement is expected to improve user satisfaction, decrease workarounds, and improve safety outcomes. 30 This is also critical when planning for the introduction of new CDS tools to ensure they are compatible with existing workflows. 24

The scale of this CDS error was significant, with the existing interruptive alert for missing staging information being silenced over 2,600 times in 2022, prior to project implementation. The alert fired for any oncologist or fellow any time the patient's chart was opened, regardless of patient location or provider role on the primary team. For example, the alert would fire for the on-call fellow during evaluation of an unplanned emergency room visit. Although we are not able to accurately quantify how many alerts were fired for the primary oncologist that led to completion of the staging form, we expect that the vast majority were inappropriate.

Turning off the alert did not lead to any significant decrease in staging documentation completion, as the alternative strategies put into place over the course of our project helped promote a sustainable mechanism without the use of the interruptive alert. We suspect that removing this alert helped to limit the habitual alert overriding that may occur when there is repeated overriding of the same alert, allowing for more adequate processing if the critical alerts remained. 18

There are some limitations to this study. First, this project focused on only one clinical section of our department. Although the hematologic malignancies section is the largest section, future endeavors will target the other clinical sections to share improvement across the department. Second, our institution does not have staging forms for patients with relapsed disease, so these patients were not included in the patients who required staging. Third, as our data collection methods utilized the presence of a treatment plan to determine the total number of patients requiring staging completion, few patients who are not managed with chemotherapy may have been excluded from this data. We do not expect it would have a notable impact on our findings, and it would be an important consideration for future endeavors in other clinical sections where non-chemotherapy management may be more frequent. Finally, form completion still requires manual data entry which may contribute to provider workload and burnout in a different manner. We did not formally assess the impact of manual form completion on provider burnout or workload. Future work will aim to use NLP to generate the form content, with validation and completion of missing data completed by clinicians to eliminate the manual data entry required by clinicians.

In summary, using quality improvement methodologies, we were able to improve the completion of cancer staging forms by increasing the identification of the responsible staff and fellow, educating the clinical teams on the importance and value of staging, and providing individual user feedback on their progress. We were able to maintain this success following the removal of a CDS malfunction. This highlights the importance of iteratively evaluating alerts to identify if they are serving their intended purpose, forcing a behavior for example in the case of our alert prompting to complete the staging form. It is important to consider alternative strategies for prompting the intended behavior, as the interruptive CDS alert was not helpful in this case.

Clinical Relevance

Discrete documentation of cancer staging facilitates a shared collaborative care model and improved data quality for clinical research and regulatory reporting. Interruptive CDS alerts were an ineffective method to encourage clinician completion of a cancer staging form. Alternative strategies including education, clear delineation of staging responsibilities, and audit and feedback were effective in improving and sustaining proper completion of staging documentation.

Multiple-Choice Questions

  1. What is an unintended consequence of interruptive alerts in the EHR?

    1. Improved medication safety

    2. Awareness of a critical condition

    3. Habitual alert overriding

    4. Avoidance of order duplication

    Correct Answer : The correct answer is option c. Frequent interruptive alerts may lead to habitual overriding of all alerts, regardless of its associated level of urgency. This may result in missing critical information in some alerts and highlights the importance of judicious use of interruptive alerts and exploration of alternative strategies for those that are identified as frequently overridden or are not resulting in the intended behavior.

  2. Which of the following is NOT mentioned as a strategy for improving cancer staging documentation?

    1. Education by clinical champions

    2. Audit and feedback to providers on staging metrics

    3. Natural language processing (NLP)

    4. Mandatory weekly tumor board meetings

    Correct Answer : The correct answer is option d. Education of providers that complete staging forms and providing feedback on their personal patient metrics were helpful strategies to improve the completion of cancer staging forms. There is ongoing exploration of leveraging natural language processing to extract the relevant data directly from patient charts to help minimize the manual input of staging data into the EHR. Although weekly tumor board meetings are commonly used to support a shared care model and are often attended by data managers for the manual collection of data for submission to cancer registries, they do not improve documentation of cancer staging information in the patient chart and involve significant resource utilization.

Funding Statement

Funding None.

Conflict of Interest None declared.

Protection of Human and Animal Subjects

No human subjects were involved in the project.

References

  • 1.Greenberg M L, Barr R D, DiMonte B, McLaughlin E, Greenberg C. Childhood cancer registries in Ontario, Canada: lessons learned from a comparison of two registries. Int J Cancer. 2003;105(01):88–91. doi: 10.1002/ijc.11004. [DOI] [PubMed] [Google Scholar]
  • 2.Gupta S, Pole J D. The validity of pediatric cancer diagnoses in a population-based general cancer registry in Ontario, Canada. BMC Cancer. 2016;16(01):885. doi: 10.1186/s12885-016-2931-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Becker M, Böckmann B, Jöckel K H et al. Mapping patient data to colorectal cancer clinical algorithms for personalized guideline-based treatment. Appl Clin Inform. 2020;11(02):200–209. doi: 10.1055/s-0040-1705105. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Yoo S, Yoon E, Boo D et al. Transforming thyroid cancer diagnosis and staging information from unstructured reports to the observational medical outcome partnership common data model. Appl Clin Inform. 2022;13(03):521–531. doi: 10.1055/s-0042-1748144. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Cecchini M, Framski K, Lazette P, Vega T, Strait M, Adelson K. Electronic intervention to improve structured cancer stage data capture. J Oncol Pract. 2016;12(10):e949–e956. doi: 10.1200/JOP.2016.013540. [DOI] [PubMed] [Google Scholar]
  • 6.Filson C P, Boer B, Curry J et al. Improvement in clinical TNM staging documentation within a prostate cancer quality improvement collaborative. Urology. 2014;83(04):781–786. doi: 10.1016/j.urology.2013.11.040. [DOI] [PubMed] [Google Scholar]
  • 7.Lee J H, Mohamed T, Ramsey C et al. A hospital-wide intervention to improve compliance with TNM cancer staging documentation. J Natl Compr Canc Netw. 2021;20(04):351–3600. doi: 10.6004/jnccn.2020.7799. [DOI] [PubMed] [Google Scholar]
  • 8.Warner J L, Levy M A, Neuss M N, Warner J L, Levy M A, Neuss M N.ReCAP: Feasibility and accuracy of extracting cancer stage information from narrative electronic health record data J Oncol Pract 20161202157–158., e169–e7 [DOI] [PubMed] [Google Scholar]
  • 9.Abedian S, Sholle E T, Adekkanattu P M et al. Automated extraction of tumor staging and diagnosis information from surgical pathology reports. JCO Clin Cancer Inform. 2021;5:1054–1061. doi: 10.1200/CCI.21.00065. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Hochheiser H, Finan S, Yuan Z et al. DeepPhe-CR: natural language processing software services for cancer registrar case abstraction. JCO Clin Cancer Inform. 2023;7(07):e2300156. doi: 10.1200/CCI.23.00156. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Leibel S, Weber R.Utilizing a physician notification system in the EPIC electronic medical record to improve pediatric asthma control: a quality improvement project Clin Pediatr (Phila) 201958(11-12):1334–1337. [DOI] [PubMed] [Google Scholar]
  • 12.Sangillo E, Jube-Desai N, El-Metwally D, Hughes Driscoll C. Impact of a clinical decision support alert on informed consent documentation in the neonatal intensive care unit. Pediatr Qual Saf. 2024;9(01):e713. doi: 10.1097/pq9.0000000000000713. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Procop G W, Keating C, Stagno P et al. Reducing duplicate testing: a comparison of two clinical decision support tools. Am J Clin Pathol. 2015;143(05):623–626. doi: 10.1309/AJCPJOJ3HKEBD3TU. [DOI] [PubMed] [Google Scholar]
  • 14.Paterno M D, Maviglia S M, Gorman P N et al. Tiering drug-drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc. 2009;16(01):40–46. doi: 10.1197/jamia.M2808. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Galanter W L, Polikaitis A, DiDomenico R J. A trial of automated safety alerts for inpatient digoxin use with computerized physician order entry. J Am Med Inform Assoc. 2004;11(04):270–277. doi: 10.1197/jamia.M1500. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc. 2006;13(02):138–147. doi: 10.1197/jamia.M1809. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Gregory M E, Russo E, Singh H. Electronic health record alert-related workload as a predictor of burnout in primary care providers. Appl Clin Inform. 2017;8(03):686–697. doi: 10.4338/ACI-2017-01-RA-0003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Baysari M T, Tariq A, Day R O, Westbrook J I. Alert override as a habitual behavior—a new perspective on a persistent problem. J Am Med Inform Assoc. 2017;24(02):409–412. doi: 10.1093/jamia/ocw072. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Chaparro J D, Hussain C, Lee J A, Hehmeyer J, Nguyen M, Hoffman J. Reducing interruptive alert burden using quality improvement methodology. Appl Clin Inform. 2020;11(01):46–58. doi: 10.1055/s-0039-3402757. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Fallon A, Haralambides K, Mazzillo J, Gleber C. Addressing alert fatigue by replacing a burdensome interruptive alert with passive clinical decision support. Appl Clin Inform. 2024;15(01):101–110. doi: 10.1055/a-2226-8144. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Thayer J G, Franklin A, Miller J M, Grundmeier R W, Rogith D, Wright A. A scoping review of rule-based clinical decision support malfunctions. J Am Med Inform Assoc. 2024;31(10):2405–2413. doi: 10.1093/jamia/ocae187. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Wright A, Ai A, Ash J et al. Clinical decision support alert malfunctions: analysis and empirically derived taxonomy. J Am Med Inform Assoc. 2018;25(05):496–506. doi: 10.1093/jamia/ocx106. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Potashner R, Jessa K, Meyer N, Patterson E, Yan A P.Special issue on CDS failures: a case study: optimizing CDS for pediatric oncology trials by transitioning from interruptive to passive alertsAppl Clin Inform2025 [DOI] [PMC free article] [PubMed]
  • 24.Yarahuan J, Kandaswamy S, Ray Eet al. Special issue on CDS failures: Clinical decision support to reduce hospital length-of-stay for cancer patients with fever and neutropenia Appl Clin Inform 2025. Epub ahead of print 10.1055/a-2540-2349 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Provost L P, Murray S.The Health Care Data Guide 1st ed.Hoboken: Jossey-Bass; 2011. Accessed at:http://ebooks.ciando.com/book/index.cfm/bok_id/504198 [Google Scholar]
  • 26.Fretheim A, Tomic O. Statistical process control and interrupted time series: a golden opportunity for impact evaluation in quality improvement. BMJ Qual Saf. 2015;24(12):748–752. doi: 10.1136/bmjqs-2014-003756. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.PythonAccessed at:https://www.python.org/downloads/release/python-31012/
  • 28.KnowWare International . QI macros for excel 2024. 2024.
  • 29.Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. BMJ Qual Saf. 2016;25(12):986–992. doi: 10.1136/bmjqs-2015-004411. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.NIHR ePrescribing Programme Team . Cresswell K M, Lee L, Mozaffar H, Williams R, Sheikh A. Sustained user engagement in health information technology: The long road from implementation to system optimization of computerized physician order entry and clinical decision support systems for prescribing in hospitals in England. Health Serv Res. 2017;52(05):1928–1957. doi: 10.1111/1475-6773.12581. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from Applied Clinical Informatics are provided here courtesy of Thieme Medical Publishers

RESOURCES