Optimizing efficiency of a custom clinical decision support tool improves adult congenital heart disease care

Abstract

Study objective

Improve the efficiency of an inpatient clinical decision support tool (CDS) for patients with adult congenital heart disease (ACHD).

Design

The efficiency of a CDS was evaluated across two time periods and compared.

Setting

An academic, tertiary care center.

Participants

ACHD patients roomed in an inpatient setting.

Intervention

Plan-Do-Study-Act (PDSA) methods were applied starting in 2021 and included refinement of diagnostic codes and the addition of department encounter codes.

Main outcome measures

True positive and false positive CDS alerts.

Results

Baseline data from 2017 had a median (IQR) of 38 (17) and 2019 baseline data had 65 (19) total alerts per month. Combining both baseline data years, the median true positive CDS alerts was 47.3 %. There were 71 (6) total alerts per month for the 2021–2022 time period and with ongoing PDSA cycles and optimization in the CDS the true positive alerts improved substantially resulting in a shifting of the median to 78.9 % within 9 months.

Conclusion

CDS can efficiently notify providers when an ACHD patient is encountered. The use of ICD 10 codes alone to identify ACHD patients has limited accuracy with a high proportion of false positives. Ongoing revision of the CDS system methods is important to improving efficiency and minimizing provider alert fatigue.

1. Introduction

Guidelines for the management of adults with congenital heart disease (CHD) recommend a multidisciplinary team that includes adult congenital heart disease (ACHD) specialists [1]. Most ACHD patients have sequelae from their native disease or previous repairs, which can take decades to manifest. Specialized ACHD care is associated with improved outcomes for both complex and simple congenital heart disease [1,2]. Access to specialized cardiovascular care, however, presents several challenges including lack of transition from pediatric to adult care, gaps in care and limited awareness by patients and providers regarding the manifestations of disease sequelae and ACHD care recommendations. Moreover, CHD makes up a small fraction of the range of adult cardiac disease, and providers may have limited exposure to the discipline. Improving knowledge about the care for a growing but infrequent patient population in complex medical systems is challenging, particularly given the multiple points of entry. Timely provider awareness about ACHD care can facilitate patient reconnection to routine outpatient CHD care and engagement of the appropriate clinical teams for inpatient and outpatient management.

Clinical decision support tools (CDS) can provide patients and providers with timely knowledge and person-specific information to enhance health and health care. CDS have been shown to significantly improve clinical practice in randomized controlled trials [3]. Electronic medical record (EMR) interventions can facilitate increased provider awareness of clinical protocols and care resources and can lead to improved outcomes, such as vaccine administration or hypertension control [4,5]. Adult patients with CHD have increased need for specialized medical care but are at high risk for gaps in cardiac care and being in medical situations with limited ACHD understanding [[6], [7], [8]]. CDS have been shown to improve quality metrics for care of adults with CHD in pediatric intensive care units [9]. CDS tools may also be useful to address the care and knowledge gaps for ACHD. A fishbone diagram of causal analysis is shown in Fig. 1.

Fig. 1.

Fishbone diagram.

An EMR-based CDS was designed to facilitate inclusion of adult congenital cardiology specialists in the peri-hospital care of adult congenital cardiology patients. Specifically, an automated EMR CDS tool was developed and implemented to notify adult congenital cardiology providers when adult congenital cardiology patients are roomed in an inpatient setting, emergency department, or operating room. The specific aims of the project were to develop an automated, prompt notification system that captures all ACHD patients encountered in a specific, designated clinical encounter settings with low proportion of false positive alerts. A key driver diagram of the project aims is shown in Fig. 2. The CDS was refined over time based on inappropriate notifications and missed notifications as well as feedback about the usefulness of the tool from the ACHD team. In this study, key implementation process outcomes were evaluated including accuracy of the CDS tool and the impact on clinical management.

Fig. 2.

Key Driver diagram.

2. Methods

In an effort to improve hospital care for adult patients with CHD in an academic, tertiary care center, discussions between clinical leads for ACHD, cardiovascular medicine, and emergency medicine began in 2015. The initial goal of the intervention was to direct admissions of ACHD patients to the designated cardiovascular medicine admitting service. The clinical directors assessed the institutional systems and deemed that a CDS would best achieve the overarching goal. The ACHD providers worked with Information Technology Liaisons from the Center for Clinical Knowledge Management to design a custom CDS to notify the adult congenital cardiology providers when adults with CHD were roomed in an inpatient setting regardless of the reason for admission.

Patients were identified by meeting three criteria: Criterion 1 - a patient age equal to or greater than 18 years, Criterion 2 - a diagnosis of CHD defined by ICD 10 code [10] recorded in the problem list, the medical history, or billing for the individual patient, and Criterion 3 – the patient was roomed in an inpatient setting, defined as the Admission, Discharge, Transfer category equals Currently Admitted Bed. The notification is automatically triggered when the three criteria are met and populates in a folder in the EMR In Basket of the ACHD providers and nursing coordinator.

The CDS alert was silently trialed by the Information Technology Liaisons for two months. The results were presented to the ACHD team and the Clinical Decision Support team and the concept was approved. The CDS was fully developed and implemented in 2017. The alert was initially delivered to ACHD cardiologists via pager. This notification method was converted to an EMR In Basket notification after one month due to dissatisfaction of cardiology providers due to a subjectively high occurrence of erroneous pages. As the intervention was used, the goals of the CDS alert matured. In addition to directing admission to the correct team, the goal evolved to incorporation of the ACHD clinical team into the patient care in line with published guidelines [1].

After CDS implementation in 2017, the alert was adjusted periodically by adding or removing specific ACHD ICD 10 codes. Prior to 2021, additional adjustments were prompted by subjective concerns about inefficiencies in the system, typically a high number of false positive alerts and incidental knowledge of patient encounters where an alert was not triggered, a false negative. Beginning in early 2021, a more focused assessment applying quality improvement processes was initiated with monthly prospective data collection. Retrospective baseline data were analyzed for the prior selected time periods to help guide areas for improvement. Plan-Do-Study-Act (PDSA) cycles were performed to improve the true positive alerts and simultaneously decrease the false positive burden, with an objective goal of improving the proportion of true positive alerts to 85 %. Reviews included discussions with the Information Technology Liaison and ACHD clinicians including the nurse coordinator, advanced practice provider, and medical director. Analyses of where improvements could be implemented centered on example cases of false positive alerts and false negative encounters. Feedback from the full ACHD team was incorporated into the reviews, particularly with regards to alert fatigue. Elements identified in the errant alerts centered on ICD 10 codes.

The above elements suggested that a new method was needed to help capture true positives. In August 2021, Encounter Department was incorporated into Criterion 2 to identify ACHD patients with a prior encounter in one of the ACHD departments satisfying Criterion 2. This was done to capture patients with 1) unusual ACHD diagnoses and 2) appropriate patients who have common diagnoses such as bicuspid aortic valve. Additionally, in order to delineate between the atrial septal defect ICD 10 codes, Systematized Nomenclature of Medicine Clinical Term (SNOMED) Logic was used to omit secundum atrial septal defects, but continue alerts for the diagnoses of primum atrial septal defect and sinus venous atrial septal defect. A timeline of the CDS development and updates is shown in Fig. 3.

Fig. 3.

Timeline of CDS development

Steps in the development and quality improvement of the CDS by time are listed. Electronic medical record (EMR), patent foramen ovale (PFO), bicuspid aortic valve (BAV), Plan-Do-Study-Act (PDSA), atrial septal defect (ASD).

Measures used to assess the efficiency of the intervention included 1) notification appropriately populates a specified EMR In Basket folder reflecting that an ACHD patient has been roomed in an inpatient setting, termed true positive (Fig. 4.), 2) notification of a patient without CHD or a simple cardiac diagnosis, such as patent foramen ovale, termed false positive, 3) absence of a notification of a patient with CHD, termed false negative, and 4) the burden of the notifications. These measures were chosen based on primary aim of accurate notification of the ACHD team when an ACHD patient is encountered in an inpatient setting and the secondary aim of minimizing alert fatigue and inefficiencies in the system. In summary, the goal was to accurately capture all ACHD patients while minimizing disruptions and inappropriate alerts.

Fig. 4.

In Basket folder for CDS alerts

Anonymized image of a clinician's EMR In Basket folder of CDS notifications that include the patient name, date and time of alert, and links to access the patient's chart.

A year's worth of data from two time periods were used as baseline data. These include 2017, which is the first year of CDS use, and 2019, which is the year after major changes were made to the CDS algorithm. No major changes were made to the algorithm in 2018, and data from 2020 was avoided to preclude unexpected effects from the COVID pandemic. Starting in 2021, focused PDSA cycles were initiated to optimize the CDS with the goal to improve true positive and decrease false positive alerts. Monthly assessment of data including total alerts, percent true positive, percent false positive, and any identified false negatives from incidental discovery were assessed and plotted on a run chart to assess for improvement from the interventions. Also, monthly assessment in which the ACHD team knowledge of the patient by the CDS changed patient management was recorded. To assess burden of the alert, two providers independently quantified the daily time required to review CDS prospectively.

Descriptive data on monthly median alerts including total, false positive, true positive, and patent foramen ovale/small secundum atrial septal defect were calculated. Monthly true positive alerts for the studied time periods between January 2017 and December 2022 were plotted in an annotated run chart. Additionally, a graph was created to show the monthly total number of alerts between April 2021 and December 2022 including the percentage of ACHD patients consulted, admitted, and if the CDS alert changed clinical management.

3. Results

The total number of alerts monthly increased from 38 (17) in 2017 to 65 (19) in 2019 (median (IQR)), suggesting that adjustments to the ICD 10 codes increased capture of ACHD patients. However, despite these adjustments, the percentage of alerts that were true positives remained low across both time periods, with a median 57.5 % in 2017 and 44.7 % in 2019 (combined baseline years median 47.3 %), and false positive alerts remained high. Data for monthly total, true positive, and false positive CDS alerts for the three time periods are summarized in Table 1. Data are also shown for alerts for patent foramen ovale/small atrial septal defect, a form of false positive.

Table 1.

The number of CDS alerts per month during the three time periods.

Year	2017	2019	2021–2022a
Alerts per month median (IQR)
Total	38 (17)	65 (19)	71 (6)
True positive	21 (17)	27 (9)	54 (11)
False positive	16 (7)	35 (12)	15 (7)
PFO/small ASD	12 (5)	30 (13)	0 (0)

^aData from September 2021 through December 2022 reflecting the time frame after department code was added to Criterion 2.

Following the changes implemented in August 2021, there was a significant increase in true positive CDS, which has sustained, and the median of percent true positives increased to 78.9 %, an improvement from the baseline median of 47.3 %. The last four months of 2022 have met the goal of 85 % true positives and if similar results persist, the median line will be able to be increased further. A run chart of true positive alerts is shown in Fig. 5.

Fig. 5.

Run chart of true positive CDS alerts for 2017, 2019 and 2021–2022.

Data from 2017 and 2019 were used to calculate the baseline with a median true positive alert of 47.3 %. Changes prior to 2021 made little improvement in true positive results. After QI team formation and evaluation for improvement opportunities, a significant change was implemented in August 2021 with immediate and sustained improvement in true positive CDS alert and a new median of 78.9 %.

With the implementation of PDSA, the number of false positive alerts dropped from 35 (12) alerts per month to 15 (7) (median (IQR)). The addition of the department code to identify ACHD patients in Criterion 2, however, introduced new source of false positive alerts. Patients encountered in the congenital cardiology department who do not have congenital heart disease also triggered the CDS. This group of patients accounted for 7 (4) alerts per month or half of the false positive alerts.

Since prospectively tracking CDS alerts between April 2021 and December 2022, 48 patient encounters resulted in a change of care management. Care changes include establishing or re-establishing care with the ACHD program, scheduling an ACHD appointment or cardiac imaging, adjustment to inpatient medical care, or prompting earlier evaluation. One example of the CDS impacting patient care is the notification of a patient arrival to the Emergency Department that triggered immediate evaluation by the ACHD team. The evaluation resulted in emergent cardiac catheterization and institution of extracorporeal membrane oxygenation in a patient with complex congenital heart disease presenting in shock [11]. Of the 48 encounters that resulted in a change in medical care, 68.8 % were lost to follow-up according to 2018 AHA/ACC ACHD guidelines [1] and had not received guideline-recommended cardiac care. A chart summarizing CDS alerts by month including total number of CDS alerts, and percentages where the ACHD team was consulted, the patient was admitted to the hospital, and the CDS changed clinical management is shown in Fig. 6.

Fig. 6.

Graph of CDS alerts per month with ACHD consultation, admission and change in clinical management. Total number of CDS alerts (blue line) per month with percentage of those patients with an ACHD consult (green bar), and admitted to the hospital (gray bar). The CDS alert changed management (black line) in a percentage of ACHD patients in most months. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

The burden of the alert process on the team was assessed. The time required to triage the CDS alert In Basket daily was assessed. Two providers separately quantified the time per day required to review the alert folder, prospectively for 30 days. The median time spent reviewing the alerts per day was 3 min (IQR 3.5 min). Duplicate alerts contributed to this workload burden. A patient can have multiple inpatient rooming events per day or admission. For example, an alert would trigger when the patient is roomed in the emergency department and then again when they are roomed in an inpatient ward. The median number of duplicates per month remained similar over the time periods; 9 (IQR 4) in 2017, 12 (IQR 5) in 2019 and 8 (IQR 5) in 2021 with no change with the August 2021 update.

There were unexpected benefits of the CDS alerts. The alerts served as a resource to locate patient details for provider clinical utility and to facilitate intra-EMR communication and placement of orders. In some instances, the alert assists in monitoring ongoing patient care and assurance that planned clinical studies are being completed and provides information of ACHD patient's access to care which may be useful to guide the ACHD program where to focus future training initiatives. Additionally, the steady, cased-based communication between the ACHD team and providers also helped improve general awareness of ACHD.

A limitation of the process and analyses is the lack of reliable mechanism to identify false negatives when no notification occurs. Recognition of false negative alerts is limited to incidental recognition by ACHD providers that an alert did not trigger. This is typically recognized during a subsequent ACHD encounter. Alert fatigue is a concern as well. This is mitigated, in part, by decreasing the false positive alerts and rotating the responsibility of alert review. With the improvement in the true positive percentages, ACHD providers report improved engagement with the CDS.

4. Conclusion

Congenital heart disease is a relatively infrequent form of adult cardiovascular disease limiting provider exposure. Clinical Decision Support can be effectively applied to provide timely knowledge and person-specific information to improve clinical care for less frequent diseases. A previous study in ACHD demonstrated that use of a CDS improved performance measures in adults recovering from cardiac surgery in a children's hospital, improving compliance from 50 % to greater than 80 % [8]. The custom ACHD CDS analyzed in this study required ongoing optimization to achieve appropriate alerts and the least disruptive workflow. Refinement of primary evaluation metrics and implementation of QI processes improved the efficiency of the CDS. With the application of PDSA cycles, the true positive alert percentage achieved the target of 85 % for each of the last four months of data analysis. Future ongoing PDSA cycles and optimization of the CDS may be able to further improve the true positive percentage and decrease the new source of false positive alerts from the addition of the department code to Criterion 2.

ICD 10 codes, as previously shown, have limited accuracy in identifying ACHD patients [12,13]. Coding errors and nuances of certain diagnoses, particularly clinically relevant forms of atrial septal defects, are difficult to distinguish. Incorporation of additional data elements to identify patients with CHD, such as department codes for encounters, was valuable to improve accurate disease categorization. Review of sentinel cases was important to informing the optimization of the CDS. Consistent advocacy for the CDS from the ACHD and Information Technology teams was important. Incomplete or inaccurate EMR documentation limits the reliability of data elements available to identify ACHD patients. Quantifying the accurate number of false negative patients is unsolvable. Despite the limitations, further work is needed to decrease false negative encounters where patients were not captured by the alert. Cross referencing a dashboard of patients active in the ACHD program may be a future opportunity.

The CDS was initially designed to appropriately direct admissions from the emergency department by facilitating individual patient communication between the emergency medicine and ACHD providers. This goal was achieved, although not through the anticipated means. The improvement was less through patient-specific care and more by exposing opportunities for improvement in the clinical workflows and better integration of systems. Specifically, findings from the CDS led to refinement of the System of Care Playbook, a center-specific tool that provides general guidance on the preferred placement of patients requiring inpatient care. The CDS also fostered collaboration leading to a general increase in inter-team communication and familiarity. Though these effects, the CDS helped improve the care of patients with ACHD. Six years after implementation, the CDS continues to be valued by the ACHD team. The value was enhanced by continued optimization and attention to the most appropriate and least disruptive workflow.

CDS is useful tool to help improve care patients with ACHD, a disease that has generally low provider familiarity and high rates of lapses in guideline-recommended cardiology care. The incorporation of QI processes was important to improving efficiency and relevance of the tool. Clinical and informatics champions are beneficial to the process. Approaches such as this may be useful to other populations that experience health care transition or clinical populations who warrant specialized care.

Declaration of competing interest

The authors have no conflicts of interest.