Adaptation of a Published Risk Model to Point-of-care Clinical Decision Support Tailored to Local Workflow

Abstract

Electronic clinical decision support can bring newly published knowledge to the point of care. However, local organizational buy-in, support for team workflows, IT system ease of use and other sociotechnical factors are needed to promote adoption. We successfully implemented a multi-variate cardiac risk stratification model from another institution into ours. We recreated the model and integrated it into our workflow, accessing it from our EHR with patient-specific data and facilitating clinical documentation if the user accepts the model results. Our clinical leaders championed the change and led educational dissemination efforts. We describe the ad-hoc social and technical collaboration needed to build and deploy the tool. The tool complements a clinical initiative within a community of practice, and is correlated with appropriate use of nuclear imaging.

Introduction

Electronic clinical decision support (CDS) shows promise for hastening the use of the latest evidence in clinical practice, but factors in addition to the clinical content influence adoption, including integration with workflow, presentation, and ease of use^1,2. The challenge of localization means that CDS built at one institution is often not directly transferable to another. Furthermore, the adoption of a tool depends on local champions, support, and change management, components of the sociotechnical approach to information systems^1,3.

We report on the successful adaptation of a published multi-variate cardiac risk model, the Cleveland Clinic treadmill mortality risk calculator⁴ to multiple medical centers in Kaiser Permanente (KP) Northern California. KP Northern California is an integrated healthcare delivery system providing comprehensive inpatient and outpatient care for more than 3.6 million members. KP physicians practice as part of a Permanente Medical Group, with peer-to-peer collaboration through Chiefs Groups. Each KP medical center has Chiefs of various areas, who have responsibility for oversight and improvement of clinical practice within their area. Most areas are clinical specialties, though there are Chiefs Groups for areas like Technology as well.

The Chiefs of Cardiology identified the need to better assess cardiac risk in patients during treadmill testing so that nuclear imaging tests, with their balance of harms and benefits, could be used more appropriately. They reached agreement about the best evidence-based clinical model, collaborated with KP’s Technology Chiefs and KP’s Clinical Library (CL) to create CDS that fit into daily workflow, and led integration and use by clinicians. We describe the sociotechnical steps we took to assure implementation, and report on use and associated outcomes.

Methods

Clinical need and clinical content

KP Chiefs Groups meet together regularly several times a year, led by a designated Chair. These peer groups act as change agents for general KP-wide initiatives, and also elevate specialty-specific concerns in order to draw upon KP-wide resources. The approximately 150 cardiologists in KP Northern California are represented through the Cardiology Chiefs. The Cardiology Chiefs reviewed region-wide utilization and determined that there was unwarranted variation in referrals to nuclear imaging after treadmill testing, that a common evaluation algorithm (the Duke TM score⁵) categorized low-risk patients as intermediate risk, and that nuclear imaging test results were not always needed to make the treatment decision. Nuclear imaging tests carry a risk of patient harm through unnecessary exposure to radiation. In addition, mis-categorizing patients at low risk of coronary artery disease can lead to further invasive evaluations or unwarranted medical therapy, where the risks outweigh the benefits. High-risk patients do often benefit from invasive procedures (e.g., cardiac catheterization), but low-risk patients can and should be effectively treated through medical management.

With the 2007 publication of a novel treadmill risk model⁴, the Chair of the Cardiology Chiefs identified an opportunity to improve and align KP’s evaluation algorithm so that more low risk patients could be appropriately identified without need for nuclear imaging. The algorithm gives mortality prognosis to patient and providers, taking into account functional capacity and other proven parameters including heart rate recovery. The algorithm is appropriate for patients with a normal baseline ECG, no known coronary disease, cardiac or renal transplantation, pacemaker or defibrillator placement, or end-stage renal disease. This algorithm had improved prognostic accuracy, allowing for appropriate conservative management. The algorithm could be encapsulated into a CDS model with the data available at the time of the treadmill test. Furthermore, it had been tested and validated in a population that included KP patients. Before implementation, the algorithm was further evaluated on a historical sample of KP patients. The algorithm is not meant to be a replacement for clinical judgment, and predicts mortality but not coronary artery disease.

Technology infrastructure

KP uses an electronic health record (EHR) by Epic Systems⁶ with integrated CDS. We also have robust CDS that interfaces with the EHR but lives outside it, allowing us greater flexibility in programming logic and maintenance. Clinical calculators comprise one such category of CDS, hosted through KP’s web-based Clinical Library. We extracted patient-specific data from the EHR and replicated the calculations reported in the literature using Javascript. In testing, we found that our initial results did not match those reported in the literature. Subsequently, we contacted the original researchers at Cleveland Clinic. They assisted us in translating among programming languages and more accurately matching our model parameters to their model; limitations of paper journal publication prevented reporting of the entire model. Additional testing was performed to confirm model accuracy. The cardiac risk calculator is available to all KP clinicians through the intranet at KP facilities, individual log-on to CL, or a KP-specific Toolbar on the top of every EHR screen. The Toolbar view is specific to the user’s clinical role (in this case, the Treadmill department) and integrates calculator access into EHR workflow.

Collaboration and Implementation

An ad-hoc group of clinicians and technologists collaborated to build the calculator. The Cardiology Chiefs directed the clinical algorithm selection and the target users, as well as the form of the CDS. They elected to implement it as a consultation tool to complement clinical judgment rather than, for example, a required step of the workflow⁷. These decisions led to conversations with different technology groups about CDS possibilities, including KP’s EHR development team, KP’s Toolbar team, and KP’s CL team. The calculator was tested with Cardiologists for ease of use, and functionality was built to show clinical definitions upon hovering over terms, and to generate a model report in a format that meets clinical documentation needs. The documentation was designed so the user could enter it into the patient chart with one click. The CDS calculator was developed over approximately 6 months. The algorithm endorsed by the Cardiology Chiefs and the calculator were publicized through regular meetings of the Chiefs Group and in-person conferences. The Cardiology Chiefs and other subject matter experts then carried the message to their own medical centers through verbal and written communication and specific in-person tutorials. The Cardiology Chiefs regularly review data on imaging use. Model use is tracked using Webtrends.

Results

The risk calculator became available to all KP users through Clinical Library in early 2010 (Figure 1), and is featured on the KP-specific Toolbar within the EHR for clinicians in the Treadmill department. It takes as input patient characteristics of age, sex, history of angina, diabetes, hypertension, smoking, treadmill test results of total METs, and ECG results. The age and sex can be pre-populated from the EHR, and the other fields are entered by the physician or clinician during the patient’s treadmill test. The model calculates 3, 5 and 10-year survival. The calculator generates a summary report which is shared with the patient and easily copied to their electronic chart as well. The workflow efficiency of an automatically generated report makes clinical documentation faster, more efficient, and more complete in contrast to manual transcription methods.

Figure 1.

Treadmill calculator on KP Clinical Library.

From 2010 to 2014, the calculator was accessed approximately 42,800 times, over 500 times a month (Figure 2). We count a visit as complete after 15 minutes of inactivity; if the page was accessed multiple times within 15 minutes, it is counted as one visit. We did not rigorously count the total number of eligible patients for the calculator, in part because, like all decision support, we would not necessarily expect it to be consulted for all eligible patients. A rough sense of the eligible population is the total number of outpatient treadmill referrals per month, approximately 3500 referrals; this referral volume includes patients who would be excluded from the algorithm, and does not include inpatient referrals. The Cardiology Chiefs’ communication, even before go-live, and the ease of access contributed to a large volume of views, achieving 500 views per month after the first three months. The calculator complements improvement initiatives to deliver patient-specific precision medicine, reduce unnecessary and potentially harmful testing, and spread best practices throughout KP. From 2006 to 2011, use of nuclear myocardial perfusion imaging declined by 51%⁸, after which imaging use remained flat. Over that same interval, KP maintained nation-leading outcomes in cardiac health⁹. The Cardiology Chiefs continue to receive and monitor imaging use data as a component of their quarterly dashboards.

Figure 2.

Number of annual calculator visits since deployment.

Discussion

Electronic clinical decision support can be effective at bringing knowledge to the point of care, but several factors influence its success, including whether the CDS supports a broader change initiative. Our sociotechnical experience of bringing a published model into our institution reveals several lessons.

People

Identifying, building, and deploying a model of care required ad-hoc collaboration among many groups, even in an organization that has many established structures for collaboration. This effort included collaboration between multiple clinical and technology teams. KP’s Cardiology Chiefs were a critical driving force in leading this effort. The Chair of Chiefs provided the intellectual and clinical case for change. Indeed, the momentum for alignment around one algorithm preceded and fueled the creation of an electronic tool, especially as the discussions around tool creation took many months. The tool was a solution to the desire to make it easy for practicing clinicians to align with the recommended algorithm; the algorithm, while considered better than alternatives, primarily served to ensure consistency of care and reduce harm. The Cardiology Chiefs determined that the most effective target audience was cardiologists rather than primary care physicians, due to the volume of candidate patients each specialty was likely to see. They directed how the model would fit into clinical workflow, how users were likely to access the system, and the simplicity required in the user interface to the model. They also directed how the interface should yield further information about the model details in order to increase confidence in the model. Most importantly, they had the trust of their peers and understood their roles as champions of change. The tool became available in the midst of a campaign for change, and joined the arsenal of methods to create and sustain recommended referral criteria.

Replicating a published model was not completely straightforward. In re-building their model, we contacted the original researchers for their model parameters as well as technical mathematical implementation details that differed between programming languages. We sincerely thank the researchers for their generosity in sharing their model and their time.

Technology

We chose to replicate the model despite the researchers providing a publicly accessible web-based version. Before our tool was ready, the Cardiology Chiefs provided the external URL in their presentations to align care for nuclear imaging. We have found that externally hosted models are not reliably accessible, and indeed the original URL of the published model is no longer valid. A replicated model also allows us to create tighter data interfaces to our EHR. The technical replication of a model is not entirely straightforward. Our EHR does not have an easy interface to build, modify, customize or maintain an algorithmic calculator, so we decided to build it in CL, which hosts several KP-built calculators. Interoperability between the EHR and external CDS is a challenge. We initially built screen scraping functionality to extract patient age and sex from the EHR; of late we have been able to use Epic Systems’ ClinKB function to extract patient-specific data to our new calculators. We find that some data elements cannot be pulled discretely from the patient record, and had to provide the ability for users to enter the data quickly and easily into our calculator. By building CDS outside the EHR, we were able to add functionality that improved clinical adoption, including explanations of rationale and tailored clinical documentation.

Process

The collaborative designers of the tool followed principles associated with good clinical decision support, including ease of use, workflow integration, meeting a user need, and getting feedback. User-entered data is not written back into the EHR from the calculator. We automatically copy calculator results in summary form to the computer desktop clipboard, which users can then paste into a patient note. We believe this dramatically increased adoption because the function was easy to use and made it quicker to document the encounter, overcoming the inherent inertia associated with a change (Figure 4). We find that performing calculations outside of the EHR has several benefits. We have seen the calculator used as a simulation and patient education tool, where users enter hypothetical parameters to see the results. For example, the calculator can show patients the effect of quitting smoking on their mortality risk, an often powerful addition to a smoking cessation conversation. Having the calculator outside the EHR also allows clinicians to exercise clinical judgment about when the model applies.

Figure 4.

Access from EHR through KP Toolbar.

The Chiefs Group endorsed and created awareness of the tool, in concert with the changes associated with appropriate imaging use. This involved duplicative communication: presentation at regular bi-monthly meetings, presentation at KP’s annual cardiology conference, and local peer-to-peer teaching. It also relied on clinicians’ preexisting knowledge of where such tools are found: KP’s Clinical Library and the KP-developed toolbar within our EHR. The Chiefs Group regularly reviews quality outcomes including the appropriate use of potentially harmful imaging modalities. This pre-existing organizational structure and process allowed us to achieve steady-state use within months, made organizational deployment relatively easy, and is reusable for CDS addressing other topics. Intangible, but critical, is a physician leadership culture that supports patient-centered decisions, continuous improvement, reduction in unnecessary interventions, quality, and safety.

Limitations

The deployment of our calculator was designed to complement a broader clinical initiative, so it is difficult to isolate the effect of the tool itself on the observed outcomes. Nuclear imaging use has been on the decline throughout KP since 2006, and the calculator is one of several tools to support clinical judgment. This aligns with published evidence that multi-modal interventions are more effective than CDS alone at changing clinical practice¹⁰. The literature supports our hypothesis that peer-based communication is effective in changing practice, but we did not measure outcomes, such as intent-to-change self-assessments, associated with these communications.

Conclusion

KP is a large integrated delivery system with robust communities of medical sub-specialties, and established channels for delivering CDS. The replication and uptake of a new calculator was markedly easier because we did not have to create sociotechnical mechanisms specifically for one CDS tool. We find the adage, “If you build it, they will come” is true with qualifications. Building a tool, even a published one, is not entirely straightforward. Vigorous communication through peer communities, support of workflow, and ease of use are critical ensuring that CDS affects patient outcomes.

Figure 3.

Sociotechnical components of successful replication.