Identifying Best Practices for Clinical Decision Support and Knowledge Management in the Field

Abstract

To investigate best practices for implementing and managing clinical decision support (CDS) in community hospitals and ambulatory settings, we carried out a series of ethnographic studies to gather information from nine diverse organizations. Using the Rapid Assessment Process methodology, we conducted surveys, interviews, and observations over a period of two years in eight different geographic regions of the U.S.A. We first utilized a template organizing method for an expedited analysis of the data, followed by a deeper and more time consuming interpretive approach. We identified five major categories of best practices that require careful consideration while carrying out the planning, implementation, and knowledge management processes related to CDS. As more health care organizations implement clinical systems such as computerized provider order entry with CDS, descriptions of lessons learned by CDS pioneers can provide valuable guidance so that CDS can have optimal impact on health care quality.

Introduction

Clinical decision support (CDS) provided through computerized provider order entry is essential to enhancing health care quality [1,2]. We define CDS broadly to include “passive and active referential information as well as computer-based order sets, reminders, alerts, and condition or patient-specific data displays that are accessible at the point of care” [3]. Federally funded incentives are sparking intense interest among most health care organizations to implement clinical information systems that include some CDS over the next few years [4]. At present, only 10 to 20 percent of U.S. hospitals have CPOE available [5,6]. Community hospitals account for 86% of the 5708 hospitals in the U.S. [7] However, only 6.9% of community hospitals report having even a basic level system [6]. Little research about CDS has been conducted in community hospitals, and in fact fully 25% of studies included in Chaudhry’s systematic review about the impact of health information technology on quality were from just four academic institutions [8]. In ambulatory settings, although 17% of physicians report that they use clinical information systems, only 4% use systems that include CPOE and CDS [9]. A major criticism of current health information technology (HIT) articulated in a recent report is that HIT fails to provide cognitive support to clinicians when they need to make decisions [10]. In fact, it can produce many unintended adverse consequences [11], especially related to clinical workflow [12]. Because many problems with CDS relate to behavioral, organizational, and cognitive issues [13,14], the Provider Order Entry Team (POET) based at Oregon Health & Science University in Portland, OR, U.S.A. is conducting two multi-site ethnographic studies about these issues, one in community hospitals and the other in ambulatory clinics and vendor environments. The purpose is to identify best practices for CDS implementation and knowledge management. We define best practices broadly to include widely recognized and agreed on procedures and practices that have been shown to improve internal processes.

The following sections summarize foci, methods, and results.

Methods

We adapted a form of ethnography which has been used successfully in international health for the rapid assessment of complex site-based situations, the Rapid Assessment Process (RAP) [15]. By using structured assessment instruments, expert interviews, field surveys, and intensive site visits by a multidisciplinary research team, we minimized the need for lengthy periods of time in the field (a characteristic of traditional ethnographic fieldwork). We also held a conference of CDS experts to help us verify what we found in the field.

Sample Selection

Because we aimed to gather information about best practices, we selected clinical sites with reputations for excellence in the use of clinical systems and CDS. Table 1 outlines attributes of our study sites. The hospitals were community hospitals with different commercial systems [16]. The ambulatory sites were all members of the Clinical Decision Support Consortium [17]. Four different commercial systems and three locally-developed systems are used at our selected clinical sites. During visits to clinical sites with commercial systems, the team realized that we could not gain a complete picture of knowledge management processes unless we could learn more about the companies that provide CDS content made available through commercial electronic medical record (EMR) systems. We selected two companies to visit first because they provide much of the CDS available through popular EMRsystems; other vendor visits are planned.

Table 1 –

Sites and data collected

	Providence Portland Medical Center	El Camino Hospital	Partners HeaithCare	Wishard Memorial Hospital Clinics	Roudebush Veterans Health Administration	Mid-Valley IPA	RWJ Medical Group	Zynx	First DataBank
Location	Portland, OR	Mountain View, CA	Boston, MA	Indianapolis, IN	Indianapolis, IN	Salem, OR	New Brunswick, NJ	Los Angeles, CA	San Francisco, CA
Characteristics of setting	Community hospital	Community hospital	Academic and community outpatient	Academic and county clinics	VA outpatient clinics	Community outpatient	Academic outpatient	Content vendor	Content vendor
Type of system	Commercial	Commercial	Locally developed and commercial	Locally developed	Nationally developed	Commercial	Commercial	NA	NA
Date of visit	12/07	2/08	6/08	9/08	9/08	12/08	2/09	7/09	10/09
Hours observing	36	26	37	20	25	33	26	NA	NA
Individuals observed	10	12	17	16	17	27	17	NA	NA
Number of clinics observed	NA	NA	9	6	5	9	6	NA	NA
Number of interviews	15	12	13	9	9	9	12	6	6

We selected as study sites two community hospitals, five ambulatory settings, and two companies that provide CDS content. Providence Portland Medical Center in Portland, OR is an urban community hospital, part of a larger hospital system, using a commercial system. El Camino Hospital in Mountain View, CA is a stand-alone suburban hospital with the oldest CPOE implementation in the world, also using a commercial system. Ambulatory sites included Partners HealthCare in the Boston, MA area, which primarily uses a locally developed system but also includes some sites with commercial EMRs; Wishard Memorial Hospital, a county hospital in Indianapolis, IN, which uses the locally developed Regenstrief Medical Record System; the Roudebush Veterans Affairs Hospital in Indianapolis, IN which uses the VA’s nationally developed CPRS system; the Mid-Valley Independent Practice Association (MVIPA) in the Salem, OR area, which uses a commercial system; and the Robert Wood Johnson (RWJ) Medical Group, in New Brunswick, NJ, which also uses a commercial system. The vendor sites were Zynx Health in Los Angeles, CA, which provides evidence-based order sets and interdisciplinary plans of care and First DataBank in South San Francisco, CA, which provides medication information. We received human subjects approval from each investigator’s home organization and from each appropriate site.

Within each organization, we selected subjects with the assistance of a local sponsor. For interviews, we purposively selected a broad spectrum of users who ranged from champions to skeptics, individuals who either develop or customize CDS, support staff, administrators, quality officers, and IT staff members. We observed users entering orders in many areas of the hospitals and in a broad spectrum of primary care and specialty clinics. At vendor sites, we interviewed CEOs, individuals who develop content, technical staff who work with EMR vendors to make content available electronically, training and support staff, and marketing managers. Experts for the expert conference were selected based on their experience with community hospital and ambulatory CDS.

Data Collection

Figure 1 shows the progression of data gathering and reporting that occurred between 2007 and 2009, including the focus of our work, activities, study sites, and publications. After several months of preliminary preparation, including development of an inventory of CDS types and knowledge management practices [18] and a demonstration of each system for each site, four to seven researchers spent three to five days at each clinical site. We visited Zynx and First DataBank for less time. The nature of these last two site visits was quite different since the focus was the vendor perspective and the visits did not involve clinical observations.

Figure 1–

POET research overview 2007–2009

*Foci, activities, and sites as they were approached over time.

For the community hospitals and the ambulatory settings, we asked questions and observed individuals and committees at work to learn more about: Policy, governance, control, culture; User issues including usability, training, and impact on workflow; CDS content/knowledge management practices; and Technology and infrastructure issues. While visiting the content vendors, we asked about: The marketplace, their niche and relations with EMR vendors; CDS content/knowledge management processes; Customers—types, frustrations, feedback, challenges; and Use of the product.

Data Analysis

The nature of the Rapid Assessment Process is such that we needed to analyze the data quickly so that we could provide feedback to the site in the form of a report, receive comments back from subjects as a form of “member checking” [19], and then prepare for the next site based on what we had learned. We did this expedited analysis by dividing into pairs to scan transcripts and field notes to identify user, administrative, and IT perspectives at each site to write sections of each report. As we did this, we noted general themes, which we began to identify during debriefings held twice a day during site visits. We developed a codebook and used a template method [20] to roughly code the data. For each report, we identified best practices (e.g. what they did especially well) and challenges in addition to themes. We requested feedback about the reports from sites as a form of member checking. Once the reports were written, we began using a more classical qualitative research approach that was inductive and interpretive. This involved content analysis using the words and actions of the subjects to identify patterns and themes to modify and augment the already-identified codes. We discussed the best practices identified across sites with our panel of experts and the CDSC research team.

Iterative Methods

Although we had carefully developed a field guide containing a site inventory profile about CDS at the site, interview question guides, observation guides, and even guides to debriefing, we found that we needed to modify each instrument for each site visit [21]. We made major changes when we started visiting outpatient sites and again when we started vendor visits. The three types of settings vary greatly in their approaches to CDS. Below we describe a number of best practices for optimizing the value of CDS that emerged from our analysis.

Results

Best Practices

Five large categories of best practices emerged from our analysis:

Best Practice 1: View CDS broadly.

Experts and users have widely differing definitions and understandings of CDS, causing a barrier to optimization of CDS use and value. The users appreciate CDS that helps them get through their day, a kind of CDS we call “inline” because it is integrated with their workflows. Experts usually describe CDS in terms of sophisticated alerts and reminders. The vendors providing content view CDS as a way to foster the practice of evidence based health care. There is a danger in defining CDS too narrowly in terms of sophisticated alerts and reminders that are available only at elite institutions. While it is useful to learn from their experiences because they are on the cutting edge and can provide models of CDS use for the future, they are not typical of the majority of sites that provide health care nationally. By defining CDS broadly so that it includes passive elements such as default values and workflow enhancement features such as templates to assist with data gathering, documentation, and clinical reports, all stakeholders will be continuously reminded that the clinicians who are the recipients of CDS are the ultimate customers who need to accept and use these features.

Best Practice 2: Move forward with simple CDS no matter what your size.

If an organization has the wherewithal to implement CPOE, it can also use CDS. There is some “low hanging fruit” among CPOE-delivered types of CDS. All of our study sites worked hard to review and select appropriate order sets so that they were available electronically when CPOE was introduced. Order sets assist workflow because they streamline ordering and at the same time they help decision-making. Well-designed checklists and templates can provide guidance at the moment the clinician needs it.

Best Practice 3: Focus on “inline” CDS.

We use this term to refer to CDS that does not interfere with a clinician’s workflow. One type to consider is “background CDS” which works behind the scenes to consider data related to an individual patient along with rules for good care and makes recommendations such as what antibiotic to order. Another kind of inline CDS is that which notifies an intermediary such as a nurse or pharmacist rather than a physician. These strategies can be especially effective in community settings where interrupting the physician in private practice can be particularly burdensome.

Best Practice 4: Use what is available from your vendor but plan to customize the CDS.

Each hospital and clinic has its own culture and need for CDS. Sites with locally-developed CDS tend to identify a clinical need and then design a CDS intervention to address it. Organizations with commercial systems may not have the staff to do this, but they can sometimes purchase appropriate CDS content through their EMR vendors. EMR vendors usually do not develop their own CDS. Instead, they rely on content development companies to write content and partner with the EMR vendor to provide it electronically. Usually both the content and EMR vendors provide a wide range of potential CDS interventions, leaving it to the customer (the hospitals and purchasing organizations) to filter out those that are not wanted. This is a necessary, but time and resource-consuming step and often more difficult than expected. Customers must have clinical leaders and skilled “analysts” on staff who understand both clinical work and information technology.

Best Practice 5: Plan knowledge management processes early.

The sites we studied that had locally-developed systems are struggling to catch up with their knowledge management needs. Each one found itself at a point where so much CDS had been developed and implemented that it was hard to keep track of it for the purpose of updating and maintenance. Organizations that have implemented commercial EMRs with CPOE more recently can take advantage of knowledge management capabilities available through vendors [22]. First, the EMR vendors track changes made at the vendor level so there is an audit trail available. A wise organization tracks its customized changes as well so that as time passes, a history of modifications is always available. Second, tools are available from content development vendors specifically designed to help with knowledge management. They can assist with the decision-making and consensus development process, notify customers when updates are needed, and keep an audit trailof decisions. Good knowledge management practices depend on three foundations: the availability of skilled information systems staff, the existence of a well-developed consensus development and decision making structure, and robust computer tools.

Discussion

We have seen that organizations can do this right [23] and can serve as models of best practices. Because we talked with representatives of many stakeholder groups interested in CDS, because we observed clinicians using CDS, because our study sites included a wide variety of sizes and types of health care delivery organizations, and because we verified a number of best practices outlined by others [2,24], we believe the five best practices categories described above represent a high level view that can be useful to all types of organizations planning to implement CPOE with CDS. All of these best practices depend on planning ahead, ideally prior to CPOE implementation. Best Practice 2, moving forward regardless of size, might entail numerous clinics organizing into a larger organization for the purpose of purchasing and maintaining an information system. This is a large undertaking, requiring significant planning, but it can possibly deliver other financial advantages beyond information systems implementation. All of the best practices also involve the availability of skilled informatics specialists. Two of our sites outsourced the customization and maintenance functions to their EMR vendors by contracting with their service organizations. Availability of a clinician leader is absolutely necessary, even if some activities are outsourced. Study limitations include restricted fieldwork and as yet incomplete vendor visits.

Conclusion

The use of ethnographic techniques to study best practices for CDS and knowledge management in the field allowed us to understand and synthesize a broad range of perspectives. Our study sites included many types of organizations; all of them can be considered pioneers because wide use of CPOE with CDS is still years in the future. Lessons these pioneers teach us now can provide valuable guidance so that CDS can eventually have optimal impact on health care quality worldwide.