The state of the art in clinical knowledge management: An inventory of tools and techniques

Abstract

Purpose

To explore the need for, and use of, high-quality, collaborative, clinical knowledge management (CKM) tools and techniques to manage clinical decision support content.

Methods

In order to better understand the current state of the art in CKM, we developed a survey of potential CKM tools and techniques. We conducted an exploratory study by querying a convenience sample of respondents about their use of specific practices in CKM.

Results

The following tools and techniques should be priorities in organizations interested in developing successful computer-based provider order entry (CPOE) and clinical decision support (CDS) implementations: 1) A multidisciplinary team responsible for creating and maintaining the clinical content; 2) An external organizational repository of clinical content with web-based viewer that allows anyone in the organization to review it; 3) An online, collaborative, interactive, internet-based tool to facilitate content development; 4) An enterprise-wide tool to maintain the controlled clinical terminology concepts. Even organizations that have been successfully using Computer-based Provider Order Entry with advanced Clinical Decision Support features for well over 15 years are not using all of the CKM tools or practices that we identified.

Conclusions

If we are to further stimulate progress in the area of clinical decision support, we must continue to develop and refine our understanding and use of advanced CKM capabilities.

Introduction

Since the Institute of Medicine’s release of “To Err is Human: Building a Safer Health System” in 1999 [1] and “Crossing the Quality Chasm: A New Health System for the 21st Century” in 2001 [2], there has been increasing pressure on everyone from healthcare providers working in solo practices to large nationwide, healthcare delivery networks to implement state of the art clinical information systems. Recent reports have reiterated the fact that, for these clinical information systems to deliver their expected improvements in quality, they must be equipped with state of the art, real-time, point-of-care, clinical decision support (CDS) features [3]. Unfortunately, the successful design, development, implementation, maintenance and evaluation of these advanced clinical decision support features is in itself a complex undertaking that is not well understood [4]. These advanced clinical decision support systems generally depend on large knowledge bases of clinical knowledge. For example, a drug-drug interaction checker requires a database of drug names and their interactions. This database must be kept up-to-date as new drugs become available, as new drug interactions become known and as our understanding of existing interactions evolves. Manual systems can be used for this task when the amount of clinical knowledge to manage is small (i.e., less than 100 interactions), but as the size of these databases grows, clinical knowledge management (CKM) tools become necessary to ensure that these clinical knowledge bases are correct, consistent, complete and current. In fact, the area of research and development referred to as “clinical knowledge management” will soon become vitally important as organizations struggle to develop, implement, maintain, and evaluate their clinical decision support efforts.

The need for high quality, collaborative, CKM practices is often overlooked even in the most advanced healthcare organizations [5]. The need to develop high quality, evidence-based CDS interventions, coupled with the need to obtain widespread, peer review of this work, makes the use of an internet-based, collaborative, CKM tool almost mandatory. Moreover, this clinical content must be kept up to date as the existing knowledge changes, new clinical knowledge is gained, and underlying changes take place in the clinical information system [6]. This places enormous burdens on the informaticians responsible for maintaining all of the clinical applications that rely on this knowledge including: Computer-based Provider Order Entry (CPOE), clinical results review, clinical documentation, and medication administration systems, for example.

In a recent article, we identified “disseminate best practices in Clinical Decision Support (CDS) design, development, and implementation” as one of the “grand challenges” in CDS [7]. A key component of any CDS initiative is how the organization develops, disseminates, maintains, and evaluates its clinical knowledge content, or its Clinical Knowledge Management practices [8].

In an attempt to explore the current state of the art in CKM, we developed an extensive survey of potential CKM tools and techniques (see Appendix). This manuscript describes the content of the CKM survey and reports our initial findings from administering it to a convenience sample of six organizations with substantial experience in the development and use of advanced clinical information systems with state-of-the-art, real-time, clinical decision support systems.

Methods

Survey Development and Testing

We developed a survey of clinical knowledge management tools and techniques, which we call the clinical knowledge management inventory, based on our extensive review of the CPOE and CDS-related literature (see [9,10] for links to an online bibliography of these two domains), interviews with experts and our own experiences in working in the field for several decades. Please see Table 2 for a list of questions.

Table 2.

Questions used to identify current practices in clinical knowledge management.

1: Do you have a dedicated, multidisciplinary team of individuals responsible for creating and maintaining your clinical knowledge and clinical decision support interventions [39, 40]?

2: Do you modify the external clinical knowledge (content) you use before implementing it [41]? If so, how?

3: Do you store your clinical knowledge externally using one of the existing clinical knowledge representation standards (Arden, GLIF, GEM [42])?

4: Does your clinical information system have the capability of using one of the knowledge representation standards (Arden, GLIF, InfoButton, etc.) to execute its clinical decision support [43,44,45,46,47,48]?

5: What types of people do you have to help manage your clinical knowledge [49]?

6: Does each piece of clinical content (i.e., alert, order set, and patient education material) have an individual responsible for monitoring the underlying clinical knowledge and maintaining the CDS intervention [50]?

7: Can your system send automatic reminders that content review date is approaching to individuals responsible for maintaining the content [51]?

8: Do you have a system in place to send automatic notifications to key stakeholders when clinical guidelines underlying CDS change [52]?

9: Do you have a system in place to “learn” new clinical knowledge from your patient database [53]?

10: Do you purchase, or share, clinical knowledge from external sources, e.g., reference information, drug information (dose ranges, interactions, patient handouts, indications, etc.), order sets, alerts / reminders [54]?

Survey Site Selection

We identified six geographically (2 Northwest, 2 Midwest, 1 Southwest, and 1 Northeast) and organizationally (a mid-size community hospital, a large nationwide health maintenance organization, two large integrated delivery networks, a county hospital affiliated with an academic medical center, and a Veteran’s Administration hospital) diverse healthcare organizations. Four of the six have been recognized for their excellence in implementation and use of advanced clinical information systems including use of clinical decision support features and functions [11,12]. In addition we sought to identify sites representing healthcare organizations with both commercially acquired and internally developed clinical information systems.

Survey Administration

Following Institutional Review Board (IRB) approval, we pre-tested the CKM site inventory with two individuals in the Portland, Oregon area. After refining several questions, we then distributed it, via email, to a single individual at each of the study sites who was directly involved in CPOE / CDS and CKM activities. If this individual had any specific clarifying questions regarding particular survey items, we answered them promptly via email or telephone.

Results

We have divided the results into three sections. In the first section, we present characterizations of the six organizations’ use of computer-based provider order entry (CPOE), various CDS intervention types, and their clinical knowledge management activities. The second section describes the content of the CKM site inventory including: 1) an overview of the CKM tools we identified as keys to success and 2) questions we asked at each site to help us better understand current practices in clinical knowledge. In the third section, we present our findings from administering this survey within six disparate organizations.

Results: 1. Characterizations of the six organizations

Three of the six sites used commercially available clinical information system products (McKesson, San Francisco, CA; Epic Systems, Verona, WI; Cerner, Kansas City, MO) while three used in-house designed and developed systems. The sites had been live with their CPOE / CDS systems from 2 – 22 years (median: 13 years). The number of order sets each site had in their production environment ranged from 185 – 801 (median: approximately 325) while the number of alerts in the production environment ranged from 40 – 7,120 (median: less than 100). In terms of the number of people (e.g., clinical domain experts, knowledge engineers, nurses, evidence-based medicine experts, informaticians, information technology professionals) involved in discussions regarding specific clinical decision support interventions, the range was between 5 and 15 people (median: 7).

Results: 2. Content of the Clinical Knowledge Management (CKM) Site Inventory

Through interviews with experts and a review of literature, we identified four tools that appear to be useful for CKM. We describe these tools in the following sections.

Tool 1: External repository of clinical content with web-based viewer

Clinical decision support content is often embedded directly in the logic of clinical information systems. This can be expedient, but, when intermingled, the content can be much more difficult to locate and maintain. While only a few of the organizations we surveyed have fully implemented such an external repository, all agreed that the ideal practice is to maintain the clinical knowledge separate from the code that implements the information system in an external repository that is both browsable and searchable. Often this repository consists of either a database or file storage system with a dedicated application interface or a browser-based interface. The key here is not the format or access mechanism for the knowledge, but, rather that storing the knowledge in an external repository, rather than just the clinical information system, facilitates the ability of clinicians to maintain, share, and review the content. With such a system, a knowledge engineer can quickly answer the question, “What is our organization doing to address the Joint Commission’s new Venous Thromboembolism (VTE) Core Measure [13]?” by simply searching the repository for all CDS content containing the term “venous thromboembolism”.

Tool 2: Online, collaborative, interactive, intranet-based tool to facilitate content development

Developing or modifying new clinical decision support content within an organization is always a challenge. A variety of people are usually involved in content development, including physicians, nurses, pharmacists, informaticians, software developers and quality improvement professionals. Bringing all of these participants together for in-person meetings is challenging. Allowing content developers and users to have an asynchronous discussion regarding the pros and cons of specific CDS interventions can be a valuable method of both developing new clinical content and gaining organizational consensus. Many organizations are beginning to experiment with new Web 2.0 technologies such as wikis, discussion boards, or listservs to facilitate distributed and asynchronous creation, consensus-building, and review of clinical content [14]. These tools are enabling such organizations to include more people in the conversation and to reach consensus more quickly than ever before.

Tool 3: Enterprise-wide tool to maintain controlled clinical terminology concepts

One of the most important prerequisites to the development and use of advanced, patient-specific CDS features and functions is the availability of a robust, controlled clinical terminology (e.g., SNOMED for problems, LOINC for lab results andICD-9 for billing diagnoses, etc.). Without such a controlled clinical terminology, CDS developers will have difficulty using the “freetext” clinical concepts within their logic. In addition, many controlled terminologies include some sort of semantic network that maintains various types of relationships among the clinical terms. These relationships can greatly simplify the construction and maintenance of the CDS. For example, a knowledge engineer might want to encode the following logic, “if the patient is on an ACE inhibitor, then…”. Without a controlled terminology that includes hierarchical relationships, the knowledge engineer would be required to list all of the Angiotensin Converting Enzyme inhibitors (i.e., ACE inhibitors) currently in use within the organization, and then update this list whenever a change was made to the organization’s formulary. Using such a tool (see figure 4), a knowledge engineer would simply add the new medication name, or a synonym for it, to the terminology and note that it is a “child” of the term ACE inhibitors. Then in the rule logic, the knowledge engineer would call the “ACE inhibitor query” (i.e., rather than listing all the potential ACE inhibitors by name) which would find all the medications in the ACE inhibitor category and compare the patient’s current medications to that list and return TRUE if one exists, FALSE otherwise.

Figure 4.

Screen print showing the tool that Kaiser Permanente uses to manage term relationships within their convergent medical terminology. (Courtesy of Robert H. Dolin, MD)

Tool 4: Tool for CDS users to provide feedback regarding specific CDS interventions

No matter how hard an organization works prior to deployment of new CDS interventions, there will always be clinicians who find justifiable exceptions to any type of CDS (e.g., alert, order set, condition-specific results display, default medication dose, etc.) developed. In addition to regularly scheduled meetings and clinical analysts who frequently interact with the clinicians at their work site, many organizations have developed a simple on-line method of gathering feedback from users while they are actually using the CDS intervention. These systems often make use of commonly available computer-based communication tools such as email, discussion groups, or databases. Often these tools capture some basic information about the user, the patient, and the clinical context in which they were working in addition to the user’s complaint(s) or suggestions. It is critical that organizations have an organized and efficient means of responding to these “suggestions” or “complaints”.

Current practices for clinical knowledge management

In addition to asking about the availability and use of CKM tools, our site inventory also asked participants about their use of specific practices for CKM. We developed this list (see Table 2) based on an extensive review of the literature surrounding clinical decision support development and maintenance in addition to interviews with experts in the field and observations conducted at many of the leading academic medical centers in the United States. While we do not yet have randomized clinical trial level evidence that any of these features, functions, or organizational practices results in higher quality clinical decision support, our experience leads us to believe that these ten items are at the very least key issues to consider when developing and maintaining high-quality clinical decision support systems.

Results: 3. Findings from administering the CKM site inventory

The first part of the CKM site inventory asked organizations to describe whether they had any of the following CKM tools (See Table 3). If they responded positively, we than asked them to describe how they were using each tool.

Table 3.

A List of clinical knowledge management tools available at each participating institution.

Participating organizations	Tool 1: External repository of clinical content with web-based viewer	Tool 2: Online, interactive, internet-based content development tool	Tool 3: Enterprise-wide tool to maintain controlled clinical terminology concepts	Tool 4: Tool for CDS users to provide feedback regarding CDS interventions
Kaiser Permanente, Northwest	Method to extract and download human-readable versions of alerts and order sets to repository	Yes, via intranet, Kaiser National website, & Epic Community library [14]	Internal, Convergent Medical Terminology controlled at National KP level 55	E-mail address for CDS-related feedback exists.
Providence Health System, Portland	Limited availability via order set library, pdf of what is built in system.	Zynx author space [56] and viewspace; and Sharepoint site for new content development and vetting by clinicians	No.	Complaint button
Regenstrief Institute, Inc.	No external repository, because CDS knowledge is integrated into the Gopher Order Entry system.	No	Internal dictionary of 33,845 terms, mapped to standard terminologies. Terms grouped, with several levels of hierarchy	Users e-mail system developers with feedback.
Partners HealthCare System	Content is developed, reviewed and curated in “Documentum’s eRooms” using semi-structured knowledge specifications.	Yes, eRooms [57],	Combination of proprietary and standard terminologies with terminology services, hierarchy and subsets.	Interventions have a feedback button and pager number.
Roudebush VAMC	No	Web-based work order system to request new tools.	CPRS/VistA for controlling ICD- 9, CPT, & DRG groups	Tool on user interface menu to document problems & suggestions
Memorial Hermann	Yes. use Sharepoint to maintain CDS inventory of CDS. Have separate databases for Order sets, Alerts/Reminders.	Zynx View Space for Order sets	No	Yes, for order sets. users can provide feedback via email or production support remedy tickets

Tool 1: External repository of clinical content with web-based viewer

These clinical knowledge repositories are being used in two different ways at four of the six sites. For example, at Partners, the CKM repository (see figure 1) documents the results of the on-line and in-person discussions and agreements regarding the proposed clinical content. The content of the CKM repository is then used by the programmer/analysts as they manually implement the logic within their clinical information system. On the other hand, within Kaiser Permanente Northwest, content is created directly in the clinical information system and the CKM repository is built by extracting the clinical knowledge from the clinical information system into a intranet-accessible directory which can be viewed, searched, or browsed via a simple web interface (see figure 2). Both of these methods of creating a CKM can create synchronization problems. For example at Partners, the clinical programmer/analysts may have to make slight modifications to the logic when they implement it within the CIS. If they do not simultaneously make these changes to the documents contained in the CKM, a synchronization problem occurs. Likewise, with the Kaiser Permanente Northwest solution, the content of the CKM is only as current as the last extraction. As shown in figure 2, which was created on August 26, 2008, the content was last extracted on August 14, 2008. While 12 days may not seem like a long time lag, it is quite possible that the clinical content may have changed since the last update. Again, the point is that any manually maintained external repository of clinical content can easily become out of sync with what is being used in the production-level clinical information system.

Figure 1.

Screen print showing the search interface to the Partners Knowledge Management Portal.

Figure 2.

Screen print showing table of contents of all clinical knowledge that has been extracted from the clinical information system.

Tool 2: Online, collaborative, interactive, intranet-based tool to facilitate content development

All, but one, of the sites are using some sort of on-line, collaborative, intranet-based tool to facilitate content development. Figure 3 shows a recent example use of Partners’ eRooms for discussion of how the outpatient EHR should behave when a clinician indicates that a patient has had a colonoscopy examination when the patient reports that he/she had it “done elsewhere”. The developers of their electronic medical record system had received complaints from end users who felt they were being inundated with “false-positive” reminders. The Partners Knowledge Management staff created a new discussion area within eRoom and invited members of the primary care subject matter expert panel to share their thoughts. Users discussed the various reminder intervals and participated in a poll. Ultimately, the decision was made to delay the reminder for 5 years in the event that either a colonoscopy or sigmoidoscopy had been completed and for 1 year if the patient had a stool guaiac test performed.

Figure 3.

Example of an online, guided discussion in Partners’ eRoom (Note: Clinician identifying information has been obscured for privacy.)

Tool 3: Enterprise-wide tool to maintain controlled clinical terminology concepts

As mentioned above, controlled clinical terminology is critical to successful implementation and maintenance of large-scale (i.e., more than 100 rule-based alerts or over 100 order sets) clinical decision support operations [15, 16]. In order to maintain such a terminology, four of the six organizations have implemented sophisticated computer-based tools that allow knowledge engineers and terminologists to create new terms and manage the relationships between these terms as well as hierarchical concept groupings (see [17] for a more complete list of requirements for these tools). Figure 4 shows a screen print of Kaiser Permanente’s tool for managing their convergent medical terminology [18]. In addition to this “in-house” developed example, there are several commercially available products that are available, however none of the organizations we studied were currently using any of these products, which include: Apelon (Ridgefield, CT) with their Distributed Terminology System (DTS) which recently became available as an open-source product [19]; Intelligent Medical Objects (Northbrook, IL) IMO® Health Objects: Clinical Interface Terminology; and Health Language® (Aurora, CO) (HLI) Language Engine™. The reason that none of the organizations we surveyed were using any of these commercially available terminology maintenance tools is based on their history and not from any failing on the part of these vendors. The four organizations with locally developed tools are among the early leaders in the field of clinical decision support and they identified the need to have these sorts of tools long before any commercially available tools were available. To date, they have all made the decision that the benefits of moving to one of these new commercially available tools are outweighed by the costs in money, time, effort, and potential loss of idiosyncratic functions that switching would inevitably entail. Finally, many of the commercial electronic medical record vendors either directly provide or collaborate with companies like those listed above to offer very similar types of tools.

Tool 4: Tool for CDS users to provide feedback regarding specific CDS interventions

All of the sites surveyed had on-line tools to facilitate the collection and analysis of user feedback. Figure 5 shows an alert for Deep Venous Thrombosis prophylaxis from the Brigham & Women’s Hospital in Boston, MA. If the user wishes to communicate with the system developers regarding this alert, they can click on a <Comment> button which brings up a feedback screen. In this instance the developers have created both a structured as well as a freetext feedback form. All feedback is sent directly to the specific “rule owner” for each rule.

Figure 5.

Top: Shows an alert for Deep Venous Thrombosis prophylaxis. Clicking on the >Comment< button at bottom right of the Alert screen (top) brings up theAlert Evaluation screen (bottom). Input to section 1 (check boxes) and section 2 (small comment box) are captured and stored for later reporting and review by rule owner. Input to section 3 (large comment area) is sent to the rule owner and any other person designated via email. Recipients are specific to each rule.

Current practices for clinical knowledge management

The final portion of the CKM site inventory asked participants about their use of specific practices for CKM.

Question 1

Do you have a dedicated, multidisciplinary team of individuals responsible for creating and maintaining your clinical knowledge and clinical decision support interventions?

All organizations had such a team and all agreed that these individuals were the most essential component of their CDS success. In addition, they stated that a transparent governance structure for all content-related decision making was also important. They felt that transparent governance structures were necessary to ensure that all relevant stakeholders had a say in the final version of the content. Further, transparency was important when consensus could not be achieved. This became more common as the scale of the organization grew, for example in disagreements between individual provider vs. practice/clinic level vs. institution vs. integrated delivery network; and in the future, regional and national networks.

Question 2

Do you modify the external clinical knowledge (content) you use before implementing it? If so, how?

Many organizations purchase or otherwise bring in externally a portion of the clinical knowledge they use. In our surveys, it appeared that drug knowledge bases were the most widely purchased assets. However, the current state of the art in purchasing or sharing external clinical knowledge requires at least a minimal, and potentially, a great deal of modification. In addition, once the content is deemed acceptable by the local organization, it must still be translated into a computable form (e.g., the Gopher Care language in use at Regenstrief or Epic’s internal clinical decision support structures) by local physicians, nurses, or pharmacists with informatics training or sometimes informaticians without specific clinical training or experience. Some sites reported making substantial modifications to the clinical content, for example, selecting a small subset of the most clinically significant drug-drug interactions for use in alerting [20].

Question 3

Do you store your clinical knowledge externally using one of the existing clinical knowledge representation standards (Arden, GLIF, GEM [21])?

None of the sites contacted in this survey used any of these standard representation formats.

Question 4

Does your clinical information system have the capability of using one of the knowledge representation standards (Arden, GLIF, InfoButton, etc.) to execute its clinical decision support?

None of the sites contacted in this survey used any of these representation standards. There were many reasons for this including: the CIS vendors did not support these standards; the standards were lacking in one or more key features that the organization felt was critical for their success [22], and many sites had a significant number of similarly functioning clinical decision support interventions developed within their systems before any of these relatively new standards were developed (see for example, McDonald, 1976 [23]. Furthermore, several organizations have implemented “infobutton-like” functionality by hard-coding uniform resource locators (URLs) into their system to provide users with key patient-specific or context-sensitive links to reference information directly from the screen that presents the patient’s current medications (though none are using the HL7 Infobutton manager standard) [48].

Question 5

What types of people do you have to help manage your clinical knowledge?

All of the organizations studied had one or more of the following types of people involved in the CKM process:

• Pharmacists with formal informatics training (e.g., Masters or Doctorate in Medical Informatics or Informatics fellowship) or extensive clinical informatics experience to develop and maintain pharmacy content

• Physicians with informatics experience to translate clinical guidelines and study protocols into CDS interventions;

• Doctoral-level medical informaticians.

• Registered nurses (RNs) with informatics experience;

• Dedicated software developers and project managers without a clinical background.

Question 6

Does each piece of clinical content (i.e., alert, order set, and patient education material) have an individual responsible for monitoring the underlying clinical knowledge and maintaining the CDS intervention?

All organizations used some sort of distributed CKM maintenance system. For example, all organizations had a pharmacist informatician responsible for developing and maintaining the content related to medications (indications, interactions, side effects, monitoring, default dosing, preferred medications and formulations). In addition, they each had physician informaticians with broader clinical oversight roles.

Often content specific to a particular clinical department (e.g., procedure or problem-based order sets) is managed by a clinical champion or medical informatician in that department.

Question 7

Can your system send automatic reminders that content review date is approaching to individuals responsible for maintaining the content?

Even though none of the organizations surveyed currently had this capability, all of them recognized the problems associated with keeping their clinical content current.

Question 8

Do you have a system in place to send automatic notifications to key stakeholders when clinical guidelines underlying CDS change?

Only one organization, which contracted with an external organization, Zynx, had this capability, though use of this functionality has been very limited. On the other hand, as the reliance on clinical decision support imbedded within clinical information systems increases all organizations realize that monitoring the clinical guidelines upon which the decision support is based will become even more important. Our experience has been that finding the resources to continuously monitor and maintain CDS interventions can tax even the most careful and financially secure organizations. Over the next several years we expect that more and more organizations will begin “out-sourcing” this work to commercial organizations that designed specifically to address this issue.

Question 9

Do you have a system in place to “learn” new clinical knowledge from your patient database?

None of the organizations in this study had this capability in place, although several have conducted specific research projects to begin experimenting with this type of capability [24]. Once again, over the next several years, most of the organizations expect this capability to become more common as they continue to develop large clinical data repositories and their informatics’ researchers develop the techniques necessary to accomplish this task [25].

Question 10

Do you purchase, or share, clinical knowledge from external sources, e.g., reference information, drug information (dose ranges, interactions, patient handouts, indications, etc.), order sets, alerts / reminders?

All sites except for one relied on an external drug information database for medication-related information. All sites surveyed had sophisticated internal systems in place to develop and manage CDS knowledge in response to clinical user feedback, as well as guidelines published in the medical and pharmacy literature, although we do not think that these practices are common at other organizations. All organizations either were using, or were considering the purchase of externally developed clinical order sets.

All of the sites also reported involvement in one or more efforts to share their content externally with other clinical sites or the general public. These sharing efforts included federally funded contracts to develop systems to share data within a small group of consortium members [for example, 26] as well as efforts to share content with other organizations using the same clinical information system vendor [14].

Discussion

We identified a variety of tools and current practices for CKM. After reviewing all of the responses to our survey and discussing the summarization of the data, we have identified the following tools and practices as the most widely used in organizations with successful CPOE and CDS implementations:

1. A multidisciplinary team responsible for creating and maintaining the clinical content.

2. An external repository of clinical content with web-based viewer that allows anyone to review it.

3. An online, interactive, internet-based tool to facilitate content development and collaboration.

4. An enterprise-wide tool to maintain the controlled clinical terminology concepts. Although these tools and practices are the most widely used, support was also seen for many of the other practices and tools we identified, even if adoption of them was, at present, more limited. We anticipate that use of these tools will increase over time.

Challenges and Opportunities for Improving Clinical Knowledge Management Capabilities

In addition to the issues surrounding the general lack of use of existing tools to support the work of clinical knowledge management, there are also several organizational issues that must be addressed. For example, the lack of money to hire additional appropriately trained clinical informaticians to serve as knowledge engineers results in existing personnel being forced to work in unfamiliar territory without the necessary time or understanding to “do the job right”. Also, involvement of users who are clinical specialists on content development teams is often hard to maintain without sufficient monetary incentives. Further, the rapidly expanding regulatory reporting and compliance requirements along with increasing emphasis on quality measures (e.g., The Joint Commissions’ CORE Measures [27] and National Patient Safety Goals [28] or Medicare’s Physician Quality Reporting Initiative (PQRI) [29], to name just a few) are placing tremendous demands on already over-extended CDS design and development human resources.

Finally, many organizations are having difficulty encouraging those working to develop clinical decision support interventions to begin using their new “web 2.0” advanced collaboration technologies. These workers continue to rely on email, with all its known problems, as their primary method for collaboration. This continued reliance on email hinders the ability of multiple workers to efficiently and effectively collaborate since some people continue to carry out discussions in emails, while other people carry out discussions on wikis, which leads to parallel systems of communication – which is more confusing, not less!

Taken together these socio-technical challenges and opportunities for improving the use of clinical knowledge management tools and techniques that we have identified lend further support to the work of many socio-technical-oriented knowledge management researchers in the business community [30]. For example, Damodaran et al. found that “the development and systematic application of effective techniques for addressing the ‘soft issues’ (i.e. the human, cultural and organizational issues) relevant to knowledge management systems” are critical to their widespread use and ultimate success [31]). Furthermore Hlupic et al. identified an additional area that must be addressed for successful implementation and use of knowledge management tools and techniques, namely philosophical or ‘abstract’ issues (e.g., fundamental ontological and epistemological issues about the notion and approach to the study of knowledge management) in addition to the technical ‘hard’, and organizational ‘soft’ issues [32].

Future Work

Clearly this study represents only a small sample of all organizations involved in clinical decision support and hence clinical knowledge management activities. The next steps are to conduct a similar survey across a wider range of institutions. Then, if these currently identified practices are shown to be widespread and useful, it would be interesting to attempt to carry out a series of clinical trials to assess the impact of the use of these tools and techniques on the overall success of an organization’s clinical decision support program.

Conclusion

The field of CKM is in its infancy, yet holds great promise. Even within organizations that have been successfully using CPOE with advanced CDS features for well over 15 years, we did not find any site that was using all of the practices in CKM that we identified. If we are to speed progress in the area of clinical decision support, we must continue to develop and refine our understanding, implementation and use of advanced clinical knowledge management capabilities.

Summary Table.

What was already known on the topic?

• For clinical information systems to deliver their expected improvements in health care quality they must be equipped with state of the art, real-time, point-of-care, clinical decision support (CDS) features.

• The successful design, development, implementation, maintenance and evaluation of these advanced clinical decision support features is a complex undertaking that is not well understood.

• The need for high quality, collaborative, clinical knowledge management tools and practices is often overlooked even in the most advanced healthcare organizations.

What this study added to our knowledge?

• The following tools and techniques should be priorities in organizations interested in developing successful computer-based provider order entry (CPOE) and clinical decision support (CDS) implementations:

  • A multidisciplinary team responsible for creating and maintaining the clinical content.
  • An external repository of the organization’s clinical content with a web-based viewer that allows anyone in the organization to review it.
  • An online, collaborative, interactive, internet-based tool to facilitate content development.
  • An enterprise-wide tool to maintain the controlled clinical terminology concepts.

Table 1.

Healthcare organizations involved in the clinical knowledge assessment study.

Participating organizations	Geographic region	Organizational structure	Clinical Information System	Year of first use of CPOE	# Ordersets in use	# Alerts in production	Median # participants in specific CDS intervention discussions
Kaiser Permanente, Northwest	Northwest	Part of a large nationwide health maintenance organization	Epic Systems, Madison, WI	1994 [33]	300 in out- patient setting	50	10
Providence Health System, Portland	Northwest	a mid-size community hospital associated with a 26 hospital regional network	McKesson Horizon Expert Orders, San Francisco, CA	2005 [34]	400	40	6
Regenstrief Institute, Inc.	Midwest	county hospital affiliated with an academic medical center,	Regenstrief Medical Record System (RMRS)	1984 [35]	190 scenario- based order sets (e.g., post-op SICU orders) and 611 problem- based treatment menus (e.g., treatment suggestions for CHF, UTI, etc.)	Drug-drug & drug- disease interactions: 821 Blocking rules: 843 Consequent orders: 784	5
Partners Healthcare System	Northeast	large regional integrated delivery network	Multiple homegrown sys. (BICS, LMR)	1993 [36]	351	7,120	5
Roudebush, VAMC	Midwest	U.S. Veteran’s Administration hospital	Open source system: VistA / CPRS	1996 [37]	300+	100+	8
Memorial Hermann	Southwest	large regional integrated delivery network	Cerner Millenium, Kansas City, MO	2006 [38]	185	40	15

The state of the art in clinical knowledge management: An inventory of tools and techniques

Types of clinical decision support to consider:

• Order sets

• Alert logic and suggestions

• Clinical reference information

• Patient education information

Description of Knowledge management features:

External repository of clinical content

The clinical knowledge used to create the clinical decision support within the clinical information system should be maintained in an external repository. This repository could consist of either a database with an application interface or as a collection of files stored in a dedicated hierarchical directory structure and accessible through a browser-based interface. By storing the knowledge in an external repository, rather than only in the clinical information system, the organization can more easily maintain, share, and allow clinicians to review the content.

Online, collaborative, interactive, internet-based tool to facilitate content development

Developing new clinical decision support content within an organization is always a challenge. Allowing content developers and users to have an asynchronous discussion regarding the pros and cons of specific CDS interventions can be a valuable method of both developing new clinical content and gaining organizational consensus. Many organizations are beginning to experiment with new Web 2.0 technologies such as wiki, discussion boards, or listservs to facilitate distributed creation, consensus-building, and review of clinical content

Enterprise-wide tool to maintain controlled clinical terminology concepts

One of the most important prerequisites to the development and use of advanced, patient-specific CDS features and functions is availability of a robust, controlled clinical terminology (e.g., SNOMED, LOINC, ICD-9, etc.). In order to maintain such a terminology, many organizations have implemented sophisticated computer-based tools that allow knowledge engineers to create new terms and manage the relationships and hierarchical groupings of these concepts.

Tool for CDS users to provide feedback regarding specific CDS interventions

No matter how hard an organization works prior to deployment of new CDS interventions, there will always be clinicians who find justifiable exceptions to any rule developed. Many organizations have developed a simple on-line method of gathering feedback from users while they are actually using the CDS intervention. These systems often make use of commonly available computer-based communication tools such as email, discussion groups, or databases. Often these tools capture some basic information about both the user, the patient, and the clinical context in which they were working in addition to the user’s complaint(s).

Web-based clinical content viewer

Even organizations that do not have an interactive tool to facilitate CDS content development often have a web-based tool that allows interested clinicians and administrators to review all clinical decision support interventions outside of the clinical information system. Many times these tools simply provide a browser front end to a database of clinical content that was automatically extracted from the clinical information system.

Knowledge Management Tools Available…	What does your organization have?
External repository of clinical content
Online, collaborative, interactive, internet-based tool (e.g., Web 2.0 technologies such as wiki, discussion boards, or listservs) to facilitate distributed creation, consensus-building, and review of clinical content
Enterprise-wide tool to maintain controlled clinical terminology concepts, relationships, and hierarchical groupings
Tool or method to allow CDS users to provide feedback regarding specific CDS interventions (e.g., email, discussion group, database, etc.)
Web-based clinical content viewer
Measures of Clinical Content & Knowledge Management System Use	What does your organization have?
Percentage of clinical content reviewed in last year
Order sets
Alert logic and suggestions
Reference information
Patient Education information
Regular reports of clinical content usage available
Number of order sets available for use
Number of alerts in production
Ad hoc query facility available
Number of users who login to the KM system each week (or month or quarter)
Median number of participants in specific CDS intervention discussions
Best Practices for Clinical Knowledge Management	What does your organization do?
Do you modify the external clinical knowledge (content) you use before implementing it? If so, how?
Do you store your clinical knowledge externally using one of the knowledge representation standards (Arden, GLIF, GEM, MEL, etc.)?
Does your clinical information system have the capability of using one of the knowledge representation standards (Arden, GLIF, InfoButton, etc.) to execute its clinical decision support?
What types of people do you have to help manage your clinical knowledge?
Do you have specific individuals responsible for individual pieces of clinical content?
Does each piece of clinical content (i.e., alert, order set, and patient education material) have an individual responsible for maintaining it?
Automatic reminders that content review date is approaching sent to individuals responsible for maintaining the content
Automatic notifications when clinical guidelines underlying CDS change
System in place to “learn” new clinical knowledge from your patient database
Do you purchase, or share, clinical knowledge from external sources…
• Reference information
Drug information (dose ranges, interactions, patient handouts, indications, etc.)
• Order sets
• Alerts / Reminders