Automating Guidelines for Clinical Decision Support: Knowledge Engineering and Implementation

Abstract

As utilization of clinical decision support (CDS) increases, it is important to continue the development and refinement of methods to accurately translate the intention of clinical practice guidelines (CPG) into a computable form. In this study, we validate and extend the 13 steps that Shiffman et al.⁵ identified for translating CPG knowledge for use in CDS. During an implementation project of ATHENA-CDS, we encoded complex CPG recommendations for five common chronic conditions for integration into an existing clinical dashboard. Major decisions made during the implementation process were recorded and categorized according to the 13 steps. During the implementation period, we categorized 119 decisions and identified 8 new categories required to complete the project. We provide details on an updated model that outlines all of the steps used to translate CPG knowledge into a CDS integrated with existing health information technology.

Introduction

Clinical decision support (CDS) has been shown to improve the quality of healthcare delivered.^1,2 With rising utilization of an electronic healthcare record (EHR) in clinical practice, there are increasingly more opportunities to incorporate CDS into additional aspects of the healthcare workflow. In order to improve the effectiveness, adoption, and sustainability of CDS implementations, there has been an increased interest in implementing systems that encapsulate more medical knowledge.^3,4

Clinical practice guidelines (CPG) continue to hold promise as a source of evidence-based medical knowledge, and there has been significant work aiding the translation of this knowledge into software systems.^5–7 However, guidelines often contain gaps and ambiguities, making it difficult to implement or operationalize the knowledge into a computable format.⁵ These shortcomings result in varied interpretations of the same recommendation and decreased adoption.^8,9 With the aid of clinical experts, it is possible to interpret and complete the knowledge required to implement intentions of the guidelines completely; however, this process is not yet standardized and there are many aspects to consider for a successful end product.³

Shiffman et al. formalized steps for translating the knowledge contained in guideline text into a computable format that could be operationalized into a CDS specification.⁵ Since the publication of the seminal paper, most CDS implementations must consider integration directly with another system, and knowledge modeling techniques have continued to develop. In this study, we validate and update the steps in the Shiffman model, and we extend them to apply to the full process of implementing a CDS system based on CPG knowledge. We show that, in addition to deriving CDS knowledge completely from guideline documents, implementing a CDS system often requires changes to the design specification.

Methods

Clinical Decision Support System

We conducted this study during a modified implementation of ATHENA-CDS, an automated CPG-based CDS system that was developed at the Veterans Health Administration (VHA) of the Department of Veteran’s Affairs (VA) Palo Alto Health Care System. ATHENA-CDS uses CPG knowledge encoded in Protégé, a knowledge acquisition program developed at the Stanford Center for Biomedical Informatics Research (BMIR).¹⁰ The CPG Knowledge Base (KB) and patient EHR data are processed by the BMIR EON Guideline Interpreter system, which generates patient-specific conclusions, including recommendations for management, evidence base for the recommendations, and a summary of patient data that is relevant to decision-making.¹¹ Clinical modules were developed, modified, or updated to provide CPG recommendations, in the context of a clinical dashboard, for the management of five chronic diseases domains: hypertension (HTN), chronic kidney disease (CKD), heart failure (HF), hyperlipidemia (HLD), and glycemic control in type 2 diabetic patients (DM).

Implementation Context

Within Veterans Integrated Service Network (VISN) 21 of the VHA, the Pharmacy Benefits Management (PBM) group has developed a primary care clinical dashboard built on top of the VA clinical data warehouse. The dashboard provides tools for clinic team members to monitor VA clinical performance measures. ATHENA-CDS was modified to generate and display guideline-based management recommendations within the clinical dashboard.

Implementation Decision Categorization

During the development of the KBs from CPGs and the integration of ATHENA-CDS into the clinical dashboard, we recorded major implementation decisions that required interdisciplinary discussion or impacted parallel or downstream project activities. Three project members (CO, SM, KWY) extracted the decisions from meeting notes and created a spreadsheet that detailed the clinical domain, date, topic, references, issue/questions, and disposition of each decision. One project member (SWT), an experienced knowledge engineer, evaluated each decision to categorize the decisions into one of the 13 steps defined in the Shiffman model (Table 1). A senior clinician (MKG) evaluated the initial categorization, and working with the knowledge engineer, developed a consensus categorization of the decisions and proposed new categories for the list of decisions that could not be categorized. The project team, which included clinicians, knowledge engineers, and system implementers, agreed upon the final list of new categories.

Table 1.

Definitions of Shiffman’s Steps⁵

Decision Category	Definition
Select Guidelines	Choice of specific guidelines and choice of specific recommendations within the selected guidelines to be implemented
Markup	Identification and tagging of guideline knowledge components relevant to operationalization
Atomize	The process of extracting and refining single concepts from the narrative text recommendations
Deabstract	The process of adjusting the level of generality at which a decision variable or action is described to permit operationalization
Disambiguate	The process of establishing a single semantic interpretation for a recommendation statement
Build Executable Statements	Arrangement of the atomized, de-abstracted, and disambiguated decision variables and actions into logical statements that can be translated readily into computable statements
Verify Completeness	The process to make sure that each recommendation provides guidance in all situations that a clinician is likely to face
Add Explanation	A facility to describe the reasoning behind recommendations
Identify Origin	Identifying a source or origin in the clinical environment for each decision variable
Insert Recommendations	Identifying an insertion point in the care process for each recommended action
Define Action Type	Categorizing guideline-recommended activities according to predefined action types
Define Associated Beneficial Services	Linking action types to associated beneficial services that offer design patterns for facilitating clinical care
Design User Interface	Selecting and grouping user interface elements to best deliver CDS output

Results

We identified 119 decisions made at project meetings over a 29-month period (Examples in Table 2). During the study period, CPGs for HTN¹², DM¹³, CKD¹⁴, HLD¹⁵, and HF¹⁶ were selected to be encoded for the KBs. Eighty of the decisions were categorized within the Shiffman model. Table 3 shows the incidence of ATHENA-CDS developmental decisions that had been categorized for each step in the model. The remaining 39 decisions were analyzed and, through consensus, placed in new categories (Table 4).

Table 2.

Sample of implementation questions and issues and resulting decision

Clinical Domain	Topic	Issue /questions	Decision
All	Drug hierarchy	Drugs added in ad hoc fashion	Reorganize drugs into same hierarchy as VA NDF-RT.
Diabetes,	Pregnancy	Difficult to identify currently	Issue primary message to women ages
Hyperlipidemia	exclusion	pregnant women	18-50 that pregnant women and nursing mothers are out of scope.
All	Medication possession ratio (MPR)	Include in knowledge base logic or use in post-processing in dashboard logic?	We are not going to use MPR to suppress recommendations, but use it to generate additional recommendations.
Hyperlipidemia	Clinical dashboard vs. Stone 201415	Stone: Treat patients w/ LDL>190 w/o risk factors Dashboard: Out of scope	Align treatment goals with Pharmacy Clinical Dashboard

Table 3.

Incidence of decisions for Shiffman’s steps

Decision Category	Incidence
Select Guidelines	14
Markup	0
Atomize	1
Deabstract	4
Disambiguate	0
Build Executable Statements	0
Verify completeness	19
Add Explanation	2
Identify Data Origins	2
Insert Recommendations	3
Define Action Type	5
Define Associated Beneficial Services	0
Specify User Interface	30
Total	80

Table 4.

Incidence of new implementation categories

New Decision Category	Incidence
Reconcile Multiple Guidelines	7
Align with Existing CDS	12
Adapt to Local System	2
Add Enhancement	2
Specify and Encode Knowledge	4
Map Terminology	2
Test CDS	8
Manage Project	2
Total	39

Implementation Model

From our study, eight new categories of steps in the implementation process were identified and defined (Table 5).

Table 5.

Definitions of new implementation steps

#	New Step	Description
1	Reconcile Multiple Guidelines	Reconciliation of recommendations from different guideline sources (e.g., VA versus professional society guidelines) and guidelines for related clinical domains.
2	Align with Existing CDS	Selection and alignment of recommendations details (e.g., HbA1C targets and drugs to recommend) based on the need to be consistent with existing system
3	Adapt to Local System	Modification of recommendations based on capabilities of local system (e.g., nonavailability of data)
4	Add Enhancement	Addition of beneficial services based on the availability of local resources
5	Specify and Encode Knowledge	Decisions about design, organization, and conventions used in representation of guideline knowledge. Encoding of knowledge base.
6	Map Terminology	Mapping terminology used in data sources to terminology used in guidelines
7	Test CDS	Simulation of patient data to verify correctness of guideline encoding and the completeness and clinical appropriateness of system-generated recommendations
8	Manage Project	Decisions about the scope of the project and the organization and timing of tasks

Most of the new steps are generalizable to all CDS implementation projects. One of the categories, Reconciliation of Multiple Guidelines, is especially derived for CDS that utilizes multiple knowledge sources. Using these observations from our study, we extend and update the Shiffman model, incorporating the eight additional steps (Table 4) that were required to implement ATHENA-CDS. Figure 1 contains the activity flow for the proposed model with 21 steps. The diagram shows that not all of the steps need to be performed in a specific order.

Figure 1.

Activity diagram of updated CDS implementation model.

Here we will describe all of the 21 steps in more detail. Starred steps are new additions to the model.

1. Knowledge Preparation

1.1. Guideline Selection

Many guidelines from different organizations/writers can exist on the same topic. Choosing which guideline to implement is an important step in the development process that can affect many of the subsequent steps. In choosing which guidelines to implement, we considered several factors: validity, level of evidence, applicability in the clinical context, institutional recommendations, local practice variation, and ease of operationalization. Furthermore, within each guideline, individual recommendations also need to be selected for implementation. For some of the clinical domains, we had the choice of using recommendations or sections of recommendations from either VA/Department of Defense CPGs or those from professional societies. Whether a recommendation could be operationalized and applied to the clinical context was an important factor in the selection process.

1.2. Guideline Reconciliation*

Reconciling multiple guidelines is a new proposed step in CDS implementation. If multiple CPGs or other knowledge sources are used within the knowledge base of the CDS, differences in recommendations from each guideline must be reconciled to prevent interactions or conflicts in the CDS logic. An important feature of ATHENA-CDS is taking account of comorbid conditions when making recommendations for the index condition.¹⁷ For the five clinical domains in ATHENA-CDS, many of the CPGs discussed overlapping topics (eg. blood pressure management). Reconciliation was necessary to prevent conflicting recommendations from the CDS. We found that reconciliation between the guidelines was most important when there was a possibility of drug-drug interactions or different usages of the same medication class. For ATHENA-CDS, domain experts worked with knowledge engineers to ensure that the resulting knowledge corpus was complete and accurate.

1.3. Markup

In this Shiffman step, a guideline is converted from text into a semi-structured representation in order to identify and tag components relevant to operationalization of a CPG.¹⁸ In his paper, Shiffman describes an XML-modeling approach that decomposes a CPG into Guidelines Elements Model (GEM) component elements.^5,19 For our purposes, an XML approach that marks up documents individually did not facilitate translation of the guideline knowledge into a Protégé knowledge base, so we took a different approach. We used a collaborative team approach for organizing the information in terms of the guideline model used in ATHENA-CDS KBs. The resulting representation facilitates the operationalization process as an intermediate view that focuses on elements important to the final knowledge encoding, such as: eligibility criteria, clinical algorithms, goals, definitions, concept hierarchies, clinical scenarios and decisions, interventions/drugs, and schemas for rating evidence quality and recommendation strength.

2. CDS Knowledge Specification and Encoding

2.1. Atomization

In order to execute the recommendations in a CPG, the recommendations in the narrative text need to be extracted and reduced to computable forms. Atomization achieves this by removing unnecessary text and identifying and normalizing the key concepts. Extraction of the guideline concepts can be performed manually or through natural language processing and text-mining tools. These concepts can then be normalized to terminologies in a custom or local library or to standardized terminologies such as Logical Observation Identifiers Names and Codes (LOINC). As an example, in the HF CPG, the selection criteria for the recommendation of an aldosterone receptor antagonist states that a patient should have:

Serum creatinine <2.5 mg/dL (or an estimated glomerular filtration rate >30 mL/min/1.73 m²) without recent worsening and serum potassium <5.0 mEq/L without a history of severe hyperkalemia

The atomization process extracted the following concepts from the statement: serum creatinine, estimated glomerular filtration rate, recent worsening, serum potassium, history of, and severe hyperkalemia. The criteria of <2.5 mg/dL, >30mLmin/m², and <5.0mEq/L were also extracted.

2.2. Deabstraction

Deabstraction is the process of converting generalized or high level concepts into specific concepts that can be operationalized. In the aldosterone receptor antagonist example, “severe hyperkalemia” needs to be converted to a more specific concept such as “serum potassium level above 1.5meq/L above normal.” While this example is relatively straightforward, other concepts can be more complex. For example, the atomized concept “metoprolol” can be further specified to individual formulations such as metoprolol tartrate or metoprolol succinate. The latter is reported to be more effective than the former in clinical trials and would be preferred.¹⁶ Therefore, careful attention is necessary to ensure adherence to evidence-based practice in the deabstraction process.

2.3. Disambiguation

In some CPGs, decision variables in recommendations can be vague and not mutually exclusive. For Disambiguation, Shiffman gives an example of this involving asthma exacerbation severity in which the descriptions for different severity levels contain ambiguous terms that could overlap.⁵ Ambiguity in a CPG can be the result of limited supporting evidence or lack of consensus. While the guidelines recommendations encoded into ATHENA-CDS did not require this type of disambiguation, there were recommendations that contained decision variables that have multiple incompatible interpretations (e.g., different cutoff values for abnormal laboratory test results). However, the translation of these variables into operational concepts was categorized as a deabstraction since they did not require delineating two mutually inclusive recommendations.

2.4. Verification of Completeness

This step ensures that recommendations provide guidance for all clinical scenarios. The goal of verification is to identify gaps in the decision criteria or in the recommended actions. The deabstraction and disambiguation process often elucidates these gaps. In the example above, the criteria for starting an aldosterone antagonist do not account for the scenario where the serum creatinine and estimated glomerular filtration rate (eGFR) have not been measured. In addition, while the criteria for starting the medication allow for values of either serum creatinine or eGFR, the subsequent dosing recommendations provide dosing recommendations based on eGFR but not serum creatinine.

2.5. Build Executable Statements

After the guideline knowledge has been atomized, deabstracted, disambiguated, and verified for completeness, it is ready for translation into full executable statements that can be encoded into the CDS logic. The example we have been building in the previous steps could result in refined logical criteria such as:

((serum creatinine < 2.5mg/dL or eGFR rate > 30 mL) and (absence of 20% rise in serum creatinine from the lowest recorded value in the last 90 days) and no history of serum potassium level > 1.5mEq/L above normal

This refined statement continues to have abstract or ambiguous concepts such as “normal” potassium. As can be seen, preparing guideline knowledge is often an iterative process that continues until the logic can be fully encoded.

2.6. Explanation

In addition to building executable statements, an important aspect of preparing the guideline knowledge involves providing the user with the reasoning behind each recommendation.²⁰ Healthcare providers often want to know the evidence behind a recommendation when it goes against their normal practice.²¹ Explanation can also fill gaps in provider knowledge. In addition, user interface screen real estate is often limited, preventing the display of every detail of a recommendation or the evidence.

2.7. Knowledge Representation and Encoding*

This is a new step that encompasses the process of encoding the knowledge prepared in the previous steps. Shiffman presumed that the knowledge would be encoded in rules with decision variables. We implemented ATHENA-CDS using a rich guideline model that required decisions on specifying how the knowledge will be represented (e.g., how a domain ontology that includes taxonomies of medical conditions and drug interventions should be organized and how incomplete information should qualify suggested drug recommendations). In addition to deciding how the logic will be encoded, details on standards and conventions, version control, and the organization of data are important considerations that will facilitate the implementation process as well as future maintenance and extension. This step also includes making adjustments to the CDS logic based on the CDS design specifications described below in the steps involving EHR and Local System Integration. Once the knowledge representation has been determined and the knowledge has been aligned to specifications, the knowledge encoding can be performed.

3. EHR and Local System Integration

3.1. Workflow Integration

3.1.1. Data Origin Identification

In order to understand how the CDS integrates into the workflow, it is necessary to identify the clinical data sources. Data can come from historical records stored in the data warehouse. Other data can originate from certain steps in the clinical workflow and get recorded at point of care through data entry into the CDS or EHR.

3.1.2. Recommendation Insertion Identification

Determining how users will want to use the system and where recommendations should be inserted in the workflow informs design development to enhance the adoption and effectiveness of the CDS. For example, ATHENA-CDS generates recommendations on guideline-concordant changes to a patient’s active medication regimen. As a result, the CDS was designed so that the recommendations are presented to the clinician while he/she is reviewing data at the point of care and before completing the order entry process.

3.2. CDS Alignment, Adaptation, and Enhancement

3.2.1. CDS Alignment*

In many healthcare organizations, there are multiple types of CDS integrated into the provider workflow. A successful CDS implementation ensures appropriate integration in the existing architecture. For this project, ATHENA-CDS was being integrated within a clinical dashboard showing metrics on specific performance targets. Since decision criteria and therapeutic targets drive the recommendations provided by ATHENA-CDS, they were modified to align with the numerator and denominator criteria of the performance metrics implemented by the dashboard. This alignment enables a user to have a consistent experience across the system. CDS Alignment can also involve the process of specifying the responsibilities of a CDS when there is an overlap in functionality with other systems. Unlike the “CDS Enhancements” step (discussed below), CDS alignment does not create new CDS capabilities, but utilizes existing capabilities. In contrast to the “Associated Beneficial Services” step (also discussed below), this step involves modifications to CDS capabilities so that they are consistent with existing CDS services.

3.2.2. CDS Adaptation*

Similarly, adaptation with the local EHR must be considered in the implementation of guideline recommendations. Specific components of the recommendation might need to be aligned with data available to the CDS. For example, if clinical information is not readily retrievable from the EHR, then alternative data sources must be established or limitations of the system must be acknowledged. Early consideration of the local system in the knowledge preparation process can reduce the chance of late modifications or removal of a recommendation. For example, many veterans receive obstetrics care outside of the healthcare system and pregnancy is not always documented in the EHR. As a result, a decision was made to consider pregnancy out of scope of ATHENA-CDS, and providers were advised to use their clinical judgment in women of reproductive age. CDS adaption involves changes to the implementation, not because of existing CDS capabilities that need to be aligned, but because of the characteristics of the local population and the data available in the EHR.

3.2.3. CDS Enhancements*

During the implementation and subsequent maintenance of a CDS project, new healthcare information technology (HIT) can emerge and interoperability between systems can improve. The availability of new data and enhancements to the EHR may allow for the implementation of new CDS capabilities. During the ATHENA-CDS implementation, the ability to check a patient’s adherence to prescriptions based on medication possession ratios became available through pharmacy data analytics. This new capability allowed for the implementation of CPG recommendations that could not be implemented before. In contrast to the use of associated beneficial services to complement CDS recommendations described by Shiffman, CDS enhancements entail changes in the CDS recommendations generated by the system.

3.3. CDS Delivery

3.3.1. Action Type Definition

The culminating step in CDS is an output that is ultimately used clinically. The majority of outputs can be classified in three action categories: gathering information, interpreting information, performing a task, and organizing care^5,22. Gathering information can come in the form of monitoring a clinical variable according to a specific criteria or schedule. Interpreting information usually results an output such as a diagnosis, prognosis, or clinical status. Recommended tasks can include prescribing medications, performing a procedure, documenting in the medical record, advocating a policy or practice, or preparing for a guideline-directed activity. Organizing care involves directing the flow of care for a patient in the form of deposition, follow-up, and referral.

3.3.2. Associated Beneficial Services

Shiftman describes this step as linking action types to services that offer design patterns for facilitating clinical care. Common use cases involve linking a prescription with a pharmacy system, sending an study order to a lab, or sending a referral to the designated department. These use cases facilitate the implementations of specific CDS recommendations using new or existing services, but do not alter the CDS recommendations themselves. For this project, ATHENA-CDS was being integrated with an existing clinical dashboard with an established set of associations. No additional decision regarding beneficial services arose during the development of the system.

3.3.3. Terminology Mapping*

An increasingly common task in HIT is data mapping to facility interoperability between systems that use different naming conventions. As a result, terminologies and concepts encoded into the knowledge representation such as laboratory studies and medications have to be mapped to those in other systems. For our project, a mapping table was needed for the data retrieval interface between ATHENA-CDS and the EHR. Similarly, the output of ATHENA-CDS had to be mapped to the clinical dashboard’s data repository and terminology usage.

3.2.4. User Interface Design

Best principles of user interface design must be used to make an effective and usable CDS.²³ In addition to the standard design issues such as formatting of the look and feel to ensure readability and usability, consideration must be made in terms of user workflow, balancing space-efficiency, and the number of clicks required to perform a task.²⁴ These design decisions can be constrained since most CDS are integrated into an existing system and must conform to or stay within the capabilities of the existing design.

4. Testing*

Verification and validation of the encoded system is vital before a CDS system is deployed and should be performed throughout the development process.^25,26 Verification is the process of ensuring the system operates according to requirements specifications. During this process, encoded logic and knowledge are tested to make sure they work as intended. CDS verification can be achieved though appropriate selection of real or engineered patient data with comparison to a reference standard. Validation of a CDS system ensures that the system performs what the end user requires. Validation can be achieved by quantitative and qualitative studies that measure the completeness and clinical appropriateness of system-generated recommendations.²⁷

5. Project Management*

Management of the project throughout the implementation process is important in ensuring a cohesive and complete software product. There are many decisions that are related to the project as a whole. These decisions can affect the previous steps by influencing the scope of the project as well as the organization and timing of tasks. Project management also facilitates appropriate communication and resource allocation and tracks progress. For this study, these administrative decisions often related to resources such as scope, human resources, and financing. Other project management decisions that were important in a CDS implementation process included decisions on organizational policies, information technology challenges, analytics support, deployment details, and quality/feature control.

Discussion

With the immense amount of medical knowledge available, CPGs provide clinicians with access to concise expert or evidence-based knowledge sources. However, there are numerous barriers that reduce the use of CPGs in clinical practice.^28–30 The Shiffman model describes a process for translating CPGs into a general requirement specification for a CDS that can be developed in isolation and implemented later in any setting. The resulting specification accounts for input variables from the CPG. In contrast, our model describes the extended process of taking a CPG, converting the knowledge into a computable format, and integrating that operationalized knowledge into a context-specific live user-facing system. As a result, the design specification and implementation process required consideration of variables specific to the VISN 21 environment as well as the national VA system.

In the analysis of our implementation, the incidence of decisions demonstrates that some steps of the original Shiffman model required more discussion and thought than others. It is possible that this variation reflects the importance and complexity of some of the steps in regards to their effect on the project as a whole. Three of the steps required a great deal of team discussion. Selection of an appropriate, usable, and valid medical knowledge is a fundamental step in clinical practice, and it is not surprising that the guideline and guideline recommendation selection process was a major point of discussion for ATHENA-CDS.^31,32 Similarly, the importance of usability in the adoption and effectiveness of CDS continues to be underscored, and user interface design decisions were heavily discussed for our CDS implementation.^24,33 Lastly, verifying the completeness of recommendations is a complex process that necessitates the input of both clinical and knowledge engineering experts.

Conversely, some steps, such as Markup, Atomize, De-abstract, Disambiguate, Build Executable Statements, and Define Associated Beneficial Services are either not reflected or under-represented in our formally recorded decisions. This finding can be explained by three factors. First, some of Shiffman’s and our additional implementation steps are optional. For example, the need for Shiffman’s Associated Beneficial Services and our CDS Alignment and Enhancements are contingent on the implementation context. Define Associated Beneficial Services step was performed prior to our project during the implementation of the clinical dashboard. Second, while the concepts behind the Markup and Build Executable Statement steps are still essential in implementing a narrative guideline, their definitions needed to be broadened, as there is no consensus technology used for this task. Lastly, the organization of our project team, and the limitation of using recorded implementation decisions as the basis for categorization can explain the under-representation of steps such as Atomize, De-abstract, Disambiguate. For many of the knowledge representation tasks, our experienced knowledge engineers perform these tasks without requiring significant input and are not reflected in the recorded decisions of the project.

As more CDS systems are integrated into EHRs, more attention needs to be placed on how they interact. It is not uncommon that the logic of each system is encoded by multiple different groups or vendors.³ As a result, the design specifications for these CDS do not take into account other systems. Alignment of CDS and reconciliation of multiple CDS in the same system are developing areas in informatics that will need continued investigation as CDS implementations increase.

Similarly, interactions between CPGs have been a known issue in both clinical practice and CDS.^3,34 CPGs, in general, are brief in their discussion of comorbid conditions.³⁴ In ATHENA-CDS, each chronic disease KB is derived from CPGs, and the CDS provides recommendations independent of the other KBs. The KBs were specifically encoded to reduce possible interactions produced by the system. However, additional investigation is needed to make multimorbidity support feasible in working implementations.³⁵

While we have attempted to generalize our findings, other implementation projects will likely face different challenges. As HIT continues to evolve rapidly, the methodology and complexity of each step will likely change. Continuing to define the methods and steps in translating evidence-based knowledge into CDS will help ensure successful implementations. In addition, efforts should continue to standardize formats for guidelines that facilitate their implementation both clinically and computationally.

Conclusion

We validated and extended steps in Shiftman’s approach to making guideline recommendations computable. We identified additional knowledge engineering and implementation categories needed because of multiple comorbidities and guidelines, actual system implementation, and integration with existing HIT tools.