An Integrated Process for Co-Developing and Implementing Written and Computable Clinical Practice Guidelines

Abstract

The goal of this article is to describe an integrated parallel process for the co-development of written and computable clinical practice guidelines (CPGs) to accelerate adoption and increase the impact of guideline recommendations in clinical practice. From February 2018 through December 2021, interdisciplinary work groups were formed after an initial Kaizen event and using expert consensus and available literature, produced a 12-phase integrated process (IP). The IP includes activities, resources, and iterative feedback loops for developing, implementing, disseminating, communicating, and evaluating CPGs. The IP incorporates guideline standards and informatics practices and clarifies how informaticians, implementers, health communicators, evaluators, and clinicians can help guideline developers throughout the development and implementation cycle to effectively co-develop written and computable guidelines. More efficient processes are essential to create actionable CPGs, disseminate and communicate recommendations to clinical end users, and evaluate CPG performance. Pilot testing is underway to determine how this IP expedites the implementation of CPGs into clinical practice and improves guideline uptake and health outcomes.

Introduction

The Problem

With the exponential growth of medical science, the volume of important information to incorporate into daily clinical practice often overloads clinicians.¹ Clinical practice guidelines (CPGs) contain systematically developed sets of related recommendations designed to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances.² CPGs aim to provide criteria and guide decisions regarding prevention, diagnosis, management, and treatment in specific areas of health care. However, because CPGs are typically in a written format (eg, narrative or textual documents in paper-based or web-based publications), these may not be easily incorporated into clinician workflows at the point of care.³ CPGs are often lengthy and not easy to access and use in clinical settings.⁴ Written CPGs alone are inefficient for clinicians to quickly apply in patient care. There are many other such barriers to the adoption of CPGs, such as a lack of awareness about the CPG and its evidence, the inability to overcome the inertia of previous practice, and the lack of technology to integrate CPGs into standard care practice.^5–9

Computable CPGs and Clinical Decision Support in Health Care

Evidence-based CPGs in a machine-interpretable format and readily and digitally available in electronic health record (EHR) workflows can overcome some of the limitations of written guidelines. Organizations have translated written guidelines into machine-readable formats known as computer-interpretable guidelines,¹⁰ which the authors term computable CPGs. Computable CPGs can increase the ease, speed, accuracy, and consistency of their transfer into health information technologies to enable clinical decision support (CDS), electronic clinical quality measures (eCQM), and electronic case reporting (eCR) or eCase reports reported to registries. Representative open-source computable CPGs are available for multiple subjects, including opioid prescribing, asthma, diabetes, and antenatal care.^11,12 Health information technologies (HIT) include information resources and systems connected via networks at local, regional (eg, state or province), national, and international levels. CDS is health information technology that builds upon the foundation of an EHR to give general and patient-specific information to health workers to improve care.¹³ For this article, CDS primarily refers to point of care systems that give clinicians or patients clinical knowledge and patient-related information that is presented at appropriate times to enhance patient care.¹⁴ CDS systems typically include computerized alerts and reminders, excerpts from clinical pathways, condition-specific order sets, focused patient data reports and summaries, documentation templates, and diagnostic support.¹⁵ Using CDS to deliver CPG recommendations within the clinical workflow has been identified as an important strategy to increase the use of guidelines, help providers make informed decisions at the point of care, and to improve care processes.^10,16

CDS tools can improve guideline adherence and promote recommended preventive care, particularly if CDS is integrated into clinical workflows and information is available in real-time.^17,18 A review of 22 studies suggests that using clinical dashboards or CDS systems that give clinicians immediate access to patient information can make processes more efficient and improve patient outcomes.¹⁹

Although there is sparse literature on the effects of CDS on health and economic outcomes, perhaps the most promising is the use of CDS related to medication management. When applied at the time of medication ordering or dispensing, CDS can increase the prescribing of appropriate medications and dosages²⁰ and effectively decrease antibiotic use.^21,22A review also showed that the transition from paper-based ordering to commercial computerized provider order entry systems in intensive care units was associated with a significant reduction in medication errors and mortality.²³ A 2021 systematic review report from The Community Preventive Services Task Force showed that the use of CDS systems increases HIV screening based on strong evidence of effectiveness for the general population and for people at high risk for HIV infection.^24,25 Research has indicated that poorly designed CDS, resulting in alert fatigue, can lead to medical errors.²⁶ Overall, CDS systems have been shown to improve health care process measures related to decision-making, diagnostic accuracy, and reducing unnecessary testing across diverse settings.¹⁸ Furthermore, the integration of computable CPGs into clinical workflows via CDS requires additional evaluation to assure that patient care and health outcomes will be improved.^27,28

For CPGs to be effective in guiding providers’ actions at the point of care, guidelines need to be created in a way that reinforces their intended purpose and use through HIT. In the field of informatics, Boxwala proposed 4 levels of knowledge: L1 (level 1) represents the “narrative or written” guidelines for a specific disease or condition. L2 represents a “semi-structured” version that is human-readable, often using flow diagrams or decision trees to describe the recommendations. L3 is a “structured and computer-readable (computable)” guideline recommendation or artifact that contains encoding logic, terminology, and data elements. L4 is the “executable” guideline recommendation used in a local environment that can be made available natively in the EHR, through web services, or via apps (eg, Substitutable Medical Applications and Reusable Technologies [SMART]-on- Fast Healthcare Interoperability Resources [FHIR®]). Knowledge levels.²⁹

1. Narrative text created by a guideline or CQM developer.

2. Semi-structured text that describes the recommendations for implementation in CDS.

3. Structured code that is interpretable by a computer (includes data elements, value sets, and logic).

4. Executable code that is interpretable by a CDS system at a local level. This will vary for each site.

To improve adoption and adherence to guidelines in clinical settings, the Centers for Disease Control and Prevention (CDC) facilitated an initiative to reengineer the development and implementation of CPGs using HIT. CDC plays an important role in public health by working with many other agencies to develop evidence-based guidelines to improve health and prevent and control emerging diseases, such as Coronavirus Disease of 2019 (COVID-19). The purpose of this article is to describe the process and one of the main products of this initiative. It resulted in a new 12-phase integrated process (IP) to co-develop, translate, and implement written and computable CPGs with the objective of increasing efficiencies (eg, time and effort needed to develop both written and computable guidelines and time to implement guideline uptake), increasing effectiveness (eg, clinician and patient adherence to guideline recommendations), and improving health outcomes. Core features of this IP include co-development and multidisciplinary collaboration throughout the 12 phases, which means that the guideline developers, informaticians, and implementers develop the written and computable CPGs in parallel through an iterative process rather than sequentially. Communications and evaluation activities are interwoven throughout the process. The activities in the IP tables are not prescriptive requirements. Instead, they provide information and options for decision-makers and stakeholders to consider and choose for their organizations and context. For example, federal guideline developers must comply with the Federal Advisory Committee Act when producing guidelines, such as those produced by the CDC.

Methods

In February 2018, the CDC convened a 4.5-day meeting to discuss the clinical and public health challenge of the timely creation and adoption of CPGs that began the CDC-sponsored initiative Adapting Clinical Guidelines for the Digital Age. Over 100 experts across the continuum of guideline development and implementation participated in this Kaizen event. Participants included CPG developers, informaticians, clinicians, implementers, HIT developers, communicators, evaluators, and other public health professionals. The goal of this initial event was to identify problem areas in guideline development and implementation and to propose redesigned processes to maximize its value to the ultimate customers, namely, health care providers and patients. The meeting organizers employed management techniques, such as brainstorming and value-stream mapping, to generate ideas and come up with creative solutions to problems. The latter is a lean manufacturing technique that helps identify bottlenecks, waste, and value-added steps within a flow of material and information—in this case, a clinical guideline. Participants with different areas of expertise were divided into 5 “value streams” or workgroups, charged with the following tasks:

• 1.Guideline creation—identify ways to improve developing and evaluating an evidence-based CPG,
• 2.Informatics—contribute to developing an international standard for building a computable CPG,
• 3.Translation and implementation—identify ways to improve processes for applying CPGs in local clinical settings,
• 4.Communication—develop a model communication plan to distribute CPGs, and
• 5.Evaluation—examine ways to evaluate this proposed new process.

Participants mapped the present or “current state” of the guideline development and implementation process across each of these 5 areas and identified related challenges (eg, long lag times for translating and implementing CPGs into patient care and lack of adherence to CPGs). Participants also noted that computable guidelines are typically developed in a linear or sequential fashion after the written guideline is published in a journal or on the web, which adds to the lag. In addition, participants were concerned that the promotion of the guideline and plans to evaluate the guideline’s use and impact are typically not given sufficient attention. The guideline development authority has the responsibility for studying the available evidence on a clinical topic and developing and publishing the guideline document. There may be little opportunity for guideline developers to monitor the testing or use of the CPG in practice and provide feedback for guideline improvements. Furthermore, the time spent developing the written recommendations could also be used for parallel work on the computable CPG and planning other downstream activities such as communication, implementation, and evaluation. The 5 workgroups then identified areas in the “current state” that could be improved and mapped ways to redesign the process to develop and apply guidelines into patient care more easily, quickly, accurately, and consistently.³⁰ The groups also reviewed CDS tools, HIT, and industry standards that are currently available to augment patient care and clinical knowledge, such as CDS Hooks, SMART, Health Level Seven International (HL7®) Fast Healthcare Interoperability Resources (FHIR®), Clinical Quality Language (CQL), Business Process Management for Healthcare (BPM+ Health),³¹ and the use of Application Program Interfaces. The workgroups proposed a new model for concurrently developing written and computable CPGs. Rather than a linear waterfall approach, this proposed new or “future state” model integrates the work of informatics, communication, implementation, and evaluation into the guideline development process.

Following this initial meeting, 3 years of work began to further develop the ideas produced during the Kaizen event. From 2018 through 2021, using expert consensus and available literature, the Guideline Creation Workgroup worked concurrently with members of the other workgroups to produce a model that built upon international standards as the foundation for guideline development.^2,32,33 This model diagram (Figure 1) and the corresponding 12 phases of activities allow clinicians and other multidisciplinary subject matter experts (SMEs), including guideline developers, informaticians, implementers, health communicators, and evaluators, to collaboratively develop guidelines. The Guideline Creation Workgroup integrated activities and products produced by the other 4 workgroups to refine the process for developing and implementing computable CPGs. This IP includes 12 tables with activities for guideline development, informatics, implementation, communication, and evaluation, with iterative feedback loops throughout the process. (https://stacks.cdc.gov/view/cdc/131006)

Figure 1.

Overview of Integrated Process for Developing Written and Computable Guidelines. Red italicized font represents informatic activities. Can be at local levels (eg, a single hospital system), regional level (eg, a large group practice in a particular state), or beyond (using shared resources, such as a computable guideline’s repository). COMM, communication; Dev Topic, development; EVAL, evaluation; INFO, informatics; IMPL *, implementation*; SME, subject matter expert; Subj, subject.

Results

Overview of the IP

Through interdisciplinary collaboration, the Guideline Creation Workgroup produced a 12-phase IP for developing clinical guidelines in written and computable forms (Figure 1). This IP outlines phases and activities to reengineer the guideline development process so the computable guideline is developed at the same time as the written guideline, to speed up their implementation into clinical practice using CDS systems and related tools, for example, eCQMs, and eCase reports. The IP also includes early development of communication and evaluation plans so they are ready to launch at the time the guideline is published. In each of the 12 phases, specific activities take place for each team, some separately and others together.

As shown in Figure 1, the IP for guideline production integrates the activities of 5 disciplinary teams that include experts in guideline development and the guideline topic, informatics, implementation, communication, and evaluation. The guideline steering or oversight committee orchestrates the work across the teams. The collaborative work between the guideline development, informatics, and implementation teams forms the core of the IP, as noted in the titles for each of the 12 phases. For example, the key guideline development activity in phase 1 is to “Define the clinical problem/need and conduct guideline topic selection,” whereas the key informatics/implementation activity is to “Assess the clinical need for computable CPG and additional products” (shown in red italics). Key activities for communication and evaluation are shown on the right side of the diagram. These activities occur less frequently but provide critical input to the guideline developers, informaticians, and implementers throughout the IP.

Figure 1 does not fully represent the “iterative” nature of collaborative activities among the 5 teams. The work of each team and how these teams perform their work in collaboration will be clarified in the following sections. As with all scientific work of high caliber, the exchange of disciplinary perspectives among teams enriches the quality of the products. Guideline developers do not need to follow all steps but can tailor this guidance to their needs and to the context in which they are producing guidelines and implementing key parts of the IP that are feasible. Variation in context can include issues such as the scientific domain and scope of the guideline, the urgency of its production, the disciplinary approaches of the experts involved, the amount of evidence to review, the resources available, and end-user acceptance. The objective of this future state IP is to produce a computable guideline and a written guideline simultaneously to improve the fidelity to the guideline’s intent, reduce the variability of guideline products, including the computable guideline, and, as a result, speed up their implementation into practice. The evaluation section and phase 11 in this article provide detail about assessing these benefits.

Guideline Development

Guideline development activities are aimed at synthesizing evidence in a transparent and rigorous fashion to inform the writing of clear and actionable guideline recommendations to improve the quality of care and health outcomes. The IP (Figure 1) provides an overview of parallel activities needed to develop written and computable CPGs with the goal of improving clinician behavior through CDS systems at the point of care. After setting up the operational framework during the pre-guideline preparation (phase 0), the formal guideline development process involves 12 phases of activities, as shown in Figure 1. Best practices for systematic reviews to support guideline creation^2,34,35 and for CPG development³³ have been outlined and are incorporated throughout the tables of phases and activities.

Informatics

Informatics activities use standards-based computable languages, models, and terminology systems to represent and interpret guidelines that are later automated in clinical domains.^36,37 Clinical aspects to be modeled include explicit definitions for domain concepts and domain-specific inferences, decision and orchestration logic, and high-level clinical workflows.^10,38 These work products are then applied to clinical data (eg, the patient’s context) to achieve the desired intent and outcomes of the guideline and its recommendations.

During the IP, informatics experts transform the written guideline into computable expressions and representations (artifacts) that can be faithfully interpreted by computers to process patient data from clinical information systems.^39,40 The informatics team should include experienced clinicians and informaticians with expertise in knowledge engineering, terminologies, cognitive informatics, and clinical workflow analysis and who are trained in the principles, methods, and tools of information science with specific expertise in solving health care problems.^41–43 Key informatics activities in the IP include assessing the need for a computable guideline (phase 1), developing the computable artifacts (phases 4–7), validating and translating computable artifacts into derivative products (phase 8), and applying the computable guideline in clinical settings (phases 9–11).

Developing the computable expression of the CPG uses the framework of the knowledge engineering lifecycle.^10,44–46 This lifecycle includes knowledge acquisition, translation and representation, execution and validation, delivery, and continuous improvements.³⁹ In the IP, much of this work is focused on the earlier phases of this lifecycle to capture and explicitly express the guideline’s concepts and possible recommendations. The translation is the process of faithfully and pragmatically conveying this understanding into formal, explicit expressions and representations that a computer can use to reason over clinical data to achieve the intent of the guideline recommendations.⁴⁷ The informatics team works in close collaboration with clinical domain experts to translate the resulting computable CPG into derivative forms that support improved health outcomes, such as CDS, performance and quality measures (eg, eCQMs), eCase reports (eg, for registries), and other products to enable practice insights.⁴⁸ The computable CPG and its derivatives, executed in clinical information systems, can help enable learning within and across provider organizations.⁴⁹ Once the guideline has been released, informatics work is centered on guideline implementation (phase 10), with a focus on setting-specific factors and clinical workflows.⁴⁹ In these phases, applied clinical informaticians play a critical role in assessing, enabling, and optimizing clinical operations and outcomes, including activities related to clinical workflow, practice management, and clinical information systems (eg, CDS).

Health Communication and Dissemination

An effective plan to communicate and disseminate guidelines is essential to their effective implementation. However, communication and dissemination efforts have not been well studied to improve the adoption of guidelines.^50,51 Communication strategy and tactics are found throughout the IP’s 12 phases, combining the use of traditional channels (eg, articles, professional presentations, issue briefs, white papers, television, radio, and social media platforms) to share the guidelines with their intended audience and encourage their use. Effective communication relies on a carefully designed health communication plan that starts at the beginning of the IP and can be adjusted as needed. The CDC and the Agency for Healthcare Research and Quality (AHRQ) offer communication strategies to facilitate the use of health guidance.

In phases 0 and 1, guideline developers consult with communication experts (particularly if the proposed guidelines are controversial or urgent) to manage issues and estimate the resources needed for communication strategies, tactics, activities, and products for engagement. A guideline could be controversial due to several common issues: the guideline contradicts a common practice, the committee did not include specific experts, members could profit from the recommendations, or the recommendations may not have strong evidence. A communication workgroup should be formed to assess audiences’ communication needs, discuss the science of the guidelines, the products needed, and obtain agreement on external communication standards, protocols, and approvals. The audience needs can be defined through collaboration among authors, partners, stakeholders, and the media. Communicators can also connect with informatics experts to discuss newer informatics-driven and technical information approaches, including ways to share informatics products. As early as phase 2, communicators should collaborate with guideline developers and other partners to draft a communication plan with questions to consider and an estimated timeline of activities to do before, during, and after the release of the guideline. Plain language and person-centered communication products should be developed in tandem with more technical guidelines.

The draft communication plan can be reassessed in phase 4 when the guideline scope is established. When the guideline manuscript is being written in phase 7, the communication team needs to ensure that all relevant parties agree with the communication plan, update the plan as needed, and develop the communication products. During phase 8, when the guideline is reviewed, the communication team focuses on approvals for communication products, including social media messages, to make products accessible. They assess media interest and conduct an environmental scan of potentially controversial or confusing issues before the guidelines are released. In phase 9, the communication team prepares dissemination approaches, platforms, and content for use, and coordinates with publishers, the partners’ communication team, and guideline spokespeople. Communication products are launched at the time of guideline publication, and when the computable guideline and its associated artifacts are published and disseminated. Success is measured through online tracking mechanisms, such as website visits and downloads, social media metrics, citation counts, and other means to determine the success of the guideline’s reach with the intended audience.

Implementation

For a CPG to improve the health of individual patients and the population, it must be put into practice. Clinicians in health care organizations (HCOs) and caregivers must adopt the CPGs as part of their health care delivery, and patients need to adopt CPG-related health practices in their daily lives. Important factors can influence the successful implementation of clinical guidelines, such as lack of support from clinical staff and superiors, insufficient staff and time, and lack of familiarity with the guideline content.^52–54 Phase 10 focuses on strategies to help HCOs implement and use the CPG. The scope of the HCOs involved in implementing CPGs at the point of care varies widely—from large integrated delivery systems, academic medical centers, and multi-hospital systems to smaller individual hospitals and group or solo practices. Because organizational readiness varies greatly, implementation activities can be performed with varying levels of formality, and some activities are optional. Personnel involved with implementation activities can include some or all of the following: local informatics and information technology specialists, local experts, operations leaders, and clinical governance bodies.

CDS systems have helped meet this growing need to filter and integrate health care information. Overall, guidelines-based CDS implementation involves applying the “5 rights” of getting (1) the right information (evidence-based), (2) to the right person (someone on the care team, patients, or caregivers, etc.), (3) in the right intervention format (alert, order set, reference info, etc.), (4) through the right channel (EHR, personal health record, smart device, etc.), and (5) at the right time in the workflow.^14,47,55 Incorporating the “CDS 5 Rights” into the workflow increases the likelihood that clinicians will follow the computable CPG recommendations, and the appropriate data will be collected and documented. There are various ways to influence the adoption of CPGs, including reminder systems and continuing medical educational programs combined with follow-up phone calls and presentations at meetings.⁵⁶ At a high level, delivering CPG recommendations in a HCO may require localization of the CPG’s general artifacts to specific local clinical workflows and systems. For instance, general descriptions of the CPG’s rationale may need to be refined to describe more specifically the local rationale—why implementing a particular CPG is important and valuable to the local organization, its patients, and the community. Similarly, translating CPG artifacts into functioning computer code in local information systems may require further mapping with local system records and often will require localization of where and how these are introduced into local clinical workflows with a user-friendly design.

One of the tenets of computable guidelines is that there is a trusted, valid, and current computable representation of the recommendations of a clinical guideline (L3).³⁸ A well-designed computable guideline can be implemented in various ways as health information technologies mature (L4). For example, the same computable CPG can be implemented as CDS to help apply the guideline within the clinical workflow, as eCQM to track adherence to the recommendations, and as eCase reports to send data on cases of the relevant condition to public health agencies or to condition-specific registries.

Although phase 10 focuses on local implementation, the IP includes implementation perspectives throughout all phases. During phase 1 (defining the clinical problem/need; topic selection), implementers can provide health care organization and practitioner perspectives and point of care considerations. Implementers could be invited to comment on the project management plan during phases 2 and 3 and to learn more about the guideline’s development process. They could also consult with the guideline team in phase 4 to evaluate plausible clinical workflows that affect the scope of the guideline. In phases 6–8, as guideline recommendations are drafted, implementers can help determine what data and quality measures are needed for associated CDS that are feasible to collect in clinical workflows to put the computable guidelines into practice. Implementors can also engage closely with those designing informatics artifacts, bringing representative end-user clinician perspectives and local informaticians into the development process. In phases 9–11 as the guideline is disseminated, implemented, and evaluated, implementation activities proceed as described above, localizing communications about the CPG, doing any necessary tailoring of informatics artifacts to local systems and clinical workflows, and performing local evaluations.

Evaluation

Activities for evaluation during the IP, found throughout the 12 phases, offer a systematic approach for assessing whether and how the proposed IP contributes to improving (1) processes, (2) products, and (3) outcomes during the development and implementation of written and computable guidelines. These evaluation activities stand apart as a set of key questions, methods, and measures of success during the guideline creation and implementation. Process evaluation examines whether the steps of the IP are followed to develop and implement high-quality guidelines and what barriers and facilitators are encountered. Findings from the process evaluation create an opportunity to make corrections that can improve processes for developing subsequent guidelines. IP product evaluation measures whether the written and computable CPGs are of high quality, accessible, easy to use, understandable, and useful. Outcome evaluation measures examine whether specific objectives, such as guideline adoption and improved health outcomes, are achieved once the guidance is released and implemented.

Evaluation plans should be developed early (phases 4 and 7), and the evaluation of the guideline implementation, outcomes, and impact takes place in phase 11. Local measures of CPG adoption and effectiveness can serve to evaluate success, setting goals, and measuring the impact of implementation (eg, did the computable CPG help to achieve its goals or did it add to the provider burden?). Implementers can use their local evaluation data to inform whether to adjust guideline implementation for better results. This may include examining the guideline’s effectiveness on adherence, performance measures (for public reporting, payment, and accountability), and quality measures that align with the guideline (for national, regional, or local quality improvement efforts). Evaluation results and challenges can also inform guideline revisions and updates (phase 12).

Multiple theories and frameworks informed the evaluation activities described in the IP, including the CDC evaluation framework,⁵⁷ diffusion of innovation,⁵⁸ actor-network theory,⁵⁹ normalization process theory,⁶⁰ and human, organization, and technology-fit framework.⁶¹ Established guideline development standards informed the evaluation components specific to the evidence synthesis and process for creating the guideline recommendations.² Because the IP calls for software engineering, evaluation includes standard activities associated with informatics, including user-centered design in formative stages and usability and user-acceptance testing as products are prepared for release.^62–64

Summary of Tables in the 12-Phase IP

The (https://stacks.cdc.gov/view/cdc/131006) contains 12 tables of phases providing detailed descriptions of activities during the IP for the 5 major disciplinary “lanes” described above. Each table is broken down into activities (what to do), tasks (how to do the activities), the responsible entity and expertise needed (who’s responsible), success indicators (evidence of completed activity), and examples of hyperlinked resources and tools to assist in completing the activities. The activities in each table are built on internationally accepted standards for guideline development and integrate additional activities for informatics, communication, health information technology implementation, and evaluation throughout. This IP is intended to enable the oversight committee and workgroups to have bidirectional exchanges with iterative feedback loops to improve the efficiency and production of guideline products and reduce lag time. This IP is a guidance tool but not prescriptive. Guideline teams can tailor this guidance to their needs and to the context in which they are producing guidelines to implement key parts of the IP that are feasible. Below is a brief description of each phase.

Phase 0: Pre-Guideline Development—Set up Operational Framework and an IP for Co-Developing Written and Computable Guidelines

Phase 0 or “preguideline development” describes the steps required to build the necessary organizational infrastructure to support the co-development of any written and computable guidelines and foster their timely completion and utility. By the end of phase 0, the organization should have a high-level operational framework with clear standard operating procedures for initiating, planning, and developing written and computable guidelines. The decision to develop a new or updated guideline could be triggered by new evidence for preventing or treating a condition, an emerging health condition, evidence of an excessive clinical health burden, or observed health disparities, among others.

Phase 1: Define Clinical Problem/Assess Clinical Need/ Select Topic/Assess Clinical Need for Computable CPG and Additional Products

Example of a Clinical Problem.

Cardiovascular disease (CVD) is the leading cause of death in the United States, and nearly one-quarter of deaths caused by CVD are considered preventable. Abnormal blood glucose metabolism is one of several important modifiable cardiovascular risk factors. Computable guidelines and CDS can help provide patient-centered, evidence-based information on preventive treatment options to consider based on that patient’s individual health history and risk factors.

Phase 1 begins when a compelling case can be made to develop the guideline, with the determination that adequate resources are available to conduct the work. The guideline leader works with stakeholders and experts to identify and assess the presence, size, or nature of a clinical need, gap, or problem, and prioritize topics considered to be promising solutions if developed into a new or updated computable CPG (see example).^12,65 This priority-setting phase is the foundation for selecting and creating CPGs designed to improve clinical practice, individual behaviors, and health outcomes.

During this phase, organizational managers gather information, assess guideline user needs and organizational capacity, get leadership support, engage stakeholders and partners, and develop a shared vision. They also begin assessing project feasibility and resources and making decisions about processes for integrating informatics standards and protocols in guideline development. Tools like The Decision Tool for New Guideline Development (Appendix A https://stacks.cdc.gov/view/cdc/131006) can help provide structure to decision-making processes and ensure critical issues are addressed.

By consulting with stakeholders and experts, this phase helps answer important questions, such as:

• 1.End-user need: is a computable guideline needed? How would it add value for end users?
• 2.Alignment: how is the development of a computable guideline consistent with our organization’s mission and goals? Is our organization the most appropriate one to lead the development of the proposed guideline?
• 3.Resource availability: what resources are needed (financial, human, etc.), and can our organization develop a computable guideline?
• 4.Partners: who are possible partners in this guideline development project?
• 5.Scoping: what are the boundaries of this project (what is in and out of scope)?
• 6.Sustainability: what resources and efforts are needed to sustain the computable guideline once it is developed?

Clinical needs can be assessed at the population level (eg, epidemiological data from claims and administrative databases, clinical registries, and surveillance data), organizational level (eg, computerized health records at a hospital or clinic), or the clinical provider level (eg, provider surveys, direct observation, and EHR chart audits). The strategies employed for needs analysis depend on the purpose of assessment and the type of data and resources available.⁶⁶ Often, this assessment identifies several needs or gaps. A systematic approach using explicit criteria can assist in prioritizing needs and topics for guideline development.

Once the guideline topics are determined, multiple parallel assessments can determine the (1) appropriate guideline development processes for the topic and context (eg, available evidence); (2) guideline format—written guideline and computable guideline that can be translated into CDS, eCQMs, electronic case reportings, etc. with input from informatics experts; and (3) communication needs and products. Communicators need to be aware of the guideline development timeline. If a topic is controversial, communication and media planning can be prepared in advance to avoid issues later. Once these items are complete, the next step is to get leaders to approve the topic and ensure funding and the necessary internal and partner staff are available.

Phase 2: Establish Guideline Oversight Committee/Explore international Collaboration/External Partners/SMEs

Phases 2 and 3 focus on developing an oversight committee and workgroups involved in the guideline development process. Phase 2 involves establishing the oversight committee and partners to provide guidance and a framework for conducting the work. The oversight committee includes multidisciplinary members from organizational leadership with broad stakeholder representation. This committee establishes a project management plan following the integrated process that includes time frames and milestones for a specific guideline topic, and collaboration mechanisms and processes to share knowledge. The committee establishes the workgroups that conduct the work, including its members and goals, and the criteria for if/when to include (HCOs) for implementation issues. The committee identifies and establishes partnerships, describes management and communication rules, and conducts educational training for all workgroups on how the integrated process will work. Finally, the oversight committee conducts a kickoff meeting with stakeholders to gain their support for the plan, assesses important issues among the intended audience, reviews the science underpinnings of the guidelines, and obtains agreement on the communication standards, protocols, and clearance process.

Phase 3: Establish Workgroups: Guideline Panel, Informatics, CDS Implementors, Systematic Review Team, Informatics Team/Set up Collaboration Platforms and Tools

Phase 3 focuses on who will conduct the work and how it will be performed. Engaging people with diverse and appropriate expertise is essential to applying the IP for developing guidelines. The guideline workgroups should be composed of SMEs with knowledge about evidence review and synthesis, guideline development, informatics, communication, dissemination, implementation, and evaluation. Those who have expertise in more than 1 area could be engaged in multiple aspects of the IP. It is important to assess the conflicts of interest of each expert and obtain disclosures before their participation. Such conflicts may disqualify the expert from participation or influence the nature of their involvement. For example, an expert who owns a substantial amount of stock in a pharmaceutical company would likely benefit financially from a recommendation to use that company’s drug as a first-line treatment for a disease. Consequently, the expert may be excluded from guideline development activities, particularly those for creating the recommendation. Clear communication of roles and responsibilities and the use of relevant tools for collaboration can help maximize the engagement of members and their partners. Overall, phases 2 and 3 establish the infrastructure that includes partnerships, workgroups, a project management plan, collaborative platforms, and a kickoff meeting and communication plan with questions to consider so that workgroups can begin the guideline development work. (Appendix B and C https://stacks.cdc.gov/view/cdc/131006).

Phase 4: Determine Guideline Scope and Identify Computable CPG Concepts

Example of a PICO Question and Guideline Scope.

Is there direct evidence that systematic screening (either targeted or universal) [compared to no screening (C)] for type 2 diabetes, impaired fasting glucose, or impaired glucose tolerance (I) among asymptomatic, nonpregnant adults (P) improves health outcomes (O)?

To address this and other PICO questions, a systematic review focused on: (1) the benefits and harms of screening for prediabetes and type 2 diabetes; (2) the benefits and harms of interventions (such as behavioral counseling focused on diet, physical activity, or both, or pharmacotherapy for glycemic, blood pressure, or lipid control, compared with no treatment or usual care) for screen-detected prediabetes and type 2 diabetes or recently diagnosed type 2 diabetes; and (3) the effectiveness of interventions for prediabetes to delay or prevent progression to type 2 diabetes.

The purpose of the scoping phase is to give an overview and set boundaries for what the clinical guideline will cover, the key clinical issues, and what it will not cover. A well-defined guideline scope serves as the foundation for tasks in all phases. If the project scope is nebulous, shifts, or expands as development proceeds, it reduces the likelihood of project success.^67,68 Consequently, during phase 4 scoping, SMEs, methodologists, systematic reviewers, selected stakeholders, and others identify the type of guideline (eg, new, interim, update, or adaptation of an existing guideline) and draft research questions that the guideline will tackle, typically framed in terms of discrete populations, interventions, comparisons, and outcomes (PICO).^69,70 Longer formats that incorporate additional factors of interest can also be used, such as PICOTS, for which the (T) reflects potential changes in the effects of an intervention over time and the (S) reflects a specific setting.⁷¹ Ideally, PICO elements are represented in a visual analytic model that shows the causal pathway by which an intervention is linked to outcomes of interest. This graphical representation highlights critical premises about the relationships and outcomes of interest, and whether the correct questions are being asked, which then guides the systematic review of the evidence. In addition to defining and writing research questions, experts for the guideline topic consult with informatics and implementation experts to describe clinical workflows that might affect the scope and identify the data elements, terminologies, clinical use cases, and relevant and reusable computable CPG concepts and artifacts.⁷² For the proposed guideline, evaluators then develop an overall logic model that provides a graphical depiction of the shared relationships among the resources, activities, time frame, outputs, outcomes, and long-term impact among the target audience. The logic model assists in drafting a plan that will be used to evaluate the guideline once published, by facilitating a common understanding of the intended outcome evaluation indicators. This model also informs the evaluation plan for the IP and its products. The draft communication plan is also updated in concert with the guideline scope and evaluation plan.

Phase 5: Identify, Assess, and Synthesize Evidence and Represent CPG Artifacts as Logical Representations/Expressions

This phase focuses on steps to identify, extract, assess, synthesize, and grade the evidence for guideline development. A key task is to choose the evidence review methods most appropriate for the guideline under development. For example, evidence reviews for an urgent public health need, such as COVID-19, may use methods that are adapted for a rapid development timeline. Topics with primarily nonrandomized trials may use evidence quality frameworks more appropriate for rating this type of evidence. Other reviews may synthesize mixed types of evidence, such as literature reviews, expert observations, and EHR patient data. Regardless of the type of evidence gathered, there should be transparency in reporting about review methods, including the search strategy, data extraction, parameters for evidence synthesis, and study quality assessment.

Systematic reviews may also be “living” reviews with systematically developed, evidence-based, ongoing surveillance of the literature. Living reviews support continuously updated recommendations for the diagnosis and management of medical conditions.⁷³ Furthermore, “living guidelines” (also known as dynamic or organic guidelines) can be developed using large and continuously updated data sets of real-world or near-real-time practice-based data that are included in guideline recommendations through a process of continuous surveillance, either manually, through special software or using artificial intelligence.^73–75

During phase 5, the informatics team reviews drafts of the evidence and identifies clinical concepts, potential data elements, and decisional flows that inform logic representations in computer code. The discipline of knowledge engineering comes to the fore as the guideline is structured into discrete information elements, such as clinically meaningful concepts and variables driving clinical decisions. Elements are mapped to standard terminologies used by the health industry to create clinical information systems. These activities are a precursor to creating and refining the computable CPG version of the guideline.

Phase 6: Craft the Recommendations and Computable CPG Artifacts

During phase 6, the guideline team develops, rates, and prioritizes the written recommendations, while the informatics and implementation experts conduct core activities needed to design and produce the computable CPG. Significant integration among the guideline development, informatics, and implementation experts occurs during this phase. A key informatics work product is a logical, formalized representation of the recommendations, typically in the form of logical flow diagrams, showing all clinical decisions arising from the recommendations and all clinical inputs (eg, required data elements, patient assessments, calculations) and outputs related to patient care (eg, diagnostic procedures, medical treatments, follow-up). In the informatics community, the same information can be represented in varying levels of structuredness. For example, the written guideline recommendations are referred to as “L1” (ie, narrative), and these logical representations are often referred to as “L2” (ie, semi-structured).^47,76 (Figure 2) A mid-progress review also occurs during this phase to make sure that the development of the written and computable CPGs meets expectations and to plan midcourse corrections as needed. If resources allow, the informatics and implementation experts also develop and test the computable CPG artifacts, work that continues in phase 7. These can be used by local implementers to develop CDS applications, measures of adherence to the overall guideline and health outcomes, and other derivative products.

Figure 2.

Semi-structured version of the 2015 US Prevention Service Task Force full recommendation statement for Screening for Abnormal Blood Glucose and Type 2 Diabetes Mellitus. This is a direct example from the US Prevention Service Task Force. Additional inclusion criteria outlined in this decision log are included in the section titled “Patient Population Under Consideration” of the full recommendation statement.^76,77

Example of a Written Diabetes Guideline Recommendation*.

The USPSTF recommends screening for abnormal blood glucose as part of cardiovascular risk assessment in adults aged 40–70 years who are overweight or obese.

* Note: This 2015 recommendation was superseded by 2021 recommendations.

Phase 7: Draft the Guideline Text and Article. Finalize Computable CPG Artifacts and Create Documentation

Once the guideline recommendations have been crafted, phase 7 begins with a review of the process for drafting the text for the guideline, including an outline of content and section headings, a finalized timeline of requisite tasks, and a plan for roles, responsibilities, and expectations. Additional supporting materials are developed to explain each recommendation. These materials typically include text that describes the methods, recommendations, supporting evidence and evidence tables, flowcharts, algorithms, figures, glossary of terms and abbreviations, group member conflict of interest statements, and other supplementary materials. Guideline developers conduct a quality assessment of the entire guideline document to obtain approval for the review. As the guideline text is written, informatics and implementation experts validate and finalize the computable CPG artifacts (L3) (ie, structured) (Figure 3) and associated documents (eg, user stories, use cases, use case diagrams, decision trees, flowcharts). Terminologies, ontologies, metadata, and other requirements are identified to support tagging and other approaches to cataloging resources on digital platforms. The guideline development group, end-user and consumer representatives, and other stakeholders also determine whether other derivative products are needed (eg, eCQMs, eCase reports). After computable CPG artifacts are completed, repositories and libraries are prepared for concepts and data elements (profiles, terminologies, value sets, libraries, groups, rules, and logic representations/expressions).

Figure 3.

Excerpt of computable diabetes screening recommendation¹²

While the draft guideline text and computable CPG artifacts are being finalized and before they are released to the public, the evaluation group updates, expands, and reviews the evaluation plan. The logic model is reviewed to determine final specific, measurable, achievable, realistic, and timely (SMART) objectives and evaluation questions, choose or develop appropriate evaluation methods and instruments to assess expected outcomes, pinpoint the intended audience and timing for evaluation, identify necessary evaluation resources, and determine how evaluation results will be used with future guideline updates. The communications team continues to collaborate with the other teams during phase 7 and makes sure they share the communication plan with other relevant staff in the organization (eg, communications, media, and policy). Communication products are developed, and items that will be published should be part of discussions to coordinate dissemination timing, design, format, branding, messages and messengers, and strategies to manage controversial topics.

Phase 8: Finalize article for Internal Review, External Review, Peer Review, and Public Comment and Validate CPG and Derivatives

Phase 8 comprises all the activities needed to prepare the guideline for its release to the public. Activities include finalizing the guideline manuscript, the computable CPG, and derivative products for review within and outside the guideline development organization. Derivative products include written or computable content derived from the guideline (eg, pocket cards, clinician mobile apps, continuing education, and computable artifacts such as info buttons and measures, including eCQMs).⁴⁸ This allows a transparent means for reviewers to examine the methods used to produce the written guideline and the computable CPG artifacts. The document undergoes public comment and peer review to allow an objective review of the guideline by outside professionals with timely revisions. The draft CPG is published online for examination to prevent the CPG from becoming outdated or irrelevant by the publication date. During this phase, the communication team engages stakeholders to promote awareness during the peer review and public comment period and ensures the communication products, messages, and social media are appropriate and easily accessible. The communication team also conducts an environmental scan to assess media interest and manage issues that may arise from the release of the guidelines. A final review of the evaluation plan is performed, so it is ready to be launched once the guideline is published. Finally, the article and other products are submitted for publication.

Phase 9: Publish and Disseminate the Guideline

Phase 9 focuses on communicating and disseminating the guideline, CPG artifacts, derivative products, and accompanying communication products at the time of publication in a peer-reviewed journal. Although phase 9 is the culmination of strategic communication planning and coordination starting from the initial decision to develop or update the guidelines, its success is predicated upon decisions and activities that take place much earlier. The communication team launches the communication plan and prepares the logistics of releasing the guidelines through the following activities: ensuring platforms and content are ready for use; notifying media of available spokespeople; coordinating with partners and the primary journal; preparing people who will speak about the guideline; releasing prepared statements and background documents; lifting embargoes; posting web content (Figure 4) and the CPG artifacts onto repositories such as AHRQ’s CDS Connect;⁷⁸ launching other dissemination activities, such as social media posts, webinars, and responding to inquiries; and tracking activities and evaluation metrics of the launched communication plan to determine if the guideline had a successful release.

Figure 4.

Example of a consumer guide fact sheet disseminated for the diabetes recommendation. Archived: Abnormal Blood Glucose and Type 2 Diabetes Mellitus: Screening | United States Preventive Services Taskforce (uspreventiveservicestaskforce.org).

Phase 10: Implement Guideline Locally

In phase 10, a health care provider organization that implements the published guideline will transform the accompanying L3 computable CPG artifacts into functioning L4 executable computer code, such as CDS (eg, order sets or advisories) and/or eCQMs within their EHR, to apply the guideline in their practice. To do so, the organization may prioritize which guideline recommendations to implement in their practice based on criteria.⁷⁹ Next, they may analyze where it is feasible to incorporate guideline recommendations into local workflows and clinical information systems, such as EHR, laboratory, and pharmacy systems. A mid-progress review is conducted with partners to review the resulting design before building and testing. This is a critical step to success. Seeking users’ perspectives for CDS usability and integration into the workflow improves decision support.⁸⁰ In 1 study, adherence to medication guidance by clinicians using CDS was enhanced by keeping alerts to a minimum.⁸¹ Once deemed feasible, implementers can begin iterative design-build-test cycles to develop or localize CDS tools, eCQMs, and other types of derivative products typically in a development environment. About midway through implementation, planning for launch should begin. In preparation for moving the new tools into the production EHR, clinical workflows, local policies, and procedures may need to be updated. Clinical users will need to be introduced to the CDS tools and trained if the new tools are complex. Once the tools are migrated to the production environment, additional refinement of local workflows and CDS tools may be needed, based on user feedback and ongoing assessments. Ultimately, the CDS tools supporting the CPG become part of established clinical workflows in the organization, setting the stage for measurement of effects on clinical practice and health outcomes and potentially for producing and sending eCase reports (eg, to disease registries). (Figure 5) Beyond clinically based CDS, the CPG may be implemented in other ways, for example, by inclusion in mobile applications (apps) or interactive care plans for patients spanning multiple care teams. (Figure 6)

Figure 5.

Example of computable diabetes recommendation implemented in a provider-facing CDS.

Figure 6.

Example of computable diabetes recommendation implemented in a patient-facing CDS mobile app.

Phase 11: Evaluate Guideline Outcomes and Impact

Example of Evaluating Diabetes Guideline Recommendations.

In the evaluation of the patient-facing CDS through the b.well mobile application, the end users generally found the educational information and game-like challenges to be useful and stated that they planned to take action on the preventive health recommendation.

In phase 11, activities are focused on evaluating whether the computable guideline is achieving its objectives, including effective implementation, adoption and regular usage, and improved health outcomes among targeted populations. After the guideline is released, the evaluation team finalizes the outcomes evaluation plan and fully implements it in conjunction with the communication plan and in collaboration with partners at local health care settings. Early data collection evaluates local awareness and perceived usability and usefulness of the guideline, as seen in the b.well mobile application example.⁸² Data collected soon after implementation may inform efforts to improve the uptake of the guideline. After the evaluation data are collected and summarized, the implementers and evaluation team closely monitor and review the results with clinicians at the local site to adjust the clinical workflow. Data collected over time is used to determine adherence to the recommendations and whether they are improving health outcomes. Guideline developers use the data and any new evidence for guideline updates, as noted in phase 12.

Phase 12: Update Guideline

Guidelines need to be periodically updated based on new information, for example, changes in evidence or values placed on outcomes, or changes in interventions such as new drugs or newly reported adverse events related to the guideline.⁸³ Updating a conventionally written guideline usually involves reviewing the available evidence, after which the guideline development group determines whether recommendations still apply or whether modifications or new recommendations are needed. Updating the computable guideline involves ensuring that new technologies and CPG components, such as CDS and eCQMs, are updated for subsequent local adaptation. Currently, the literature does not describe the processes required to update computable guidelines in response to changes in evidence. However, the HL7®-created CPG-on-FHIR® standard describes Agile processes that accommodate such updates.⁸⁴ Nonetheless, such processes will likely require careful operational planning and management by trained staff. As part of this IP, phase 12 proposes a system that supports continuous and timely feedback for updating the CPG according to established parameters. Furthermore, it describes the link between the written and computable guidelines and their associated computable CPG artifacts in a way that enables an easy and quick update and maintenance of these CDS tools, eCQMs, eCase reports, and other products after the guidelines are updated. It includes collecting information from providers and CDS users as well as nontraditional evidence such as real-time data from clinical practice and exploring and using new technologies such as artificial intelligence, text mining, machine learning, and crowdsourcing. If a “living guideline” approach is used, updating can be more dynamic with modular modifications made to accommodate the latest scientific information. Finally, it is important to have a plan to clearly communicate and evaluate the updated guideline recommendations and derivative products.

Summary

In developing the 12 phases, the authors reviewed several past and ongoing real-world clinical guideline development projects and vetted the process with informatics and implementation professionals from HL7® (www.HL7.org) to add to and validate details of the IP. This resulted in a revised diagram representing the life cycle of guideline development, as seen in Figure 7. The outer circle shows 12 phases of formal guideline development resulting in published written recommendations. The middle circle details informatics and clinical implementation activities for the computable guideline. Collaboration and iteration among teams across phases are shown as paired arrows at critical phases of the process. A third inner circle summarizes the evaluation activities to be conducted during guideline development. Finally, at the center of the process are the guideline users, who should be engaged through human-centered design processes in all phases.

Figure 7.

A 12-Phase integrated process for developing and implementing clinical practice guidelines.

The diagram also presents the life cycle within quadrants, showing the interim and final outputs of the guideline development activities, such as written recommendations, corresponding to “levels of knowledge” proposed by Boxwala et. al.⁴⁷ for engineering computer-based CDS. While either the original Figures 1 or 7 may be used to orient teams to the IP, this diagram better emphasizes the cyclical and iterative nature of creating written and computable CPGs through the 12 phases.

Case Example for COVID-19

To illustrate how the IP can be tailored to specific guideline development contexts, the authors present a short retrospective case study, referencing the cycle diagram (Figure 7) and corresponding phases of the IP. A more detailed version of this case study is in the FHIR® Clinical Guidelines (CPG-on-FHIR®) implementation guide.⁸⁵ In May 2020, during the SARS-CoV-2 pandemic, an interdisciplinary team created a computable clinical best practice tool for classifying the severity of COVID-19 cases in emergency departments and making recommendations for testing and management. This served as an informal pilot of the IP. Team members from the Adapting Clinical Guidelines Initiative joined with professionals from nearly 50 private sector organizations, including hospitals and health systems, medical specialty societies, and IT vendors, in a “collaboratory,” named the COVID-19 Digital Guideline Working Group (Figure 7, phase 1). The working group was further organized into 2 interdisciplinary Agile CPG teams, to (1) develop guidance and (2) translate the written guidance into a computable form (ie, L1, L2, and L3 artifacts) as described in detail in Figures MSC.04 and FIG MSC.07 in the HL7® COVID-19 severity classification methodology in the CPG-on-FHIR® IG (phases 2 and 3).⁸⁵ Specific teams conducted work focused on guidance development, informatics, and implementation using integrated and Agile processes. The informatics team members included experts in knowledge engineering, medical terminology (nomenclature for interoperability, eg, SNOMED, FHIR®), clinical informatics, and software engineering. The teams set up shared tools platforms to complete the work (phase 3).

Collaboratory participants reported that the Agile CPG teams, which included clinicians, informaticians, and representatives from medical professional societies and practice centers, cycled through activities in regular working sessions while emerging evidence was rapidly reviewed (phases 4–8) to produce iterations of the guidance. Interdisciplinary work included documenting evidence and detailing clinical guidance on COVID-19-related severity classification, risk assessment and prognosis, diagnostic testing, and interpretation, level of care (ie, management), and pharmacologic and nonpharmacologic treatment (phase 5). The informatics team (specifically, knowledge engineers) identified clinical concepts needed for the management tool and defined the requisite data elements and logic necessary to express those concepts (phase 5).⁸⁵ These data definitions were provided to the COVID-19 interoperability alliance and terminology vendors for explicit definitions in standard terminologies and their mappings to real-world interface terminologies in current use within EHRs to ensure usability when deployed.⁸⁶ All further informatics knowledge engineering work products are built upon these defined data elements and mapped clinical concepts. Next, the content development team created a 2-step visual representation for the triaging and testing of patients who presented in emergency settings (phase 6). This visual representation used the patient’s clinical history, monitoring, and diagnostic testing results to determine a COVID-19 severity score, risk assessment, and calculated prognosis. The severity and risk of disease progression, in turn, determine the most appropriate interventions and level of care for the patient (eg, ambulatory, inpatient, critical care, discharge, or escalation of in-hospital care). At the same time, the Agile CPG team structured the content into diagrams (L2) showing data flows, clinical workflows, and decision points, including explicit descriptions to inform the corresponding logic (L3) for the computable guideline. (Phase 6).

The work continued in a concurrent, iterative fashion in phases 6–7. As the content team finalized the management tool in written form, the informatics team refined the flow diagrams (L2) and computable expressions (L3). As the teams moved toward publication, iterative reviews of design and work products took place across the functional teams (guidance development, informatics, and implementation), experts, and end users (phases 6–8). While the final emergency department COVID-19 management tool was undergoing peer review (phase 8), experts reviewed and simulated realistic test cases of the computable models. By August 2020, 3 months after project start, the American College of Emergency Physicians disseminated the management tool to its 50,000 members worldwide for immediate use and the initial “Draft for Trial Use” computable format was made available (phases 9–10).

In 2021, members of the COVID-19 Digital Guideline Working Group continued their work to evaluate the use of the management tool in clinical settings and to update it with emerging evidence and best practices (phases 11–12). Although the development of this management tool was atypical and driven by the urgent need to respond to the pandemic, there were clear benefits in using integrated and Agile processes⁸⁴ to develop the written and computable guidance at the same time, as reported by working group members. These benefits include increased speed of development and determination of guidance, prompt clarification of content-related terms and logic, and streamlined production of computable artifacts for use in CDS systems, apps, registries, and more. Such an approach further makes possible clinical feedback and mechanisms for rapid learning and improvement.

Discussion

To advance the process of producing guidelines in the digital age, an interdisciplinary team produced and proposed this 12-phase IP for co-developing, communicating, implementing, and evaluating written and computable CPGs. While keeping true to the methodological standards of guideline development, the team incorporated information technology support, evaluation results, and novel dissemination and publication methods, as well as rigorous quality assurance throughout the guideline life cycle. Compared to a traditional “waterfall” or sequential approach, this IP is expected to result in a more synchronized and Agile approach for the development, communication, implementation, adoption, and evaluation of guidelines in clinical practice settings. The 12 phases provide a clear roadmap of activities, resources, and tools in the tables for conducting this interdisciplinary work to translate new guideline knowledge into practice more efficiently. The process builds on contemporary practices for guideline development, including the GIN-McMaster Development Checklist, and integrates internationally vetted and published HIT standards (ie, CPG-on-FHIR®).^{2,30,32,33,87} It provides a framework for guideline developers to: (1) facilitate early engagement and collaboration among interdisciplinary experts and partners in co-developing and implementing written and computable CPGs, to allow a better understanding of guideline nuances and pilot testing in a health care environment; (2) develop communication plans early in the guideline development process designed to increase awareness and stimulate adoption of the guideline; and (3) develop evaluation plans, from start to finish, that use ongoing feedback loops to support rapid improvements of the CPG, its use, and resulting health outcomes. Developing communication and evaluation plans early in the IP can aid in the faster promotion to the intended audience and evaluation soon after the CPG guideline and related artifacts are published.

Key Results Within the Relevant Literature

Recent evidence shows promising results of CDS systems combined with the integration of computable CPGs into clinical workflows to further improve patient care and health outcomes.^27,28,88 Although CDS systems have been shown to improve health care process measures related to decision-making, diagnostic accuracy, and appropriate testing across diverse settings, effects on clinical and economic outcomes remain sparse.¹⁸

Project Limitations

This preliminary effort to develop an IP for written and computable guideline development has several limitations. First, this IP for guideline development and implementation has not been fully applied, tested, and evaluated. Nevertheless, there are instances, such as the COVID-19 case study, in which parts of an IP have been enacted. More examples are found in the HL7® CPG-on-FHIR® implementation guide.¹¹ These examples informed the development of this process. Second, there has been no published work on an efficient approach for developing computable CPGs along with earlier planning of communication, implementation, and evaluation activities. Therefore, this IP was developed primarily based on a literature review and the expertise of a multidisciplinary group with experience in guideline development, informatics, CPG implementation, communication, and evaluation. Third, this innovation assumes that the necessary diverse subject-matter expertise will be available to the organization when following this process. However, this may not be feasible in all cases, particularly when organizations have focused on the traditional forms of guideline development and implementation. Finally, while this process is intended to shorten the overall time to guideline implementation, it may lengthen the development time and require greater costs and staff time during earlier phases of guideline development because of its multidisciplinary approach.

Implications for Current Practice and Policy

In 2016, the AHRQ launched CDS Connect, a web-based platform for sharing interoperable CDS. The Office of the National Coordinator for Health Information Technology, the principal federal entity charged with supporting a nationwide HIT infrastructure, recently recommended improving HIT and reducing documentation burden, time inefficiencies, and hassle for health care providers.⁸⁹ There are several implications of this 12-Phase IP for improving HIT and current practice and policy. First, awareness of this process can spur guideline developers to include other relevant expertise. This IP can also be a tool to educate interdisciplinary teams, including informaticians, implementers, evaluators, and communicators about the guideline development process.

Second, the adoption of this IP may facilitate a more efficient approach to guideline development, decrease the time to implementation, and provide timely updates. New evidence should be able to be more quickly integrated into computable CPGs fostering increased consistency and accurate implementation across HCOs. Further, the IP supports the integration of large-scale data from patients into computable CPGs. For example, the rapid cycle of knowledge gained during the COVID-19 pandemic is an example of the speed at which “big data” from clinical practice, in combination with literature evidence, can inform computable CPGs.

Third, incorporating informatics, evaluation, and communication activities earlier in the guideline development process will require resources beyond those typically allocated in developing a written guideline. However, co-development, rather than sequential development of the written and computable CPGs may provide substantial savings in overall costs and time during both the development and implementation phases. A further advantage is that communication and evaluation activities will be properly budgeted and executed efficiently. An Evaluation Framework companion article in this supplement provides a means to evaluate the benefits of the IP.

Finally, this 12-Phase IP is intended to offer flexible and Agile guidance. Guideline developers can tailor it to their needs and implement some or all parts of the IP depending on what is deemed important or feasible. There could also be different entry points to the IP if the outputs of some phases have already been produced through other processes. For example, the CDC Division of HIV Prevention Guideline is using select components of earlier phases of the IP to update their HIV Screening Guidelines (phase 12), which were originally developed as narrative guidelines. The COVID-19 Digital Guideline Working Group, working with the American College of Emergency Physicians, also used a tailored IP to develop a severity risk score for patients with COVID-19.⁸⁵

Recommendations for Further Research or Policy and Practice

Components of this IP have already been applied in practice, as demonstrated with the quick development of the COVID-19 severity risk tool and currently to updating the CDC HIV guidelines. Pilot testing, evaluation research, and tracking metrics are needed to demonstrate if and how well this 12-Phase IP improves the current state of guideline development and use.

The IP may be well suited for a living guidelines approach due to its iterative and cyclical phases and flexibility to tailor it for use. Living guidelines are typically modular, with sections updated as warranted by new information, thereby shortening the production and publication time required for journal-based guidelines. Tools are available to enable the dynamic updating of guidelines.⁹⁰ Guideline developers would likely shift their guideline development methods if the use of this 12-Phase IP, either alone or in combination with a living guidelines approach, resulted in reduced times to guideline implementation, more efficient use of resources, improved guideline adherence, clinical care, or patient health outcomes.

Interdisciplinary work with the HIT industry on this IP has already advanced the production of written and computable guidelines and products, for example, diabetes, opioids, and a COVID-19 severity risk screening tool. As more interdisciplinary teams engage in this IP, lessons will be learned about what works, and improvements will be made to yield faster implementation of guidelines into clinical information systems and practice.

Conflicts of interest

The authors have no conflicts of interest to disclose.