Knowledge creation: synthesis, tools and products

The foundation for any activity related to knowledge translation is knowledge; it’s hard to do KT without the “K.” In the first article of this series, we introduced the knowledge-to-action cycle, which provides a framework for strategies in knowledge translation. In the centre of this cycle is the knowledge funnel, which represents knowledge creation.¹ As knowledge moves through the funnel, it is refined and, ideally, becomes more useful to end-users of the knowledge. These end-users can include clinicians, researchers, policy-makers and the public. During each phase of knowledge creation, producers of knowledge tailor their activities to the needs of these end-users.

First-generation knowledge is that which is derived from primary studies such as randomized trials. Knowledge synthesis represents second-generation knowledge; systematic reviews are examples of this knowledge-related product. Third-generation knowledge includes tools and products such as decision aids and educational modules whose purpose is to present knowledge in user-friendly, implementable formats.

Implementation of evidence from individual studies may be misleading because of bias in the conduct of studies or random variations in findings. This fact has led to greater emphasis on knowledge syntheses as the foundation of efforts to implement knowledge.² Knowledge syntheses serve to interpret the results of individual studies within the context of all of the evidence. These syntheses provide the evidence base for other tools of knowledge translation, such as decision aids for patients, clinical practice guidelines or policy briefs. For further discussion of the types of knowledge syntheses available, see a recent article by Ioannidis that describes them more fully.³ We need to avoid the “knowledge translation imperative,” or the assumption that the results of every study should be implemented. Instead, we should focus on areas where a mature and valid evidence base exists.

Knowledge tools such as guidelines for clinical practice and decision aids for patients can be used by clinicians and patients to make evidence-based decisions. The goals of using these tools are to provide an explicit, evidence-based description of the risks and benefits of an intervention and to outline areas where evidence is lacking. Many situations exist where sufficient evidence is lacking or the balance between risks and benefits is nearly even⁴ and this highlights the importance of clinical expertise and exploration of the values and circumstances of patients.

What are clinical practice guidelines?

Evidence-based guidelines for clinical practice are knowledge tools that are systematically developed statements aimed at assisting clinicians and patients to make decisions about appropriate health care for specific clinical circumstances.⁵ Clinicians are bombarded with practice guidelines, as seen in guideline databases,⁶ which contain thousands of these tools. Practice guidelines may also inform decisions made by health care policy-makers and clinical managers.⁷ In addition to many other factors (e.g., the values and preferences of patients, care providers and society) guidelines are intended to assist decision-making but not supplant it. They are not dictum-like or formulaic tactics to drive patient care. Guidelines should state whose viewpoint was considered in the development of the guidelines and provide a framework for clinicians and patients to use when deciding how to implement the guidelines.

How are practice guidelines developed?

Systematic and rigorous methods exist to ensure that high-quality clinical practice guidelines are created.^8–13 The development of guidelines requires a combination of social engagement and methodological rigour (Box 1). Ensuring that the appropriate stakeholders are involved is a key strategy to facilitate successful development and implementation of guidelines. Stakeholders may include clinicians, researchers, funders, policy-makers, patients and others. Ideally, the group carrying out the development should be composed of a multidisciplinary team of stakeholders, including clinical and content-specific experts, methodologists and other end-users, including patient representatives.¹⁴ The external review of guidelines is also a form of social engagement. The process of external review can create accountability between developers and users of the guidelines, and provides a forum through which stakeholders can be engaged in the endorsement and implementation of the guidelines. It is only with the engagement of relevant stakeholders that a viable strategy for implementation of the guidelines can be developed.^8,9

Box 1. Steps in the development of clinical practice guidelines

• Establish multidisciplinary guideline team.

• Identify clinical question that explicitly defines the patients, intervention or exposure, comparisons (if relevant), outcomes of interest and setting.

• Conduct a systematic review of evidence.

• Appraise and interpret evidence and come to consensus on its meaning.

• Draft guideline recommendations that align with evidentiary base.

• Complete an external review of draft report among intended users and key stakeholders.

• Revise the guideline in response to external review.

• Prepare the final guideline report for distribution and dissemination

• Prepare strategy for implementation.

Methodologically, clinical practice guidelines begin with a clinical question informed by a clinical or care-related problem. The PICO (Patient, Intervention, Comparison, Outcome) format can be used to formulate the question. This question informs the inclusion and exclusion criteria that will be used to complete a systematic review of the evidence. Ideally, guidelines should include all relevant strategies (e.g., strategies for screening, diagnosis, and treatment or prognosis) and the full range of outcomes (both good and bad) that are important.

Valid guidelines create their evidence-related components from systematic reviews of relevant, worldwide literature.¹⁵ This evidence base serves as the foundation for developers to make clinical recommendations. The reviews are “necessity”-driven and synthesize the best evidence (even if it’s of shaky quality) that can be found to guide a decision. Thus, some recommendations in a guideline may be derived from high-quality evidence and others from evidence that is much more liable to error.¹⁵ Given that the quality of evidence may vary, it is useful to have the strength of the recommendations graded on the quality of evidence found. Although variability exists in the methodological approaches to this grading, developers and users of guidelines agree that a need exists for a clear link between recommendations and the evidence that supports them.^16,17

The GRADE (Grading of Recommendations Assessment Development and Evaluation) method is an emerging strategy favoured by those who seek a structured, categorical framework for grading the level of evidence and strength of recommendations.^18,19 A clear articulation of how the developers interpreted the evidence and reached their conclusions is important. In addition to an evidence-specific component, guidelines should have a detailed instructional component about what to consider in individual patients. Although much of this instructional component needs to be individualized to local settings and patients, general frameworks can be provided that consider local variation in prevalence, feasibility, potential barriers and other variables.

The next step in developing a guideline is an external review by key stakeholders and intended users of a draft version of clinical practice guidelines.^8–11 This review can improve the quality of guidelines by identifying evidence that was missed. It can also enable stakeholders to endorse the interpretation of the evidence by the developers or offer alternative interpretations. Furthermore, this review provides an opportunity to explore the “implementability” of recommendations, including barriers and enablers to their application.

Although several resources are available to guide in the development of original clinical practice guidelines,^8–11,13 recent efforts by a group of international researchers known as Adapte have resulted in a rigorous methodology to enable clinical practice guidelines from one jurisdiction or context to be adapted for use in another.²⁰ The process of adapting guidelines is discussed in a later article in this series.²¹ Adaptation is identified as an important step toward reducing duplication in the development of guidelines, which can result in a tremendous cost in financial and human resources.

Do clinical practice guidelines work?

The impact of clinical practice guidelines on practice and outcomes is complex. Systematic reviews by Grimshaw and others^22,23 suggest that interventions to implement clinical practice guidelines do, on average, influence both the processes and the outcomes of care, although the effect sizes tend to be modest at around 10%. Interventions for implementation ranged from mass-media interventions to use of local opinion-leaders and include interventions targeted toward the public, health care professionals, managers and others.²³ However, the potential benefits of clinical practice guidelines are only as good as the quality of the guidelines themselves. Interestingly, Grimshaw and colleagues²³ found in their review of 235 studies that only 3 guidelines were explicitly evidence-based.

Although faithfulness to evidence-based principles is important, other nonscientific factors influence the uptake of guidelines. These factors include the perceptions by the adopters of the guidelines’ characteristics and messages and of the development process, and factors related to norms and context.²⁴ The likelihood of success is increased by a strategy for implementation that includes analysis of enabling factors and barriers, selection of appropriate and feasible interventions related to knowledge translation, and indicators to measure impact. These issues are addressed in later articles in this series.^1,8,25–29

How do we determine the quality of clinical practice guidelines?

Not all clinical practice guidelines are created equal. The quality of these tools can be extremely variable and often falls short of basic standards. For example, when Graham and colleagues²⁷ appraised the quality of 217 Canadian clinical practice guidelines for drug therapy that were produced between 1994 and 1999, they found that less than 15% of those reviewed met 50% or more of the 20 criteria assessing the rigour of development. The mean proportion of criteria that were met was 30%. Before efforts at implementation are undertaken, it is critical that a high-quality guideline be developed or adapted. Not unusually, different professional organizations may provide different recommendations based on the same evidence.³⁰ These differences can result from differences in methods (e.g., search criteria, the way evidence is appraised and the way recommendations are created), in the interpretation of evidence and in health care settings.

In response to this challenge, the AGREE (Appraisal of Guidelines Research and Evaluation) Collaboration was established to develop a valid and reliable instrument for assessing the quality of guidelines (Box 2). The AGREE Collaboration defines quality of guidelines as the level of confidence that can be placed in the minimization of biases linked to the rigour of development, presentation and applicability of a clinical practice guideline and in the clear reporting of each step of the development process.^12,13 The AGREE instrument is composed of 23 items answered using a four-point Likert scale (i.e., ranging from “strongly disagree” to “strongly agree”), one item for global endorsement answered with a four-point categorical scale, and a User Guide.^12,13 The AGREE Next Steps initiative builds on this work to improve the reliability and establish the validity of the instrument.

Box 2. Instrument domains of the AGREE (Appraisal of Guidelines Research and Evaluation) Collaboration

• Scope and purpose of the guideline

• Involvement of stakeholders in development of the guideline

• Rigour of the process of development

• Clarity and presentation of the summary of evidence and recommendations

• Applicability of the guideline

• Editorial independence

The AGREE instrument provides important criteria upon which to evaluate clinical practice guidelines. But the clinical validity, appropriateness of recommendations and thorough analysis of the capacity to implement recommendations are not within its scope. Indeed, although no gold standards for these measures of quality currently exist, some advances have been made. For example, the GLIA (Guideline Implementability Appraisal) tool provides a unique perspective by assessing the extent to which recommendations are implementable (Box 3).^28,29

Box 3. Elements of the GLIA (Guideline Implementability Appraisal) tool for assessing guidelines²⁹

• Decidability: precisely under what circumstances to do something

• Executability: exactly what to do under the defined circumstances

• Effect on process of care: the degree of impact of the recommendation on the usual workforce

• Presentation and formatting: the degree to which the recommendation is recognizable and succinct

• Measureable outcomes: the degree to which the guidelines identify markers or end-points for tracking the effect of implementation of the recommendation

• Validity: the degree to which the recommendation reflects the intent of the developer and the strength of the evidence

• Novelty and innovation: the degree to which the recommendation proposes behaviour considered unconventional by clinicians or patients

• Flexibility: the degree to which the recommendation permits interpretation and allows for alternatives in its execution

• Computability: the ease with which the recommendation can be operationalized in an electronic information system (only relevant if electronic implementation is planned)

What are decision aids for patients?

Decision aids for patients translate evidence into patient-friendly tools to inform patients on their options, help them clarify the value they place on benefits versus harms, and guide them in the process of decision-making.³¹ As mentioned earlier, knowledge syntheses are the foundation for these knowledge tools. Specifically, evidence included in decision aids for patients is defined as up-to-date scientific information on options, benefits and risks of options, and associated probabilities.³² Formats for these tools include paper-based booklets, video/DVDs, decision boards and Internet-based materials. Decision aids for patients are used as adjuncts to counselling by practitioners for decisions for which the best choice depends on how patients weigh the benefits, risks and scientific uncertainty (e.g., birth control, genetic testing, breast and prostate cancer treatment, options for menopause symptoms, back pain, osteoarthritis or level of care at end of life). These decision aids differ from educational materials by not only providing information on options but also tailoring it to patients’ clinical risk profiles and guiding patients to define and express their personal values. Items from a sample page of a patient decision aid are provided in Appendix 1, available online at www.cmaj.ca/cgi/content/full/cmaj.081230/DC1. Others are available elsewhere online at www.ohri.ca/decisionaid.

How are decision aids developed?

High-quality decision aids for patients are developed using a systematic process. We briefly summarize the methods of development here. Explicit guides to the process are available elsewhere.³² The first step is to determine the decision-making needs of potential users (e.g., patients and clinicians). Needs assessments focus on the users’ perceptions of the decision (i.e., options, outcomes, values), perceptions of others involved in the decision (i.e., decisional roles, opinions, pressures) and resources needed to make or implement the decision.³³ Second, the patient decision aid is based on a synthesis of the evidence and includes essential elements (Box 4). To minimize bias and improve the ability of patients to understand the probabilities of outcomes, evidence-based criteria are available for displaying probabilities within decision aids (Box 5). This is a crucial and complex step; we refer readers to the handbook by Jacobsen and O’Connor³³ for specific details on how to frame the evidence. Third, the decision aid is reviewed by a panel of experts external to the development process. The panel may include clinicians, researchers, patients and others. Finally, the decision aid is evaluated by end-users. Defining a “good decision” is a challenge when no single “best” therapeutic action exists and choices depend on how patients value benefits versus harms. The International Patient Decision Aids Standards Collaboration has reached consensus on criteria for judging the quality of decisions (i.e., decisions should be informed and preference-based) and criteria for processes leading to high-quality decisions (i.e., recognize that a decision needs to be made, know options and their features, understand that values affect the decision, be clear about the options and features that matter most, discuss values with the clinician and become involved in preferred ways).

Box 4. Essential elements of decision aids for patients

• Evidence-based information on the condition, options and outcomes relevant to the patient’s clinical situation

• Risk-based communication of the probabilities of potential outcomes and the levels of scientific uncertainty of each

• Clarification of values to ascertain which benefits, harms and scientific uncertainties are most important to the patient

• Structured guidance in the steps of deliberating and communicating with practitioners and significant others

Box 5. Criteria of IPDAS (International Patient Decision Aids Standards Collaboration) for the presentation of the probabilities of potential outcomes of different options³³

• Use event rates that specify the population and period

• Compare probabilities of potential outcomes using the same denominator, time period and scale

• Describe uncertainty of probabilities

• Use multiple methods to view probabilities (words, numbers, diagrams)

• Allow patients to view probabilities based on their own situation

• Place probabilities in context of other events

• Use both positive and negative frames

Do decision aids work?

A recent review of 10 systematic reviews of decision aids for patients showed that these knowledge tools improve patients’ participation in decision-making, patients’ knowledge of options, and agreement between patients’ values and the subsequent treatment or screening decisions.³⁴ Impact on clinical outcomes is less clear.³⁴ When probabilities of outcomes are presented, patients have more realistic expectations of the chances of benefits and harms. The use of elective surgery (e.g., hysterectomy, prostatectomy, mastectomy, coronary bypass surgery, back surgery) decreased in favour of more conservative options without apparent adverse effects on health-related outcomes or anxiety. One trial evaluated the cost-effectiveness of patient decision aids for women experiencing benign abnormal uterine bleeding.³⁵ Although no consistent effect on health status was observed, women who received information plus coaching to help them express their preferences had lower rates of hysterectomy compared with women in the control group (odds ratio [OR] 0.60; 95% confidence interval [CI] 0.38–0.96) and women who received information but no coaching (OR 0.52; 95% CI 0.33–0.82]). Costs to the health system were lower for those who received education and coaching related to preferences.

A systematic review of 28 studies found that barriers to implementing decision aids for patients in clinical practice include practitioners’ perceptions of the readiness of patients to use the aids, forgetting to offer them to patients, too complex or too simple content of thought of practitioners, time required to make them available, outdated evidence, cost and limited accessibility.³⁶ The review also found that decision aids for patients are more likely to be used when they result in positive effects on patient outcomes or the clinical process, when patients prefer to actively participate in decision-making and when health professionals are motivated to use them.

How do we determine the quality of decision aids?

Although many decision aids for patients are available, they are, as with clinical practice guidelines, of variable quality.³⁷ As a result, the International Patient Decision Aids Standards Collaboration was established to reach agreement on criteria for developing these decision aids and appraising their quality.³² The International Patient Decision Aids Standards checklist has domains that include essential content (i.e., providing information, presenting probabilities, clarifying values and guiding deliberation and communication), development (systematic process of development, balance, evidence base, plain language, disclosure) and evaluation (quality of decisions).^32,38 This checklist is used to appraise decision aids available for patients in the Cochrane Decision Aid Inventory and the A-to-Z Inventory at a publicly available website, (www.ohri.ca/decisionaid). Validity and reliability testing of an International Patient Decision Aids Standards instrument is underway.

Evidence-related gaps

Research should focus on a number of areas to advance our understanding of knowledge tools. These areas include strategies to enhance understanding and implementation of systematic reviews (syntheses) and knowledge products such as guidelines and decision aids for patients. More research is needed to understand the impact of decision aids for patients on important clinical outcomes. Studies of the validity and reliability of the International Patient Decision Aids Standard are necessary.

The book Knowledge Translation in Health Care: Moving from Evidence to Practice, edited by Sharon Straus, Jacqueline Tetroe and Ian D. Graham and published by Wiley-Blackwell in 2009, includes the topics addressed in this series.

Articles to date in this series

• Straus SE, Tetroe J, Graham I. Defining knowledge translation. CMAJ 2009;181:165–8.

Key points

• Systematic reviews form the foundation of many activities related to knowledge translation.

• High-quality clinical practice guidelines and decision aids can improve the quality of decisions and care.

• Ongoing efforts to improve the application of clinical practice guidelines and decision aids are warranted.