Abstract
Context: Barriers to the use of systematic reviews by policymakers may be overcome by resources that adapt and present the findings in formats more directly tailored to their needs. We performed a systematic scoping review to identify such knowledge-translation resources and evaluations of them.
Methods: Resources were eligible for inclusion in this study if they were based exclusively or primarily on systematic reviews and were aimed at health care policymakers at the national or local level. Resources were identified by screening the websites of health technology assessment agencies and systematic review producers, supplemented by an email survey. Electronic databases and proceedings of the Cochrane Colloquium and HTA International were searched as well for published and unpublished evaluations of knowledge-translation resources. Resources were classified as summaries, overviews, or policy briefs using a previously published classification.
Findings: Twenty knowledge-translation resources were identified, of which eleven were classified as summaries, six as overviews, and three as policy briefs. Resources added value to systematic reviews by, for example, evaluating their methodological quality or assessing the reliability of their conclusions or their generalizability to particular settings. The literature search found four published evaluation studies of knowledge-translation resources, and the screening of abstracts and contact with authors found three more unpublished studies. The majority of studies reported on the perceived usefulness of the service, although there were some examples of review-based resources being used to assist actual decision making.
Conclusions: Systematic review producers provide a variety of resources to help policymakers, of which focused summaries are the most common. More evaluations of these resources are required to ensure users’ needs are being met, to demonstrate their impact, and to justify their funding.
Keywords: Systematic reviews, decision making, databases, knowledge translation
The systematic review of research on a specific topic has become widely recognized as an aid to evidence-informed decision making in health care (Centre for Reviews and Dissemination 2009). Substantial public funding of synthesized evidence and guidance has been used to support health care decision making around the world, and in the United Kingdom, this investment has been described as National Health Service (NHS) research and development's most important contribution to the global science base (Horton 2006). Despite this investment, the availability of a systematic review does not necessarily ensure that decision makers know about it or can make sense of the findings (Centre for Reviews and Dissemination 2009).
There has been much recent interest in the usefulness of systematic reviews in policymaking contexts (Lavis et al. 2005; Sheldon 2005), despite the documented barriers to using systematic reviews to inform national, local, or organizational policy decisions. These barriers include difficulties in locating and appraising reviews; the review reports’ lack of timeliness or user friendliness; and the real or perceived failure of reviews to address relevant questions, contextualize the findings, or make actionable policy recommendations (Innvaer et al. 2002; Lavis et al. 2005; Petticrew et al. 2004; Sheldon 2005). While recognizing that the real world of policymaking reflects a complex interaction among economic, political, and social factors; research evidence; and the ideologies of those making the decisions (Klein 2003), it also is clear that the potential impact of systematic reviews on decision-making processes is not yet fully achieved.
One way in which these barriers can be overcome is through the provision and use of resources that adapt for and present to policymakers the findings of systematic reviews in a more directly useful form. In a recent discussion paper, Lavis distinguished among three types of review-derived products (summaries of systematic reviews, overviews of systematic reviews, and policy briefs) aimed at policymakers and other stakeholders and gave examples of each (Lavis 2009). Summaries encapsulate systematic reviews’ take-home messages and may add value by, for example, evaluating their methodological quality or assessing the findings’ local applicability. Overviews of systematic reviews systematically and transparently identify, select, appraise, and synthesize systematic reviews in a given topic area. Policy briefs based on systematic reviews systematically and transparently identify, select, appraise, and synthesize systematic reviews, other research studies, and context-specific data to address all aspects of a policy question (Lavis 2009). In this article, we use the term knowledge-translation resources to describe all three of these products and services. As far as we are aware, no systematic search for such products has been undertaken.
As part of a larger research project (Hanbury et al. 2010), we are providing a knowledge-translation service to local decision makers in the Leeds–Bradford area of England, translating existing sources of synthesized and quality-assessed evidence into actionable messages. To inform the development and maximize the impact of this service, we performed a systematic scoping review of knowledge-translation resources as just defined. Utilizing Lavis's taxonomy, we identified and described existing products and approaches. We were particularly interested in any evidence regarding whether some ways of presenting evidence were more effective than others in their perceived usefulness to policymakers and whether this might translate into differences in their use of systematic review evidence.
Methods
We systematically identified resources that translated the results of systematic reviews into forms useful for policymakers.
Inclusion and Exclusion Criteria
Resources were eligible for the review if they were based exclusively or primarily on systematic reviews and included a statement or other clear evidence that they were aimed at health care policymakers, particularly those at the national or local level responsible for buying services or meeting the service needs of a population or coverage/reimbursement decisions. Resources had to have a stated focus (e.g., by subject or relevance to a particular geographical area).
To be included, resources had to contain an element of “translation” or repackaging of the review content. Simply making available systematic review reports with executive summaries written by the review's authors was not enough to qualify for inclusion. For practical reasons, we used only English-language resources that continue to be regularly or periodically updated. We also included studies of any design that evaluated resources cited in the review that used a sample of the intended user group or administrative data.
Finally, we excluded resources making systematic review evidence accessible only to patients/consumers, those for health professionals making decisions about individual patients (unless relevant to higher-level decision making), and clinical guidelines based on systematic reviews.
Identification of Knowledge-Translation Resources
We searched electronic databases (MEDLINE, EMBASE and CINAHL) from 1990 to October 2009 for papers describing or evaluating knowledge-translation resources. (Detailed search strategies are available on request from the authors.) We screened the websites of International Network of Agencies for Health Technology Assessment (INAHTA) members, resources that Lavis mentioned (Lavis 2009), and other sites known to us. We also contacted the Cochrane Collaboration. We sent an email survey to the contact address for each website, requesting information about and evaluations of the organization's knowledge-translation activities, mainly to locate unpublished evaluations. We also screened and contacted resources drawn to our attention by survey respondents and scanned reference lists of retrieved articles to find more studies.
In addition, we screened abstracts of the Cochrane Colloquium (2000 to 2009) and HTA International (2007 to 2009).
Selection and Data Extraction
First, one of the reviewers screened websites to eliminate those with no English-language content. The remaining sites were evaluated against the inclusion criteria by two reviewers independently. Decisions on inclusion of studies identified by the literature search were made by at least two reviewers independently.
We classified resources as summaries of systematic reviews, overviews of systematic reviews, or policy briefs using Lavis's definitions (Lavis 2009). Our synthesis of the resources was mainly descriptive and focused on their “value-added” features in facilitating policymakers’ access to evidence; we did not try to assess the resources’ quality.
We extracted data from the evaluation studies identified by the literature search using a separate form. For studies that evaluated knowledge-translation resources in a user survey, the extracted data included answers to three key questions for surveys (who was studied, how the sample was obtained, and what the response rate was) (Rithalia et al. 2009). (A full description of the review methods is available from the authors on request.)
Results
Knowledge-Translation Resources
Our searches identified twenty knowledge-translation resources that met our inclusion criteria, of which eleven (e.g., Evidence Update, Health-evidence.ca, and AHRQ's Effective Health Care Program policymakers’ summaries) were classified as summaries of systematic reviews (see Table 1), six (including ARIF reports, Evidence Essentials, and SUPPORT summaries) as overviews of systematic reviews (see Table 2), and three (including McMaster Health Forum evidence briefs) as policy briefs (see Table 3). The details of the included resources are listed alphabetically in appendix 1, and those resources that narrowly failed to meet the inclusion criteria are listed in appendix 2.
Table 1.
Characteristics of Summaries of Systematic Reviews
| Name | Produced by | Subject and/or Geographical Focus | Product Description | Methods Described | Features |
|---|---|---|---|---|---|
| Cochrane-Related Resources | |||||
| Evidence Update | Effective Health Care Research Consortium, led by the Liverpool School of Tropical Medicine, UK | Health care interventions relevant to people in low- and middle-income countries | Two-page summaries of Cochrane reviews | No | 1 |
| Health Knowledge Network bulletins | Centre for Health Communication and Participation, Latrobe University, Australia | Evidence-based strategies for consumers’ and caregivers’ engagement in health | Five-to-eight-page bulletin summarizing findings of a single Cochrane review | Yes | 1 |
| Policy Liaison Initiative | Policy Liaison Initiative (a collaboration between the Australian government's Department of Health and Ageing and the Australasian Cochrane Centre) | N/A (range of topics/interventions covered) | Brief summaries of Cochrane reviews | No | 1 |
| Other Systematic Review Producers’ Resources | |||||
| Database of Abstracts of Reviews of Effects (DARE) | CRD, University of York, UK | Effectiveness of interventions | Database of critical abstracts of quality-assessed systematic reviews | Yes | 1, 2, 3 |
| Health-evidence.ca | McMaster University, Canada | Public health and health promotion | Database of health promotion and public health systematic reviews | Yes | 1, 2 |
| Rx for Change | CADTH (Canadian Agency for Drugs and Technologies in Health) | Interventions to improve drug prescribing and use | Summaries of individual reviews plus brief overviews by type of intervention | Yes | 1 (for topic areas), 2 |
| Effective Health Care Program Policymaker summaries | Effective Health Care Program (AHRQ); existing summaries prepared by John M. Eisenberg Center and Oregon Health and Science University, USA | Comparative effectiveness research in USA | Four-page “summary for clinicians and policymakers” | No | 1, 7 |
| “Alert report” summaries | SBU (Swedish Council on Health Technology Assessment) | Swedish HTA reports | Short (one- or two-page) summary with extensive reference list | Yes | 1, 7 (sometimes), 8 (sometimes) |
| Community Guide | U.S. Task Force on Community Preventive Services with support from the Centers for Disease Control and Prevention | Interventions to promote health and/or prevent disease in community settings; USA | Web-based summary of systematic review findings and task force's recommendations relating to a specific intervention | No | 1 |
| WHO-Related Resources | |||||
| Reproductive Health Library | Department of Reproductive Health and Research, World Health Organization | Sexual and reproductive health, especially in developing countries | A combination of IMRAD-style commentaries summarizing Cochrane reviews and DARE abstracts for non-Cochrane reviews | Yes (instructions for commentary) | 1, 2, 4, 5 (for Cochrane) |
| Health Evidence Network (HEN) evidence summaries of network members’ reports | HEN, part of WHO Europe | Health policy in Europe | Summaries of systematic review reports produced by HEN-member organizations | No | 1, 4 |
Notes: 1 =“bottom line” summary of key points; 2 = evaluation of methodological quality; 3 = evaluation of reliability of conclusions; 4 = evaluation of generalizability of results; 5 = expert commentary (by independent subject experts); 6 = includes other types of research evidence alongside systematic reviews; 7 = addresses local context for implementation or decision making; 8 = produced in response to decision makers’ requests.
Table 2.
Characteristics of Overviews of Systematic Reviews
| Name | Produced by | Subject and/or Geographical Focus | Product Description | Methods Described | Features |
|---|---|---|---|---|---|
| ARIF reports | Aggressive Research Intelligence Facility, Public Health, Epidemiology & Biostatistics, University of Birmingham, UK | NHS decision makers in West Midlands region (England) | “On demand” summary reports for health care decision makers in West Midlands | Yes | 1, 2, 3, 4, 8 |
| Evidence in brief (Alberta Ambassador Program) | Institute of Health Economics, Alberta, Canada | Pain management in primary care; Canada (Alberta) | Brief tabular summaries of evidence from systematic reviews | Yes | 1, 2 |
| Evidence Essentials | Royal Australasian College of Surgeons: ASERNIP-S | Surgical procedures and devices; Australia | Reports “designed to inform on the existence and findings of high-level evidence such as systematic reviews and health technology assessments” and provide rapid and timely information to decision makers | Yes | 2, 6, 7, 8 |
| Evidence Notes | NHS Quality Improvement Scotland | Scotland | Four-page summaries of systematic reviews’ findings on particular topics or clinical areas, along with brief information about local prevalence and any safety or economic implications | No | 1, 4, 7, 8 |
| Supporting Translation of Evidence into Policy and Practice (STEPP) | South African Cochrane Centre in collaboration with Western Cape Provincial Department of Health (DoH), South Africa | South Africa | One-page summaries mainly based on Cochrane review findings, along with brief information about local applicability | Yes | 1, 7, 8 |
| SUPPORT summaries | SUPPORT Collaboration, led by Norwegian Knowledge Centre for the Health Services | Reviews of interventions to improve maternal and child health in low- and middle-income countries (LMICs) and of health systems interventions relevant to LMICs | Structured summaries of relevant reviews that include background information needed to understand the findings, a detailed summary of the review's main findings (strength of evidence rated by GRADE), and an assessment of the findings’ relevance to LMICs that cover applicability, equity issues, and any economic considerations | Yes | 1, 2, 3, 4, 7 |
Notes: 1 =“bottom line” summary of key points; 2 = evaluation of methodological quality; 3 = evaluation of reliability of conclusions; 4 = evaluation of generalizability of results; 5 = expert commentary (by independent subject experts); 6 = includes other types of research evidence alongside systematic reviews; 7 = addresses local context for implementation or decision making; 8 = produced in response to decision makers’ requests.
Table 3.
Characteristics of Policy Briefs
| Name | Produced by | Subject and/or Geographical Focus | Product Description | Methods Described | Features |
|---|---|---|---|---|---|
| EVIPNet (Evidence-Informed Policy Network) policy briefs | EVIPNet and associated organizations; EVIPNet is supported by the World Health Organization (WHO) | Health policy; low- and middle-income countries | Website includes documents produced by various organizations and in various formats. Methodology as described meets criteria for policy briefs based on systematic reviews. Briefs vary in the extent to which they are based on systematic reviews. | Yes | 1, 2, 3, 4, 5, 6 |
| Health Evidence Network (HEN) evidence reports | Health Evidence Network, part of WHO Europe | Health policy; Europe | Evidence reports are described as “syntheses of the best available evidence in response to policymakers’ questions.” They vary in the extent to which they are based on systematic reviews. | Yes | 1, 2, 4, 8 |
| McMaster Health Forum evidence briefs | McMaster Health Forum, McMaster University, Canada | Health policy; Canada | Evidence briefs summarize research evidence (drawn from systematic reviews when possible) about a problem, plus three options for addressing the problem and key implementation issues. They do not contain recommendations. | Yes | 1, 2, 4, 6, 7 |
Notes: 1 =“bottom line” summary of key points; 2 = evaluation of methodological quality; 3 = evaluation of reliability of conclusions; 4 = evaluation of generalizability of results; 5 = expert commentary (by independent subject experts); 6 = includes other types of research evidence alongside systematic reviews; 7 = addresses local context for implementation or decision making; 8 = produced in response to decision makers’ requests.
Literature Search for Evaluation Studies
The literature search identified 5,121 records, of which 82 were ordered for full-text evaluation or as background papers. We included four published evaluation studies that met our criteria (Handoll and Madhok 2001; Packer and Hyde 2000; Rashiq et al. 2006; Thornton-Jones, Hampshaw, and Soltani 2002). We excluded a study by Ciliska and colleagues (Ciliska et al. 1999) because it appeared to be evaluating the systematic reviews themselves rather than a resource or service derived from them.
Our screening of Cochrane Colloquium abstracts and contacts with the authors identified three unpublished studies that met our inclusion criteria (Dilkes, Hill, and Ryan 2008; Rosenbaum, Glenton, and Oxman 2008; Young et al. 2005). One of these evaluations was subsequently accepted for publication (Rosenbaum et al. 2011). We also found an unpublished evaluation of the excluded Evidence Aid service (Turner et al. 2008). Our screenings of the HTA International's proceedings did not identify any relevant studies.
Characteristics of Summaries of Systematic Reviews
We identified eleven resources that provided value-added summaries of systematic reviews. Six of these resources reported on methods, for example, how reviews were selected for summarizing and the criteria used to assess the quality of the methodology. In addition to a summary of key points, the most common features were evaluations of the review's methodological quality and/or its generalizability across settings (Table 1). Two services (Effective Health Care Program policymaker summaries and SBU Alert report summaries) appeared to relate the evidence to the local context in a few but not all cases. One service (the WHO Reproductive Health Library) included concise commentaries on Cochrane reviews written by independent subject experts.
Characteristics of Overviews of Systematic Reviews
The resources classified as overviews of systematic reviews (Table 2) generally had more features than did the summaries and were more likely to describe methods (five out of six did so). Four of the resources produced overviews in response to requests from policymakers. Although classified as primarily a summary source, the Rx for Change database also provides brief overviews of systematic review evidence for each type of intervention within its scope.
Characteristics of Policy Briefs Based on Systematic Reviews
We included three series of policy briefs (Table 3), although only those by the McMaster Health Forum were unambiguously based primarily on systematic reviews. The EVIPNet and HEN products were described as using the best available evidence and varied in their use of systematic reviews. Only one of the three series was produced in response to policymakers’ requests, and one addressed the local context for implementation.
Evaluations of Knowledge-Translation Resources
Of the seven resources evaluated, two produced summaries of Cochrane reviews (Dilkes, Hill, and Ryan 2008; Handoll and Madhok 2001), and the remainder were classified as overviews. Five of the evaluations were surveys of users (see Table 4), but only three offered response rates. The SUPPORT evaluation differed from the others in that representatives of the intended user group were interviewed during the development process rather than after the service was up and running (Rosenbaum, Glenton, and Oxman 2008). Two evaluations took the form of descriptive reports (Thornton-Jones, Hampshaw, and Soltani 2002; Young et al. 2005), and only the evaluation of the Alberta Ambassador Program was conducted independently of the service provider (Rashiq et al. 2006).
Table 4.
Evaluation Studies
| Name of Resource | Study Design | Population of Interest | How Sample Was Obtained | Response Rate |
|---|---|---|---|---|
| Summaries | ||||
| Cochrane Database of Systematic Reviews (summaries of Cochrane reviews) (Handoll and Madhok 2001) | User survey | Decision makers in UK local health authority | Review summaries “sent for feedback and discussed with key people involved locally in those areas” | Not reported |
| Health Knowledge Network (HKN) Evidence Bulletins (Dilkes, Hill, and Ryan 2008) | User survey | Members of HKN (consumer groups, health professionals, and policymakers) | All members invited to participate | 41/170 (approx.); ≈ 24% |
| Overviews | ||||
| Alberta Ambassador Program (Rashiq et al. 2006) | User survey | Health professionals attending workshops on the Ambassador Program | All workshop participants were surveyed | 79/130 (61%) for postworkshop questionnaires |
| ARIF (Packer and Hyde 2000) | User survey | Representatives of health authorities in West Midlands (UK) | Respondents selected individually | Not applicable |
| Systematic reviews and other sources used to describe “best practice” in decision sheets (Thornton-Jones, Hampshaw, and Soltani 2002) | Descriptive report of project methods and outcomes | Decision makers in UK local health authority | Not applicable | Not applicable |
| SUPPORT summaries (Rosenbaum, Glenton, and Oxman 2008) | User testing via semistructured interviews | Health policymakers in low- and middle-income countries | Purposive sampling to include policymakers and managers at different levels from five countries | 13/21 (62%) for follow-up questionnaire |
| STEPP (Young et al. 2005) | Review of project's progress | Policymakers in South Africa | Not applicable | Not applicable |
The McMaster Health Forum policy briefs have been evaluated, but the results are not available for wider circulation. At the time of this writing, a report summarizing the evaluations of the briefs produced so far was planned (J. Lavis, personal communication, 7/28/2010). Regular evaluations of the Policy Liaison Initiative summaries also have been conducted, but they are the property of the Australian Department of Health and Aging and are not available for wider circulation (M. Misso, personal communication, 3/2/2010).
Evidence of Usefulness/Use
All the evaluations reported on the resources’ perceived usefulness to policymakers, and four provided evidence of actual use. Handoll and colleagues reported that a summary of a Cochrane review had been of direct use to local researchers, resulting in the routine use of the Cochrane Library (Handoll and Madhok 2001). Of the respondents to the Alberta Ambassador Program postworkshop survey, 62/79 reported that they had disseminated material to others (Rashiq et al. 2006), although this represented fewer than 50 percent of the 130 workshop participants. In their decision making, members of the East Riding and Hull Health Authority Maternity Strategy Group used the “decision sheets” produced by Thornton-Jones and colleagues, who in turn used systematic reviews and guidelines. Specifically, these decision sheets informed a recommendation for the early alignment of services for Down's syndrome that was subsequently implemented (Thornton-Jones, Hampshaw, and Soltani 2002). Finally, the study of seventeen requests for information from the ARIF service found that actions following the appraisal of evidence included a new service developed and put in place (1), influenced primary care purchasers not to purchase a service (1), did not succeed in influencing primary care purchasers (2), and a topic taken up by a regional group (2). No action was taken in nine cases, although further work was planned or in progress in eight of these (Packer and Hyde 2000).
Challenges
The evaluations also found a number of challenges in translating systematic review evidence for policymakers. Two studies reported that some users found the review summaries/overviews too long and complex (Dilkes, Hill, and Ryan 2008; Rosenbaum, Glenton, and Oxman 2008). A third found only limited support for summaries of Cochrane reviews (Handoll and Madhok 2001). This was attributed in part to “policy overload” in the NHS, which reduced interest in resources not directly related to current national policies and the existence of other products covering similar ground (e.g., Effective Health Care), as well as increasing local access to the full text of reviews via the Cochrane Library.
The STEPP program in South Africa, which produces overviews of systematic review evidence in response to policymakers’ requests, received only one request in a year. Again, this dearth of requests was attributed to policymakers’ occupation with implementing policies. A lack of culture and mechanisms for raising questions about the effects of interventions also was thought to inhibit policymakers’ use of the service (data from unpublished reports).
Rashiq and colleagues (Alberta Ambassador Program) and Thornton-Jones and colleagues (decision sheets) saw as an issue the amount of work required to translate systematic review evidence for policymakers (Rashiq et al. 2006; Thornton-Jones, Hampshaw, and Soltani 2002). Thornton-Jones and colleagues pointed out that much of this work was developing methods at the outset of the project. The Alberta team also reported some disagreements among team members about the content of the summaries.
Discussion
Main Findings
To the best of our knowledge, ours is the first systematic review of this field. The results reveal a wide range of systematic review–based services for policymakers. Summaries were the most common, followed by overviews. The few policy briefs may reflect the labor- and knowledge-intensive nature of this type of service. A number of new EVIPNet policy briefs on improving access to malaria treatment in Africa were published while we were working on this review (EVIPNet 2010), and such products may become increasingly available in the future. The summaries and overviews varied in their features and target audience.
We were surprised by the few published evaluations that we were able to locate. Most of the studies reported on perceived usefulness, although there were some examples of review-based resources being used to assist actual decision making, for example, decisions to develop a new service or to reject a proposal to purchase a service (Packer and Hyde 2000; Thornton-Jones, Hampshaw, and Soltani 2002). The study by Handoll and Madhok (2001) suggested that users’ perceptions of the relevance of the underpinning reviews limited the use of value-added summaries of Cochrane reviews. This work, however, was completed more than ten years ago, and by now there are likely to be many more policy-relevant Cochrane reviews. None of the evaluations assessed the cost-effectiveness of the services provided, a key question in a setting in which the resources for knowledge management are finite.
The evaluations also emphasized the challenges of translating evidence into a form that policymakers would find useful and of engaging with them to provide the most relevant and timely evidence (Thornton-Jones, Hampshaw, and Soltani 2002; Young et al. 2005). Lavis presented a model of policy briefs in a 1:3:25 format (one page of key messages, a three-page executive summary, and a twenty-five-page report, with a longer technical report if necessary). This format follows policymakers’ stated preferences (Lavis et al. 2005) and has been used by the McMaster Health Forum, but no evaluation results are available at present. The forthcoming report of an initial evaluation of the McMaster policy briefs (J. Lavis, personal communication, 7/28/2010) will be awaited with interest.
We located a number of unpublished evaluations, suggesting the possible existence of publication bias affecting the literature in this area. Accordingly, we urge service providers who evaluate their services to improve internal quality to make these publicly available whenever possible, either through formal publication or via the organization's website.
Strengths and Limitations
We cannot rule out the possibility that our search missed some relevant resources, although our systematic approach picked up some resources not cited in Lavis's paper (Lavis 2009). Our Internet search covered major providers and was supplemented by an email survey.
Even though we defined our inclusion and exclusion criteria in advance, we had some issues in applying them. Some of our exclusions, for example, Evidence Aid and possibly Health Systems Evidence, may surprise some readers, but we felt that these services were providing an index to other resources rather than translating or repackaging content. Our decision to include the Community Guide but to exclude the products of the U.S. Preventive Services Task Force (USPSTF) also was difficult, but we felt that the latter was definitely providing guidelines and hence our protocol excluded it.
Although Lavis's classification helped us synthesize the findings, his categories may not be mutually exclusive. For example, the SUPPORT summaries have characteristics of both summaries and overviews, and Rx for Change provides both overviews and summaries of systematic reviews. The term policy brief or something similar has been applied to many different products. Lavis and colleagues from the SUPPORT collaboration published a detailed description of systematic review–based policy briefs (Lavis et al. 2009), but as yet there are few published examples that meet their specification. Further development of ways of classifying knowledge-translation resources may be useful.
Implications
We recognize that research evidence is only one of many factors in decision making, especially at higher levels of policy (Klein 2003), and that most policymaking processes are best understood not as organized around discrete decisions but as a continuous flow of “problems,” possible “solutions,” and political circumstances/opportunities that often occur without obvious endpoints or decisions (Davies, Nutley, and Smith 2000). This does not preclude, however, the need for knowledge providers to work hard to make the evidence accessible and, when possible, to evaluate the success of their efforts.
The number of systematic reviews published annually has been increasing each year, and this upward trend looks likely to continue (Booth et al. 2010). As the numbers increase, so too does the need for local adaptation and interpretation, as busy decision makers are faced with making sense of multiple reviews assessing different interventions for the same condition or assessing the same intervention but involving subtly different combinations of population, comparator, and/or outcomes. For example, recent systematic reviews of population screening for breast cancer have produced different conclusions about the balance of benefits and harms and the robustness of the evidence base (Gotzsche and Nielsen 2009; Nelson et al. 2009). Policy briefs or overviews of systematic reviews could help policymakers interpret these findings and address conflicting findings, although a combination of methodological and clinical expertise would be required for more complex cases.
More evaluations of services are required to ensure that the end users’ needs and preferences are being met, to demonstrate their impact, and to justify their funding. Comparisons of different formats for systematic review–based products, as Rosenbaum and colleagues did (Rosenbaum, Glenton, and Oxman 2008), also could be useful.
Other useful features of the knowledge-translation services we reviewed are applicability of the systematic review results to the local context and implementation issues. Policymakers also are likely to want information about the cost-effectiveness of both interventions and policies (i.e., taking into account the costs and effects of the implementation strategies as well as the interventions themselves [Mason et al. 2001]), including the degree of uncertainty surrounding these issues. We propose incorporating these features as much as possible into the evidence briefings that we produce in the future. Our briefings will be close to Lavis's model of policy briefings, but shorter and addressing smaller topics. For example, we recently produced a briefing on models of service provision for adolescents with eating disorders to inform a possible reconfiguration of services (Chambers et al. 2010). From the outset, we planned to engage decision makers in producing briefings and to make their evaluation an integral part of the process.
Conclusion
Systematic review producers provide a variety of resources to help policymakers, of which focused summaries are the most common. The extent to which these resources are used and are found useful by policymakers is unclear. More evaluations of these resources are required to ensure that users’ needs are being met, to demonstrate their impact, and to justify their funding.
Acknowledgments
This work was done as part of the TRiP-LaB (Translating Research into Practice in Leeds and Bradford) project funded by the National Institute of Health Research (NIHR) as part of the Collaboration for Leadership in Applied Health Research and Care (CLAHRC) for Leeds, York, and Bradford (LYBRA). The views expressed in this article are those of the authors and not necessarily those of the NIHR or the CLAHRC. We are grateful to all those who responded to our survey and/or supplied copies of unpublished reports.
Appendix 1 Details of Included Resources
| Alberta Ambassador Program. Available at http://www.ihe.ca/research/ambassador-program/(accessed May 25, 2010). |
| ARIF (Aggressive Research Intelligence Facility). Available at http://www.arif.bham.ac.uk/(accessed May 27, 2010). |
| ASERNIP-S (Australian Safety and Efficacy Register of New Procedures—Surgical) Evidence Essentials. Available at http://www.racs.edu.au/racs/research-and-audit/asernip-s (accessed January 18, 2011). |
| Community Guide. Available at http://www.thecommunityguide.org/index.html (accessed May 28, 2010). |
| Database of Abstracts of Reviews of Effects (DARE). Available at http://www.crd.york.ac.uk/crdweb/(accessed May 24, 2010). |
| Effective Health Care Program policymaker summaries. Available at http://effectivehealthcare.ahrq.gov/index.cfm (accessed May 21, 2010). |
| Evidence update. Available at http://www.liv.ac.uk/evidence/evidence.htm (accessed May 25, 2010). |
| EVIPNet (Evidence-Informed Policy Network). Available at http://www.evipnet.org/php/index.php (accessed May 26, 2010). |
| health-evidence.ca. Available at http://www.health-evidence.ca/(accessed May 27, 2010). |
| Health Evidence Network (HEN). Available at http://www.euro.who.int/en/what-we-do/data-and-evidence/health-evidence-network-hen (accessed May 26, 2010). |
| Health Evidence Network (HEN): Evidence summaries of network members’ reports. Available at http://www.euro.who.int/en/what-we-do/data-and-evidence/health-evidence-network-hen/publications/evidence-summaries-of-network-members-reports (accessed June 21, 2010). |
| Health Knowledge Network bulletins. Available at http://www.latrobe.edu.au/chcp/hkn/bulletins.html (accessed May 28, 2010). |
| McMaster Health Forum. Available at http://www.mcmasterhealthforum.org/about_products.php?cat=product (accessed June 24, 2010). |
| Policy Liaison Initiative: Relevant and accessible evidence from The Cochrane Library. Available at http://www.cochrane.org.au/ebpnetwork/(accessed May 27, 2010). |
| Rx for Change database. Available at http://www.rxforchange.ca (accessed May 24, 2010). |
| SBU Alert. Available at http://www.sbu.se/en/About-SBU/SBU-Alert/(accessed May 25, 2010). |
| Scottish Health Technologies Group (SHTG) evidence notes. Available at http://www.nhshealthquality.org/nhsqis/6588.html (accessed May 28, 2010). |
| SUPPORT: Supporting Policy relevant Reviews and Trials. Available at http://www.support-collaboration.org/index.htm (accessed May 27, 2010). |
| Supporting Translation of Evidence into Policy and Practice (STEPP). Available at http://www.mrc.ac.za/cochrane/stepp.htm (accessed May 28, 2010). |
| WHO Reproductive Health Library. Available at http://apps.who.int/rhl/en/(accessed May 28, 2010). |
Appendix 2 Selective List of Excluded Resources
| Organization | Name of Resource | Reason for Exclusion |
|---|---|---|
| AHRQ | U.S. Preventive Services Task Force | Guidelines |
| HSAC: Health Services Assessment Collaboration | Summary report | One-off project |
| VATAP–VA Technology Assessment Program | Short reports | Not updated (last was 2004) |
| Centers for Disease Control and Prevention | Prevention Research Synthesis | No translation or repackaging |
| Cochrane Collaboration | Evidence Aid | Insufficient translation or repackaging |
| Cochrane HIV/AIDS group | Rapid advice | No translation or repackaging |
| Robert Wood Johnson Foundation | RWJF Synthesis Project | Not primarily based on systematic reviews |
| McMaster Health Forum | Health Systems Evidence | Insufficient translation or repackaging |
| Social Care Institute for Excellence (SCIE) | Research briefings | Not primarily based on systematic reviews; focus on social care rather than health care |
References
- Booth A, Clarke M, Ghersi D, Moher D, Petticrew M, Stewart L. An International Registry of Systematic-Review Protocols. The Lancet. 2010 doi: 10.1016/S0140-6736(10)60903-8. July 12 [epub ahead of print]. DOI: 10.1016/S0140-6736(10)60903-8. [DOI] [PubMed] [Google Scholar]
- Centre for Reviews and Dissemination. Systematic Reviews: CRD's Guidance for Undertaking Reviews in Health Care. York: University of York; 2009. [Google Scholar]
- Chambers D, Grant R, Warren E, Pearson S-A, Wilson P. Use of Systematic Review Evidence to Inform Local Decision-Making in the National Health Service: A Case Study of Eating Disorders. Abstracts of the Joint Cochrane and Campbell Colloquium. October 18–22, Keystone, Colorado. Cochrane Database of Systematic Reviews. 2010 Supplement 2010: Art No.: CD000002. DOI: 10.1002/14651858.CD000002. [Google Scholar]
- Ciliska D, Hayward S, Dobbins M, Brunton G, Underwood J. Transferring Public-Health Nursing Research to Health-System Planning: Assessing the Relevance and Accessibility of Systematic Reviews. Canadian Journal of Nursing Research. 1999;31:23–36. [PubMed] [Google Scholar]
- Davies H, Nutley S, Smith P. What Works? Evidence-Based Policy and Practice in Public Services. Bristol: Policy Press; 2000. [Google Scholar]
- Dilkes H, Hill S, Ryan R. Abstracts of the 16th Cochrane Colloquium. Freiburg: 2008. Evidence for Improving Communication and Participation: Evaluation of Evidence Bulletin Knowledge Transfer [P02-13] October 3–7. [Google Scholar]
- EVIPNet (Evidence-Informed Policy Networks) EVIPNet Policy Briefs on Malaria Treatment. 2010. Available at http://www.who.int/rpc/evipnet/policybriefs/en/index.html (accessed November 8, 2010)
- Gotzsche PC, Nielsen M. Screening for Breast Cancer with Mammography. Cochrane Database Systematic Review. 2009 doi: 10.1002/14651858.CD001877.pub3. CD001877. [DOI] [PubMed] [Google Scholar]
- Hanbury A, Thompson C, Wilson PM, Farley K, Chambers D, Warren E, Bibby J, Mannion R, Watt IS, Gilbody S. Translating Research into Practice in Leeds and Bradford (TRiPLaB): A Protocol for a Programme of Research. Implementation Science. 2010;5:37. doi: 10.1186/1748-5908-5-37. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Handoll H, Madhok R. Utility of the Cochrane Database of Systematic Reviews for Evidence-Based Health Policy and Practice: A Case Study. Journal of Clinical Excellence. 2001;3:59–68. [Google Scholar]
- Horton R. Health Research in the UK: The Price of Success. The Lancet. 2006;368:93–97. doi: 10.1016/S0140-6736(06)68974-5. [DOI] [PubMed] [Google Scholar]
- Innvaer S, Vist G, Trommald M, Oxman A. Health Policy-Makers’ Perceptions of Their Use of Evidence: A Systematic Review. Journal of Health Services Research & Policy. 2002;7:239–44. doi: 10.1258/135581902320432778. [DOI] [PubMed] [Google Scholar]
- Klein R. Evidence and Policy: Interpreting the Delphic Oracle. Journal of the Royal Society of Medicine. 2003;96:429–31. doi: 10.1258/jrsm.96.9.429. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Lavis JN. How Can We Support the Use of Systematic Reviews in Policymaking? PLoS Medicine. 2009;6:e1000141. doi: 10.1371/journal.pmed.1000141. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Lavis JN, Davies H, Oxman A, Denis J-L, Golden-Biddle K, Ferlie E. Towards Systematic Reviews That Inform Health Care Management and Policy-Making. Journal of Health Services Research & Policy. 2005;10(suppl. 1):35–48. doi: 10.1258/1355819054308549. [DOI] [PubMed] [Google Scholar]
- Lavis JN, Permanand G, Oxman AD, Lewin S, Fretheim A. SUPPORT Tools for Evidence-Informed Health Policymaking (STP) 13: Preparing and Using Policy Briefs to Support Evidence-Informed Policymaking. Health Research Policy and Systems. 2009;7(suppl. 1):S13. doi: 10.1186/1478-4505-7-S1-S13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Mason J, Freemantle N, Nazareth I, Eccles M, Haines A, Drummond M. When Is It Cost-Effective to Change the Behavior of Health Professionals? JAMA. 2001;286:2988–92. doi: 10.1001/jama.286.23.2988. [DOI] [PubMed] [Google Scholar]
- Nelson HD, Tyne K, Naik A, Bougatsos C, Chan BK, Humphrey L. Screening for Breast Cancer: An Update for the U.S. Preventive Services Task Force. Annals of Internal Medicine. 2009;151:727–37. W237–42. doi: 10.1059/0003-4819-151-10-200911170-00009. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Packer C, Hyde C. Does Providing Timely Access to and Advice on Existing Reviews of Research Influence Health Authority Purchasing? Public Health Medicine. 2000;2:20–24. [Google Scholar]
- Petticrew M, Whitehead M, Macintyre SJ, Graham H, Egan M. Evidence for Public Health Policy on Inequalities: 1: The Reality According to Policymakers. Journal of Epidemiology & Community Health. 2004;58:811–16. doi: 10.1136/jech.2003.015289. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Rashiq S, Barton P, Harstall C, Schopflocher D, Taenzer P. The Alberta Ambassador Program: Delivering Health Technology Assessment Results to Rural Practitioners. BMC Medical Education. 2006;6:21. doi: 10.1186/1472-6920-6-21. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Rithalia A, McDaid C, Suekarran S, Norman G, Myers L, Sowden A. A Systematic Review of Presumed Consent Systems for Deceased Organ Donation. Health Technology Assessment. 2009;13:1–95. doi: 10.3310/hta13260. iii, ix–xi. [DOI] [PubMed] [Google Scholar]
- Rosenbaum S, Glenton C, Oxman AD. Abstracts of the 16th Cochrane Colloquium. Freiburg: 2008. Summaries of Evidence for Health Policymakers in Low and Middle Income Countries (LMIC) [O07–25] October 3–7. [Google Scholar]
- Rosenbaum S, Glenton C, Wiysonge C, Abalos E, Mignini L, Young T, Althabe F, et al. Evidence Summaries Tailored for Health Policymakers in Low- and Middle-Income Countries. WHO Bulletin. 2011;89(1):54–61. doi: 10.2471/BLT.10.075481. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Sheldon TA. Making Evidence Synthesis More Useful for Management and Policy-Making. Journal of Health Services Research & Policy. 2005;10(suppl. 1):S1–5. doi: 10.1258/1355819054308521. [DOI] [PubMed] [Google Scholar]
- Thornton-Jones H, Hampshaw S, Soltani H. Evidence, Policy Making and the Art of the Possible: A Methodology Applied to Screening. Journal of Clinical Excellence. 2002;4:9–16. [Google Scholar]
- Turner T, Clarke M, Green S, Tharyan P, Lumbiganon P, Garner P, Smith H, Clark K. Abstracts of the 16th Cochrane Colloquium. Freiburg: 2008. Evaluating Evidence Aid [O08–32] October 3–7. [Google Scholar]
- Young T, Olivier L, Arens F, Oliver J, Cloete K, Volmink J. Abstracts of the 13th Cochrane Colloquium. Melbourne: 2005. STEPP: Supporting Translation of Evidence into Policy and Practice [O46] October 22–26. [Google Scholar]