Improving Utility of Evidence Synthesis for Healthy Public Policy: the Three Rs (Relevance, Rigor, and Readability [and Resources])

Abstract

Systematic reviews have the potential to promote knowledge exchange between researchers and decision-makers. Review planning requires engagement with evidence users to ensure preparation of relevant reviews, and well-conducted reviews should provide accessible and reliable synthesis to support decision-making. Yet, systematic reviews are not routinely referred to by decision-makers, and innovative approaches to improve the utility of reviews is needed. Evidence synthesis for healthy public policy is typically complex and methodologically challenging. Although not lessening the value of reviews, these challenges can be overwhelming and threaten their utility. Using the interrelated principles of relevance, rigor, and readability, and in light of available resources, this article considers how utility of evidence synthesis for healthy public policy might be improved.

Systematic reviews can provide invaluable support to decision-making by providing readily prepared syntheses of quality-appraised evidence. Many research funders regard evidence synthesis as a mechanism for knowledge exchange.¹ However, there are conflicting reports from decision-makers about the use and value of evidence syntheses.^2,3 To improve the utility of systematic reviews, Lavis et al.³ have called for more innovation in the preparation of systematic reviews. Specifically, they highlight the need for improved relevance and accessibility of reviews, while maintaining rigor that is the underpinning value of a systematic approach.

Evidence synthesis for healthy public policy is typically complex and methodologically difficult, presenting challenges to the preparation of reviews that are relevant, rigorous, and readable (or accessible). Although policy relevant, the questions are often broad and imply inclusion of diverse data types and sources that can be difficult to synthesize.^4,5 Consequently, rigorous reviews of broad complex policy relevant questions are costly and can take a long time, sometimes years, to complete.^6–8 Complexity does not diminish the importance of the review questions, but decision-makers need evidence that is timely as well as relevant.^9,10 Regardless of how rigorous and comprehensive a review is, syntheses that are not timely may no longer be relevant once completed.¹¹ Moreover, long delays in dissemination of syntheses may perpetuate avoidable harm.¹² The nature of available evidence may also present challenges to the rigor and readability of complex reviews. Statistical synthesis is rarely appropriate, the resulting narrative syntheses are often lengthy, and the link between the synthesis and the conclusions can be opaque. In addition, the dearth of well-conducted studies assessing the health impacts of public policies, such as housing investment, welfare changes, etc., will often restrict the review findings to establishing uncertainty and may question the utility of such reviews.¹³

This article describes the specific challenges of complex reviews, and what they can contribute to policymaking. The principles of relevance, rigor, and readability are presented, along with examples of how these principles can be incorporated into complex reviews for healthy public policy to help improve their utility. Finally, the value for money of long complex reviews is considered. Justification for, as well as ways and implications of, limiting a review are described.

WHAT MAKES HEALTHY PUBLIC POLICY REVIEWS COMPLEX

Complexity in a broad review may arise for a number of reasons (see the box at the top of the next page). The topic or intervention of interest to healthy public policy are often complex, whether generating new evidence in a primary study or synthesizing existing evidence. Interventions of interest may comprise multiple components that vary by setting or individual, and may involve multiple outcomes at different levels, presenting potential for dynamic and important but immeasurable interactions.^14–16

Characteristics of Complex Reviews

• Address complex topics with multiple potential, and poorly understood, interactions between intervention, mediators, and outcomes.

• Broad inclusion criteria introduce multiple sources of heterogeneity, for example, diverse contexts, populations, interventions, outcomes, study methods, and data types (qualitative, quantitative, theoretical).

• Address multiple questions. For example: is an intervention effective? Are there differential effects across different populations or contexts? What are the key mediating factors? Is the intervention cost effective? What is the impact on proxy measures of longer term outcomes?

Broad review questions, typical within healthy public policy reviews, imply a broad scope as defined by the review inclusion criteria or population, intervention, comparison, outcomes, context, and study design (PICOCS). Reviews with broad inclusion criteria produce highly heterogeneous data that are an important and methodologically challenging source of complexity in reviews of healthy public policy.⁴ The breadth of the review may be limited by prioritizing what will be useful to evidence users (see the section on Relevance) and what is manageable and affordable (see the section on Resources), but may also be shaped by the nature and volume of available evidence. Over the past decade, the dearth of well-conducted cohort studies assessing the health impacts of social interventions has been well documented. The lack of high-quality evidence may require further broadening of inclusion criteria to allow the review to establish what is known, the nature of evidence available on a topic, and immediate evidence priorities. For example, it may be useful to consider data from uncontrolled studies or interrupted time series, or to include interventions that have some key similarity to the intervention of initial interest to the review.¹⁷ Reviews may not be restricted to empirical data. When the links between the dependent and independent variables are poorly understood, a systematic review of existing theory may help prioritize what empirical evidence should be subsequently reviewed.¹⁸

An additional source of complexity in a review may be the number of questions addressed. Questions that go beyond “what works?” enhance the usefulness and interpretation of effectiveness reviews and are increasingly recommended by both the Cochrane and Campbell Collaboration.^19,20 Additional questions might investigate differential effects across population subgroups, contexts, or cost effectiveness.²¹ Reviews of healthy public policy interventions may also benefit from including synthesis of shorter term impacts on socioeconomic determinants of health and health outcomes.¹⁶ Public policy interventions are not regularly evaluated for health impacts; in addition, health impacts may not be realistically expected within the timeframe of an evaluation. Reviews that incorporate synthesis of proximal impacts on determinants of health may provide the best available evidence to indicate the potential for longer term health impacts, and may provide valuable data on poorly understood pathways to health impacts.^6,22

TOO COMPLEX TO BE USEFUL OR SYNTHESIZABLE

The potential for intangible interactions between intervention and outcome, together with the dearth of evidence able to attribute impacts to interventions, is a common feature in many healthy public policy reviews. This can result in extreme levels of heterogeneity, and presents challenges to preparation of meaningful synthesis. The heterogeneity of study characteristics not only contraindicate statistical synthesis, but may also result in a narrative synthesis that is limited to summarizing the findings of individual studies. It might be questionable whether these complex data can or should be synthesized. The complexity in the resulting “syntheses” may be considered to be intolerable, serving only to confirm uncertainty and what is not known.^11,13,23,24 However, the extent of complexity is a matter of perspective, and establishment of unknowns is important. There are few research or review questions that cannot be broken into many additional questions. Even within the relatively tidy realms of clinical randomized controlled trials (RCTs), there may be few interventions that are truly simple.²⁵ The heterogeneity of interventions as an obstacle to useful synthesis may also be a matter of perspective. Heterogeneity within a review may even be desirable where a configurative, rather than aggregative, approach to synthesis is necessary to investigate commonality across apparently diverse settings or interventions.²⁶ Aspects of construct validity and external validity,^27,28 or function²⁹ may be common across nonidentical interventions. Synthesis at the level of commonality may be both conceptually appropriate and valuable, especially where there is little or no evidence on the specific intervention of interest.³⁰ It has been argued that the development of theories or hypotheses about the mechanisms for the effects of complex interventions (social intervention theory), which draws on common features of multiple studies, may be a more valuable contribution to policy development than numerical approximations of effects from single studies.^30,31

WHAT COMPLEX REVIEWS CAN CONTRIBUTE TO HEALTHY PUBLIC POLICY

Even the best available evidence on the health impacts of public policy is likely to be limited with respect to attributing impacts to an intervention, and the conclusions will often highlight uncertainty.^11,13 This can be frustrating, but citing Popper, Pawson et al.²⁴ draw attention to the innate uncertainties within science, advocating increased acceptance and acknowledgment of the limitations of evidence. Well-conducted reviews play an essential part in establishing what is known, differentiating levels of uncertainty, and establishing research and knowledge priorities.

A complex review, although likely to be limited with respect to effectiveness, should be able to contribute more than establishing uncertainty (see the box at the bottom of this page).¹³ Even when standardized effect sizes are not available or synthesizable, a review can map out the nature and quality of the best evidence available to address a question, along with details (e.g., relevance to the local context). Further mapping can establish the nature and direction of impacts, and mapping of intermediate impacts can help explain the absence of hypothesized health impacts. The nature of available evidence will shape what questions a synthesis can usefully address and the type of synthesis possible,⁴ and review methods can be applied across the spectrum of evidence sources to include empirical, theoretical, and methodological questions.^32,33 Some examples of these are provided as data available as a supplement to this article at http://www.ajph.org.

What Complex Reviews Can Contribute to Healthy Public Policy

• Identify appropriate questions for future systematic review and primary studies, and prioritize future research needs

• Map nature of best available evidence by location, intervention, study methods, and study quality

• Establish existence, nature, and direction of reported impact, including intermediate and adverse impacts

• Map existing theories to help develop related hypotheses for future review or study

• Develop typology of interventions

• Develop typology of methods

• Develop empirically supported theory

Note. Example reviews are available as a supplement to the online version of this article at http://www.ajph.org.

CONDUCTING A SYSTEMATIC REVIEW: RELEVANCE, RIGOR, READABILITY

It is easy for broad, complex reviews to become overwhelming for the reviewer and the reader. A clearly defined and focused question is an essential part of any review. However, even apparently clear review questions can spawn multiple unseen difficulties. Many reviews have been abandoned because the review team, or their resources, have been exhausted by the volume of literature, the multiple emerging questions of interest, and the difficulties in analyzing the complex data in a way that is useful and maintains the integrity of the systematic review approach. In addition to rigor, systematic reviews must be relevant and readable if they are to be useful. These 3 principles (the 3 Rs) are interrelated and should be considered together when planning a review, and will likely require to be discussed in light of the fourth R (resources). The remainder of this article outlines how these 4 Rs may be applied to complex reviews for healthy public policy as a way to enhance utility of syntheses while maintaining the value of a transparent, systematic approach.

Relevance

Devoting time to developing a clearly defined and focused question is central to planning reviews that are relevant, as well as conceptually appropriate and manageable. This first requires a scoping process to identify the heterogeneity of the PICOC implied in the initial question and the consideration of factors that may mediate the impact of the intervention or independent variable. This process can help establish the potential scale of the initial review question and uncover multiple additional questions of interest that are not apparent at first. Logic models are recommended as a way of mapping the outcome of these discussions. The use of a visual map of the potential scope of the initial review question can help to refine the review question by prioritizing what is useful and also manageable within the available time (see the section on Resources).¹⁶ The timeliness of a review is also pertinent to relevance; relevance may diminish with increased time lags between initiation of a question and a completed review.

Discussion with topic experts and scoping searches to assess the nature and quantity of evidence that is likely to be available to answer a review question can contribute to development of the review scope and promote relevance. For example, advance knowledge that there are few RCTs of the intervention of interest may indicate the need for broader inclusion criteria to make use of the best available evidence and avoid the review process leading to an empty review. Similarly, knowledge of important differences in interventions and context across countries or over time may provide a justification for limiting the review scope by prioritizing what is most relevant.

Relevant reviews should reflect the needs of the end user of the review.³⁴ Relevance of reviews can be enhanced by engaging with users when agreeing with the review question and scope, both at the start and throughout the review process. Evidence users can also advise on the interpretation and generalizability of the review findings, and appropriate dissemination to other evidence users. Involvement of evidence users in reviews can take many forms.³⁵ Both the Cochrane and the Campbell Collaborations advocate user involvement in reviews as a way to promote the quality and relevance of reviews.^36,37

Rigor

The key components of a systematic review are searching and identification of evidence, appraisal of the quality of evidence, and finally synthesis of evidence. Regardless of the methods used for each step, transparency is central to the systematic approach that aims to minimize bias in the selection and interpretation of evidence.³³ Just as in primary research, the aim (in theory) is that the reader could replicate the methods and draw the same conclusions. Methods for searching, appraisal, and meta-analysis of effectiveness evidence from RCTs are well developed to maximize transparency. However, evidence on the health impacts of public policy is often difficult to identify, rarely comes from RCTs, and is rarely amenable to meta-analysis.⁴ Moreover, much of the published guidance on the conduct of systematic reviews focuses on questions of effectiveness.^36,38

Identifying evidence for healthy public policy requires professional input from an information scientist to develop and conduct sophisticated, multipronged search strategies covering the scope of the PICOCS, as well as covering a wide range of sources beyond peer-reviewed journal publications and electronic bibliographic databases.^39–41 Clear records of the search strategy and identified hits should accompany the published review to ensure transparency of the search scope and study selection.

Inclusion of nonrandomized studies (NRS) in systematic reviews, although common, continues to be the source of much debate.^13,42 There are undoubted difficulties in interpreting evidence from NRS. Nevertheless, in the absence of RCTs, it is helpful to establish what is known drawing on the best available evidence, even if this is limited with respect to knowledge of effectiveness. When data are synthesized, assessment of validity is essential regardless of study design, and there are many tools available to assess data from nonexperimental quantitative and qualitative studies.^42–45

The heterogeneity of data and study design alone mean that evidence syntheses for healthy public policy will often require the data to be synthesized narratively. There is very little guidance on the process of, or agreed standards for narrative synthesis; consequently, the potential for bias in the process of interpreting the synthesized data are difficult to quantify. In any systematic review, it is important that the final synthesis reflects the quality of the data, that no study of similar quality to another receives undue emphasis, and that the links between the data and the conclusions of the synthesis are made clear. Popay et al.⁴⁶ have identified common elements of narrative synthesis and recommend the following steps: development of a theory for the review to test; preliminary synthesis; and then an iterative process exploring relationships within and between studies and testing the robustness of the final synthesis. Although this is useful, the issue of transparency of the interpretation persists. Careful tabulation of the included evidence can promote transparency. When standardized effect sizes are available, forest plots provide a useful visual summary of included data, even if meta-analysis is not appropriate. When standardized effect sizes are not available to represent all the included quantitative data, an effect direction plot may be useful (Figure 1).⁴⁷ Logic models may also be a useful tool to map included evidence. Figure 2 illustrates how data from qualitative and quantitative studies of warmth improvements were mapped in a recent systematic review of housing improvement. The final map represented agreement of preliminary versions prepared by 2 independent reviewers and included an indication of the data source and quality to maintain transparency and the integrity of the systematic approach.

FIGURE 1—

Example effect direction plot of health impacts for 3 categories of housing improvement.

Source. Reprinted with permission from Thomson and Thomas.⁴⁷

FIGURE 2—

Example logic model mapping qualitative and quantitative data included in a systematic review of the health impacts of warmth and energy efficiency housing improvements.

Source. Reprinted with permission from Thomson et al.⁶

Readability

Readability of a review is important for both relevance and transparency (rigor). For a review to be relevant to evidence users, it must be accessible and the rationale for the synthesis’ conclusions must be clearly communicated.^3,48 Preparation of plain language summaries that avoid esoteric methodological terms are useful but require careful writing to avoid misinterpretation by readers. When a review is unable to reach a clear conclusion, it is critical that the reason for and implication of the uncertainty is clearly communicated. In the messy world of social interventions, the limitations of science and evidence need to be better understood by both the reviewers and evidence users hoping for certainty.²⁴

Transparency of the review process and the links between the data and the conclusions of the synthesis are essential. However, presenting lengthy text and data tables of complex data are unlikely to be accessible to anyone other than the most committed reader. There are too few examples of how data can be summarized, but the examples provided here (see the section on Rigor; Figures 1 and 2) demonstrate that 1-page summaries of complex data can be prepared. Further innovations to develop visual displays of complex data that retain the integrity of the systematic review process are required to promote both the rigor and utility of evidence synthesis of broad and complex topics, such as healthy public policy.

Resources

Available resources will shape the scope of a review, and prior consideration of appropriate use of resources is necessary to ensure a review is timely and focuses on what is relevant. Rapid reviews are one way of ensuring a timely product. Although, as Abrami et al.⁴⁹ suggest, the term “brief” may be more appropriate to emphasize limited scope as well as time, as opposed to a rapid review where intensive use of multiple reviewers may not reduce expense.

When the volume of studies is overwhelming, limiting the inclusion criteria to the better quality studies provides a clear justification for limiting a review. Within healthy public policy, the volume of high quality studies will rarely be overwhelming. Rather, what may be overwhelming is the complexity, such as inclusion of multiple outcomes and multiple questions. Ensuring a review is manageable within available time and resources needs discussion between reviewers and end-users about the review’s scope to prioritize what is feasible and also relevant. This may require prioritizing breadth over depth, or vice versa.³² Reviews that prioritize depth over breadth most often restrict the inclusion criteria (PICOCS) in some way. The appropriateness and usefulness of narrow inclusion criteria will depend on relevance, as well as volume and quality, of available evidence.

Many healthy public policy issues point to the need for broad reviews, but limits for broad reviews can be tricky to negotiate with evidence users. Constraining the review depth may require limiting the volume of included studies or data analysis. Sensitive and sophisticated search strategies are still recommended, but limiting the number of search sources may reduce costs without significant impact on search results.⁴¹ Sampling from comprehensive searches can provide a useful way to limit the volume of included studies while still ensuring a representative or purposive sample.⁵⁰ Prioritization of exactly what will be useful with respect to data extraction and critical appraisal may also reduce resource demands. Detailed evidence appraisal is an essential component of effectiveness reviews, but this may be of less value when the aim is to establish the nature and range of available evidence, or where the key interest is to establish the way in which an intervention has been evaluated (e.g., to identify study designs used and outcomes assessed).^51,52 Additional time calculating standardized effect sizes and perform meta-analysis may be of questionable value when amenable data are not available across all included studies. In such a review, it may be of more use to tabulate the nature and direction of effects.

Pragmatic decisions to limit a review will lead to a tradeoff between preparation of a review that is timely and useful, and that is in-depth and comprehensive. Central to the rigor of the systematic review approach is transparency. When the volume of relevant evidence threatens preparation of a review that is both timely and useful to evidence users, it may be that transparency trumps comprehensiveness. As in any study, the limits and accompanying justification need to be clearly reported and also reflected on in terms of potential bias and shortcomings in the review conclusions, as well as the potential value of a more comprehensive review.

CONCLUSIONS

The breadth and complexity of synthesis for healthy public policy can be overwhelming, even for experienced reviewers. The competing need for rigor and timeliness is challenging and underlines the need for careful thought about how syntheses can best meet the needs of evidence users and also maintain the benefits of a rigorous, systematic approach. Restricting the coverage of a review will introduce limitations that need to be reflected on in the review’s conclusions, but may help produce timely syntheses. A review that is less comprehensive but is relevant and rigorous may be of greater use than a review that is produced years after it was needed, and may provide robust justification for resources to conduct a more comprehensive review. When sufficiently rigorous, restricted reviews may still merit dissemination in peer-reviewed journals,^53,54 and may provide examples of innovations to improve utility of evidence synthesis for decision-makers. The interrelated principles of the 3Rs can help maximize the utility of evidence syntheses while retaining the integrity of a systematic approach. The fourth R may be a further requirement, not simply because of funding constraints, but also to promote timeliness (relevance).