Doing More with More: How “Early” Evidence Can Inform Public Policies

Calls for government-funded activities to be “evidence based” are ubiquitous. “Gold standard” studies, including randomized controlled trials and systematic reviews (Isett, Head, and VanLandingham 2016), have expanded the availability of evidence-based programs and practices (VanLandingham and Silloway 2016). However, because of their complexity, large-scale policies (comprising services, laws, rules, and regulations implemented at the population level) are more difficult to study experimentally, resulting in evidence gaps.

Public policies should be informed by the best information available. This article focuses on the utility of early evidence assessment and provides an example of one approach called the Quality and Impact of Component (QuIC) Evidence Assessment. This approach provides a systematic and timely method for policy analysis that can be applied to many types of emerging and complex public policies.

Gaps in the Evidence to Inform Policy

Public policies rarely have the same kind of rigorous, “gold standard” evidence as the pilot interventions they intend to scale up. Policies are often complex, containing multiple components, and implemented on multiple levels over time, so extracting component-specific effects proves difficult. Other challenges to conducting rigorous policy impact studies include sparse data (Isett, Head, and VanLandingham 2016), poorly matched communities for controlled comparisons, and complex, macro-level forces (Wilt et al. 2008).

The complexities that accompany major public policies inevitably create gaps in empirical evidence. Evidence gaps have been used to justify inaction, but inaction is unacceptable for practitioners given the urgency and/or magnitude of many public problems. A systematic approach to assessing all available evidence can inform policy; this article describes how applying such an approach can help fill gaps.

Addressing Gaps with Early Evidence Assessment

Public decision makers need to know which policies are feasible and most likely to achieve the desired impact. When the evidence base for a policy does not include impact studies, an approach for assessing all relevant, available evidence is needed to inform policy decisions in the short term and research studies in the longer term. Figure 1 provides a process for policy research beginning with early evidence assessment.

Figure 1. Division for Heart Disease and Stroke Prevention’s Policy Research Continuum.

Early evidence assessment is completed simultaneously with policy surveillance, which is the systematic collection, analysis, and interpretation of policies. When policy implementation, rating, and impact studies become feasible, they may validate the results of early evidence assessment. Once enough high-quality policy impact studies are complete, a systematic review can provide an authoritative recommendation for policy dissemination and implementation.

Integrating many types of evidence from diverse sources during early evidence assessment provides the breadth of information that is needed for policy decisions (Bowen and Zwi 2005). For example, practice-based knowledge, including professional judgment and an understanding of context, can be gleaned from policy statements, guidelines, and briefs by professional and nonprofit organizations; formative and process evaluation studies; and journal articles. This knowledge can help build theory. Additionally, studies examining outcomes of discrete interventions, alone or in combination with other interventions, can inform whether policies applying similar approaches could be scaled to the population level.

Early Evidence Assessment with QuIC

A challenge in expanding the evidence base for policy decisions is that existing methods for assessing diverse evidence, such as integrative literature review, lack rigor (Whittemore and Knafl 2005). QuIC applies a systematic assessment to the early evidence aligned with individual components of a multicomponent public policy. For example, state laws to promote health in the workplace have components addressing different diseases and risk factors, incentives, awareness, and evaluation (VanderVeur, Gilchrist, and Matson-Koffman 2017). Because there are no studies of the impact of a state workplace health promotion law, QuIC was useful in assessing and prioritizing its many components.

QuIC works by broadening the scope of usable evidence and completing high-level assessments of evidence bases relevant to policy components. Published and gray literature about a policy topic is collected to identify policy components, and then each item of evidence relevant to a component is coded for (1) potential impact and (2) quality. Coding results are aggregated in order to score each evidence base with the QuIC rubric.

The QuIC rubric includes four criteria to assess evidence for potential impact: “effectiveness” assesses improvements to primary outcomes, “equity and reach” assesses effectiveness for target populations, “efficiency” assesses economic and quality-related outcomes, and “transferability” assesses effectiveness across diverse settings. To date, effectiveness has focused on health-related outcomes, as QuIC has only been applied to public health policies. Four additional criteria assess the quality of an evidence base: “evidence types” assesses the rigor of study designs in the evidence base, “evidence sources” assesses the credibility of evidence publishers, and “evidence from research” and “evidence from translation and practice” assess the amounts of evidence from these approaches. Each criterion is scored “very strong,” “strong,” “moderate,” or “weak.” Criteria scores are summed to generate a potential impact score and a quality score for the evidence base aligning with a component.

These two scores are considered together to rank an evidence base as “best,” “promising (quality),” “promising (impact),” or “emerging.” For example, best evidence bases have very strong or strong potential impact and very strong or strong quality, while promising (impact) evidence bases have very strong or strong potential impact but moderate or weak quality. A best evidence base would likely include several published experimental studies from research, quasi-experimental studies from practice, and supportive policy recommendations, resulting in the higher quality score. These studies would have found mostly positive health and economic outcomes for target populations across diverse settings, resulting in the stronger potential impact score.

This evidence can educate public decision makers as they consider policy options that align with best evidence. Reports on QuIC assessments present components in order of evidence level: those aligning with best evidence are presented first, followed by those aligning with promising and emerging evidence. Reports also provide a summary of the evidence aligning with each component, describing the interventions, populations, settings, and outcomes studied.

Impact of QuIC

The QuIC framework was first published in 2015 (Barbero et al. 2015) and applied to early evidence relevant to 14 components of state law pertaining to community health workers; results were used to assess the extent to which existing laws aligned with evidence (Barbero et al. 2016). In 2016, a revised version of the QuIC tool was applied to early evidence relevant to 21 components of state law about workplace health promotion, 7 about public access defibrillation, and 2 about defining and establishing standards for the community health worker workforce (see table 1).

Table 1.

Snapshot of Three QuIC Evidence Assessments Completed in 2016

Policy:	State law to support workplace health promotiona	State law to support public access defibrillationb	State law to define and establish standards for the community health worker workforcec
QuIC assessment results	13 best	3 best	2 best
	4 promising	4 promising	0 promising
	4 emerging	0 emerging	0 emerging
Example of a policy component aligning with best evidence	Workplace interventions targeting obesity	Training anticipated rescuers	Establishing a community health worker certification process
Example of a policy component aligning with promising evidence	Tax credits for workplace health promotion programs	Ongoing quality improvement in programs	None
Example of a policy component aligning with emerging evidence	Raising awareness for workplace health promotion	None	None

^aWorkplace health promotion refers to a coordinated and comprehensive set of strategies that include programs, policies, benefits, environmental supports, and links to the surrounding community designed to meet the health and safety needs of all employees.

^bPublic access defibrillation engages the public to respond to cardiac arrest with automated external defibrillators.

^cCommunity health workers are frontline public health workers who bridge communities and health care systems.

QuIC’s strength lies in its ability to translate a large and diverse evidence base addressing multiple policy options into reports that can be easily understood by decision makers. Many scholars and practitioners have requested QuIC presentations and some have used QuIC to inform strategic planning. In 2015, the American Heart Association applied QuIC to evidence relevant to local policies ensuring access to clean drinking water (Labarthe et al. 2016).

Because evidence drawn from practice-based knowledge is included, the results of QuIC assessments have generally aligned with recommendations from practice. The Community Health Workers Section of the American Public Health Association found that 2014 QuIC results for state community health worker laws aligned with their policy position, so they presented results directly to state policy makers attending a 2014 Milbank Memorial meeting. The QuIC results for state community health worker laws were also summarized by the California Health Workforce Alliance in its 2015 publication “Community Health Workers in California: Sharpening Our Focus on Strategies to Expand Engagement” and the U.S. Department of Health and Human Services in its 2016 issue brief “Community Health Workers: Roles and Opportunities in Health Care Delivery System Reform.” QuIC assessment results from 2016 (table 1) are currently being disseminated.

Because an early evidence base does not include policy impact studies, one important limitation of QuIC is that it only assesses the potential for policy impact, based on evidence for the types of interventions addressed in a multicomponent policy. Broadly defined policy components limit the extent to which QuIC results can be generalized and attributed to specific policies. Furthermore, a QuIC assessment does not describe how implementation strategies or contextual factors affect policy outcomes.

Nevertheless, early evidence assessments like QuIC could provide a basis for future policy research. In 2016, the Centers for Disease Control and Prevention’s QuIC for state public access defibrillation laws helped design a policy rating and impact study, and its QuIC for state community health worker laws helped identify research questions that informed a policy implementation case study. While these studies are underway, QuIC results are being disseminated, so policy makers have the best available information for decision making.

QuICs Relevance to Public Administrators

Public administrators could use QuIC to accelerate the translation of knowledge into informed policy decisions. The QuIC Handbook (available by contacting the authors) provides a systematic procedure, typically requiring less than four months, for identifying policy components, collecting and coding evidence, assessing reliability, scoring and summarizing evidence, and translating results into reports. While expertise and access to evidence are needed to guide a QuIC assessment, evidence coding can be completed by junior research staff.

Overall, QuIC sets a higher standard for assessing early evidence than its main alternative of integrative literature review, which has historically lacked rigor in analysis, synthesis, and conclusion drawing (Whittemore and Knafl 2005). A coding scheme and rubric make QuIC systematic and transparent. Criteria germane to policy analysis make QuIC versatile and potentially applicable to many different types of policies.

Conclusion

Policy can be an important tool for scaling up interventions addressing public problems. Early evidence assessment tools like QuIC can leverage best available evidence to support timely, evidence-informed policy decisions.

The field of public administration could work toward building consensus on standards for early evidence. Identifying a hierarchy of policy decisions and acceptable levels of uncertainty may help. For example, while policies directly impacting health, such as vaccination laws, require a level of certainty only achieved by rigorous studies, some policies to improve health services and systems could be sufficiently informed by practice-based evidence including expert opinion.

Engagement of practitioners in validating and identifying new sources of early evidence will be crucial. Scholars should integrate early evidence into frameworks and develop more practical tools for appraisal and application. Increased collaboration between scholars and practitioners should lead to more evidence, more evidence-informed policy, and better health and well-being for the population.