An Evidence Integration Triangle for Aligning Science with Policy and Practice

Abstract

Over-reliance on decontextualized, standardized implementation of efficacy evidence has contributed to slow integration of evidence-based interventions into health policy and practice. This article describes an “evidence integration triangle” (EIT) to guide translation, implementation, prevention efforts, comparative effectiveness research, funding, and policymaking. The EIT emphasizes interactions among three related components needed for effective evidence implementation: (1) practical evidence-based interventions, (2) pragmatic, longitudinal measures of progress, and (3) participatory implementation processes. At the center of the EIT is active engagement of key stakeholders and scientific evidence, and attention to the context in which a program is implemented. The EIT model is a straightforward framework to guide practice, research and policy toward greater effectiveness, and is designed to be applicable across multiple levels—from individual-focused and patient-provider interventions, to health systems and policy-level change initiatives.

Introduction

Translation of research evidence to widespread application in practice has variously been conceptualized as a linear process—a “pipeline” or “roadmap” that unfortunately is slow, uncertain, and incomplete.^1,2 The dominant conceptualizations of translation of science into practice begin with research products developed by investigators, and then go through various sequential steps to the eventual routine use by practitioners. This type of scientific evidence, however, developed in isolation from its projected users, often fits uncomfortably in the settings and populations where it is intended to be applied. The art of policy and practice involves reconciling the strength of published evidence with its relevance based on the experience of those who know, live and work with the problem that the evidence is designed to solve.³

The Roadmap for Medical Research by the NIH⁴ suggests a progression from T1 research (basic discovery), to T2 research (evaluation of efficacy). Recent contributions have expanded this to T3 research (evaluation of implementation in practice), and T4 research (assessing the impact on population health).^5,6 T1 and T2 research, with their emphasis on bringing basic research to clinical trials, dominate biomedical funding but are not enough. The current complex health and healthcare challenges require complex, multilevel solutions tailored to the specific settings in which they are applied.^7,8 The limited effect of research on population health argues for increasing the current low levels of investment in T3 and T4 investigation to enhance the success of prevention and implementation science. The research, policy and funding communities cannot keep relying on the same highly controlled efficacy research, pushed into the same unidirectional and leaky implementation pipeline, while expecting different outcomes.^9,10

To increase the relevance, application and impact of scientific investigation, researchers, practitioners, community members and policymakers need a straightforward and systematic way to understand the pathway from research discovery to population health outcomes.¹¹ Evidence, practice and policy must begin with the end goal in mind to foster adoption, implementation adaptation, and sustainability.^12,13 The traditional linear approach to research translation has been critiqued by many, including the authors, but few clear, feasible alternatives have been proposed.^7,10,14,15

Several research translation models have been employed productively, but they are often found to be too complex, academic or time-consuming for clinicians, community members and health systems.^16–18 This paper describes a three-pronged model called the Evidence Integration Triangle (EIT) (Figure 1) that captures essential dimensions of an effective interaction between research and its practice/policy translation. The EIT builds on, and attempts to distill, the critical elements of these important predecessor models. It is designed to be more intuitive and readily applied by stakeholders, including practitioners, policymakers, and citizens to foster high-impact knowledge implementation by research-practitioner-community partnerships.

Figure 1.

Evidence Integration Triangle (EIT) Model

CBPR, community-based participatory research

The purposes of this paper are to: (1) describe the EIT as a model to help optimize practice through research evidence and speed integration of science, policy and practice;¹⁹ (2) suggest practical actions and keys to success within and across the three domains; (3) provide examples of application of the EIT; and (4) discuss implications for researchers, practitioners, and policymakers.

The Framework

The EIT depicts in a simple framework the complex multilevel contextual factors affecting the integration of scientific knowledge into practical applications. As shown in Figure 1, bringing together evidence and relevant stakeholders is central. Interactions among the three main evidence-based components—intervention program/policy, implementation processes, and measures of progress—empower these stakeholders to use scientific evidence to maximize positive health impact and value and encourage development and sharing of new knowledge to inform future interactions.

As depicted at the bottom of Figure 1, context is pivotal to the EIT. The multilevel context— conditions surrounding health problems and intervention opportunities in a particular place with a particular population—is a key starting point. Context also changes over time, giving a temporal and recursive aspect to the EIT, with context continually informing the other key components. The multilevel aspect of EIT aligns it with the growing emphasis on ecologic models of organizational and community assessment, systems approaches,^17,20–24 program planning and evaluation,^16,17 and with reorientation of “the clinical effectiveness research paradigm” toward greater recognition of “innovation and practice-based approaches” to evidence.³ Keeping an eye on contextual factors allows evidence to be made and kept relevant. The other EIT model components are described below.

Intervention Program/Policy

Intervention programs and policies need pragmatic evidence relevant to the stakeholders who must implement them. This focus on external validity is challenging because the standards for rigor in most scientific evidence emphasize internal validity.^14,18 Published recommendations from systematic reviews rely primarily on RCTs of efficacy (www.cochrane.org).²⁵ These, however, have been slow to translate and are perceived by many as lacking relevance to their setting or population.^10,15 Consideration of external validity necessitates that research be more transparent about issues of recruitment, context, settings, capacity, and representativeness, and that the primary questions that need to be addressed for translation and implementation are of the realist variety,^26,27 which focus on questions of the form “which intervention factors are most effective—for which patient subgroups, when administered by what staff, under which conditions, for what outcomes”.

Public health and policy experts have advocated for expanded use of practice-based evidence and insisted that research designs should fit the question and context rather than vice versa.^9,28,29 Both the IOM³⁰ and Etheredge³¹ have stressed the need for “rapid learning evidence,” a medicalized portion of which is increasingly available from electronic health records. Such learning uses close to real-time data on hundreds of thousands of real-world patients experiencing interventions delivered by practicing clinicians in real-world delivery systems. Simulation modeling also has experienced substantial advances in computing power, which can be used to provide tests of concept and potential outcomes prior to investment in long, expensive trials.^17,32

The types of evidence being recommended here involve marrying rigorous design focused on internal validity and theory-driven hypotheses with an increased focus on external validity, contextual considerations, and stakeholder relevance.^9,33 Relevance is achieved by attending to the contexts in which they will be implemented.³⁴ Context includes multilevel factors including the historical, political, economic, social, environmental and cultural settings in which a program is being implemented (Figure 1). Programs need to be practical and efficient so that they are capable of having a broad reach, especially to settings and people most in need or at highest risk. Whenever feasible, the ideal interventions are ones demonstrated to be generalizable across diverse settings and under diverse conditions of implementation, with minimal adaptation.³⁵

Practical Measures for Monitoring Progress

Standardized, practical measures are needed to evaluate progress toward goals and objectives. At the national level, measuring progress toward the accomplishment of Healthy People 2020 objectives (www.healthypeople.gov) has focused efforts on programs and policies that make the greatest difference in the health of populations. Far less attention has been focused on identifying “best practical measures” that are feasible for practitioners, health systems and policymakers to assess progress on the outcomes they address. Some of the key implementation successes have come from very simple innovations such as surgical checklists and provider reminder systems, which focus attention on key implementation issues.³⁶

Implementation success needs to be monitored and frequent feedback provided so that adjustments can be made if desired outcomes are not achieved in local implementation. Choosing the best metrics involves trade-offs to find the best balance among criteria such as those outlined in Table 1. These criteria combine traditional psychometric concerns of scientific rigor with practical considerations of relevance, feasibility, and in particular, being actionable in typical settings.

Table 1.

Recommended characteristics for practical measures and assessments

Characteristic	Recommended Criteria
Reliable	Especially test–retest (less on internal consistency)
Valid	Construct validity, criterion validity, established norms
Sensitive to Change	Appropriate for longitudinal use, goal-attainment tracking, repeated administration
Feasible	Brief (generally three items or less); easy to score/interpret
Important to Clinicians	Indices for health conditions that are prevalent, costly, Challenging
Public Health Relevance	To address without measures, in primary care domain, related to Healthy People 2020 goals
Actionable or feasibility of developing recommended clinical decision support	Realistic actions, reliable referral, immediate discussion, online resources, how easy or difficult would it be to develop a clinical response “toolkit” to act on the resulting data
User-friendly	Patient interpretability; face validity; meaningful to clinicians, public health officials, community members and policymakers
Broadly applicable	Available in relevant languages; validated in various cultures and Contexts
Low-cost	Publicly available or very low cost to promote widespread use
Enhances Patient Engagement	Having this information is likely to further patient involvement in their care and decision-making

To optimize these criteria, those who monitor implementation often face the choice of using off-the-shelf measures that have been validated but not exactly right for a given application versus developing new measures specifically for a given evaluation. Between these extremes is a middle ground that includes using the most-relevant items from previously validated measures alongside new purpose-developed measures. The aims of relevance, engagement, and ongoing learning can be met by complementing quantitative measures with qualitative assessment and analysis. Such mixed methods³⁷ can be particularly helpful for assessing meaning from the perspective of participants, discovering new constructs, assessing unanticipated outcomes, and providing narrative meaning to numeric results.

Measures ideally should meet standards of reliability and validity, but also be practical, normative, sensitive to change, usable longitudinally, available in relevant languages, have face validity for stakeholders and practitioners, and cause only modest staff and patient/population burden. Such indices are critical because an intervention is seldom implemented in practice exactly as it was in research. Data-based adjustments are usually required. Relevant and timely information is necessary to create rapid learning healthcare systems.^31,38

Partnership Implementation Process

Moving an intervention from one setting to another requires recognition that different practitioners and stakeholders hold more or less authority, varied opinions, and more or less inclination, capacity and resources to support its implementation. The most common perception practitioners hold of experimental evidence is that it was generated in a system with far more resources, and on people that are carefully selected, compared to actual implementation settings.¹⁵ Guidelines for evidence-based practices that seem to deny or disparage professional or personal judgment, or to limit discretion in applying new methods to local situations, can arouse defensiveness or resentment. Evidence-based programs and practical measures alone are insufficient.

To succeed, interventions must be implemented with methods that engage the partners and multiple stakeholders, and that treat their varied perspectives with consideration and respect. The top-down “we are the experts” attitude has been a source of many failures. With a growing emphasis on participatory approaches,²⁵ an increasing number of researchers and organizational leaders give lip service to egalitarian processes of evidence development, adaptation and implementation, but the participation they invite is often perfunctory and cosmetic. Much research continues to produce rather sterile, decontextualized answers to the question of “what” needs to be implemented, and little on “how” best to implement the evidence-based interventions and measures in relevant settings and populations. Needed are approaches that employ the principles of community-based participatory research^39–41 and team science,^42,43 that take stakeholder and local perspectives seriously and treat all collaborators as valued “experts” on their domains of interest.⁴²

Various forms of evidence are essential to involve stakeholders from the beginning and throughout all phases of project planning, implementation, management, and evaluation. For research to influence implementation, planners and decision makers must take these key issues into consideration in recommending evidence-based practices.⁴⁴

Multilevel Context and Interactions Among Components of the Evidence Integration Triangle

Each of the individual components of the EIT—evidence-based intervention, practical longitudinal assessment, and a partnership implementation approach—becomes necessary, but not sufficient, for successful integration of research, practice and policy. The specific elements of the EIT require attention if research is to influence practice in ways that improve population health. This paying attention⁴⁵ involves iterating between the big picture and the particulars of the multilevel context, ^8,24 working to assure that activities are coordinated to support each other, and are sensitive to and fit the implementation context. Applying the EIT, then, involves developing the three main components based on relevant evidence and interactions with key stakeholders, while periodically raising and lowering the gaze to pay attention to the multilevel context.⁴⁶

Opportunities to Apply the Evidence Integration Triangle to Improve Prevention and Health Care

If national policymakers continue to require that state and local programs be evidence-based, even when such evidence does not exist, then the evidence to be considered must be expanded to take the implementation and partnership processes into account. Recommendations also must emphasize not just “best practices” from evidence-based reviews of controlled trials, but also “best processes” of assessing needs, joint decision-making, planning, management, and ongoing evaluation in partnership with stakeholders (Palmer VJ et al., in preparation). The EIT suggests a path to accomplish this through the iterative feedback process across the triangle components. Feedback from implementation/assessment to the evidence produces “practice-based evidence.”⁹

In community, regional or national implementation contexts, participatory research, both practice-based⁴⁷ and community-based, is needed.⁴⁸ This research strategy, which has a growing emphasis in primary health care,⁴⁰ clinical trials research,⁴⁹ and public health,⁵⁰ has been stimulated by practice-based research networks, and research funding opportunities provided by the Agency for Healthcare Research and Quality, the CDC, the Kellogg Foundation, and the Robert Wood Johnson Foundation. In recent years, participatory research increasingly has been advocated in the name of transdisciplinary research and team science⁴³ and as part of the NIH Roadmap in Community Translational Science Awards.⁵¹

Rapid Learning Organizations

One important implication of the ongoing and iterative nature of the EIT is that it fosters the creation of rapid learning organizations.^31,52,53 Keeping the EIT components and the larger context in view over time results in an ongoing cycle of knowledge generation, implementation and measurement.^23,31 This iterative process can be entered at any point in the triangle. For example, the intervention and evaluation design considerations become modified by assessments of progress; learning about what works in the implementation process may require modifications of the intervention in collaboration with local stakeholders and those affected by the program, and the addition of new measures.⁵⁴

For example, an ongoing investigation to understand the complexity of primary care practice and community settings, found them to be dynamic adaptive systems with the capacity to “learn,”⁵⁵ and then used that understanding to design tailored implementation processes resulting in sustained improvement.⁵⁶ Ongoing measurement and evaluation involving both quantitative and qualitative assessment has fostered rapid cycle learning.⁵⁵

For complex issues that have eluded solutions, such as the “wicked problems” (Palmer et al., in preparation) of obesity, violence, or health inequities, application of the EIT can foster a transdisciplinary approach⁴³ in which people bring their diverse training and backgrounds to work together to make sense⁴⁵ across usual boundaries^46,57 to develop mutual understanding. Wicked problems “are ill-formulated, where the information is confusing, where there are many clients and decision makers with conflicting values, and where the ramifications in the whole system are thoroughly confusing.”⁵⁸ The EIT can guide practical interventions that are sources of learning in real time. For example, the model could inform initiatives of the Centers for Medicare and Medicaid Services (CMS) Innovation Center, and many natural experiments occurring as part of transformations efforts to establish patient-centered medical homes ^59,60 and accountable care organizations.^61,62

Public Health and Policy Opportunities

After over a decade of following the hierarchy of evidence-based medicine,⁶³ systematic reviews of community preventive services and lifestyle interventions frequently found a relative paucity of evidence, and often an impossibility of conducting RCTs on populations, leading repeatedly to conclusions of “insufficient evidence”.⁶⁴ The urgency of action needed in the face of epidemics in HIV/AIDS, H1N1 influenza virus, food-borne diseases, and obesity, has forced a greater appreciation of the wide range of other types of evidence that can and must inform policy action.⁶⁵

The successful tobacco control experience in reducing U.S. smoking prevalence illustrates what can be accomplished by paying attention to and working on the multiple components of the EIT. A focus on practical measures produced a renaissance in the priority given to surveillance data and analysis of population trends in systematic evaluation of the natural experiments of policy and broad program innovations. This, in turn, is having a transformative influence on public health thinking about evidence in general as a guide to public health practice. Other important advances will be driven by the dramatically increased availability of community-level data on health, health behaviors, and health determinants, as well as many other community attributes available via community health indicators, and the ever-increasing GIS databases. When interpreted through “dashboards” and other applications that can clearly and compellingly display complex and interrelated data sets, these data have considerable potential to inform public health action policies and campaigns.

Research Applications

To provide the information needed to apply the EIT, research methods need to be more rapid, practical, transparent, and relevant to stakeholders. These suggestions are congruent with recent movements supported by AHRQ as practical trials,⁶⁶ and by the CONSORT working group on pragmatic trials.⁶⁷ These groups, along with the new Patient-Centered Outcomes Research Institute (www.pcori.org) emphasize research that uses practical designs to produce results that are relevant to real-world settings, and that study complex multimorbid patients in challenging settings, and address issues such as implementation and generalizability of results.

The EIT implies that research designs and evaluations should be iterative and dynamic.²⁴ Application of the EIT has potential to stimulate creative evaluation methods and designs that use practical measures of progress to provide rapid feedback to inform adjustments and mid-course corrections using partnership principles.⁶⁸ The considerations raised by the EIT also provide opportunities for comparative effectiveness research across the prevention and disease-management spectrum^69,70 — focusing on interactions that may explain substantial variance in why interventions differ in their effectiveness, as well as why the same intervention is successful in some settings, and not in others.²⁶

Example Application

An example of how the EIT can be applied to increase the frequency of evidence-based health behavior change counseling in primary care settings. This project, described in detail elsewhere,⁷¹ is an ongoing effort among the NIH and multiple professional and consumer organizations to facilitate the delivery of patient-centered approaches to health behavior and psychosocial issues.⁷² The EIT elements of this effort include: (1) engaging stakeholders including primary care organizations (e.g., American Association of Family Medicine, Society of General Internal Medicine, AHRQ) and consumer-focused groups (e.g., Center for the Advancement of Health and Consumers Union) throughout the process (stakeholders); (2) attending to the larger context, which includes the advent of the primary care medical home^73–76 and the meaningful use of the EHR (context); (3) achieving consensus on a core set of standard, brief, actionable, patient-reported items on health behaviors and psychosocial issues that are both scientifically sound as well as actionable and feasible to implement longitudinally (practical process measures for monitoring progress); (4) creating decision aids to provide feedback to both patients and healthcare teams on issues for discussion and goal setting/action planning, and to connect with evidence-based health behavior change strategies recommended by the U.S. Preventive Services Task Force (evidence-based intervention program/policy)⁷⁷; and (5) an iterative process for identifying and field-testing recommended items, soliciting feedback from expert panels, numerous organizations and constituents via an interactive web-based wiki process, and pilot-testing in diverse primary care organizations that collaborated on study design (partnership implementation process).

The iterative nature of the EIT process is illustrated by the feedback provided from the common data elements (practical measures), which will inform adaptations at both multiple levels. Although this project is still ongoing, it is apparent that the local context, including clinic patterns of patient flow and level of EHR integration are critically important for implementation delivery.

Discussion

The EIT framework suggests several testable hypotheses that could inform implementation science. One key hypothesis is that programs that incorporate all three evidence-based components of: (1) an effective program collaboratively selected and adapted, (2) practical longitudinal measures for rapid feedback on progress, and (3) true partnership approaches to implementation that pay attention to contextual factors, should be superior to programs that focus on fewer components. A more subtle hypothesis is that programs that pay attention to EIT model features iteratively—that adapt initial interventions using feedback on progress, team science principles that involve transdisciplinary interactions,⁴² and shared decision-making among stakeholders—should perform better in the long term than those that focus predominantly on continued fidelity to an original set of intervention activities.

Since funding and research emphasis has focused predominantly on identifying evidence-based interventions, greater attention is needed to the other two components of the EIT—practical indicators of progress and the participatory implementation process.⁷⁸ Research on the EIT and could benefit from measurement of the extent to which the three areas of the EIT align with and support each other. This concept of “alignment” has also been discussed as a key to the success of the Chronic Care Model⁷⁹ and multilevel intervention programs.²⁴ We are not aware of such alignment measures, and at present the construct is probably initially best approached qualitatively.

Both the EIT and the parent field of implementation science¹⁸ could benefit from practical demonstrations and assessments of the multilevel concept of “partnership implementation approach.” To capture patient–practitioner interactions, conceptually related but lengthy measures of slightly different constructs have been developed at the individual/dyadic level for patient-centered health care.⁸⁰ Also, the EIT can aid the operationalization of community-based participatory research principles.^81,82 Finally, use of the EIT can inform evolving literature on the “team science” of how transdisciplinary groups from varying perspectives can best work together constructively.⁴²

Conclusion

Many of the needs for prevention, health care, and population health solutions involve complex problems in complex community and healthcare environments, faced by complex patients, settings and cultures. These challenges demand complex interventions, which are unlikely to be immediately successful when initially applied.⁸³ Application of the EIT, and approaching improvement efforts as complex adaptive systems,⁸⁴ can help guide us toward solutions to these “wicked problems (Palmer et al, in preparation).”⁵⁸

Addressing the EIT components and interactions from the outset of research initiatives can maximize the yield of investment in science by guiding strategic decision-making about research areas to pursue and how evidence can inform health promotion and healthcare-quality research. Considerations raised by the EIT also can inform comparative effectiveness research, quality improvement interventions, evidence implementation, and policy decisions about resource allocation. In the current resource-challenged environment, society cannot afford to invest in knowledge generation that is uninformed by its evidence integration and application in context.