A Framework for Enhancing the Value of Research for Dissemination and Implementation

Abstract

A comprehensive guide that identifies critical evaluation and reporting elements necessary to move research into practice is needed. We propose a framework that highlights the domains required to enhance the value of dissemination and implementation research for end users. We emphasize the importance of transparent reporting on the planning phase of research in addition to delivery, evaluation, and long-term outcomes. We highlight key topics for which well-established reporting and assessment tools are underused (e.g., cost of intervention, implementation strategy, adoption) and where such tools are inadequate or lacking (e.g., context, sustainability, evolution) within the context of existing reporting guidelines. Consistent evaluation of and reporting on these issues with standardized approaches would enhance the value of research for practitioners and decision-makers.

A major challenge for practitioners and policymakers is that most evidence-based interventions are not ready for widespread dissemination.¹ Not only does most research on evidence-based interventions not translate into practice or policy; but also, if it does, it usually takes an extraordinarily long time.^2,3 This matters for public health practice and policy because the knowledge generated from taxpayer expenditures on research are not reaching the public, and especially not those most in need.⁴ This reflects poor return on investment, suboptimal health outcomes, and significant opportunity costs. Thus, there has been greatly increased attention to dissemination and implementation (D&I) research in the past few years both in the United States and internationally.^5,6 For present purposes, we adopted the definitions in the National Institutes of Health (NIH) program announcement on D&I research.⁷ Dissemination is defined as the targeted distribution of information and intervention materials to a specific public health or clinical practice audience. The intent is to spread knowledge and the associated evidence-based interventions. Implementation is the use of strategies to adopt and integrate evidence-based health interventions and change practice patterns within specific settings.⁷

There are many reasons for this slow and incomplete translation,¹ including research methods and reporting standards that do not seem relevant to the situations and decisions faced by practitioners and policymakers.^8–10 To address these issues, the NIH and the Veterans Administration held a series of invited state-of-the-science meetings in late 2013 and early 2014 to address key gaps and opportunities in D&I research. Three separate assembled working groups focused on 1 of the following 3 issues: (1) training, (2) study design, and (3) reporting and measurement.

The goals of the working group focused on reporting and measurement were to identify key areas in need of better measurement and reporting at all stages of research for dissemination and implementation. We describe a framework developed by this working group. The working group included 23 D&I researchers, practitioners, and decision-makers from the United States and Canada. At the meeting, there was considerable discussion around guidelines for research reporting and their impact, and whether the D&I field was ready for Consolidated Standards of Reporting Trials (CONSORT)–like reporting guidelines. The consensus was that, given the plethora of existing guidelines and reporting criteria,¹¹ it was premature to propose a specific set of guidelines until more was known about whether there are D&I-related gaps in existing guidelines.

Participants decided that, to advance the field and state of knowledge, a reasonable and important first step was to construct a framework that could serve as a guide to researchers to enhance D&I evaluation and reporting relevant to stakeholders. Such a framework is not intended as a formal theory or another model of D&I research; currently, more than 60 such models exist, with many overlapping constructs.¹² Rather, the purposes of the proposed framework are to (1) focus attention on needs and opportunities to increase the value and usefulness of research for end users, and (2) identify key needs for evaluation, before issuing formal reporting guidelines for D&I research. Whereas previously published reviews of D&I models have developed strategies to select or use models for research or practice,^12–15 here we provide a comprehensive framework to guide researchers across the different phases of research.

The purposes of this article are threefold: (1) to present and discuss implications of the framework, organized by different steps in the research process; (2) to highlight areas that are underreported, but would substantially enhance the value of research for end users with the end goal of improving population health; and (3) to compare concepts in existing reporting guidelines to our framework.

The target audience for this article includes D&I and comparative effectiveness researchers¹⁶ and those who are users of D&I evidence, who might consider asking the questions identified here when reviewing research reports or considering adoption of programs and policies. In addition, researchers at earlier stages of the translation cycle (efficacy researchers) could likely benefit from attention to these issues if their goal is to have the products of their research advance to policy or practice. We realize that efficacy research is quite different and are not implying that these issues need to be addressed at that stage. We do think, however, that it is never too early to begin considerations of translation and “designing for dissemination.”^17,18 If the products of efficacy or effectiveness research are substantially misaligned with conditions, resources, and policies that have an impact on real world public health and health care delivery contexts, it is very unlikely that such interventions or guidelines will ever be adopted, or if adopted, will be implemented with quality or will be sustained.¹⁹ Therefore, the issues in this article should be relevant to both D&I and comparative effectiveness researchers and those seeking to develop, or select and make decisions about real-world use of interventions, programs, and guidelines.

FRAMEWORK FOR DISSEMINATION AND IMPLEMENTATION REPORTING

The meeting planning group (R. E. G., R. C. B., C. R. C., B. A. R., G. N.) developed an initial framework based on their expertise, as well as input from all expert working group members via structured premeeting telephone conversations with the objective to identify essential elements and recognized gaps in existing D&I models and frameworks in the area of measures and reporting. This early version of the framework was presented at the start of the meeting and strongly endorsed by the participants. Following this introductory session presenting the framework, participants discussed at length and reached consensus on key framework components and subcomponents (Figure 1).

FIGURE 1—

Framework for enhancing the value of research for dissemination and implementation.

Note. D&I = dissemination and implementation; PRECIS = Pragmatic Explanatory Indicator Summary. For a glossary of terms, please see Rabin et al.²⁰ as well as appendix of additional definitions (available as a supplement to the online version of this article at http://www.ajph.org).

^aUnderused or underreported elements despite published assessment tools.

^bEmerging areas of interest with limited or no published assessment tools.

Figure 1 represents the various interrelated phases of an efficacy, effectiveness, or D&I research project in the columns, identifies key reporting elements in each phase, and highlights crosscutting issues (i.e., multilevel context, stakeholder perspectives, and costs). Detailed definitions of framework components are included in an appendix (available as a supplement to the online version of this article at http://www.ajph.org). As indicated by the title and the “Goals” column, the ultimate goal is to increase the usefulness of research for end users to enhance equitable, cost-efficient population health. In the next sections we highlight some crosscutting themes and important elements within each phase. We recognize that it may not be feasible to incorporate all of these elements in every study, and a programmatic approach may be more appropriate. But most, if not all, of these elements should be considered to ensure that the results of research efforts not only accelerate the use of a specific intervention but also advance the field in general and provide information to improve translation.

Crosscutting Themes

Several crosscutting themes apply to most or all phases of research, as indicated at the bottom of the figure. For example, applied research is almost always multilevel and crosses social–ecological levels.^21,22 Unfortunately, it is not always conducted or reported that way and, in particular, information related to multilevel context, such as historical context, policy climate, and incentives, as well as organizational settings and persons delivering and receiving interventions, is often lacking or missing important elements.^23,24 As indicated in multiple columns, the issue of “fit” or alignment (or lack of fit) between an intervention program or policy and its context is one of the key elements of our framework; thus, reporting on context is essential for end users to understand the potential fit for a given intervention.

Context is one of the most important and least often reported elements in research.^25,26 As conceptualized here, context is multilevel, and cuts across domains including economic, political, social, and temporal factors. At present, there is no agreed-upon method for assessing context. The most frequent attempts to capture basic structural characteristics of context, such as study population size and demographics, are seldom the most relevant underlying causes of important outcomes.²⁷ Moreover, structural characteristics fail to capture other important contextual factors that may greatly influence adoption, implementation, and sustainment such as organizational capacity for change, presence of an opinion leader, communication and feedback strategies, and the nature of the provider–end user relationship.

Other factors involve clarity of health information exchange between provider and end user within the patient’s personal goals of care, ensuring sufficient explanation of acceptable alternatives and best-case–worst-case scenarios. Interspecialty characteristics of a profession or individual professional such as training and credentialing, risk aversion, or society guidelines also influence the rapidity and penetration of evidence into practice. In present day health care, context can change rapidly; thus, it is important that context be assessed longitudinally, not just at the beginning of a study. There are ongoing efforts to specify organizational characteristics related to setting context and several instruments drawn from the management and business literature,²⁸ but many such instruments are so lengthy as to be impractical or not helpful for analyses because they are so general. At present, we strongly recommend using mixed-methods approaches to report on context.^29–32

A second crosscutting issue concerns the importance of considering and reporting information relevant from multiple stakeholder perspectives. A large body of literature calls for the need to make research more relevant to stakeholders, including practitioners and other end users of research (e.g., patients, community members).^1,33–37 Less frequently addressed, but also critical perspectives include those of policymakers and administrators who must make difficult investment decisions. In particular, and as indicated as a third crosscutting issue, information on costs and resources required to deliver an intervention are essential, and often the first question asked by potential decision-makers. All too often, there are no data to answer this question,³⁸ despite existing methods,^38,39 and, when present, cost-of-intervention information, for example, is often nonstandardized and often does not take the perspective of a potential adopting organization.⁴⁰

Planning

Information from the planning phases of research needs to be reported more frequently. By planning, we mean activities that precede actual piloting or project activities that involve delivery of services. This ideally involves a series of interactive meetings with stakeholders to understand and address the elements listed in the “Planning” column of Figure 1. It is often stated that one needs to plan for dissemination (or sustainability, or scale-up) from the beginning,¹⁷ but, despite these entreaties, it is usually not at all apparent how, if at all, research and intervention planners took such issues into consideration in the planning phase. Systematic approaches such as intervention mapping, a process for developing theory and evidence-based interventions, takes the planner through an explicit process to plan implementation interventions during the initial intervention development phase. Planners are guided to consider context and players responsible for program adoption, implementation, and sustainability, to consider what has to be done as well as the factors influencing specific adoption, implementation, and maintenance behaviors, and to change methods and strategies to increase use.⁴¹ We also recommend efforts to help specify the intervention theory or logic model that specifies how a program is intended to work.^42,43 It is important for end users and other researchers to understand how a program developer thinks that change will come about. Less often reported, even in D&I research, is the implementation strategy employed to deliver the intervention. Proctor et al. recently published proposed guidelines for reporting elements of implementation strategies.⁴⁴ They suggested

naming, defining, and operationalizing implementation strategies in terms of seven dimensions: actor, the action, action targets, temporality, dose, implementation outcomes addressed, and theoretical justification.^44(p1)

Among the setting characteristics that are relevant in this phase are organizational readiness to change, setting capacity, and resource constraints. Investigators purporting to use community-engaged, stakeholder-engaged, or patient-centered research methods should report on the specific partnership strategies used, including the frequency and structure of meetings, and the amount of decision-making and budget authority that patient and community partners have. Finally, costs and resource considerations are important in the planning phase through use of evaluability assessment,⁴⁵ which is a pre-evaluation activity designed to maximize the chances that any subsequent evaluation will result in useful information.⁴⁶ This can determine the probability that an intervention has even a remote chance of being adopted or successfully implemented in real-world settings, even if effective.

Delivery

In the delivery or implementation phase, some of the same issues are important, but in slightly different ways. Of paramount importance in the delivery phase is reporting on the evolution of both intervention components and the implementation strategy. It is rare for either to be implemented exactly as planned, and transparent reporting on how these strategies evolved and how the evidence-based program was adapted to fit local contexts is crucial for understanding implementation.^45,47,48 Recent publications on how to report on the balance between fidelity and adaptation^49,50 provide helpful discussions and guides for reporting. Historically, failure to report on adaptations has been one weakness in D&I research, possibly because unwavering fidelity to an intervention protocol was considered paramount to good science,⁵¹ and, hence, researchers may not report adaptations out of concern that this would undermine the credibility of (and ability to publish) their study. But data on adaptations and local customization is the type of information that end users and potential adopting organizations most need. Over time, information on which elements of an intervention are essential (core) and which can be modified to local context should accumulate and guide future adaptation efforts.⁵²

The costs of intervention delivery³⁸ are also important to report. We recommend use of standard procedures to estimate costs, reporting from the perspective(s) of the organization delivering the program and from the patient perspective.⁴⁰ In some cases, it may also be important to report costs from the societal perspective for large-scale public programs, or to estimate cost–benefit or downstream costs. Costs that are frequently excluded yet important to know are recruitment or promotion costs, and replication costs.³⁸ Researchers often omit these expenses because they think of recruitment as a package that includes application of research inclusion and exclusion criteria and informed consent procedures. Although some of these are research costs that would not be required in standard care (e.g., informed consent), it is usually necessary to advertise and enroll participants into a real-world, nonresearch program, and these often substantial costs can be separated from research costs.

Finally, both reach and adoption need to be reported more consistently.⁵³ Recent reviews of both publications and grant applications reveal that reach among individual participants (patients) is reported increasingly often as the percentage of persons approached or invited who participate, but the representativeness of these participants is reported far less often.^53,54 Adoption among potential settings and staff is reported much less often, and again the characteristics of those taking part versus not are even less often reported.^53,54

Results Reporting

The information in the results (and also the methods or appendix) section of an article is what is traditionally thought of as evaluation, process, and outcomes reporting. Our recommendations are to consider more broadly the outcomes of a program, including both intended and unintended consequences. This is not inconsistent with requirements to a priori identify and report on a primary outcome for a project, but recognizes that programs have more than just a single impact. Findings over the past couple decades have clearly established that interventions have both intended outcomes, and usually also other corollary and often unintended consequences (which can be positive or negative).^55–58 If one considers the perspective of practitioners and policymakers who must choose among interventions for very diverse conditions, it is also helpful to include impact on broader or more general outcomes such as health-related quality of life.^59,60

As indicated in the “Results Reporting” column of Figure 1, this broader focus includes reporting on both the robustness of outcomes (e.g., across different subgroups of patients, staff, settings, and time) and considering external validity and generalizability as well as internal validity issues.²⁴ To address generalizability, researchers need to describe the intervention and contextual factors in sufficient detail to allow local decision-makers to make their own judgments about the applicability of the findings to their own settings and context. These questions (e.g., Will this intervention work in settings like mine? With patients like mine? With resources available to me?) are the types of issues most important to practitioners, decision-makers, and ultimately patients.⁹ This contextual perspective is aligned with the goals of both the precision or personalized medicine movements in the United States^61–63 and realist perspectives internationally.⁶⁴

Possibly the reporting change most currently ready to implement is the use of the Pragmatic Explanatory Indicator Summary (PRECIS) criteria, which could serve as one component of a future D&I reporting guideline.⁶⁵ Developed by the CONSORT consortium to complement the CONSORT reporting criteria on pragmatic trials, the current PRECIS criteria include information on the extent to which a given study is pragmatic versus “explanatory” (or efficacious) on each of 10 dimensions (e.g., participant eligibility criteria, experimental intervention flexibility, experimental intervention practitioner expertise).

Long-Term Outcomes

One of the least reported outcomes across almost all health literatures is the extent to which an intervention or policy is maintained.⁵³ There are multiple reasons for this, including lack of planning for sustainability, publication challenges, and limited funding periods, but in many ways the fundamental evaluation question from a stakeholder perspective is “Can this program be sustained over time (and under what conditions)?” Emerging literature is also documenting that an intervention almost never remains the same as it was during initial intervention, but changes over time.^66,67 The irony is that to maintain or sustain results, an intervention may well need to change over time given that many aspects of context change. This recognition is starting to appear in the literature and has variously been termed evolvability or dynamic sustainability,⁶⁶ and, when applied to different settings, transportability.^68–70

Related to the theme of external validity,²⁴ there is a need to report on replication of a program in different settings. Failure to replicate has been identified as a critical issue at all phases of research from basic discovery to dissemination,^71,72 and understanding the context and conditions under which results are replicable or not is one of the most important tasks of D&I research.³³

Reporting data on longer-term costs and benefits of programs and policies, and especially the cost–benefit value of different variations of interventions, is an important ultimate outcome, as indicated at the bottom of the “Long-Term Outcomes” column in Figure 1.

Summary of Figure and Framework

The issues identified in the framework are all important. In Figure 1, the expert working group highlighted with footnotes those issues most in need of attention to report on to advance the value of research for D&I. Footnote “a” denotes elements that are underreported, but for which there are published assessment tools (e.g., cost of intervention, implementation strategy, adoption).^{39,43,44,73–78} Footnote “b” denotes emerging areas of interest, for which there are currently no or few published assessment procedures, and little agreement on how that issue should be reported (e.g., context, sustainability, evolution).

A final point is that these issues are not independent, but rather interrelated. Many reflect a needed change in perspective from a reductionistic focus on a single thing at a time holding everything else constant to one of studying events and interventions in context, and understanding interactive and contextual effects.^6,7,31 There needs to be more consistent reporting, using standard methods to significantly advance the field, and these issues need to be reported transparently and in connection with each other. Reporting on any 1 or 2 elements or issues in the framework will not substantially improve our knowledge or help end users. A concerted effort to address most of these issues in concert and transparently discussing issues such as adaptations made and lessons learned would substantially increase the value of D&I research.

EXPLORING THE NEED FOR REPORTING GUIDELINES

Although its underpinnings have been around since the 1970s,⁷⁹ formal recognition of the field of D&I research in the United States has been present for about a decade. As noted by Chambers,⁸⁰ the first program announcement for National Institutes of Health D&I research was published in 2002, attendance at the annual NIH conference on D&I research has quadrupled from 2007 to 2011, and the volume and quality of D&I research has increased considerably. The framework described in this article is seen as one important step to provide background to consider the need for a set of reporting guidelines for D&I research.

Part of the next stage in the evolution in D&I research involves reporting guidelines, which provide methodological structure for the design, conduct, and reporting of research. The Enhancing the Quality and Transparency of Health Research (EQUATOR) network was established in 2008 to raise awareness of existing reporting guidelines, while serving as a central repository and general resource for scientific authors, peer reviewers, and journal editors.^81,82 Their process includes an assessment of existing guidelines for a research domain and the overall quality of published articles without a directly applicable guideline. The guidelines in EQUATOR provide a checklist to frame the conduct and reporting of research. For example, guidelines exist for randomized controlled trials,⁸³ diagnostics,⁸⁴ observational trials,⁸⁵ quality improvement,⁸⁶ cost-effectiveness,⁸⁷ case reports,⁸⁸ and systematic reviews.^89,90 Furthermore, some guidelines have multiple extensions including the CONSORT recommendations for harms,⁹¹ noninferiority,⁹² cluster-design,⁹³ pragmatic trials,⁹⁴ herbal studies,⁹⁵ and nonpharmacological treatment interventions.⁹⁶ These reporting guidelines often enhance the quality of research reporting,⁹⁷ but the current guidelines have been slow to penetrate into individual journals’ author instructions.⁹⁸ One pragmatic obstacle to widespread dissemination and utilization of existing reporting guidelines is the continually expanding number of these recommendations, which create an atmosphere of “guideline fatigue.”

Some D&I researchers have highlighted the need for explicit scientific methods to study and report complex interventions.⁴² In related work, the Workgroup for Intervention Development and Evaluation Research checklist was developed and tested to evaluate the quality of reporting in knowledge translation systematic reviews.⁹⁹ A structured reporting guidelines framework for D&I research derived by experts with widespread endorsement by D&I researchers, publishers, and policymakers may be needed to improve the quality and transparency in reporting of D&I research. However, our working group decided that proceeding to a set of D&I-specific reporting guidelines was premature without a systematic evaluation of existing guidelines to evaluate their overlap with our framework. Therefore, we reviewed the existing guidelines with the most likely overlap germane to D&I research, as judged by our consensus work group.

To identify existing reporting guidelines with potential relevance or overlap with D&I research, we used 2 approaches. First, one author (C. R. C.) contacted the EQUATOR network in January 2013 to simultaneously report our plans to organize a consensus conference to formulate D&I reporting guidelines and to ascertain whether any other groups were working on similar projects. This author also reviewed all of the published and in-development guidelines archived on the EQUATOR network Web site. The EQUATOR network subsequently connected us with the workgroup for the Standards for Reporting Phase IV Implementation Studies initiative in the United Kingdom. We consulted with the workgroup’s principal investigators to identify overlap and unique attributes of the 2 projects, as well as deficiencies in existing reporting guidelines. The second strategy that we used to identify existing reporting guidelines with potential relevance was to confer with participants of our NIH work group to solicit their awareness of pertinent, potentially overlapping reporting guidelines. One participant and coauthor (J. M. G.) listed the 6 published guidelines that we assessed. These existing reporting guidelines were CONSORT cluster randomized trial,⁹³ CONSORT pragmatic trial,⁹⁴ CONSORT nonpharmacological,⁹⁶ PRISMA-health equity extension,¹⁰⁰ Standards for Quality Improvement Reporting Excellence,⁸⁶ and the complex interventions in health care (CReDECI) checklist.¹⁰¹

The results of our preliminary review of existing reporting guidelines overlap with our D&I framework are summarized in Table 1. This table was produced by one author’s (C. R. C.) review of the existing reporting guidelines’ recommendations within the context of the framework. Notably, none of the existing guidelines incorporate all of the key domains of our D&I framework. The CReDECI checklist incorporates the most elements of our framework by including 5 of 9 domains, but does not provide recommendations for scalability, pragmatic assessment, evolvability, or replication. None of the existing publications provides a recommendation to assess pragmatic versus explanatory design with an instrument such as the PRECIS tool.¹⁰² The CReDECI is the only guideline that recommends a description of the theoretical basis for the intervention tested.¹² None recommends description of the conceptual basis of the implementation strategies employed. The Standards for Quality Improvement Reporting Excellence criteria are the only guideline to advocate a description of scalability, but this element is vaguely described as

paying particular attention to components of the intervention and context factors that helped determine the intervention’s effectiveness (or lack thereof), and types of settings in which this intervention is most likely to be effective.⁸⁶

TABLE 1—

Overlap Between Dissemination and Implementation Framework and Existing Reporting Guidelines

Guideline	Program Logic	Context	Scalability	Adoption	Robustness	Pragmatic Criteria	Sustainability or Evolvability	Replication	Economic Evaluation
Criteria for Reporting the Development and Evaluation of Complex Interventions (CReDECI)101	+	+	−	+	+	−	−	−	+
Consolidated Standards of Reporting Trials (CONSORT) cluster93	−	?	−	+	−	−	−	+	−
CONSORT nonpharmacological96	−	?	−	?	−	−	−	+	−
CONSORT pragmatic94	−	−	−	−	−	−	−	+	−
Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)-EQUITY100	−	+	−	?	−	−	−	−	−
Standards for Quality Improvement Reporting Excellence (SQUIRE)86	−	+	?	+	−	−	+	+	+

Note. + = clear description suggesting that authors use guideline to incorporate this domain into their research report; ? = equivocal in that the guideline does not explicitly advise authors to incorporate this domain into their research report; − = no clear description suggesting that authors use guideline to incorporate this domain into their research report.

Our conclusion is that existing reporting guidelines most applicable to D&I researchers and peer reviewers do not address or lack precision around several essential D&I elements of our framework, so further exploration of D&I-specific reporting guidelines with EQUATOR methodology is warranted.

The next step in evaluating the value of distinct D&I reporting guidelines will be to contact the author groups for the existing guidelines summarized in Table 1 to ascertain plans for future or ongoing extensions to their guidelines into the realm of D&I research, with the assistance of the EQUATOR network. In addition, several other groups are already working on additional reporting guidelines that may be applicable to D&I research. The Standards for Reporting Phase IV Implementation Studies initiative in the United Kingdom seeks to improve the quality of reporting of implementation studies in real-life settings.¹⁰³ In addition, the Reporting Manualised Interventions for Dissemination and Evaluation statement,¹⁰⁴ Template for Intervention Description and Replication,¹⁰⁵ and CONSORT extension for social and psychosocial interventions¹⁰⁶ are all in development. These author groups will be contacted to assess the degree of overlap between their reporting guidelines and our D&I framework.

One limitation of our approach to the framework is that the expert panel was populated mainly by US-based scientists, so we have likely missed some key points from knowledge translation, knowledge exchange, and D&I research in a global context. In addition, the “underused” and “lacking” labels are based on expert judgment and not a systematic review of the literature.

Whether to proceed with the development of distinct D&I reporting guidelines based upon our framework depends upon a more in-depth assessment of the benefits of a separate guideline weighed against the pragmatic threat of “guideline fatigue.” One of the major benefits of distinct D&I reporting guidelines is the opportunity to explicitly define the key elements of D&I research in 1 article, along with their associated measures. The major barrier to reporting of these measures is the lack of practical, validated, well-accepted instruments and analytic approaches. However, highlighting the need for such tools and strengthening guidelines that require that future D&I studies attempt to assess these elements should enhance the scope and value of D&I research.