Standards for Reporting Implementation Studies (StaRI) Statement

Abstract

Implementation studies are often poorly reported and indexed, reducing their potential to inform initiatives to improve healthcare services. The Standards for Reporting Implementation Studies (StaRI) initiative aimed to develop guidelines for transparent and accurate reporting of implementation studies. Informed by the findings of a systematic review and a consensus-building e-Delphi exercise, an international working group of implementation science experts discussed and agreed the StaRI Checklist comprising 27 items. It prompts researchers to describe both the implementation strategy (techniques used to promote implementation of an underused evidence-based intervention) and the effectiveness of the intervention that was being implemented. An accompanying Explanation and Elaboration document (published in BMJ Open, doi:10.1136/bmjopen-2016-013318) details each of the items, explains the rationale, and provides examples of good reporting practice. Adoption of StaRI will improve the reporting of implementation studies, potentially facilitating translation of research into practice and improving the health of individuals and populations.

Summary box

• Underpinning the 27 item StaRI Checklist is the concept of dual strands describing (a) the strategies used to promote implementation and (b) the intervention being implemented

• The expectation is that the authors will clarify both (a) how they anticipate that the strategies employed are likely to promote implementation and (b) explain the underpinning premise of why implementation of the intervention may be expected to improve healthcare or health outcomes

• Unlike most reporting standards that apply to a specific research methodology, the StaRI Statement and accompanying Checklist refers to the broad range of study designs employed in implementation science

• The requirement for extensive description of context, implementation strategies, and interventions, as well as reporting a broad range of effectiveness, process, and health economic outcomes, will challenge journals operating strict word limits for research papers and may require (innovative) solutions and use of supplementary online materials

Globally, healthcare systems are struggling to deliver the benefits of research to their populations.^{1 w1-w3} Increasingly, it is recognised that translation from “bench to bedside to community”^w4 is often ineffective and inefficient. The scientific community needs to focus on how effective interventions are disseminated and implemented across the spectrum of contexts and settings in order to improve individual and population health.^w5 Against this background, implementation science has emerged as an important discipline for developing the evidence base on how to translate research findings into routine care.^{1 2 3 4}

Implementation studies are, however, often poorly reported and indexed, making it difficult to find, reproduce, or synthesise the evidence from relevant studies.⁵ More specific criticisms include poor (or absent) descriptions of conceptual frameworks underpinning the research,^{5 6} inadequate description of context, and incomplete information about how the intervention was promoted and implemented (or not) in the different settings.^{6 w6 w7} Similar concerns with, for example, the reporting of randomised controlled trials (RCTs) led to the introduction of the Consolidated Standards of Reporting Trials (CONSORT) checklist,^w8 with evidence of subsequent improvement in reporting standards.^w9-w11 There have been calls for the development of similar standards for transparent and accurate reporting of implementation studies.^{5 6 7} The Standards for Reporting Implementation Studies (StaRI) initiative aimed to address this need.

Scope and relationship with other reporting standards

Implementation science encompasses a broad range of methodologies applicable to improving the dissemination, implementation, and scaling up of effective behavioural, clinical, healthcare, public health, global health, and educational interventions⁸ (or discontinuation of ineffective or harmful practices^w12) with a view to improving quality of care and health outcomes. Although this document is set within the context of healthcare and population health, there are parallels in other domains (such as educational initiatives).^{9 w13} The StaRI Statement and Checklist may thus have resonance outside healthcare.

Understanding of the position that implementation studies hold in the science of developing, evaluating, disseminating, and implementing healthcare interventions has been evolving over recent years. The UK Medical Research Council (MRC) Framework for Development and Evaluation of Complex Interventions¹⁰ emphasises the need to disseminate and implement findings of complex interventions trials but offers no advice on how to achieve this. PRECIS-2 conceives trial design on an “explanatory-pragmatic” spectrum¹¹ but does not project beyond pragmatic trials to implementation in routine practice. Neither of these frameworks addresses the need for research to explore how interventions shown to be effective in trials require adaptation if they are to align with the routines of practice and be successfully implemented into “usual care” settings.^{12 w14-w17} Implementation science undertakes studies that explore healthcare contexts, develop and evaluate strategies for implementing effective interventions that address local realities, can be implemented at scale and are potentially sustainable.^{2 w18} Proposed frameworks have added an “implementation cycle” to complement the MRC’s complex intervention cycle,¹³ or extended a linear spectrum (see fig 1).^{7 14} Others have emphasised the potential overlap between effectiveness and implementation research and described “hybrid” designs.¹⁵

Fig 1 Positioning of implementation studies and the focus of StaRI reporting standards (adapted from fig 12.1. in Brownson et al⁸)

The StaRI Statement and Checklist are intended to improve reporting of implementation studies, employing a range of study designs to develop and evaluate implementation strategies with the aim of enhancing adoption and sustainability of effective interventions.¹⁶ Implementation studies may be distinguished from quality improvement reports that describe system level initiatives, typically in the context of a specific problem within a specific healthcare system,^{16 w19} and the World Health Organisation guidelines, which focus on improving reporting of their fieldwork.^w20

Methods

We followed the methodology described in the Developing Health Research Reporting Guidelines.^w21 Our full protocol is available on the EQUATOR website.^w22 After a systematic literature review, we recruited international multidisciplinary experts (including healthcare researchers, journal editors, healthcare professionals and managers, methodologists, guideline developers, patient organisations, and funding bodies) to participate in an e-Delphi exercise.¹⁷ Of 66 experts approached, 23 contributed suggestions for the Checklist, 20 completed the first scoring round, and 19 completed the second scoring round. Of 47 potential items, 35 reached the a priori level of consensus for inclusion—that is, 80% agreement with priority scores 7, 8, or 9—and 19 items achieved 100% agreement. All these items, with their final priority scores, were taken forward as candidate items for inclusion in the StaRI Checklist.

In April 2015, we convened a two-day consensus working group in London attended by 15 international delegates (UK or Europe=11, US or Canada=4) drawn from multiple disciplines. Delegates included healthcare researchers (n=9), journal editors (n=6), healthcare professionals (n=8) and managers (n=1), methodologists (n=4), guideline developers (n=2), and funding bodies (n=2) (several participants had more than one role). This group discussed the candidate items and agreed the first draft of the StaRI Checklist. The discussions were informed by the outcome of the e-Delphi exercise (see appendix 1 for the e-Delphi results as provided to the workshop delegates), but items were also considered in the context of other published reporting standards and the wider literature, and the working group’s expertise in implementation science. After general discussion on key defining concepts (informed by points raised in the e-Delphi¹⁷), each candidate item was considered in turn. Agreement was reached by discussion rather than by consensus scoring. The initial draft statement and documents were subsequently developed iteratively by email discussion.

Constructive feedback on a penultimate draft of the StaRI Statement from colleagues working internationally in the field of implementation science, healthcare researchers, clinicians, and patients was used to help shape the final version of the paper. In addition, we presented the concepts to and sought feedback from several workshops, conference discussions, and implementation project steering groups.

Defining concepts

There are two defining concepts underpinning the StaRI Statement and Checklist. The first is the dual strands of describing, on the one hand, the implementation strategy and, on the other, the clinical, healthcare, global health, or public health intervention being implemented.^{3 w23} For example, an implementation strategy (staff training, changes to invitation letters and appointment systems, development of computer templates, ongoing audit, etc) might support an intervention (such as offering the option of telephone consultations) with the aim of improving access to routine asthma care.^w24 These strands are represented as two columns in the Checklist (see table 1). The primary focus of implementation science is the implementation strategy,^w25 and the expectation is that the items in column 1 will always be fully completed with details of how the intervention was implemented and the impact measured as an implementation outcome. The second strand (column 2) expects authors to complete items about the impact of the intervention on the health of the target population. This may be measured as a health outcome, or it may be more appropriate to cite robust evidence to support known beneficial effects of the intervention on health of individuals or populations (such as reducing smoking prevalence). Even when evidence is strong, the possibility that the impact of an intervention may be attenuated when it is implemented in routine practice needs to be considered. Although all items are worthy of consideration, not all items will be applicable to, or feasible in, every study; a fully completed StaRI Checklist may thus include a number of “not applicable” items.

Table 1.

Standards for Reporting Implementation Studies: the StaRI Checklist of items to be reported

Checklist item		Implementation strategy	Intervention†
Title	1	Identification as an implementation study, and description of the methodology in the title and/or keywords
Abstract	2	Identification as an implementation study, including a description of the implementation strategy to be tested, the evidence-based intervention being implemented, and defining the key implementation and health outcomes
Introduction	3	Description of the problem, challenge, or deficiency in healthcare or public health that the intervention being implemented aims to address
	4	The scientific background and rationale for the implementation strategy (including any underpinning theory, framework, or model, how it is expected to achieve its effects, and any pilot work)	The scientific background and rationale for the intervention being implemented (including evidence about its effectiveness and how it is expected to achieve its effects)
Aims and objectives	5	The aims of the study, differentiating between implementation objectives and any intervention objectives
Methods: description	6	The design and key features of the evaluation (cross referencing to any appropriate methodology reporting standards) and any changes to study protocol, with reasons
	7	The context in which the intervention was implemented (consider social, economic, policy, healthcare, organisational barriers and facilitators that might influence implementation elsewhere)
	8	The characteristics of the targeted “site(s)” (locations, personnel, resources, etc) for implementation and any eligibility criteria	The population targeted by the intervention and any eligibility criteria
	9	A description of the implementation strategy	A description of the intervention
	10	Any subgroups recruited for additional research tasks, and/or nested studies are described
Methods: evaluation	11	Defined pre-specified primary and other outcome(s) of the implementation strategy, and how they were assessed. Document any pre-determined targets	Defined pre-specified primary and other outcome(s) of the intervention (if assessed), and how they were assessed. Document any pre-determined targets
	12	Process evaluation objectives and outcomes related to the mechanism(s) through which the strategy is expected to work
	13	Methods for resource use, costs, economic outcomes, and analysis for the implementation strategy	Methods for resource use, costs, economic outcomes, and analysis for the intervention
	14	Rationale for sample sizes (including sample size calculations, budgetary constraints, practical considerations, data saturation, as appropriate)
	15	Methods of analysis (with reasons for that choice)
	16	Any a priori subgroup analyses (such as between different sites in a multicentre study, different clinical or demographic populations) and subgroups recruited to specific nested research tasks
Results	17	Proportion recruited and characteristics of the recipient population for the implementation strategy	Proportion recruited and characteristics (if appropriate) of the recipient population for the intervention
	18	Primary and other outcome(s) of the implementation strategy	Primary and other outcome(s) of the intervention (if assessed)
	19	Process data related to the implementation strategy mapped to the mechanism by which the strategy is expected to work
	20	Resource use, costs, economic outcomes, and analysis for the implementation strategy	Resource use, costs, economic outcomes, and analysis for the intervention
	21	Representativeness and outcomes of subgroups including those recruited to specific research tasks
	22	Fidelity to implementation strategy as planned and adaptation to suit context and preferences	Fidelity to delivering the core components of intervention (where measured)
	23	Contextual changes (if any) which may have affected outcomes
	24	All important harms or unintended effects in each group
Discussion	25	Summary of findings, strengths and limitations, comparisons with other studies, conclusions and implications
	26	Discussion of policy, practice and/or research implications of the implementation strategy (specifically including scalability)	Discussion of policy, practice and/or research implications of the intervention (specifically including sustainability)
General	27	Include statement(s) on regulatory approvals (including, as appropriate, ethical approval, confidential use of routine data, governance approval), trial or study registration (availability of protocol), funding, and conflicts of interest

*Implementation strategy refers to how the intervention was implemented.

†Intervention refers to the healthcare or public health intervention that is being implemented.

Note: A key concept is the dual strands of describing (a) the implementation strategy and (b) the clinical, healthcare, or public health intervention that is being implemented. These strands are represented as two columns in the checklist. The primary focus of implementation science is the implementation strategy (column 1) and the expectation is that this will always be completed. The evidence about the impact of the intervention on the targeted population should always be considered (column 2) and either health outcomes reported or robust evidence cited to support a known beneficial effect of the intervention on the health of individuals or populations. While all items are worthy of consideration, not all items will be applicable to or feasible within every study.

The second concept is that, unlike most reporting standards that apply to a specific research methodology, StaRI applies to the broad range of research methodologies employed in implementation science (for example, cluster RCTs, controlled clinical trials, interrupted time series, cohort, case study, before and after studies, as well as mixed methods for quantitative or qualitative assessments).³ Authors are referred to other reporting standards for advice on reporting specific methodological features—for example, randomisation in cluster RCTs,^w26 matching criteria in cohort studies,^w27 or addressing reflexivity in qualitative research.^w28

The StaRI Checklist

The StaRI Checklist comprises 27 items, of which 10 items expect authors to consider the dual strands of the implementation strategy and the intervention (see table 1). Details about each of the Checklist items is provided in the accompanying Explanation and Elaboration document published in BMJ Open.¹⁸ Appendix 2 is a version of the Checklist for completion by authors submitting an implementation paper. It is strongly recommended that authors using the StaRI Checklist read the detailed document that explains the rationale for each item and provides examples of good practice.

Three overarching components are emphasised in the Checklist:

1. The expectation is that authors have an explicit hypothesis (we use the term “logic pathway”) that spans both how the implementation strategy is expected to work and the mechanism by which the intervention is expected to improve healthcare (see Explanation and Elaboration document¹⁸ for a table of alternative terminologies and a link to a detailed description of “logic models”). This logic pathway should reflect the rationale presented in the introduction, determine the approach to implementation, dictate implementation, health, and process outcomes, and provide insights into why and how the implementation strategy and intervention worked (or not).

2. The balance between fidelity to, and adaptation of, the implementation strategy and intervention is of particular interest in implementation science. Fidelity refers to the degree of adherence to the described implementation strategy and intervention; adaptation is the degree to which users modify the strategy and intervention during implementation to suit the local needs (see table 2 for further description and examples). Insufficient fidelity to the “active ingredients” of an intervention may dilute effectiveness,^w29 whereas insufficient adaptation or tailoring to local context may inhibit effective implementation.¹⁹ An approach to reporting these apparently contradictory concepts is to define the core components of an intervention (ideally related to the logic pathway) to which fidelity is expected, and those aspects that may be adapted by local sites to aid implementation.^{19 w30}

3. Successful implementation of an intervention into practice is a planned, facilitated process involving the interplay between individuals, intervention or new ways of working, and context to promote evidence-informed practice.²⁰ A rich description of the context is critical to enable the reader to assess the external validity of the reported study,^w18 and to decide how the context in the study compares to their situation and whether the implementation strategy can be directly transposed or will need adapting.^w31 Similarly, social, political, and economic context influence the “entrenched practices” that hinder evidence-based de-implementation of unproven practices or interventions.^{w12 w32}

Table 2.

Terminology: definitions and illustration

Terminology	Definition	Illustration using a study implementing supported self management for asthma
Implementation science	The scientific study of methods to promote the systematic uptake of evidence-based interventions into practice and policy and hence improve health4	Improving implementation in routine practice of evidence-based supported self management for asthma
Implementation strategy	Methods or techniques used to enhance the adoption, implementation, and sustainability of an under-utilised intervention15 w26	A programme of professional training, templates for reviews, access to resources, facilitation, audit, and feedback
Intervention	The evidence-based practice, programme, policy, process, or guideline recommendation that is being implemented8	Provision of asthma self management in routine asthma reviews, including completion of action plans
Implementation outcome	Process or quality measure to assess the impact of the implementation strategyw24	Proportion of people with asthma who have an action plan
Health outcome	Patient-level health outcomes for a clinical intervention, such as symptoms or mortality; or population-level health status or indices of system function for a system/ organisational-level intervention15	Proportion of people with asthma requiring unscheduled care for asthma or patient reported asthma control
Logic pathway	The way(s) in which the implementation strategy and intervention are hypothesised to operate	An organisation that prioritises self management encourages or enables trained professionals to provide asthma action plans; self management improves asthma outcomes
Fidelity	The degree of adherence to the described implementation strategy and/or the degree to which an intervention is implemented as prescribed in the original protocolw29	Uptake of professional training, utilisation of review templates (implementation fidelity), and assessment of adequacy of education and completion of action plans (intervention fidelity)
Adaptation	The degree to which the strategy and intervention are modified by users during implementation to suit local needs19	Use (or not) of telehealth to deliver reviews or provide action plans. Different professionals (doctors, nurses, pharmacists) with primary responsibility for self management education

Discussion

Implementation science is an emerging and rapidly evolving field. The StaRI Statement and Checklist should therefore be seen as an evolving document, and potentially a catalyst for discussing and defining how implementation studies are conceived, planned, and reported.⁵

We hope that the concept of dual strands will resonate with researchers designing and reporting implementation science studies. We appreciate that the distinction will not always be as unambiguous as it seems in the StaRI checklist, but we suggest that considering the design and evaluation of implementation studies in these two strands is helpful and aids clarity of study design and reporting. We also recognise that not all studies will measure health outcomes, though consideration of the ultimate goal of improving health through implementing an evidence-based intervention would seem a reasonable requirement. Feedback on this underpinning concept will be valuable for future iterations of the StaRI Statement and Checklist.

There are two practical challenges for the application of StaRI that warrant discussion. First, implementation science uses diverse methodologies that need to be accommodated in the reporting standards. One option is to incorporate relevant items from other checklists, but this may be perceived as limiting the methodological options. StaRI therefore advises authors to consult methodological checklists for reporting design-specific aspects of their chosen study design. By doing this, we have implicitly prioritised the concept underpinning implementation studies, though this should not be interpreted as undermining the rigour of reporting the chosen study design.

The second challenge is the requirement for extensive description of context, implementation strategies, and interventions as well as reporting a broad range of primary effectiveness, process, health, economic, and implementation outcomes.^w25 This requirement will stimulate debate about word counts, supplementary material, and additional publications in order to accommodate journal requirements, author needs, and reader preferences. This tension is further discussed in the Explanation and Elaboration document,¹⁸ and some practical approaches are suggested for summarising information in tables or figures. We look forward to learning how authors and journals work with these challenges and the (innovative) solutions that they adopt (such as appendices, supplementary online files, and additional publications).

Conclusion

The StaRI Statement is registered with the EQUATOR Network (www.equator-network.org), and the Checklist (for completion by authors) is freely available from bmj.com (appendix 2). We invite editors of all journals publishing implementation research to consider requiring submission of a StaRI Checklist, and authors reporting their implementation studies to use the Checklist. In the future we would like to work with authors as they apply the Checklist to their papers, “road testing” the standards and enabling iterative development.

Previously published reporting guidelines have been instrumental in improving reporting standards,^{6 w8-w10} and our hope is that StaRI will achieve a similar improvement in the reporting of implementation strategies that will facilitate translation of effective interventions into routine practice, ultimately to benefit the health of individuals and populations.

Linked information

• StaRI website. www.equator-network.org/reporting-guidelines/stari-statement/

• Enhancing the QUAlity and Transparency Of health Research (EQUATOR). www.equator-network.org/

• Consolidated Framework For Implementation Research (CFIR). www.cfirguide.org/imp.html

• Logic models. www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide

• Dissemination and implementation models in Health Research and Practice. www.dissemination-implementation.org/index.aspx

• Process evaluation of complex interventions. www.ioe.ac.uk/MRC_PHSRN_Process_evaluation_guidance_final(2).pdf

• Reach Effectiveness Adoption Implementation Maintenance (RE-AIM). www.re-aim.hnfe.vt.edu

Web Extra.

Extra material supplied by the author

Appendix 1: e-Delphi results as provided to the workshop delegates

Appendix 2: Version of StaRI Checklist for completion by authors submitting an implementation paper

Supplementary references w1–w32