Development and psychometric evaluation of the Implementation Science Research Project Appraisal Criteria (ImpResPAC) tool: a study protocol

Abstract

Introduction

The need for quantitative criteria to appraise the quality of implementation research has recently been highlighted to improve methodological rigour. The Implementation Science Research development (ImpRes) tool and supplementary guide provide methodological guidance and recommendations on how to design high-quality implementation research. This protocol reports on the development of the Implementation Science Research Project Appraisal Criteria (ImpResPAC) tool, a quantitative appraisal tool, developed based on the structure and content of the ImpRes tool and supplementary guide, to evaluate the conceptual and methodological quality of implementation research.

Methods and analysis

This study employs a three-stage sequential mixed-methods design. During stage 1, the research team will map core domains of the ImpRes tool, guidance and recommendations contained in the supplementary guide and within the literature, to ImpResPAC. In stage 2, an international multidisciplinary expert group, recruited through purposive sampling, will inform the refinement of ImpResPAC, including content, scoring system and user instructions. In stage 3, an extensive psychometric evaluation of ImpResPAC, that was created in stage 1 and refined in stage 2, will be conducted. The scaling assumptions (inter-item and item-total correlations), reliability (internal consistency, inter-rater) and validity (construct and convergent validity) will be investigated by applying ImpResPAC to 50 protocols published in Implementation Science. We envisage developing ImpResPAC in this way will provide implementation research stakeholders, primarily grant reviewers and educators, a comprehensive, transparent and fair appraisal of the conceptual and methodological quality of implementation research, increasing the likelihood of funding research that will generate knowledge and contribute to the advancement of the field.

Ethics and dissemination

This study will involve human participants. This study has been registered and minimal risk ethical clearance granted by The Research Ethics Office, King’s College London (reference number MRA-20/21-20807). Participants will receive written information on the study via email and will provide e-consent if they wish to participate. We will use traditional academic modalities of dissemination (eg, conferences and publications).

Strengths and limitations of this study.

• Input from a multidisciplinary, international expert group will inform the development of ImpResPAC.

• Our definition of ‘experts’ in this study could exclude the perspectives of other stakeholder groups that could be useful and how the tool might be valued by groups excluded in the initial development process.

• ImpResPAC will enable users to undertake a comprehensive, transparent and fair appraisal of the conceptual and methodological quality of implementation research.

• Some limitations to the study design include the lack of public and patient involvement, due to lack of funding to involve patient and the public in the research.

Introduction

High-quality research is critical to knowledge accumulation and the advancement of scientific fields. Over the past decade, Implementation Science (IS) has benefited from notable efforts to advance the conceptual clarity of fundamental IS concepts and methodological guidance and recommendations to support applied health researchers and practitioners working within the field to design high-quality implementation research.^1–5 Such advances include, but are not limited to, the proposal of an effectiveness-implementation hybrid design typology,¹ an implementation theory and framework comparison and selection tool,⁶ a working taxonomy of implementation outcomes,³ taxonomies of implementation strategies,^{4 5 7} guidance to identify, select and tailor implementation strategies⁸ and repositories of implementation outcome instruments.^9–13

Despite these advances, however, practical guidance consolidating IS concepts and methodological guidelines and recommendations (eg, design decisions to inform the appropriate hybrid design selection) until recently was lacking. This gap, in part, is likely to have contributed to poor quality implementation research.^{14 15}

Recently, the Implementation Science Research Development (ImpRes) tool and supplementary guide were developed, with the explicit aim to address this gap,¹⁵ ImpRes was intended to support applied health researchers and those working within the field to design high-quality implementation research, and consequently help educate the next generation of IS researchers and build capacity within the field.¹⁵ Based on key conceptual and methodological literature containing design guidance and recommendations, and an expert consensus-building brainstorming process, ImpRes incorporates core IS principles and concepts that researchers should consider when designing IS research—including application of appropriate theories and/or frameworks, selection of implementation and other types of outcomes, development of stakeholder informed implementation strategies and evaluation of health economic elements of implementation efforts. Initial usability testing with end-users (ie, researchers with varying degrees of IS knowledge/expertise) showed that the ImpRes tool is useful for identifying project areas where implementation research is lacking and for improving the quality of implementation research.¹⁵

While ImpRes has the potential to contribute to filling a much-needed capacity-building gap, the need for a quantitative tool to appraise the quality of implementation research has recently been highlighted as a further area for development of the field.¹⁴

Practical tools to improve the quality of reporting have been shown to improve research reporting (eg, the development of the Consolidated Standards of Reporting Trials checklist, for the reporting of randomised controlled trials.^16–18 Research appraisal tools allow research stakeholders (eg, research grant panels and educators) to undertake a standardised, transparent, objective and fair appraisal.¹⁹

A previous attempt to use the traditional National Institutes of Health (NIH) scoring criteria to evaluate grant applications for implementation and improvement sciences projects, identified the need for evaluation criteria capable of identifying specific strengths and weaknesses of implementation studies.¹⁴ An initial effort to address this gap has recently been reported by Crable et al,¹⁴ who developed a scoring system, ImplemeNtation and Improvement Science Proposals Evaluation CriTeria (INSPECT), based on Proctor’s 10 key ingredients in high-quality implementation research grant proposals, to identify common deficiencies in implementation and improvement science research proposals from a grant application perspective.¹⁴

Another example of prior efforts to quantify the quality of implementation research, by some of the authors of this paper (CS, LG and LH), reported the initial development of a quantitative appraisal tool, based on the ImpRes tool and supplementary guide^{20 21} as part of a master’s dissertation project. Due to time constraints and scope of the master’s dissertation project, this initial development work focused on five of the 10 ImpRes domains: (1) implementation research characteristics; (2) implementation theories, frameworks and models; (3) determinants of implementation; (4) implementation strategies and (5) implementation outcomes. These domains were considered to be most relevant and specific to implementation research, whereas the other domains (eg, service and patient outcome), while still relevant to implementation research, overlap over research types (eg, effectiveness research).

This quantitative appraisal tool, structured as a rubric, applied analytic scoring to study protocols, published in Implementation Science, using a 4-point scale (ranging from ‘1’ indicating that the protocol is lacking detail and of suboptimal conceptual and methodological quality to ‘4’ indicating that the protocol provides explicit descriptions, justifications and citations from the literature and is of excellent conceptual and methodological quality). Initial development included applying the appraisal criteria to 16 implementation research protocols, published in Implementation Science, where all cumulative scores were expressed as a percentage of the total achievable score for that protocol, to indicate and allow IS protocols to be compared based on conceptual and methodological strength. The resulting intraclass correlation coefficient (ICC) was in the excellent inter-rater reliability (IRR) range: ICC: 0.85.²²

Here we build on this early-phase study by Sweetnam et al,^{20 21} and report a study that will develop a complete and comprehensive tool to appraise the conceptual and methodological quality of implementation research, termed the Implementation Science Research Project Appraisal Criteria (ImpResPAC) tool. The study aims to develop appraisal criteria for the remaining five ImpRes domains: (1) service and patient outcomes; (2) unintended consequences; (3) economic evaluation; (4) stakeholder involvement and engagement; (5) patient and public involvement and engagement and to refine the existing criteria developed by Sweetnam et al.^{20 21}

The specific objectives of the research are as follows:

1. To formulate an ImpResPAC expert advisory group to contribute to the refinement and content of ImpResPAC.

2. To develop a comprehensive and in-depth quantitative appraisal tool to be used by implementation research funders to appraise the conceptual and methodological quality of IS research: ImpResPAC.

3. To evaluate the psychometric properties (reliability and validity) and usability, including the acceptability, feasibility and appropriateness, of ImpResPAC.

ImpResPAC will complement but extend recent efforts by Crable et al,¹⁴ who developed and evaluated the ‘INSPECT’ tool. While overlap between INSPECT and ImpResPAC will exist, the two appraisal systems will differ notably in focus, depth of appraisal and the foundations on which they are based. For example, INSPECT primarily focuses on fundability because it is based on grant proposal criteria, whereas ImpResPAC, based on the ImpRes tool and guide, focuses on conceptual and methodological quality of implementation research. Furthermore, INSPECT operationalises the ‘key ingredients’ to writing implementation research grant proposals developed by Proctor et al,¹⁹ which operates specifically within the NIH proposal scoring framework,²³ whereas ImpResPAC will not be developed within the constraints of a single grant proposal scoring framework, thus its applicability will not be limited in this way.

Methods and analysis

We will conduct a multistage, mixed-methods study to develop, refine, and evaluate the psychometric strength of ImpResPAC.

Stage 1: ImpResPAC development (September 2021–November 2021)

ImpResPAC will map onto the 10 domains of the ImpRes tool and supplementary guide (see figure 1).

Figure 1.

ImpRes domains to be represented in ImpResPAC.¹⁵

As part of a previous study, five of the ImpResPAC domains were developed and IRR was assessed.²⁰ Formal quantitative psychometric testing of the content validity and concurrent validity of ImpResPAC was beyond the scope of this previous work. In this research, the five previously developed domains will be subject to refinement within the tool development stage of this study, and the remaining five domains will be developed by the ImpResPAC development/research team.

Stage 2: ImpResPAC content validation and refinement (December 2021–December 2022)

To ensure that ImpResPAC is face and content valid, we will use purposive sampling to form an ImpResPAC expert advisory group, consisting of a number of eminent academics across the world that have made a significant contribution to the conceptual and methodological advancement of one or more of the ImpResPAC domains. Experts in each domain will be asked to review and provide feedback, including modifications and suggestions for improvement, on the ImpResPAC domain(s) that they have expertise in.

We define an expert as ‘someone widely recognized as a reliable source of knowledge, technique, or skill whose judgment is accorded authority and status by the public or his or her peers’.²⁴ The ImpResPAC development/research team will generate a list of experts that meet the above criteria, based on our collective knowledge. Once experts have agreed to participate in the study, we will encourage them to nominate additional experts, that is, snowballing technique, whose contribution would be valuable. Once experts agree to participate, they will have the option to be recognised as a contributor in the study or for their participant to remain anonymous. We expect to identify 70–100 experts globally in the field of implementation science. We hope experts, both academics and practitioners, working in high-income, middle-income and low-income countries will participate.

Using surveys, the expert advisory group will review ImpResPAC domain(s) and items for content, style and comprehensiveness. Members of the expert advisory group will be presented with an overview of ImpResPAC, ImpResPAC user instructions, the ImpResPAC domain(s) that they are an expert in, survey instructions and survey questions. The survey will be attached in an email to experts.

Experts will be asked to review the overview of ImpResPAC, ImpResPAC user instructions and ImpResPAC domain(s) and associated items for the domain(s) that they agree they are ‘experts’ in. Members of the expert advisory group will have 4 weeks to complete the survey. A reminder email will be sent 2 weeks after the survey is first sent and 1 week before the 4-week deadline.

The development/research team will collate and review all comments and suggested refinements to ImpResPAC and refinements will be decided via group discussions until consensus is reached. Once ImpResPAC is finalised, we will quantitatively assess the acceptability, appropriateness and feasibility of ImpResPAC. All members of the ImpResPAC expert advisory group will be invited to review the refined version ImpResPAC and provide feedback on the acceptability, appropriateness and feasibility of ImpResPAC (all domains) via a follow-up survey. Experts will be given the option of providing feedback on the domains that they provided feedback on in stage 1 (survey A) or if they wish, providing feedback on the entire tool. See online supplemental additional file 1 for survey questions.

Stage 3: Application and psychometric evaluation of ImpResPAC (January 2023–July 2023)

ImpResPAC, developed in stage 1 and content validated and refined based on expert feedback in stage 2, will be applied to 50 research protocols published in Implementation Science to evaluate its psychometric strength.

Two of the study authors (CS and LH), with expertise and experience in implementation and improvement science research, will independently appraise the conceptual and methodological quality of the 50 most recently published research protocols published in Implementation Science, using ImpResPAC. We decided to appraise research protocols published in Implementation Science as it is the most well established (since 2006), highest impact factor journal in the field and regarded, by researchers, practitioners and funders as a key source for dissemination and implementation research in health.²⁵ Furthermore, Implementation Science publishes research covering a broad array of content areas and settings, making it an ideal test bed for ImpResPAC.

Inclusion criteria

Study protocols that describe the following:

1. Effectiveness-implementation hybrid design studies (ie, a study design that takes a dual focus in assessing clinical effectiveness and implementation).¹

2. Implementation research studies (ie, research focused on the adoption or uptake of clinical interventions by providers and/or systems of care).¹

Exclusion criteria

Study protocols/proposals that describe the following:

1. Theoretical or methodological research (eg, theory development and measurement development), where implementation of an evidence-based intervention is not planned

2. Deimplementation studies of interventions found to be of low value, wasteful or clinically ineffective. The field of deimplementation is expanding rapidly, and although there have been recent attempts to theorise the deimplementation process,²⁶ and the field is still in infancy.²⁷ As such consensus regarding deimplementation and research guidance is lacking and further methodological development is still necessary.²⁸ For this very reason, this subsection of IS was not included in the ImpRes tool and guide and will also not be included in ImpResPAC.

Assessment of the validity and reliability of ImpResPAC

We will employ an item exploratory factor analysis (EFA) to the polychoric matrix of the 10 ImpResPAC domains to determine and confirm scale factor structures (construct validity). A varimax rotation will be applied to improve the interpretability of the factors obtained. We will use three criteria to select the final factors: (1) the scree plot (2) eigenvalues>1 and (3) >90% of total variance explained by the factors. ImpResPAC will be applied to 50 protocols for pragmatic reasons, as this equates to the minimum number of observations (50), required when conducting EFA.²⁹

Convergent validity will be further examined by estimating the correlation between the global ImpResPAC dimension with the global scores of INSPECT¹⁴ as both scoring criteria rate the quality of proposed implementation science research. Spearman’s correlation coefficients will be calculated and interpreted as follows: >0.90: excellent relationship, 0.71–0.90: good, 0.51–0.70: fair, 0.31–0.50: weak and<0.30: none.³⁰

We are expecting fair to good correlations, as excellent correlations would indicate that ImpResPAC is a duplication of INSPECT. A comparison of ImpResPAC and INSPECT domains, presented in supplementary material, indicates clear similarities between a number of domains (eg, ‘Theories, frameworks and models’ domain of ImpResPAC and ‘Conceptual model and theoretical justification’ element of INSPECT), a degree of similarities between some domains (eg, Determinants of implementation: contextual factors’ domain of ImpResPAC and ‘Feasibility of proposed research design and methods’ element of INSPECT) and no apparent similarities between some domains (eg, ‘Patient and Public Involvement’ domain of ImpResPAC, which has no similarities to INSPECT elements). Given the varying degrees of content overlap between ImpResPAC and INSPECT domains, as described in detail above, we hypothesise that there will be a fair to good relationship (correlation coefficient r: 0.31–0.70) between global ImpResPAC and INSPECT scores.

Cronbach’s alpha coefficient will be used to evaluate the reliability (internal consistency) of the 10 domains of ImpResPAC, as it evaluates the extent to which the domains within a scale are intercorrelated with one another and thus seem to measure the same concept. Its value ranges from 0 to 1 and internal consistency is suggested to be acceptable when Cronbach’s alpha is at least 0.70.³⁰ Inter-rater reliability will be assessed using Criterion of Lin’s ρ≥0.70 to indicate acceptable reliability. A weighted kappa score will also be calculated for each ImpResPAC domain to provide details on the test–retest and inter-rater reliability. A criterion of weighted kappa≥0.40 will be used to indicate acceptable domain level reliability. Precision will be assessed to test how well each domain fits within its proposed scale.³⁰ Corrected domain-total correlations of <30 will indicate poor fit of items within the ImpResPAC total score.³⁰ Each ImpResPAC item will be correlated both with its own global domain score total and with the other global domain totals. Each component will require higher correlation with its own domain than other ImpResPAC domains to demonstrate precision.

Patient and public involvement

Patients or the public were not involved in the design, conduct or reporting plans of this research.

Discussion

This study will develop, refine, content validate and evaluate the psychometric strength (ie, the reliability and validity) of an expert derived tool, ImpResPAC, to appraise the conceptual and methodological quality of implementation research. The proposed research will fill an important gap in our ability, as a field, to conduct a comprehensive, transparent, systematic and in-depth quantitative appraisal of implementation research. Purposively sampling experts to form an international ImpResPAC expert advisory group to refine and content validate ImpResPAC, will ensure appropriate appraisal criteria, relevant to the conceptual and methodological quality of implementation research, is developed, which will allow an in-depth, comprehensive appraisal of implementation research. Feedback on the acceptability, feasibility and appropriateness of ImpResPAC will also be sought from the ImpResPAC expert advisory group.

Previous research suggests that researchers seeking to design implementation research find it challenging to distinguish between implementation research and efficacy and effectiveness research and consequently fail to design high-quality implementation research.⁴ With the availability of the ImpRes tool and supplementary guide, consolidating methodological guidelines and recommendations, researchers, practitioners and students are better equipped to design high-quality implementation research proposals. We envisage ImpResPAC primarily being used by funding bodies as a standardised and transparent method to differentiate high-quality and low-quality implementation research and identify areas for improvement before funding decisions are made. In addition, we also envisage that ImpResPAC will be useful to educators who are tasked with appraising implementation projects submitted by students/learners, especially in educational settings where the ImpRes tool and guide informed the curriculum. We plan to explore whether another potential application of ImpResPAC would be for implementation researchers, practitioners and students/learners to use ImpResPAC as a quality assurance step, to self-assess a funding application or implementation project, prior to submission.

Although INSPECT already exists as a standardised appraisal tool for implementation research proposals, we plan to develop a complementary, yet conceptually distinct tool that focuses exclusively on conceptual and methodological quality of IS research proposals. As such, ImpResPAC scoring domains will differ to INSPECT domains, as highlighted in supplementary material (online supplemental additional file 2). For example, team experience with setting, treatment and implementation process is one of the 10 elements of the INSPECT tool, however the ImpRes tool and supplementary guide, and consequently ImpResPAC, will not contain criteria measuring this domain as team experience is not a direct measure of conceptual or methodological quality of IS research. Similarly, ImpResPAC will contain criteria that INSPECT does not explicitly appraise. For example, ImpResPAC will appraise whether research teams plan to evaluate unintended consequences of implementation in addition to exploring and quantifying the anticipated benefits of implementation. Furthermore, the level of detail at which implementation research will be appraised using the two scoring systems will differ substantially. For example, INSPECT provides an overall appraisal of the measurement and analysis of IS research proposals, however the ImpRes guide, and consequently ImpResPAC, will contain three domains relating to measurement and analysis; (1) service and patient outcomes; (2) implementation outcomes and (3) economic evaluation, providing a much more detailed and focused appraisal of the outcomes typically assessed in implementation research. The initial mapping of the ImpRes tool and supplementation guide to develop the ImpResPAC tool (stage 1) and a detailed comparison of ImpResPAC tool domain items (initial mapping) and the INSPECT tool element items can be found in supplementary material (online supplemental additional file 2).

INSPECT operationalised grant proposal criteria proposed by Proctor’s et al ‘key ingredients’, which were developed nearly a decade ago (ie, 2012),¹⁹ whereas ImpResPAC will identify conceptual and methodological strengths and weakness in IS projects taking account of the conceptual and methodological developments that have taken place in more recent years. As such, ImpResPAC will include and operationalise key methodological guidelines and recommendations that simply did not exist nearly a decade ago.1 8 10 31–37 ImpResPAC will operationalise, for example, the key methodological and conceptual guidelines and recommendations that have been described in the ImpRes tool and guide, as well as guidelines suggested by our international expert advisory panel, and key literature published since the development of the ImpRes tool and guide.

This study has a number of limitations. We acknowledge the importance of public and patient involvement in the design of implementation research, but the study we report here is not funded and did not have the funds to involve patient and the public in the research. We strongly recommend that any future ImpResPAC research, including further validation and utilisation, includes patient and public involvement. Second, we acknowledge that in order to truly test the value of ImpResPAC, it will be preferable to seek feedback from implementation research stakeholders who have had the opportunity to apply the tool in practice, but this is beyond the scope of this research. Future studies should evaluate the value of ImpResPAC with implementation research stakeholders who have applied the tool. Third, our definition of ‘experts’ (someone widely recognised as a reliable source of knowledge, technique or skill whose judgement is accorded authority and status by the public or his or her peers) could exclude useful perspectives of stakeholder groups. Finally, although the implementation research protocols that will be appraised, using ImpResPAC, will cover a broad range of content areas and settings, appraising protocols published in Implementation Science is likely to positively skew the results (ie, it is fair to assume that only high-quality IS protocols will have been published in Implementation Science). This is a specific and inherent challenge with the planned research, as access to implementation research protocols rejected from journals and unsuccessful grant proposals submitted to funding bodies are not publicly available and unattainable for obvious reasons.

High-quality implementation research is key to advancing the field and improving the adoption, implementation, sustainment and scale-up of evidence-based interventions. This research will advance the field by developing a quantitative appraisal tool, which we believe will be of immediate use and value to IS research stakeholders (eg, grant reviewers and educators), to undertake a comprehensive, transparent and fair appraisal of the conceptual and methodological quality of implementation research.

Ethics and dissemination

This study will involve human participants. This study has been registered and minimal risk ethical clearance granted by The Research Ethics Office, King’s College London (reference number MRA-20/21-20807). Participants will receive written information on the study via email and will provide e-consent if they wish to participate. We will use traditional academic modalities of dissemination (eg, conferences apublications).

Ethics statements

Patient consent for publication

Not required.