Identifying optimal frameworks to implement or evaluate digital health interventions: a scoping review protocol

Abstract

Introduction

Digital health interventions (DHIs) are defined as health services delivered electronically through formal or informal care. DHIs can range from electronic medical records used by providers to mobile health apps used by consumers. DHIs involve complex interactions between user, technology and the healthcare team, posing challenges for implementation and evaluation. Theoretical or interpretive frameworks are crucial in providing researchers guidance and clarity on implementation or evaluation approaches; however, there is a lack of standardisation on which frameworks to use in which contexts. Our goal is to conduct a scoping review to identify frameworks to guide the implementation or evaluation of DHIs.

Methods and analysis

A scoping review will be conducted using methods outlined by the Joanna Briggs Institute reviewers’ manual and will conform to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews. Studies will be included if they report on frameworks (ie, theoretical, interpretive, developmental) that are used to guide either implementation or evaluation of DHIs. Electronic databases, including MEDLINE, EMBASE, CINAHL and PsychINFO will be searched in addition to grey literature and reference lists of included studies. Citations and full text articles will be screened independently in Covidence after a reliability check among reviewers. We will use qualitative description to summarise findings and focus on how research objectives and type of DHIs are aligned with the frameworks used.

Ethics and dissemination

We engaged an advisory panel of digital health knowledge users to provide input at strategic stages of the scoping review to enhance the relevance of findings and inform dissemination activities. Specifically, they will provide feedback on the eligibility criteria, data abstraction elements, interpretation of findings and assist in developing key messages for dissemination. This study does not require ethical review. Findings from review will support decision making when selecting appropriate frameworks to guide the implementation or evaluation of DHIs.

Strengths and limitations of this study.

• To our knowledge, this is the first scoping review to identify frameworks to implement or evaluate digital health interventions on a broad scale.

• The study protocol was informed by rigorous and established methods as suggested by the Joanna Briggs Institute approach for scoping reviews and adheres to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews.

• Digital health knowledge users, such as policymakers, researchers, clinicians and developers have been engaged in the design and development of the review since its inception to ensure relevance and scope of project.

• This scoping review will not examine the usability of the frameworks, as such our findings will be limited to descriptive syntheses.

• Findings stemming from this review will provide practical guidance for digital health knowledge users and enable them to use evidence informed approaches to select optimal frameworks to implement or evaluate digital health interventions.

Introduction

Frameworks help to systematically organise and link research objectives or constructs, and provide useful insights in quantitative and qualitative analyses, which can inform interpretation or decision making.^{1 2} The Medical Research Council (MRC) categorises frameworks into four distinct groups: (1) development frameworks, which can model processes and outcomes; (2) feasibility frameworks, which can guide pilot testing of an intervention; (3) implementation frameworks, to guide evidence into clinical practice and (4) evaluation frameworks, to determine intervention effectiveness.³

A recent scoping review, identified over 159 knowledge translation frameworks to guide implementation and evaluation of health interventions in clinical practice settings, presenting a plethora of options for the implementation and evaluation of digital health interventions (DHIs).⁴ Implementation and evaluation frameworks present an opportunity to address gaps relating not only to whether an intervention works but provide actionable insights for how to support their uptake in practice.

DHIs differ from traditional health interventions such as implementing a new programme or evaluating drug effectiveness. DHIs include any health service or treatment delivered using technology that aims to facilitate, capture or exchange knowledge.⁵ Examples of DHIs include electronic medical records, mobile applications or wearable sensors for remote monitoring. DHIs are complex, differ both in intended functionality (eg, self-management support vs data sharing) and intended users (eg, patients vs providers). DHIs are not static; instead the interaction between the technology, end-user and the healthcare team and setting is by its nature dynamic and thus can vary substantially over time.⁶ Given the unique sociotechnical aspects of DHIs, it remains unclear which frameworks can be appropriately applied in this emerging field.

This paper outlines the protocol for a scoping review to identify frameworks to guide the implementation or evaluation of DHIs. Specifically, our objectives are to:

1. Describe the attributes of existing frameworks that have been used to guide the implementation or evaluation of DHIs.

2. Identify the proposed role of each framework, including the constructs and mechanisms they target.

3. Describe how each framework has been applied in primary studies, if applicable.

The results of this review will provide practical guidance and support for researchers, clinicians, policymakers and developers in selecting the most appropriate framework for DHIs, which will support evidence-based approaches in relation to implementation and evaluation efforts.

Methods and analysis

We will conduct a scoping review to comprehensively search the literature, ‘map’ the evidence and identify gaps in the research knowledge base.^{7 8} The study will be conducted using established methods outlined by the Joanna Briggs Institute reviewers’ manual⁷ and reporting will conform to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews.⁹ This protocol is registered on Open Science Framework (OSF) and is available at https://osf.io/8jydm/. OSF is an open source platform where researchers can share protocols, data and contributes to transparency of research.¹⁰

Eligibility criteria

Studies reporting on the development or application of frameworks (ie, theoretical or interpretive) to guide implementation or evaluation of DHIs in healthcare will be included. We will use the WHO definition of health which encompasses physical, mental and social well-being and spans across multiple disciplines such as psychology, sociology or medical sciences.¹¹ DHI was defined as any health service or treatment delivered using technology that aims to facilitate, capture or exchange knowledge (formally or informally).⁵ DHI definition was generated from a search of the literature and consultations with digital health knowledge users, including policymakers, researchers, clinicians and developers. Implementation frameworks will be operationalised according to MRC guidance, as frameworks that aim to guide research into practice, which can include development, feasibility and dissemination frameworks.³ Evaluation frameworks will be defined as frameworks that focus on determining the effectiveness of DHIs, which includes measuring outcomes and understanding processes or mechanisms of action.³ No limitations will be placed on user population, comparators, study design, publication status or geographic region to enhance the comprehensiveness of our results and avoid unintended exclusion of relevant studies. Conference abstracts/proceedings and white papers will be included. We will include studies reported in other languages and use appropriate tools (ie, Google translate, translation services, contact author) to assess inclusion. Commentaries and studies examining mathematical or statistical frameworks will be excluded.

Information sources

An experienced information specialist developed the literature search in consultation with the multidisciplinary research team. The search will be peer reviewed by a second information specialist using the Peer Review of Electronic Search Strategy (PRESS) checklist to ensure the search is comprehensive and maximises appropriate search terms.¹²

We will search MEDLINE, EMBASE, CINAHL and PsychINFO using keywords such as ‘digital health’ and ‘framework’. Additional search terms were drawn from multiple disciplines such as psychology, nursing, sociology and medicine to ensure comprehensiveness. The databases will be searched from inception to present and the search strategy is presented in online supplementary appendix 1. We chose not to use the BeHEMoTh (behaviour of interest, health context, exclusions and models or theory) approach¹³ as specified in our OSF registration. Although this approach has been successful in identifying frameworks in knowledge translation,⁴ it did not prove to be a feasible approach in our scoping review as it yielded a vast number of citations with limited specificity related to our objectives. We used a simplified heuristic, which included identifying DHIs in various healthcare contexts, adding terms for frameworks and removing exclusions such as animal studies (online supplementary appendix 1).

The search strategy will be supplemented by a search for grey literature using the checklist suggested by the Canadian Agency for Drugs and Technologies in Health.¹⁴ Specifically, we will search for white papers or benefit evaluation studies through health technology assessment agencies such as Agency for Healthcare Research and Quality and National Institute for Health and Care Excellence, Canada Infoway and other relevant organisations involved in providing guidance on delivery of healthcare services. We will use keywords such as ‘digital health’, ‘frameworks’ and ‘benefits evaluation’ to refine our supplementary search. In addition, we will also scan reference lists of included studies and conduct a forward citation search (ie, examine studies that reference the included studies) in Web of Science using the cited reference search feature. This will ensure our approach is comprehensive.

Eligibility screening process

Citations obtained from the literature search will be uploaded to Covidence,¹⁵ a systematic review software program which organises citations, enables screening of citations by multiple reviewers and identifies discrepancies. We will apply a two-step process for identifying relevant citations. At level 1, titles and abstracts will be assessed using the eligibility criteria (online supplementary appendix 2). Studies with abstracts fulfilling criteria will be passed to level 2 where the eligibility criteria will be applied to the full text articles.

Prior to screening, a pilot test will be completed using a random sample of 10% of citations or full text articles, with the expressed purpose of assessing agreement between reviewers at each level. Specifically, percent agreement will be used to assess agreement among reviewers (inter-rater reliability ≥80% will be considered adequate). If agreement is not reached, a second pilot will be conducted with another random sample of 10%. A third reviewer will mediate any disagreements. Citations and full text articles will be screened in duplicate by two reviewers.

Data items and abstraction process

Studies fulfilling the eligibility criteria will be abstracted in Excel. We will extract the following study characteristics for the identified frameworks: name, reference, theory associated with framework (if applicable), description of its components or constructs and its application in research (or stage of research to which it was applied, if applicable). For studies outlining the application of a framework, additional characteristics will be abstracted such as the type of DHI, healthcare setting, method of application and nature and directionality of the results. We will abstract information such as name of the framework, the role of framework in study (ie, development, feasibility/pilot testing, implementation, evaluation), components of the framework that were used, type of DHI, the objective of the study (if applicable) and healthcare setting from included studies.

Methodological appraisal

We will not assess the quality of included articles in the scoping review (consistent with Joanna Briggs Institute reviewer’s manual⁷ as our purpose is to gain an overview of frameworks used in relation to DHIs and not to assess the quality of their application.

Ethics and dissemination

This scoping review is focused on published reports and studies of DHI and does not involve patients or primary data collection; as such, no formal ethics approval is required.

The dissemination plan will be tailored to end-users and will include passive and interactive strategies such as peer-reviewed publications, conference events and other network events with digital health knowledge users. To ensure broader reach, we will also disseminate our findings through social media platforms, and public-facing communications such as one-page briefs released on the Women’s College Institute for Health Innovation website at Women’s College hospital.

Patient and public involvement

We employed an integrated knowledge translation strategy to engage digital health knowledge users in the review process to ensure the scope of the project met the needs of various end-users. Knowledge users are defined as individuals who are likely to use the findings to inform health decision making.² A priori, we decided to engage senior leaders and policymakers at organisations that promote or support implementation of digital health solutions, as well as researchers, clinicians and developers evaluating DHIs in real-world settings. An advisory panel of digital health knowledge users was established to provide input at strategic phases of the scoping review.

Potential panelists were identified through organisational networks and were invited to participate via email. Six members agreed to participate (CSG, TS, HCW, JZ, SM, DL) on the advisory panel. Panelists and the research team convened a meeting and discussed the strategic steps and opportunities for involvement and input in the review. Specifically, the advisory panel will support refinement of inclusion criteria, prioritisation of data abstraction elements, assist in interpretation of findings and develop dissemination strategies. Panelists have national and international networks that will ensure the scope of the review reflects the knowledge needs of a diverse audience, which is directly in line with the stated aim of providing practical guidance on the selection and application of frameworks for DHIs. As the intended audience of this paper does not include patients and members of the general public, they were not included as part of the advisory panel. The perspectives of patients and the general public will be incorporated through their participation and involvement in the respective studies included as part of this review.

Analysis

Included studies will be summarised using qualitative description, an approach that seeks to create an understanding of phenomenon through accessing the meanings ascribed by authors.¹⁶ Descriptions of individual frameworks will be organised by key categories, including study design, report type (published vs non-published), methodological approach (ie, how the framework is intended to be applied) and application papers (ie, how the framework has been applied in practice). We will then synthesise findings by mapping core components of the frameworks and examining how research objectives and type of DHIs are linked to the framework. Categorisation will use language directly from included studies, where possible, and authors will be contacted when information is not present or unclear. The advisory panel will guide the synthesis of findings by providing input on the level of detail abstracted from included articles and provide input on categorisation of frameworks, where appropriate.

Strengths and limitations

To our knowledge, this is the first scoping review to examine the use of frameworks to guide implementation or evaluation of DHIs on a broad scale. The protocol was generated using established methods for the conduct of scoping reviews and informed by input from digital health knowledge users to define scope and ensure the relevance of the project. A clear understanding of which frameworks can be used for development, feasibility, implementation and evaluation of DHIs will facilitate decision making by making evidence-based approaches available to policymakers, clinicians and developers. Additionally, this guidance will support researchers in identifying appropriate frameworks with the goal of establishing consistency across studies, minimising duplication and accelerating scientific progress.

Given the breadth of this scoping review, we anticipate a few key challenges. The first relates to the inconsistent and often ill-defined nature of DHIs and frameworks. To be inclusive, we have defined DHIs broadly as any health intervention that can be delivered using technology to ensure we capture frameworks that are currently being used across formal (eg, care delivered within the walls of a healthcare organisation) and informal settings (eg, direct to consumer technologies). Moreover, use of the term framework itself also creates challenges. For the purposes of this scoping review, we have defined a framework as a tool to systematically organise and link research questions or constructs, but a range of terms are often used synonymously (eg, models or processes). To account for this variability, we will include studies reporting on ‘models’ and work closely with the advisory panel to confirm whether the reported framework aligns with our a priori definition, as well as the needs of relevant digital health knowledge user groups.

Second, we also anticipate challenges searching the literature as a product of inconsistent terminology outlined above. We have constructed our search to balance comprehensiveness and specificity, working closely with an information specialist to ensure the number of citations are focused and feasible. Several iterations of the literature search were conducted, specifically, we added in keywords and removed them in a stepwise fashion to understand the impact on specificity and sensitivity of our search. Through this iterative process, we developed our search strategy, which was then peer reviewed using the PRESS checklist; however, we anticipate additional challenges when screening.

Third, we anticipate challenges arising from poor reporting or limited description, as evidenced by previous studies.^{17 18} Authors may not provide sufficient details on the frameworks they use or their method of application.¹⁹ To mitigate this, we will contact authors to obtain additional information whenever information is missing or unclear.

Finally, we anticipate that some included frameworks will have a dual purpose of addressing implementation and evaluation or may contain components that lend themselves to both constructs. We will convene with the advisory committee on a quarterly basis to discuss these issues as they arise and will devise the most appropriate plan for analysis through group consensus.

Overall, identification of frameworks will serve as a guide for researchers, clinicians, policymakers and developers of DHIs by providing practical guidance on which frameworks may be most appropriate for which objectives (ie, implementation or evaluation). In parallel, the results will contribute to a more nuanced understanding of how to evaluate and implement DHIs, including the identification and understanding of key constructs.