Evaluating digital health interventions: key questions and approaches

Abstract

Digital health interventions (DHI) have enormous potential as scalable tools to improve health and healthcare delivery by improving effectiveness, efficiency, accessibility, safety and personalisation. Achieving these improvements requires a cumulative knowledge base to inform development and deployment of DHI. However, evaluations of DHI present special challenges. This paper aims to examine these challenges and outline an evaluation strategy in terms of the Research Questions (RQs) needed to appraise DHIs. As DHI are at the intersection of biomedical, behavioural, computing and engineering research, methods drawn from all these disciplines are required. Relevant RQs include defining the problem and the likely benefit of the DHI, which in turn requires establishing the likely reach and uptake of the intervention, the causal model describing how the intervention will achieve its intended benefit, key components and how they interact with one another, and estimating overall benefit in terms of effectiveness, cost-effectiveness and harms. While Randomised Controlled Trials (RCTs) are important for evaluation of effectiveness and cost-effectiveness, they are best undertaken only when: a) the intervention and its delivery package are stable; b) these can be implemented with high fidelity and c) there is a reasonable likelihood that the overall benefits will be clinically meaningful (improved outcomes or equivalent outcomes at less cost). Broadening the portfolio of RQs and evaluation methods will help with developing the necessary knowledge base to inform decisions on policy, practice and research.

Background & Aims

There is enormous potential for digital health interventions (i.e. interventions delivered via digital technologies such as smartphones, website, text messaging) to provide effective, cost-effective, safe, and scalable interventions to improve health and healthcare. Digital health interventions (DHI) can be used to promote healthy behaviours (e.g. smoking cessation, ^{1, 2} healthy eating, ³ physical activity, ⁴ safer sex ⁵ or alcohol consumption ⁶), improve outcomes in people with long term conditions ⁷ such as cardiovascular disease (McClean in press), diabetes ⁸ and mental health conditions ⁹ and provide remote access to effective treatments, for example computerised cognitive behavioural therapy for mental health and somatic problems. ^10–14 They are typically complex interventions with multiple components, and many have multiple aims including enabling users to be better informed about their health, share experiences with others in similar positions, change perceptions and cognitions around health, assess and monitor specified health states or health behaviours, titrate medication, clarify health priorities and reach treatment decisions congruent with these, and improve communication between patients and health care professionals (HCP). Active components may include information, psycho-education, personal stories, formal decision aids, behaviour change support, interactions with HCP and other patients, self-assessment or monitoring tools (questionnaires, wearables, monitors, and effective theory-based psychological interventions developed for face-to-face delivery such as cognitive behavioural therapy or mindfulness training.

To date, the potential of DHIs has scarcely been realized, partly because of difficulties generating an accumulating knowledge-base for guiding decisions about DHI’s. Difficulties include the rapid change of the wider technology landscape (Patrick 2016), which requires DHI’s to constantly evolve and be updated just to remain useful, let alone improve. For example, imagine an iPhone app promoting physical activity, with development and evaluation starting in 2008. Results from a randomized controlled trial may not be published till 5–6 years later, by which time the iPhone operating system (iOS) had undergone substantial changes to functionality, design, and overall use. These operating system changes would result in the evaluated app feeling out-of-date at best and non-functional at worst. As such, the knowledge gained from that efficacy trial would be minimally useful for supporting current decisions about using that app. Other difficulties include the idiosyncratic wants and needs of users and the influence of context on effectiveness.

However, the public, patients, clinicians, policy-makers and healthcare commissioners all have to make decisions on DHI now, and researchers need to support such decision-making by creating an actionable knowledge base to identify the most effective, cost-effective, safe, and scalable interventions (and components) for improving individual and population health. These decisions are particularly important in resource-constrained contexts.

This paper explores issues that arise in developing an accumulating knowledge base around DHI, and how this knowledge can be generated in a timely manner, using scarce resources efficiently. The approach is pragmatic, with a focus on decision-making and moving the science forward, generating cumulative knowledge around identifying important components and working out how to test them with a view to improving the quality and effectiveness of DHI and the efficiency of the research process. This paper is written from the perspective of a body charged with appraising evidence for using specific DHI within a publically-funded, resource-limited health system, such as the UK National Institute for Health and Care Excellence (NICE).

This paper does not seek to provide detailed analysis of appropriate design features of evaluation studies such as choice of comparators, outcome measures, mediator and moderator variables, study samples, or the occasions when particular study designs are a better fit with the evaluation context. These are important issues for which a literature is beginning to emerge. ^{15, 16}

Paper structure

The paper starts by defining the Research Questions (RQ) which, in our opinion, should form the basis for an appraisal of a DHI (Table 1). It then considers appropriate research methods for each of these RQ. Where the appropriate methods are largely similar to those used in research of other (non-digital) complex interventions, readers are referred to the appropriate references. Where there are novel or specific issues which arise, or are particularly salient, in evaluation of digital health interventions, the main areas of consideration for each issue are outlined. Throughout, the paper emphasises that the RQ apply not just to the digital components of the DHI, but also the surrounding “delivery package”. This package will vary according to the nature and functions of the DHI, but often requires as much thought and study as the DHI itself. Example components of delivery packages could include system redesign where use of DHI becomes standard clinical practice, ¹⁷ ad hoc referral from a clinician, ¹⁸ supported access (eg. face-to-face, ¹⁹ by telephone, ²⁰ or email ²¹), hosting on a trusted portal (e.g. NHS Choices), marketing via public health campaigns, or embedding in a social network.

Table 1.

Key Research Questions for an appraisal of a DHI.

Defining the problem 1Is there a clear health need which this DHI is intended to address? 2Is there a defined population which could benefit from this DHI?

Defining the likely benefit of the DHI 3Is the DHI likely to reach this population, and if so, is the population likely to use it? 4Is there a credible causal explanation for the DHI to achieve the desired impact? 5What are the key components of the DHI? Which ones impact on the predicted outcome, and how do they interact with each other? 6What strategies should be used to support tailoring the DHI to participants over time? 7What is the likely direction and magnitude of the effect of the DHI or its components compared to a comparator which is meaningful for the stage of the research process? 8How confident are we about the magnitude of the effect of the DHI or its components compared to a comparator which is meaningful for the stage of the research process? 9Has the possibility of harm been adequately considered? And the likelihood of risks or adverse outcomes assessed? 10Has cost been adequately considered and measured? 11What is the overall assessment of the utility of this intervention? And how confident are we in this overall assessment?

Decisions to be made based on our current knowledge 12Should we change research priorities? 13Should we change clinical practice?

Research Questions

Defining the problem

• 1Is there a clear health need which this DHI is intended to address?
• 2Is there a defined population who could benefit from this DHI?

As with any complex intervention, consideration of the likely benefits of a digital health intervention starts with a detailed and often theory-based characterisation of the nature of the problem and the context in which the intervention will be used. ^22–24

Defining the likely benefit of the DHI

• 3Is the DHI likely to reach this population, and if so, is the population likely to use it?

The concepts of reach, uptake and context are particularly salient for DHI, as impact and cost-effectiveness are highly dependent on the total number of users (McNamee 2016), and effectiveness may be highly dependent on context. For example, effects seen when a DHI is used in a controlled environment (laboratory or clinical office) may not be replicated if used in the “wild”, with many competing demands on user’s attention. An important consideration is whether a DHI is accessible across a range of commonly used operating systems and devices and is interoperable with other healthcare information systems, such as electronic health records (EHRs). Hence an early component of any evaluation of a DHI should be a determination and optimisation of reach and uptake by the intended population, in the context in which the DHI will be used. This will often require iterative adaptations both to the DHI itself (e.g. to improve usability, acceptability) and to the “delivery package” around the DHI. For many DHI, ‘users’ will include healthcare professionals (HCPs) who ‘prescribe’ the DHI and monitor outcomes. Hence RQs 3 - 6 require work with HCPs as well as patients or the public.

Establishing and optimising potential reach and uptake requires methods used in engineering and computer science, collectively referred to as human-centred design ^25–27. These include concept sketching, ²⁸ co-design strategies ²⁵ and low-fidelity or “Wizard-of-Oz” prototyping, ^29,30 and user experience testing. ²⁷ In the business world there is increasing interest in “lean” principles ³¹ that specify methods for early-stage testing of features related to feasibility ³² including:

• Acceptability and usability (will the target audience (e.g. patients, HCPs) incorporate and sustain the intervention into their lives/clinical practice?);

• demand, (will relevant stakeholders use it?);

• implementation, (will it have high fidelity within real-world use?);

• practicability, (can it be delivered with minimal burden?);

• adaptation, (can it be adapted to novel contexts without compromising fidelity and integrity?); and

• integration, (can it be integrated successfully into existing healthcare systems?).

• 4Is there a credible causal explanation for the DHI to achieve the desired impact?

Establishing a credible causal explanation for the DHI is essential and must address not only the DHI, but also the “delivery package”. For example, if there is a human support element, is that element aimed entirely at improving engagement with the DHI, or will there be additional therapeutic content embedded in the human support? Are there important issues around the credibility or authority invested in those that deliver the human support? See (Hekler 2016) and (Yardley 2016) for further discussion.

• 5What are the key components of the DHI? Which ones impact on the predicted outcome, and how do they interact with each other?

Understanding which components actually have the predicted impact on the outcome, and whether and how components interact, is critical. Most DHI are highly complex interventions, containing multiple components so the development process needs to include a period of optimisation. This entails evaluating the performance of individual components of the intervention, and how the presence, absence, or setting of one component impacts the performance of another. One efficient method is the Multiphase Optimisation Strategy (MOST), ^{33, 34} which involves establishing a set of components that are candidates for inclusion, specifying an optimization criterion for the entire intervention, and then collecting experimental data to identify the subset of components that meet the criterion. Here the term “component” is broadly defined, and may refer to aspects of the content of the intervention, including any human input; ³⁵ factors affecting compliance with, adherence to, fidelity of, or scalability of the intervention; ³⁶ variables and decision rules used to tailor intervention strategy, content, or intensity to individuals; ³⁷ or any aspect of an intervention that can profitably be separated out for examination. Two example optimization criteria are: the most effective intervention that can be delivered for < £100 per participant; or most effective intervention that requires no more than one hour per week of participant time.

The experimental approaches used for optimization include full or fractional factorial experiments ^{38, 39}, the sequential multiple-assignment randomized trial (SMART), ⁴⁰ and system identification techniques ^{41, 42}. The factorial experimental design can be a useful and economical approach for examining the effects of individual intervention components, and is the only experimental design that enables full examination of all interactions. ³⁸ For further discussion see Collins et al. ^{38, 43}

• 6What strategies should be used to support tailoring the DHI to participants over time?

Where the research question focuses on tailoring the DHI to participants over time (e.g., non-responders, or daily adjustments reflecting changing needs or context) a SMART design, micro-randomized trial, or system identification experiment may be appropriate. A SMART is a special case of the factorial experiment involving randomization at several stages, where each stage corresponds to one of the decisions that must be made about adapting the intervention, and some or all of the randomization may be contingent on response to treatment. ^{34, 40, 44}

System identification approaches are used in engineering to obtain dynamic systems models; these in turn are the basis for the design of control systems which achieve optimization. ⁴⁵ System identification experiments are inherently idiographic in nature, and work best when planned changes (preferably random or pseudo-random in nature) are introduced to adjustable components of an intervention (e.g. dosages). After obtaining experimental data, the system identification methodology guides decisions of model structure, parameter estimation, and model validation prior to dictating the usefulness of the model for controller design. For examples see Timms et al. ⁴⁶ and Deshpande et al; ⁴⁷ experimental procedures involving pseudo-random multisine signals are currently being evaluated in a physical activity intervention based on Social Cognitive Theory. ⁴⁸

• 7What is the likely direction and magnitude of the effect of the DHI or its components compared to a comparator which is meaningful for the stage of the research process?
• 8How confident are we about the magnitude of the effect of the DHI or its components compared to a comparator which is meaningful for the stage of the research process?

Once RQ 3–6 have been addressed, the research team are likely to be able to estimate the direction and magnitude of the effect of the DHI. If this estimate suggests that the DHI is likely to be beneficial to individuals or a population, has sufficient acceptability and feasibility to ensure adequate reach and uptake for cost-effectiveness, and when the total treatment package (i.e. DHI + delivery package + context of use) have all been iterated and adapted to the point where the treatment package is likely to remain relatively stable over the medium term, it may be appropriate to undertake a definitive randomised controlled trial to establish the magnitude of the effect (effect size) of the DHI compared to a meaningful comparator. “Relatively stable” is a matter for investigator judgement, guided by the causal explanation and optimisation data. ⁴⁹ The wider technological landscape is likely to continue to evolve, and investigators must judge what impact this will have on the generalisability of their findings. The importance of undertaking a RCT, and not relying solely on formative studies is evidenced by the fact that RCTs have repeatedly overturned assumptions drawn from observational or non-randomised studies (e.g.^{50, 51}). Hence the assumption of equipoise, required for a trial to be ethical, does hold. Although the general principles of designing and conducting RCTs for complex interventions ²² are applicable to DHIs, there are specific features of DHI which need consideration if a trial is to provide useful evidence that supports rational decision-making. These include:

• The context in which the trial is undertaken

• The trade-off between external and internal validity

• Specification of the intervention and delivery platform

• Choice and specification of the comparator

• Establishing separate data collection methods from the DHI itself

The importance of context has been described above (RQ 3 & 5). Understanding, defining and describing the context in which an RCT is undertaken is necessary to inform judgements around the generalisability of the results outside the trial environment, particularly before implementing a DHI in a different context.

Deciding how to balance external and internal validity is a challenge for many trials ⁵², but is particularly salient for trials of DHI. External validity refers to the extent to which the results apply to “a definable group of patients in a particular setting”, while internal validity is based on how the design and conduct of the trial minimises potential for bias ⁵³. The emphasis in trials of pharmaceutical products is on internal validity and reducing bias, and extensive work has confirmed the importance of this. ⁵⁴ However, there are real questions as to how well approaches developed to reduce bias in drug trials translate to trials of complex interventions in general ⁵² and to digital interventions in particular, including concerns about the degree to which design features that enhance internal validity jeopardise external validity. For example, poor retention to the trial, leading to missing follow-up data, may be countered by boosting the human component of the trial by undertaking some of the trial activities face-to-face, or by recruiting highly motivated participants who may be unrepresentative of the people who would use the intervention in routine practice. Hence data from trials apparently at low risk of bias may paradoxically be less appropriate for informing policy than those with potentially greater risk of bias but better generalisability.

Detailed specification of the DHI is important, but may be hard to achieve, particularly where there is a high degree of tailoring, adaptive learning and user choice. By specification we mean having an agreed framework for classifying the intervention components, including the degree of human input and components which are individually tailored. Such specification is required for replication of trial results, comparison between DHI, synthesising data across trials in systematic reviews and meta-analyses ⁵⁵ and may help with determining the criteria for ‘substantial equivalence’ of digital interventions. The concept of ‘substantial equivalence’ is used for medical device and pharmaceutical regulation by the FDA and similar regulatory bodies. Essentially, if a pivotal trial exists, interventions meeting criteria for ‘substantial equivalence’ would not require further RCT evidence. For example, if a pivotal RCT (or meta-analysis) demonstrated effectiveness of a mindfulness-based digital intervention for depression, then each new mindfulness app for depression would not be required to undergo RCT testing – but to demonstrate substantial equivalence to existing ‘predicate’ interventions. ⁵⁶ The relevant data to collect would then focus on usage, adherence, demographic access parameters, user preferences etc.

The selection of a suitable comparator is determined by the research question addressed, which will vary with the stage of the research. In pragmatic trials which aim to determine the effectiveness of a new treatment compared to current best practice, the comparator is typically ‘treatment as usual’ (TAU). However, in trials of DHIs, the participants in the TAU group may have access to a myriad of other digital interventions. People accustomed to using digital interventions are often also accustomed to searching online for resources. Someone who has sought help for a particular problem, entered a trial, and been randomised to the comparator arm, and who finds the comparator intervention unhelpful, may well search online until they find a better resource. ⁵⁷ This activity may be hard to prevent or track, but risks undermining the trial.

In head-to-head RCTs, where the effects of two (or more) DHIs are compared with each other or against a face-to-face intervention, it is important to define which components of the comparator interventions are the same and which are different. Here the specification of the comparator should follow the same principles as the specification of the intervention outlined above. ⁵⁵

There is a temptation in RCTs of DHI to embed data collection into the intervention, but this may introduce systematic bias or confound the intervention with the measurement method. This bias may favour the intervention or, by more accurately recording adverse events, it may appear to show that the intervention is causing harm.

• 9Has the possibility of harm been adequately considered? And the likelihood of risks or adverse outcomes assessed?

Digital health interventions are not harm free, although to date, the data on actual harms are relatively sparse. There are various mechanisms by which DHI could result in harm. First, they could be designed to achieve an outcome which is widely viewed as harmful, for example websites which promote suicide. Secondly, DHI can make fraudulent claims, which if believed, can result in the user experiencing harm. Examples of this include apps that claim to promote safer consumption of alcohol, including providing estimates of blood alcohol concentration (BAC) to enable users to determine whether they are safe to drive, but which do not in fact have any capacity to estimate BAC. ⁵⁸ Alternatively, a DHI could contain inaccurate information or advice. Thirdly, a DHI could provide accurate information and advice, but this could be misinterpreted or wrongly applied, leading to that decisions which harm health. Alternatively, this accurate information could lead to increased anxiety or depression. Fourthly, ineffective DHI lead to opportunity costs for users, and if paid for by a health service, opportunity costs for the system. If individuals or systems put resources (funds, time, effort) into ineffective interventions, those resources are not available for effective interventions. Fifthly, individuals (and systems) may become disillusioned and despondent if they use ineffective interventions, leading to a belief that either the individual is incapable of responding to treatment, or that all DHI are useless and no further effort should be invested. Finally, DHIs may ‘leak’ personal data because of inadequate security and encryption functions. ⁵⁹

All developers of DHI should actively consider the possibility of harm and include evaluations that look for potential harms including breaches of privacy and information governance. Identification and quantification of expected harms (such as increased anxiety) can be undertaken as part of an RCT, but unexpected harms will require alternative strategies for identification and quantification. Some may emerge during the development and optimisation work, while others may require long-term observational studies during widespread implementation.

• 10Has cost been adequately considered and measured?

It is essential to consider sustainability and cost-effectiveness from the very beginning of the development of a DHI. The development phase should include consideration of the long term costs of maintenance and updating, how these costs could be met, and who will take responsibility for them. Methods for undertaking a formal health economic analysis is addressed in detail by (McNamee et al 2016).

• 11What is the overall assessment of the utility of this intervention? And how confident are we in this overall assessment?
• 12Should research priorities change?
• 13Should clinical practice change?

Answers to the previous ten questions should enable an assessment of the overall utility of the DHI (e.g., balancing its effects, usage, scalability, costs and safety), along with an estimate of how confidence in this assessment. This in turn can guide decision-making about research priorities and clinical practice. This assessment may range from considering that there is sufficient evidence of beneficial effect with sufficient confidence in the effect size, along with adequate understanding of the costs, scalability, sustainability and risks of harm for a specific DHI that it should be incorporated into routine clinical practice, to realising that a given DHI is so unlikely ever to have either sufficient clinical impact or reach that no further research resource should be invested in it.

Discussion and conclusions

This paper outlines a research question-driven approach to the evaluation of DHI, which should lead to an accumulating knowledge base around such interventions in a timely and resource efficient manner. Good research in this area requires fertile multi-disciplinary collaborations which draw on insights and experience from multiple fields including clinical medicine, health services research, behavioural science, education, engineering, and computer science. Researchers from an engineering or computer science background may be surprised by the reliance on randomised controlled trials, while those from a biomedical or behavioural sciences background may consider there is too much emphasis on methods other than RCTs. The view put forward in this paper is that definitive, well-designed RCTs remain an important part of the overall toolkit for evaluating DHI, but only one part. Researchers in this field could learn from the iterative approach adopted by engineering and computer science where interventions undergo multiple cycles of development and optimisation. A definitive trial should be undertaken only once: a) the intervention together with the delivery package around it have reached a degree of stability such that future developments can be considered relatively minor, b) there is reasonable confidence that the intervention plus delivery package can be implanted with high fidelity and c) there is a reasonable likelihood that the overall benefits will be clinically meaningful and lead to either improved outcomes or equivalent outcomes at less cost.

How best to combine rigor with efficiency in evaluating DHI requires a great deal of methodological research. Areas to explore in future methodological research include:

1. Enabling individual studies to generate more useful data:

   1. Consideration and validation of appropriate short-term proxy outcomes, together with identification of when use of these is appropriate, and when definitive outcomes such as health status are needed;

   2. Improving methods for early formative work, to make it as efficient as possible, and define if further investment in more intensive research designs and development processes is warranted.

   3. Better understanding of how to improve the internal validity of RCTs of DHI in terms of retention and follow-up, without jeopardising external validity in terms of the population recruited or impact on the intervention;

   4. Improved methods for reducing the large amounts of missing data that may occur, and addressing the inevitable biases this raises;

   5. Better methods for determining whether and how a DHI will become scalable and sustainable, including understanding how a DHI might be supported through self-sustaining business models.

2. Enabling more useful synthesis and comparison of data generated by different studies:

   1. Identification, specification and classification of important contextual factors;

   2. Specification and classification of target populations;

   3. Specification and classification of DHI, so that we can gain an understanding of the important active components, mechanism of action, replicate and synthesise evidence across DHI evaluations and begin to address the issue of determining “substantial equivalence” between DHIs;

   4. Specification and determination of appropriate comparators, according to the stage of the research process;

   5. Improved reporting of studies of DHI, building on initiatives such as the TIDIER reporting guideline ⁵⁵ and the CONSORT–EHEALTH statement. ⁶⁰

Key guidance points and priority topics for future research.

Guidance points based on existing research

1. The efficient development of safe, effective, widely accessible DHIs requires innovative research methods to generate an accumulating knowledge base that can be used to guide decision making.

2. Reach and uptake are crucial determinants of the overall impact of a DHI, and can be determined and improved using human-centred design methods.

3. Sustainability and revenue models should be considered early in the development process.

4. Defining a clear causal model that accounts for the multiple components of a DHI and the surrounding delivery package is essential.

5. Identifying the essential or active components of a DHI or its delivery package can be done using a framework derived from engineering known as Multiphase Optimisation Strategy (MOST).

6. Randomised controlled trials (RCTs) remain an important method for determining DHI impact in terms of effectiveness and cost-effectiveness, but are best undertaken once the DHI and its delivery package are stable, can be implemented with high fidelity, and are highly likely to lead to clinically meaningful benefits.

The key priority is to improve the efficiency of evaluations without jeopardising rigor. Achieving this will entail

1. Enabling individual studies to generate more useful data through: improving methods of early formative work; better understanding of when and how short-term proxy outcomes should be used and when definitive outcomes are needed; better methods for improving internal validity of trials without jeopardising external validity; improved methods for enhancing DHI uptake and minimising missing data; and better methods for considering whether and how DHI will become scalable and sustainable.

2. Enabling more useful synthesis and comparison of data generated by different studies through: improved specification and classification of context, target populations, digital health interventions and their components, using more appropriate comparators for the stage of the research process, and improved reporting of trials of DHI.