Improving eHealth intervention development and quality of evaluations

The systematic review and meta-analysis by Katrina Champion and colleagues¹ in The Lancet Digital Health included randomised controlled trials to assess the effectiveness of school-based eHealth interventions to prevent multiple lifestyle risk behaviours among adolescents. The results showed some small but significant short-term benefits on physical activity, screen time, and fruit and vegetable intake. No improvements were found for smoking or alcohol use, or consumption of fat or sugar-sweetened beverages and snacks.

The authors were constrained by the nature of the evidence available, with the quality of the randomised controlled trials identified rated as low to very low. Below we highlight four key factors that hinder the collection and reporting of good quality evidence, and suggest how digital health intervention development and evaluation could be improved.

First, the authors could not explore the size of the effect of different intervention components because most interventions and studies did not adequately express their behavioural change techniques, thus precluding meta-regression of distinct intervention components. Process evaluations are important for examining content and context (ie, the intervention components, their mapping on the behaviour change techniques, and what works best, for whom, and why) and could have partly mitigated these limitations but were lacking in the studies included in Champion and colleagues’ review. Beyond including thorough process evaluations, we suggest future research adequately expresses behaviour change techniques and their associated components, for instance, by using Michie and colleague’s taxonomy.² We also recommend that future development of interventions and evaluations should produce theories of change, to show potential pathways for change, and logic models, laying out the pathways used by a particular intervention and contextual dependencies and following the UK Medical Research Council guidance for developing and evaluating complex interventions (currently being updated).³

Second, of the 16 included interventions, a quarter of the randomised controlled trials expressed their comparison groups as receiving education as usual, with no information on what that comprised. Education as usual might have involved an evidence-based, face-to-face intervention that could have been more effective than an eHealth intervention,⁴ or might not have involved any intervention; therefore, interpretation of these results is not possible. Three of the interventions compared an eHealth intervention with an evidence-based, face-to-face intervention that might be more influential. So, fewer than half of interventions (44%) compared an eHealth intervention with assessment only, rather than another form of intervention. Hence, uncertainty exists about the effect size of eHealth interventions, with it possibly being underestimated. To avoid such uncertainty, we recommend following the extended CONSORT guidelines for reporting social and psychological interventions,⁵ which should lead to improved transparency of descriptions of comparison groups, although even greater emphasis on this transparency in future guidelines would be helpful.

Third, the authors could only say that any effects they found were short-lived, because to date the studies have only included short-term follow-up. Funders and evaluators should consider the benefit of funding calls that enable longer-term follow-up of eHealth interventions, and possible prospective cohort studies. These issues are probably compounded by the rapid pace of technological change and resonate with another key challenge identified within the field—ie, developing an accumulating knowledge base to guide digital health intervention development.⁶ To counter these shortcomings, we highlight the need for more substantive theoretical development to understand potentially generalisable mechanisms, including the interplay between individual factors, social norms, social networks, wider communities, and system context.⁷

Finally, many of the studies used measurement tools developed for surveillance only and might not be appropriate for measuring behavioural change. This factor was particularly notable for the use of self-reported physical activity measures, which are likely to be prone to measurement error and problems with recall, particularly when assessing physical activity for children and adolescents. Specifically, these instruments could insufficiently capture incidental or sporadic bouts of activity characteristic of younger people and children.⁸ These choices of measure cast additional doubt on the results and effect sizes of the studies. Future studies should be more explicit on the measures used in intervention settings. Only three interventions included some form of device-based physical activity assessment (ie, accelerometers). Device-based measures of physical activity might overcome some of the challenges associated with relying only on self-report, particularly when examining responsiveness to change in children and adolescents. Additionally, poor quality measures of screen time have been identified as a major methodological issue in research examining the influence of use of digital technology on adolescent health and wellbeing.⁹ We suggest a need exists to develop measurement tools that are valid, reliable, and sensitive to behavioural change for different age groups.

Beyond these issues, Champion and colleagues did not take the opportunity to suggest future research should include economic evaluation of eHealth interventions, which is increasingly important for policy makers and commissioners.¹⁰ A need also exists to assess health inequalities, possibly amplified by digital technology. Champion and colleagues’ Article helped highlight the lack of interventions including mobile devices and applications, given the widespread use of smartphones among adolescents, research is needed in this area.