Using digital interventions to improve the cardiometabolic health of populations: a meta-review of reporting quality

Abstract

Objectives. We conducted a meta-review to determine the reporting quality of user-centered digital interventions for the prevention and management of cardiometabolic conditions.

Materials and Methods. Using predetermined inclusion criteria, systematic reviews published between 2010 and 2015 were identified from 3 databases. To assess whether current evidence is sufficient to inform wider uptake and implementation of digital health programs, we assessed the quality of reporting of research findings using (1) endorsement of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, (2) a quality assessment framework (eg, Cochrane risk of bias assessment tool), and (3) 8 parameters of the Consolidated Standards of Reporting Trials of Electronic and Mobile HEalth Applications and onLine TeleHealth (CONSORT-eHEALTH) guidelines (developed in 2010).

Results. Of the 33 systematic reviews covering social media, Web-based programs, mobile health programs, and composite modalities, 6 reported using the recommended PRISMA guidelines. Seven did not report using a quality assessment framework. Applying the CONSORT-EHEALTH guidelines, reporting was of mild to moderate strength.

Discussion. To our knowledge, this is the first meta-review to provide a comprehensive analysis of the quality of reporting of research findings for a range of digital health interventions. Our findings suggest that the evidence base and quality of reporting in this rapidly developing field needs significant improvement in order to inform wider implementation and uptake.

Conclusion. The inconsistent quality of reporting of digital health interventions for cardiometabolic outcomes may be a critical impediment to real-world implementation.

INTRODUCTION

Digital technologies are being rapidly adopted around the world in both high-income and more resource-constrained countries. Everyday technologies and devices are “disrupting” the traditional ways in which people use information, communicate with one another, and undertake commerce and business. The number of active mobile devices (eg, smartphones, tablets) has now surpassed the world’s population (7.22 billion devices¹). China has the largest number of current users of any country in the world, and the Asia Pacific, Middle East, and African regions are predicted to account for 80% of all new subscriptions in the next 5 years. The advent of mobile devices, text and multimedia messaging, websites, mobile applications (apps), wearable devices and sensors, and social media has enabled a burgeoning new field of digital public health. This field capitalizes on the ubiquity of these platforms and devices to improve the health of defined populations (eg, those who share a medical condition, medical facility, or set of risk factors) by enabling the delivery of digital health programs to entire populations, irrespective of age, geographic location, or socioeconomic position.² Further, the rapid uptake of new technologies allows for tailoring and personalization of interventions due to significant advances in automated data analytics and health informatics. Despite these exciting advancements, there remain a number of critical gaps with respect to the quality of evidence underpinning the widespread and effective use of digital health interventions for chronic disease prevention and management and translating them into real-world settings. As the prevalence of chronic conditions requiring ongoing self-management, such as coronary heart disease and type 2 diabetes mellitus (T2DM), continues to rise globally, there is an urgent need for high quality and effective digital health interventions to improve the health of populations.

It is widely acknowledged that there is a lack of formal scientific evidence for many digital health interventions. Both the Journal of the American Medical Association³ and the British Journal of Medicine⁴ have called for better quality and reporting of evidence that advances our knowledge in this field. Specifically, many user-centered (ie, initiated by consumers rather than health professionals) digital health-related interventions remain poorly evaluated, and many wearable and other measuring devices have been inadequately validated. Many are incompatible with evidence-based health care practices, care delivery, and guidelines. This is particularly true of applications (apps). In 2015, over 40 000 health and wellness apps were available in Apple’s App Store, with only a small proportion formally evaluated, much less clinically calibrated against gold standard measurement. For example, Kumar et al.⁵ reviewed all available apps designed for hypertension management, and of those claiming to function as a blood pressure–measuring device, not one provided any strong evidence of validation compared to the gold standard. Use and engagement are also serious issues as, for example, 50% of health and wellness apps in the App Store have more than 500 downloads.⁶

There remains a lack of consensus among public health and health researchers about how best to interpret the research findings of digital health interventions that have been formally evaluated. This is partially due to a lack of standardization and therefore consistency in reporting of the use and outcomes of these interventions, which could compromise wider implementation and dissemination at the population level. While a number of barriers to widespread uptake and implementation are common in any complex intervention, some are more problematic for digital health research. For example, the rapid advancement of digital health and other technologies has exceeded researchers’ ability to understand how best to evaluate them.⁷ Therefore, any comprehensive evaluation required before scale-up leads to delays in releasing programs that have been evaluated. Furthermore, funding constraints limit the ability to develop and comprehensively evaluate a digital technology–based intervention within the same research project. Thus, putative evaluation and reporting frameworks may be incompatible or inappropriate for reporting comprehensive results of digital health trials (eg, CONSORT guidelines) or when synthesizing studies and reviews (PRISMA guidelines⁸) in order to reach evidence-based, scientifically rigorous conclusions. In their 2014 meta-review that found promising (cost-)effectiveness evidence for eHealth interventions, Elbert et al.⁹ recommended that researchers make transparent use of reporting guidelines appropriate for specific study designs. For example, the Consolidated Standards of Reporting Trials of Electronic and Mobile HEalth Applications and onLine TeleHealth (CONSORT-eHEALTH) guidelines,¹⁰ which were developed to promote better quality reporting of digital health randomized controlled trials (RCTs), could be useful to synthesize the existing evidence and help determine population impact, potential scale-up, and/or translation into clinical care. These guidelines consider critical issues related to implementation, specifically: (1) development of the intervention being evaluated; (2) participant program access; (3) description of the interventions being evaluated (model, theory, content, communications channels, prompts); (4) indication of where resources were provided to supplement the interventions; (5) data collection and storage process such as security and measures of usage (self-report vs. objective); (6) diagram showing attrition at stages of the intervention including usage, dose, engagement; (7) demographics on the digital health divide; and (8) process outcomes. Using the 8 parameters of CONSORT-eHEALTH as well as reporting of PRISMA guidelines and a quality assessment framework (eg, Cochrane risk of bias assessment tool), we conducted a meta-review of published digital health trials to assess the scientific process that guides evidence-based best practice in the field.

METHODS

Search strategies

Literature searches for systematic reviews of user-centered, technology-delivered interventions to improve type 2 diabetes and/or cardiovascular disease (CVD) were performed in the PubMed, EMBASE, Cochrane Library, and Scopus databases. Two of the authors (FC, PR) independently screened all titles and abstracts for relevance. Citations were screened through Web of Science for additional literature. In addition, hand searching of key journals and reference lists of all studies selected for inclusion was done.

Search terms

The search strategy was informed by the search query of Elbert et al.,⁹ a recently published systematic review of systematic reviews and meta-analyses to assess effectiveness and cost-effectiveness of eHealth interventions in somatic diseases. It was built to retrieve systematic reviews of technology-delivered interventions to improve lifestyle behaviors. In addition to their search query “[eHealth] AND [effectiveness] AND [systematic review OR meta-analysis],” we ran the following: “[technology-delivered] AND [systematic review OR meta-analysis],” “[technology-delivered] AND [lifestyle] OR [behavior] AND [effectiveness] AND [systematic review OR meta-analysis],” “[person-centered] AND [technology-delivered] AND [effectiveness] AND [systematic review OR meta-analysis].” Further, as this review aimed to report on how data from reviews of digital health interventions are being synthesized with respect to key evaluation parameters outlined in the CONSORT-eHEALTH guidelines since their inception,¹⁰ we used January 2010, the year those guidelines were published, as a starting point. The search was extended to include articles published until June 2015. Details about the search terms used are provided in the Supplementary material.

Inclusion/exclusion criteria

Systematic reviews on technology-delivered interventions for prevention or management of T2DM and/or CVD that were compared to a control condition were included. Interventions had to meet the following criteria: (1) published between January 2010 and June 2015; (2) involved general, not specific (eg, elderly, hard to reach), adult populations; (3) used interventions and programs focused on improving lifestyle behaviors (smoking, alcohol intake, diet, weight, physical activity, and sedentariness) to prevent or control CVD or type 2 diabetes; and (4) primarily used technology to deliver the intervention. Interventions that were not user- or consumer-centered (eg, education of medical or nursing students and health care professionals to improve health service use) or did not involve adults were excluded. We did not assess papers written in languages other than English.

Data extraction and analysis

Two reviewers (FC, PR) independently evaluated articles at the title/abstract review stage and assessed their eligibility. Full-text articles of abstracts meeting inclusion criteria were then reviewed, and, where eligible, the following data were extracted and recorded in 1 of 3 extraction tables:

1. Key characteristics: Review type (eg, meta-analysis, systematic review), year range of studies included in each review, target health behavior or risk factor, study design (eg, RCT, non-randomized trial, pre-post study, quasi-experimental design), number of studies in each review and total number of subjects in all studies included in the review, and study location.

2. Quality of evidence:

   • - Reporting the use of PRISMA guidelines;
   • - Reporting application of a quality assessment framework by which to assess bias (eg, Cochrane Risk of Bias inventory; yes/no); and
   • - Grading according to level of detail provided on the following 8 parameters from the CONSORT-eHEALTH guidelines: (1) development of the program being evaluated; (2) participant access; (3) descriptions of the digital health interventions being evaluated (model, theory, content, communications channels, prompts); (4) indication of where resources were provided to supplement the digital health interventions; (5) data collection and storage process, such as security and measures of usage (self-report vs objective); (6) diagram showing attrition at stages of the interventions including usage, dose, engagement; (7) demographics on the digital health divide; and (8) process outcomes. Development of the initial CONSORT-eHEALTH statement and instrument has been published in detail previously, but briefly, it occurred through a literature review, Delphi process, and consensus workshop.¹⁰ Two reviewers (FC, PR) graded the evidence for each parameter using the following scoring system developed by the authors: A = comprehensive data shown, B = some data shown, C = no data shown (even if available from original sources), D = no data available in original studies (unavailable from original sources). The grades were subsequently averaged for each parameter by the first author (AO) to produce an overall 12-scale score ranging across A+ (strong-level evidence), B (moderate-level evidence), C (mild-level evidence), and D (weak-level evidence). Any disagreements between authors on assessment of the included reviews were resolved by discussion with a third author (SB or AO).

RESULTS

Search results

The initial search results yielded a total of 4539 articles (see Figure 1). After removing duplicates (n = 1133) and screening the remaining papers for the a priori inclusion and exclusion criteria, 318 eligible reviews remained. Subsequent citation screening through Web of Science and hand searching of key journals and reference lists of all studies selected for inclusion resulted in 33 papers being included in this meta-review (15 with accompanying formal meta-analysis). Table 1 displays the key characteristics of the 33 reviews, according to mode of delivery.

Figure 1.

Flow diagram of results of search.

Table 1.

Key characteristics of reviews, by technology type

Author (Year)	Review type, year range of studies	Study location	Health behavior target(s)	Study designs	No. of studies(total n)
Mobile health interventions
Bort-Roig et al. (2014)11	SR (2007–2013)	Spain, USA, China, Germany, UAE, Finland, South Korea, UK, Norway, Australia, Canada, Netherlands	PA	RCTs, Pre-post	26 (n = 828)
O’Reilly et al. (2013)12	SR (2006–2012)	Not reported	PA	RCTs, QEDs	22 (n = 1988)
Stephens et al. (2013)13	SR (2006–2009)	Switzerland, USA, Finland, Korea	Weight, PA	RCTs, QEDs	7 (n = 1377)
Fanning et al. (2012)14	SR and MA (2000–2012)	Not reported	PA	RCTs	11 (n = 1351)
Lyzwinski et al. (2014)15	SR and MA (2007–2013)	Finland, UK, USA, Australia,	Weight	RCTs	17 (n = 1796)
Computer and Web-based interventions
Civljak et al. (2014)16	SR and MA (2004–2013)	USA, Norway, Netherlands, Switzerland, Germany, multinational population (not specified)	Smoking cessation	RCTs, QEDs	28 (n = 45 000)
Aneni et al. (2014)17	SR (2001–2012)	Not reported	PA, diet	RCTs, follow-up studies	29 (n = 15 261)
Pal et al. (2013)18	SR and MA (1986–2011)	USA, South Korea, Australia, UK, China	Self-management of T2DM	RCTs	16 (n = 3578)
Ramadas et al. (2011)19	SR (2000–2010)	USA, South Korea, Taiwan, Canada	Self-management of T2DM	RCTs, QEDs	13 (n = 1644)
Angeles et al. (2011)20	SR and MA (2004–2009)	Korea, Finland, Canada, USA	Self-management of T2DM	RCTs	9 (n = 926)
Pietrzak et al. (2014)21	SR (2004–2012)	USA, Canada, Netherlands, Spain, South Korea, Switzerland, Australia	Weight, diet, PA, smoking, alcohol intake	RCTs, observational, studies, pilot	23 (n = 9615)
Pereira et al. (2015)22	SR (2006–2012)	Not reported	Self-management of T2DM	RCTs, QEDs, comparative effectiveness designs	14 (n = 2802)
Levine et al. (2014)23	SR (2001–2013)	Not reported	Weight	RCTs	16 (n = 6786)
Lustria et al. (2013)24	SR and MA (1999–2009)	USA, Australia, UK, NZ, Netherlands, Canada, Belgium, Switzerland	PA, diet, smoking cessation	RCTs, QEDs, no treatment control	40 (n = 20 180)
Reed et al. (2012)25	SR and MA (1985–2009)	Not reported	Weight	RCTs	11 (n = 1866)
van Vugt et al. (2013)26	SR (2002–2012)	North America (n = 10), Europe (n = 2), Asia (n = 1)	Diet, PA, medication, smoking	RCTs	13 (n = 3813)
Vegting et al. (2013)27	SR (2003–2012)	Not reported	CVD risk factors, self-management of T2DM	RCTs	9 (n = 1995)
Yu et al. (2011)28	SR (2003–2008)	Not reported	Diabetes, CVD risk	RCTs, controlled clinical trials, pre-post, observational studies	SR, 57 (n = ), MA, 12 (n = 2731)
Harris et al. (2011)29	SR and MA (1991–2011)	Not reported	PA	RCTs	43 (n = not reported)
Foster et al. (2013)30	SR and MA (1991–2011)	Not reported	PA	RCTs	11 (n = 5862)
Buhi et al. (2013)31	SR (2004–2010)	USA, France, NZ, Norway, South Korea, Scotland, UK, Germany, Finland, India, Europe (countries not specified), Austria	Diabetes (n = 17), smoking (n = 8), weight (n = 4), preventive activities (n = 2), PA (n = 1)	RCTs, QEDs	34 (n = 6235)
Social media/social networking interventions
Toma et al. (2014)32	SR and MA (2002–2013)	USA, Korea, Canada, India, Israel, Turkey, France, Finland, Iran, Taiwan, Italy, UK	Self-management of T2DM	RCTs	34 (n = 4977)
Telehealth and/or telemedicine
Verhoeven et al. (2010)33	SR and MA (2002–2009)	Not reported	Self-management of T2DM	RCTs, QEDs, observational cohort studies	90 (n = 704)
Merriel et al. (2014)34	SR and MA (2004–2011)	USA, Belgium, Netherlands, Canada, Japan	Diet, PA, weight, smoking, BG control	RCTs	13 (n = 9500)
Munro et al. (2013)35	SR (2000–2012)	USA, Australia, Canada	PA, diet, medication	RCTs, non-RCTs	(n = 830)
Omboni (2012)36	SR and MA (1996–2011)	Not reported	Blood pressure	RCTs	23 (n = 7451)
Cassimatis et al. (2012)37	SR (2001–2011)	Partially reported USA, South Korea)	BG control, diet, PA, medication taking	RCTs	14 (n = 2300)
Combination of technologies
Connelly et al. (2013)38	SR (2003–2011)	USA, Canada, South Korea, Iran	PA, T2DM self-management	RCTs	15 (n = 4567)
Wieland et al. (2012)39	SR and MA (1988–2011)	USA, Scotland, Spain, Croatia	Weight	RCTs	14 (n = 2537)
Chang et al. (2013)40	SR (2001–2013)	USA, Canada, Australia, UK	Diet, PA, weight	RCTs	20 (n = 11 457)
Saffari et al. (2014)41	SR and MA (2005–2013)	Taiwan, Korea, USA, India, Iran	T2DM self-management	RCTs	10 (n = 960)
Bacigalupo et al. (2013)42	SR (1998–2011)	USA, Finland, UK, Germany	Weight	RCTs	7 (n = 584)
Cotterez et al. (2014)43	SR (2000–2012)	Not reported	T2DM self-management: healthy eating, physical activity, BG control, medication taking, coping/suppose, reducing risks, problem solving	RCTs, pre–post	9 (n = 1913)

SR = systematic review, MA = meta-analysis; NZ = New Zealand; UK = United Kingdom; CVD = cardiovascular disease; BG = blood glucose; PA = physical activity; T2DM = type 2 diabetes mellitus; QED = quasi-experimental design; RCT = randomized controlled trial.

The major modes of intervention delivery were: (1) mHealth, text messaging, mobile, and/or smartphone (n = 5); (2) Internet-based programs (n = 16); (3) telehealth and/or telemonitoring (n = 5); (4) social media (n = 1); and (5) a combination of technologies (n = 6). The reviews variously targeted lifestyle-related factors (smoking cessation, alcohol intake, diet, weight, physical activity, and sedentariness) individually, in combination, or as a combination of these factors with aspects of cardiometabolic disease prevention or management. Less than 5% of all studies appeared in more than one systematic review, indicating the breadth and diversity of studies reviewed. The primary and secondary results of the 33 papers are summarized in Table 2 according to mode of delivery.

Table 2.

Primary and secondary results of reviews, by technology type

Author (year)	Primary review results	Secondary review results
Mobile health interventions
Bort-Roig et al. (2014)11	Good user perceptions of smartphone interventions’ usability and usefulness.	Smartphone strategies to influence PA were ad hoc, not theory-based. Intervention effects modest at best.
O’Reilly (2013)12	In all, 75% reported significant PA, sedentary behavior changes. These studies employed SMS communication to promote PA (n = 4), PA self-monitoring through mobile journaling (n = 4), or SMS and journaling.	Usability mixed; 58% agreed easy to use. No long-term follow-up.
Stephens (2013)13	In all, 71% reported significant results in at least one outcome, physical inactivity and/or weight.	High acceptability of text messaging and smartphone applications.
+Fanning et al. (2012)14	Significant moderate effects for mHealth interventions.	Moderate to large effect for pedometer steps. Nonsignificant effects for moderate-vigorous PA duration.
+Lyzwinski et al. (2014)15	Medium significant effects favoring mHealth interventions compared to controls.	Reduced BMI, waist circumference, body fat %; improved dietary intake and self-reported physical activity.
Computer and web-based interventions
+Civljak (2014)16	Several reported success of smoking cessation ≥6 months.	Programs tailored to individual responses had higher quit rates than UC. Internet may add benefit when used with nicotine pharmacotherapy.
Aneni et al. (2014)17	No effect on PA, dietary outcomes, lipid profiles, or hypertension. Modest improvements observed in weight.	Successful interventions included “human contact” and environmental modification, or targeted specific disease entities, eg, hypertension.
+Pal et al. (2013)18	Small effect of BG control, with a larger effect in the mobile phone group.	Little evidence for improving depression, health-related QoL, or weight.
Ramadas et al. (2011)19	Goal-setting, personalized coaching, interactive feedback, online peer support all successful.	Strong theoretical basis, longer intervention duration increased success, ie, only relatively longer studies (12 weeks) reported positive findings.
+Angeles et al. (2011)20	Web-based tools better than UC for HbA1c and LDL-C.	Heterogeneity among studies with 12-month intervention.
Pietrzak et al. (2014)21	Majority of studies reported improvement in blood pressure and HbA1c in patients with T2DM.	Fewer CVD events and lower weight, improved lipid profile, eating habits, increased physical activity.
Pereira et al. (2015)22	Effective at improving BG control and diabetes knowledge compared with UC.	Interventions with a human element seen as more attractive to users.
Levine et al. (2014)23	Technology-assisted weight loss interventions compare favorably to other modalities.	Twelve (75%) interventions achieved weight loss (range: 0.08–5.4 kg) compared to controls, while 5%–45% of patients lost at least 5% of baseline weight.
+Lustria et al. (2013)24	Tailored websites and programs more effective.	Targeting general populations more effective than specific groups.
+Reed et al. (2012)25	Computer group lost significantly more weight.	Substitution studies: no difference between intervention and control.
van Vugt et al. (2013)26	Nine saw improvements in depression, diabetes distress, well-being, self-efficacy, stress, communication.	Seven grounded in theoretical model; self-regulation theory, social learning theory most common.
Vegting et al. (2013)27	Four had significant difference in BMI/weight; 2 had significant difference in SBP; 2 had significant difference in DBP.	Multiple modifiable lifestyle behavior. Internet interventions in primary or secondary care not superior to UC for CVD risk factors.
Yu et al. (2011)28	Few tools met criteria for effectiveness, usability, usefulness, and sustainability.	Need to identify strategies to minimize website attrition and enable patients and clinicians to make informed decisions about website choice.
+Harris et al. (2011)29	E-learning no more effective than other behavior change approaches to diet, reducing obesity or weight.	Heterogeneity of studies meant no firm conclusions could be drawn.
+Foster et al. (2013)30	Positive, moderate-sized effects on increasing self-reported PA and cardiorespiratory fitness at 12 months.	Effectiveness of interventions supported by moderate-high quality studies
Buhi et al. (2013)31	In all, 35% of studies focusing on diabetes and improving diabetes management reported statistically significant improvements in BG.	Using SMS with longer intervention duration led to greater improvements in BG, BP, weight, smoking; 76.5% did not use theoretical framework, most had more than 300 participants.
Social media/social networking interventions
+Toma (2014)32	Compared to controls, interventions reduced HbA1c, systolic and diastolic BP, triglycerides, TC.	Subgroup analysis: T2DM had greater HbA1c reduction than T1DM.
Telehealth and/or telemedicine
+Verhoeven et al. (2010)33	Few studies showed significant differences between usual care and intervention groups.	High degree of heterogeneity and few quality studies.
+Merriel et al. (2014)34	No evidence for overall CVD risk reduction.	Weak evidence for reduction of BP and total cholesterol, and no change in HDL or smoking rates.
Munro et al. (2013)35	Home-based CR as effective as hospital-based. May produce longer-term gains via maintenance of PA.	Results positive with regard to patient outcomes and feedback.
+Omboni et al. (2012)36	HBPT improved the physical component of QoL.	No difference was observed in the risk of adverse events.
Cassimatis et al. (2012)37	Half reported significant improvements in BG control.	In total, 5/8 studies on dietary adherence, 5/8 on physical activity, 4/9 on BG self-monitoring, 3/8 on medication taking reported significant effects.
Combination of technologies
Connelly et al. (2013)38	All reported an increase in physical activity: Web (n = 9), mHealth (n = 3), CD-ROM (n = 2), computer-based (n = 1); n = 9 reported a significant increase.	Promoting participant adherence leads to better outcomes. Logbooks, phone calls, and e-mails increased behavior change.
+Wieland et al. (2012)39	Effective compared to no or minimal (pamphlets, UC) intervention.	Smaller effect (weight loss, lower levels of maintenance) compared to in-person interventions. Only one study examined 12-month outcomes.
Chang et al. (2013)40	Social media use inconsistently reported.	Social media incorporated in online weight management interventions via message boards and chat rooms with unclear benefits.
+Saffari et al. (2014)41	Effect of interventions on glycemic control greater for text messaging and Internet (86%) than texting alone (44%).	Age, sample size, diabetes duration, period of intervention, level of HbA1c, and type of intervention may have implications for effectiveness.
Bacigalupo et al. (2013)42	Strong evidence across several high-quality RCTs of short-term weight loss due to mHealth interventions.	Moderate evidence for medium-term outcomes, none >12 months.
Cotterez et al. (2014)43	Two showed improvements in diet and/or PA; 2 had improvements in glycemic control compared to control.	Successful studies were theory-based, had interactive components with tracking and personalized feedback, opportunities for peer support.

+ = meta-analysis conducted; PA = physical activity; SMS = short messaging service; BG = blood glucose; UC = usual care; HbA1c = hemoglobin A1c; LDL-C = low-density lipoprotein; T2DM = type 2 diabetes mellitus; BMI = body mass index; SBP = systolic blood pressure; DBP = diastolic blood pressure; TC = triglycerides; CR = cardiac rehabilitation; QoL = quality of life; RCT = randomized controlled trial; CVD = cardiovascular disease; T2D1 = type 1 diabetes mellitus; HDL-C = high-density lipoprotein; HBPT = Home Blood Pressure TeleMonitoring.

Quality of reporting of digital health interventions

After applying the aforementioned 8 parameters to critique the 33 reviews, we found that:

• Only 1/33 (3%) reported comprehensive data on the development of the digital health interventions being evaluated³⁰;

• 9/33 (27%) reported comprehensive data on participant access;

• 15/33 (45%) provided a comprehensive description of the digital health interventions being evaluated (model, theory, content, communications channels, prompts);

• 10/33 (30%) indicated where resources were provided to supplement the digital health interventions;

• 5/33 (15%) detailed data collection and storage processes such as security, measures of usage (self-report vs objective);

• 2/33 (6%) provided a detailed flow chart/diagram showing attrition at stages of the interventions, including usage, dose, engagement;

• 16/33 (48%) presented comprehensive data on demographics related to the digital divide; and

• 12/33 (36%) reported comprehensive process outcomes.

When we graded the evidence from each review and provided an average grade for each parameter (Table 3), the strongest evidence (B+) existed for parameters related to participant program access, intervention description, data on demographics as related to the digital health divide, and supplementary programs. The weakest evidence (C) existed for parameters related to the intervention development process and the provision of attrition rates at multiple stages of intervention.

Table 3.

Strength of evidence presented by review, according to parameters of CONSORT-eHEALTH guidelines

			Parameters CONSORT-EHEALTH guidelines for improved and standardized evaluation reports of Web-based and mobile health interventions10
Author (year)	PRISMA guidelines applied?	Quality assessment tool used?	Program development process	Participant program access	Intervention description	Supplementary programs	Data collection and storage	Attrition	Demographic data on digital health divide	Process outcomes
Bort-Roig et al. (2014)11	No	No documented assessment of study quality	C	B	B	A	B	C	B	A
O’Reilly et al. (2013)12	No	Effective Public Health Practice Project Quality Assessment Tool44	C	C	B	C	B	C	C	B
Stephens et al. (2013)13	No	No documented assessment of study quality	C	B	B	B	C	C	B	B
Fanning et al. (2012)14	No	Quality of execution portion of Guide to Community Preventative Services data extraction form45	C	B	B	B	C	C	B	C
Lyzwinski et al. (2014)15	No	Cochrane tool for assessing risk of bias46	C	A	A	B	A	C	A	B
Civljak et al. (2014)16	No	GRADE (grading of recommendation assessment, development, and evaluation) scale47,48	C	B	A	B	B	B	A	A
Aneni et al. (2014)17	No	Methodological criteria designed by Ogilvie (2008)49	B	C	B	A	B	C	A	C
Pal et al. (2013)18	No	Cochrane tool for assessing risk of bias46	C	A	A	B	C	C	B	C
Ramadas et al. (2011)19	No	Criteria based on Cochrane Collaboration Back Review50	C	A	A	A	A	B	A	A
Angeles et al. (2011)20	No	GRADE scale47,48	C	B	A	C	B	B	C	A
Pietrzak et al. (2014)21	No	Quality assessment tool (effective public health practice project)51	C	B	A	A	C	C	B	B
Pereira et al. (2015)22	No	No documented assessment of study quality	D	B	A	C	B	B	B	A
Levine et al. (2014)23	No	Delphi and Cochrane Effective Practice and Organization of Care criteria52 PRagmatic-Explanatory Continuum Indicator Summary53	C	B	A	B	B	A	A	C
Lustria et al. (2013)24	No	No documented assessment of study quality	C	B	A	B	A	A	A	B
Reed et al. (2012)25	PRISMA8	Cochrane tool for assessing risk of bias46	C	B	B	C	B	C	A	C
van Vugt et al. (2013)26	PRISMA8	Assessment of study quality completed; tool not referenced	C	A	A	B	B	B	A	C
Yu et al. (2011)28	No	Tool developed from existing framework54	C	B	B	C	D	B	C	B
Vegting et al. (2013)27	No	No documented assessment of study quality	D	B	B	B	B	B	D	A
Harris et al. (2011)29	No	Cochrane tool for assessing risk of bias46	C	B	B	C	C	C	B	C
Foster et al. (2013)30	No	Cochrane tool for assessing risk of bias46	A	A	A	A	B	B	A	A
Buhi et al. (2013)31	No	No documented assessment of study quality	C	B	A	A	B	C	A	C
Toma et al. (2014)32	PRISMA8	Cochrane tool for assessing risk of bias46	C	A	B	A	A	C	A	C
Verhoeven et al. (2010)33	No	Five levels used to categorize methodological quality of studies55	C	B	A	A	A	C	A	A
Merriel et al. (2014)34	No	Cochrane tool for assessing risk of bias46	C	A	B	B	C	C	B	A
Munro et al. (2013)35	No	Jadad scale56	D	C	B	B	B	C	B	A
Omboni et al. (2013)36	PRISMA8	Validated quality checklist57	C	B	B	C	B	C	B	A
Cassimatis et al. (2012)37	No	Cochrane tool for assessing risk of bias46	C	C	B	A	B	C	A	C
Connelly et al. (2013)38	No	Study quality assessed Framework/tool not reported	C	A	B	B	B	C	A	B
Cotterez et al. (2014)43	No	No documented assessment of quality	C	C	B	B	B	C	B	C
Wieland et al. (2012)39	No	Cochrane tool for assessing risk of bias46	C	A	A	A	B	A	A	A
Chang et al. (2013)40	PRISMA8	Jadad scale55 RCT quality scores adapted from Norman et al. (2007)58	C	B	A	B	C	C	B	C
Saffari et al. (2012)41	No	Jadad scale55 Cochrane tool for assessing risk of bias46	C	C	B	B	B	C	B	C
Bacigalupo et al. (2013)42	PRISMA8	Van Tulder rating system (2003)59	C	C	B	B	B	C	A	C
Average evidence grading by parameter			C	B+	B+	B+	B−	C	B+	B−

A = comprehensive data shown, B = some data shown, C = no data shown, D = no data available in original studies

PRISMA guidelines, considered the gold standard for reporting of systematic reviews, were applied in 6 of the 33 reviews (18%) (Table 3). With respect to quality assessment frameworks used in the respective reviews, the Cochrane tool for assessing risk of bias was most commonly used (n = 10). Six made no mention of conducting their review in accordance with an established quality assessment framework. One stated that quality was assessed but did not report or cite the tool used.³⁸ One review reported using the PRagmatic-Explanatory Continuum Indicator Summary tool, which was originally designed to classify randomized clinical trials as being pragmatic or explanatory but has since been modified to grade systematic reviews of trials to specifically help policymakers, clinicians, and researchers assess the applicability of systematic reviews to real-world practice.

DISCUSSION

To our knowledge, this is the first meta-review to provide a comprehensive analysis of the quality of reporting of research findings for a range of digital health interventions. Our findings suggest that the evidence base and quality of reporting in this rapidly developing field needs significant improvement in order to inform wider implementation and uptake. As mentioned, inconsistencies are due to a range of factors, which include heterogeneity in study measures and reporting and limitations in conducting thorough scientific evaluations of digital health technologies. As everyday technologies advance alongside the global burden of chronic disease, patients and clinicians can harness them to provide tailored interventions and highly personalized health care. However, to effectively elucidate what works for whom and how, improvements are required in the quality and consistency of scientific reporting of evidence in this field.

Our findings build on those of Elbert et al. (2014),⁹ who conducted a meta-review of the (cost-)effectiveness of eHealth interventions for somatic diseases. The authors recommended at the time that more attention be given to developing and evaluating strategies to help introduce digital initiatives into daily practice. Due to the degree of heterogeneity in modalities, populations, and outcomes, we were unable to draw firm conclusions about the effectiveness and magnitude of the digital health interventions evaluated. We recommend that readers draw on the data from the specific reviews to gain better insights regarding the effectiveness of digital intervention modalities and target outcomes they are most interested in.

Future uptake of frameworks such as the eCONSORT reported here and, more recently, the World Health Organization’s evidence reporting and assessment checklist for mHealth⁶⁰ may reduce heterogeneity and strengthen the evidence base in this field to maximize population impact. While the current lack of uptake may reflect a lack of awareness, impracticalities, or space restrictions of this reporting by scientific journals, such guidelines represent an important approach to the reporting (and, by extension, the conduct) of digital health research in a systematic way. Reporting on the evolution of digital health interventions (formative strategies), translational processes such as preparation and optimization, piloting and pre-piloting, and consumer co-design provides valuable information that can be transferred to real-world applications and provide insights for others in the field. Further, this detailed reporting allows limitations (eg, intervention bias toward higher-income and highly educated populations) to be made transparent and addressed.

The strengths of this meta-review include the large number of reviews included, the corresponding size of the study samples, and the use of a rigorous reporting framework specifically designed for this field of research, used as a benchmark for assessing the evidence base. We do, however, acknowledge the limitations. As mentioned, these include the heterogeneity of the content of the interventions, the types of intervention (indicated, selective, universal prevention, management), and the characteristics of study samples (level of risk or disease progression), all of which prevented us from making conclusions about the effectiveness of these interventions and determining which work best for whom and under what conditions. Finally, it is plausible that evidence presented in this paper is not fully inclusive of all available programs, as many commercially available digital health products remain unpublished.

While these findings provide comprehensive information about the quality of evidence around digital health interventions and reporting them, there remain critical gaps. Key questions include: How do we best translate research outcomes into policy and practice? What theoretical models are best for the widespread scale-up and integration of these interventions and platforms into routine health care delivery? To date, very few studies have reported detailed measures of engagement or individual-level characteristics that influence engagement with technology-based interventions. Hence, little is known about the active mediating mechanisms through which these interventions exert their effect on behavioral and clinical outcomes, and this is especially true of real-world interventions.⁶¹ Therefore, future research in this area should pay special attention to the factors that influence intention to use, actual usage, and users’ reasons for using or not using these technologies, and examine and evaluate strategies to improve engagement. The processes of behavior change and clinical outcomes should also be examined. To improve the prospect of translating eHealth studies into the real world, emphasis on the quality of reporting, especially on implementation aspects, is paramount.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sector. AO is supported by a NHMRC Early Career Fellowship (1052865). AL is supported by NHMRC Centre of Research Excellence in eHealth (APP1032664).

Competing Interests

AL is a co-author of the CONSORT eHealth guidelines. All other authors declare no support from any organization for the submitted work, no financial relationships with any organization that might have an interest in the submitted work in the previous 3 years, and no other relationships or activities that could appear to have influenced the submitted work.

Contributors

AO, FC, CBT, and BO conceptualized the paper. FC, PR, and AO devised and conducted the search and analyzed the data. FC, AO, SB, and MC co-wrote the initial version of the manuscript, with AL, CBT, NK, and BO critically revising subsequent versions. All authors approved the final version of the manuscript.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.