Using the RE-AIM framework to evaluate internal and external validity of mobile phone–based interventions in diabetes self-management education and support

Abstract

Objective

We evaluated the extent to which studies that tested short message service (SMS)– and application (app)-based interventions for diabetes self-management education and support (DSMES) report on factors that inform both internal and external validity as measured by the RE-AIM (Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance) framework.

Materials and Methods

We systematically searched PubMed, Embase, Web of Science, CINAHL (Cumulative Index of Nursing and Allied Health Literature), and IEEE Xplore Digital Library for articles from January 1, 2009, to February 28, 2019. We carried out a multistage screening process followed by email communications with study authors for missing or discrepant information. Two independent coders coded eligible articles using a 23-item validated data extraction tool based on the RE-AIM framework.

Results

Twenty studies (21 articles) were included in the analysis. The comprehensiveness of reporting on the RE-AIM criteria across the SMS- and app-based DSMES studies was low. With respect to internal validity, most interventions were well described and primary clinical or behavioral outcomes were measured and reported. However, gaps exist in areas of attrition, measures of potential negative outcomes, the extent to which the protocol was delivered as intended, and description on delivery agents. Likewise, we found limited information on external validity indicators across adoption, implementation, and maintenance domains.

Conclusions

Reporting gaps were found in internal validity but more so in external validity in the current SMS- and app-based DSMES literature. Because most studies in this review were efficacy studies, the generalizability of these interventions cannot be determined. Future research should adopt the RE-AIM dimensions to improve the quality of reporting and enhance the likelihood of translating research to practice.

INTRODUCTION

Diabetes self-management education and support (DSMES) is a necessary component in diabetes care. DSMES plays an important role in glycemic management and preventing or delaying diabetes-related complications when administered alongside medical care and management.^1,2 However, many barriers such as competing priorities, transportation barriers, underperceived seriousness of diabetes, and lack of accessible services discourage patients from obtaining traditional institution-based self-management education or training.^3–6 There is a need for developing alternative DMSES interventions that are accessible and low cost for populations with diabetes.

Mobile phone technology can serve as an effective platform to deliver DSMES because of its ubiquitous availability and adoption by all populations.^7,8 The 2 most common tools used in mobile phone–based interventions are short message service (SMS) and smartphone applications (apps). SMS and app-based interventions have been applied in multiple behavioral interventions, including promoting physical activity, tracking healthy eating, monitoring blood glucose, taking medication, monitoring complications, and problem solving, and show promising results.^9–17 One recent meta-analysis and systematic review focused on multiple strategies of health technology interventions in diabetes management identified a larger effect size in mobile phone–based interventions compared with other forms of technological interventions, such as computer- or Internet-based interventions.¹⁸ However, despite the increased popularity and demonstrated efficacy of mobile phone–based interventions in diabetes management, it remains largely unknown whether these interventions can be translated beyond the research setting and be broadly adopted in clinical and other settings. Additionally, review studies in this area have concentrated on reporting issues through the lens of internal validity.^10,13 To date, no systematic review has evaluated domains of external validity and identified gaps that could inform the generalizability and translatability of mobile phone–based interventions in the specific area of DSMES.

To bridge efficacy and effectiveness, or research and practice, Glasgow et al developed the RE-AIM (Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance) framework.^19,20 The RE-AIM framework consists of 5 dimensions: reach into the target population and representativeness of the study sample; efficacy or effectiveness of the intervention on the primary outcome tested under either optimal or real-world conditions, effect on quality of life, and avoidance of unintended or negative consequences; adoption by target setting, organizations, and staff; implementation on consistency and cost of delivery; and maintenance of intervention effect over time.^19,20 The RE-AIM framework has been utilized in reporting internal and external validity across numerous behavioral intervention studies, including physical activity, weight loss maintenance, and health literacy.^21–27 These findings provide recommendations and future directions to improve the quality of reporting and to enhance the likelihood of translating research to practice. The goal of this review is to evaluate the extent to which studies, both randomized controlled trials (RCTs) and non-RCTs, testing SMS and app-based interventions to facilitate DSMES report on factors that inform both internal and external validity of the intervention. In addition to data extracted from literature databases, we also included data directly collected from study authors via emails. Recommendations to improve the quality of reporting and the likelihood of broad dissemination of effective SMS- and app-based interventions are provided based on 2 sources of data and other relevant evaluations of behaviors interventions.

MATERIALS AND METHODS

Database search and study inclusion

We adopted the validated PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) approach to conduct the literature review. PRISMA is an evidence-based minimum set of items for reporting in systematic reviews.²⁸ We systematically searched PubMed for eligible articles from January 1, 2009 to February 28, 2019, using combinations of the following MeSH (Medical Subject Headings) (M) and text word (TW) search terms: (1) Diabetes Mellitus Type 2 (M), non insulin dependent diabetes (T), T2DM (T); (2) self-management (M), self care (M); and (3) mHealth (TW), mobile health (TW), cell phone (M), cell phone (TW), mobile phone (TW), short message service (TW), and text messaging (M). Similar searches were conducted in Embase, Web of Science, CINAHL (Cumulative Index of Nursing and Allied Health Literature), and the IEEE Xplore Digital Library. We also performed supplementary searches using the reference lists of eligible articles and relevant systematic review and other review articles. We decided to search from 2009, as 2009 marked a shift in technology from basic mobile phones to smartphones, and therefore the use of apps.²⁹

Eligibility criteria

Studies were included if they were peer-reviewed studies testing the effect of SMS- or app-based interventions on DSMES among patients with type 2 diabetes. We excluded studies that did not use SMS or an app as the primary intervention (ie, used only phone calls or wearables devices for remote monitoring). In addition, we excluded qualitative research (ie, exclusive evaluations on patients’ attitudes or perceptions on interventions), studies of cost-effectiveness analyses, and protocol articles.

Study screening

We used a multistage screening process in which search results were first pooled and transferred to EndNote software (Clarivate Analytics, Philadelphia, PA) for duplicate removal. Next, 2 reviewers (Y.Y. and S.J.P.) independently screened article titles and abstracts for relevance. Studies with “type 2 diabetes” or “non–insulin dependent diabetes” in the titles and abstracts and included a mobile phone–based intervention for DSMES were selected for full text review. Finally, the 2 reviewers conducted full text reviews of selected articles to confirm eligibility. Articles extracted from reference lists underwent an identical process (Figure 1).

Figure 1.

Article screening process (PRISMA [Preferred Reporting Items for Systematic Reviews and Meta-Analyses] 2018 flow diagram). CINAHL: Cumulative Index of Nursing and Allied Health Literature.

RE-AIM coding and scoring

A 23-item validated data extraction tool based on the RE-AIM framework was used to assess the extent of reporting study elements related to internal and external validity. Table 1 presents details on components that assess the RE-AIM dimensions. “Yes” or “no” were used to code the presence or absence of the RE-AIM components outlined in Supplementary Table 2. Each article was coded by 2 researchers independently; disagreement in coding was discussed until consensus was reached (Y.Y. and S.J.P.).

Table 1.

Evaluation dimensions and measures of the RE-AIM framework

Dimension	Componenta	Internal or external validity indicator	Description22
Reach
The number, proportion, and representativeness of individuals who are willing to participate in a given intervention.	Methods to identify target population	Internal	Description of the process by which the target population was identified in the intervention
	Inclusion criteria	Internal	Description of characteristics of the target population that were used to determine if a potential participant was eligible in the intervention
	Exclusion criteria	Internal	Description of characteristics of the target population that prevent a potential participant from being eligible to participate
	Sample size	Internal	The number of people who agree to participate
	Participation rate	Internal	Sample size divided by the target population denominator
	Representativeness	Internal, external	Comparison of characteristics of the study participants in comparison to the target population
Efficacy/effectiveness
The impact of an intervention on important outcomes, including potential negative effects.	Measures/results at least 1 follow-up	Internal	The study outcomes are measured at a time point after baseline
	Intent-to-treat analysis utilized	Internal	Analyzing participants in trials in the groups to which they were randomized, regardless of whether they received or adhered to the allocated intervention
	Satisfactionb or potential negative outcomes	Internal	Measuring acceptability and usability of the intervention in participants; evaluate unintended consequences that may result from the intervention
	Attrition	Internal	The proportion that was lost of follow-up or dropped out of the intervention
Adoption
The number, proportion, and characteristics of adopting settings and interventions agents	Description of intervention locationc	Internal, external	Description of characteristics of the location of the intervention
	Description of staff who delivered the programc	Internal, external	Description of characteristics of staff who delivered the intervention
	Method to identify staff who delivered the intervention	External	Description of the process by which the staff was identified for participation in the study
	Level of expertise of delivery agent	External	Training and educational background among those who delivered the intervention
	Inclusion/exclusion criteria of delivery agent or setting	External	Description of the eligibility criteria of the setting/agent
	Adoption rate of delivery agent or setting	External	The number of participating delivery settings or agents divided by the number of eligible and approached delivery settings or agents
Implementation
The extent to which the intervention is delivered as intended (eg, information on duration and frequency of intervention, fidelity to the intervention protocol, and cost including time and money)	Intervention duration and frequency	Internal	Length of the intervention (ie, days, weeks, months, and length of each intervention contact) and number of contacts with participants
	Extent protocol delivered as intended (%)	Internal	Description of the fidelity to the intervention protocol
	Measures of cost of implementation	Internal, external	The costs including both money and time of delivery across all levels of implementation
Maintenance
The extent to which a participant maintains the change due to intervention and an intervention becomes institutionalized or part of the routine organization practices	Assessed outcomes ≥6 mo postintervention	External	Description of follow-up outcome measures of individuals at some duration after intervention was terminated
	Indicators of program level maintenance	External	Description of program continuation after completion of the research study
	Measures of cost of maintenance	External	The ongoing cost of maintaining delivery across all levels of the intervention
	Program adopted in other setting/populations	External	Description of the intervention being adopted beyond the original setting and population

RE-AIM: Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance.

^aComponents were derived from a reliable extraction tool.^18–20,23

^bComponents were included to ensure relevancy with the mobile health interventions.

^cComponents were informed by other RE-AIM framework reviews of health behavior interventions.^21,48

Missing data inquiry from study authors

After the article screening and data extraction, we contacted corresponding authors by email when interventions included in the final review had missing or discrepant information. We introduced the purpose of our study and provided inquiries about missing or discrepant data according to RE-AIM criteria as well as additional questions regarding study implementation and generalizability. If there was no response to our initial email, reminders were sent at 7, 14, and 30 days.

RESULTS

Study characteristics

Our search yielded 20 studies (Supplementary Table 2). Of these, 14 were RCTs and 2 were pragmatic trials. Four were quasi-experimental studies, among which 1 was an intention-to-treat trial. Nine studies centered on app-based interventions, and 11 focused on SMS-based or the combination of SMS and phone call–based interventions. Eight studies were conducted in urban settings and 1 included both urban and rural settings. Five studies were conducted in the United States, 3 were conducted in Europe (Italy, Spain, and England), 4 were conducted in East Asia, 2 were conducted in India, 2 were conducted in Iran, 1 had locations in both Australia and Taiwan, 1 were conducted in Canada, 1 were conducted in New Zealand, and 1 were conducted in Bahrain. A total of 12 authors from 12 studies responded to our email inquiries, with 9 addressing our inquires (Supplementary Table 2).

RE-AIM analysis

Reach

All 20 studies reported specific inclusion and exclusion criteria and methods to identify target population. All 20 studies provided sample size (median sample size = 140; range, 32-427). Five studies did not report on number of eligible participants^30–35; therefore, participation rate could not be calculated. Among 16 studies that provided the participation rates, median participation rate was 83% (range, 5.7%-100%). No studies in this review explicitly described characteristics of both participants and nonparticipants; therefore, we were unable to make conclusions about representativeness of the samples (Table 2).

Table 2.

Proportion of mobile health intervention on DSMES reporting RE-AIM framework dimensions and components (n = 20)

Dimension and componenta	Proportion (%)	Notes
Reach
Methods to identify target population	100
Inclusion criteria	100
Exclusion criteria	94.4
Sample size	100	Median sample size = 140 (range, 32-427)
Participation rate	70	Median rate 83% (range, 5.7%-100%)
Representativeness	0
Efficacy/effectiveness
Measures/results at least 1 follow-up	100
Intention-to-treat analysis utilized	45
Satisfactionb or potential negative outcomes	25
Attrition	14.8	Median attrition rate 15% (range, 2%-35%)
Qualitive methods to measure efficacy/effectiveness	50
Adoption
Description of intervention locationc	90
Description of staff who delivered the programc	0
Method to identify staff who delivered the intervention	0
Level of expertise of delivery agent	45
Inclusion/exclusion criteria of delivery agent or setting	5.5 (setting); 0 (staff)
Adoption rate of delivery agent or setting	0
Implementation
Intervention duration and frequency	100	Median follow-up 4.5 months, (range, 3-22 months)
Extent protocol delivered as intended	30
Measures of cost of implementation	15	2 studies reported monetary cost, 1 study reported time cost
Maintenance
Assessed outcomes ≥6 mo postintervention	5
Indicators of program level maintenance	0
Measures of cost of maintenance	0
Program adopted in other setting/populations	20

DSMES: diabetes self-management education and support; RE-AIM: Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance.

^aComponents were derived from a reliable extraction tool.^19–21,24

^bComponents were included to ensure relevancy with the mobile health interventions.

^cComponents were informed by other RE-AIM framework reviews of health behavior interventions.^22,36

Effectiveness

All 20 studies reported on findings for at least 1 follow-up measurement. Six (33.3%) studies had more than 1 follow-up.^31,37–42 Nine (45%) studies reported on intention-to-treat analysis.^32,38–47 Eight (40%) provided information on missing data or imputation measures.^{37,39–45,47} The median attrition rate was 15% (range, 2%-35%). Five (25%) studies reported satisfaction or potential negative outcomes.^{37,40–43,48} Ten (50%) studies reported including qualitative methods to measure efficacy or effectiveness of their studies. Among these, 7 used satisfaction surveys or patients’ feedback from real-time monitoring systems to understand participants’ satisfaction and acceptability of the intervention.^{32,33,37,38,40–45,49} One study reported intermittent technical challenges faced by few participants,³⁷ and another study reported intervention burden or poor response to participant requests as reasons for dropout from the study.⁴² Information on adverse outcomes related to communication mode such as misinformation, confusion, or data errors were not reported in any studies (Table 2).

Adoption

Ten of 20 (50%) studies provided information on adoption items.^{30,37–39,43–46,48,50} Most study participants (n = 18; 90%) were recruited in primary or secondary care settings and utilized physician or self-referral. One study recruited participants online,⁴⁴ and another identified potential eligible participants using automatic search from anonymous pharmacy data of 40 pharmacies.⁴⁵ Four studies (20%) provided brief descriptions of characteristics of the recruitment settings.^37,38,43,48

Regarding intervention delivery, 10 (50%) studies reported level of staff expertise. Three studies used diabetes educators,^39,44,50 1 used nurses,⁴⁸ 1 used physicians,⁴⁰ 1 used a combination of clinicians and diabetes educators,³⁰ 1 used a combination of nurses and research staff,⁴⁶ 1 used a combination of nurse and nutritionist,^41,42 and 2 used only research staff to facilitate intervention delivery.^38,47 No study provided characteristics, method to identify, eligibility, or adoption rate of intervention staff. Even though most studies reached patients in a primary or secondary care setting, the intervention may be delivered in nonclinical settings (eg, home) remotely though app and text. Only one study³⁹ reported on eligibility for setting selection, method to identify setting, and setting adoption rate. However, this setting was the recruitment setting rather the intervention setting³⁹ (Table 2).

Implementation

All 20 studies reported study duration (median length of follow-up = 4.5 months; range, 3-22 months). One study did not report intervention frequency or dosage.³⁸ In terms of intervention fidelity, 6 (30%) studies reported that the intervention protocol was delivered as intended.^{31,37,40–43,48} One study author replied by email that their intervention was not carried out as intended.⁵⁰ One study was an intention-to-treat trial,⁴⁴ and 10 study authors did not respond to our inquiry on adoption components. Two studies measured monetary cost of the intervention,^37,43 and 1 measured time cost.⁵⁰ Two studies reported providing incentives to intervention staff,^41,42,44 and 2 provided incentives to participants (Table 2).^43,48

Maintenance

Only one study measured outcomes at 6-month follow-up after the study completed, according to study author’s response via email.³⁷ No study reported indicators of program-level maintenance nor cost of maintenance. Authors from 4 studies informed us that their interventions have been adopted in other settings or patient populations^41–43,45 or were being considered for a large scale implementation (Table 2).³⁷

Applicability of studies conducted in the United States

Of the studies conducted in the United States, app-based intervention studies either provided mobile phones with unlimited service plans³⁹ or required patients to own an iPhone,⁴⁴ whereas SMS-based interventions required patients to own cellphones capable of receiving text messages.^48,50 Most studies had predominantly White non-Hispanic participants, with one study including predominantly Hispanic participants,⁴³ and another study had 39.3% African American participants.³⁹ One study identified that patient messages and diabetes educator responses were in the electronic health records, where primary care physicians (PCPs) could access to all messages.⁵⁰ In a study by Bauer et al,⁴⁸ physicians received information on diagnosis and management of diabetic neuropathy as they exclusively recruited participants with peripheral diabetes neuropathy; however, change in treatment secondary to SMS or app was not tracked. Quinn et al³⁹ compared the control group with 3 additional groups: (1) an app-based intervention group with no PCP involvement, (2) an app-based intervention group in which PCPs received unanalyzed raw data, and (3) an app-based intervention group in which PCPs received summarized report by fax or email with treatment recommendations. Findings indicate that addition of clinical decision support did not differ from an app-based intervention group with no physician involvement.³⁹ Two studies had no physician involvement.^43,44 Overall, mobile phone–based DSMES interventions in the United States did not provide actionable summarized data in the electronic health record for PCPs to provide clinical decision support.

Connecting mobile phone–based data to clinical management

Four studies conducted outside of the United States reported some form of provider management of mobile phone–based patient entered data. One study conducted in China provided patients with an action plan as previsit summaries for physician visits.³³ Another Chinese study also provided guidance for blood glucose monitoring, diet, and exercise.³⁵ Another study conducted in India had a web portal and smartphone app for physicians to communicate treatment titration or recommendations.³⁸ One study from Bahrain gave patients the mobile number of the diabetes educators’ and physicians’ to allow for texting between visits.³⁰ A study from Korea had medical staff from a hospital provide tailored recommendations based on patients’ self-monitoring data of glucose and blood pressure.⁴⁹

DISCUSSION

Discussion notes for main results

This study used the RE-AIM framework to systematically review the state of research on SMS- and app-based interventions targeting DSMES from both internal and external validity perspectives. Methods to target populations and selection criteria were reported across studies. The participation rate (average at 70%) is similar to other behavioral interventions (average at 76%) that include physical activity promotion or smoking cessation at individual level.¹⁹ This participation rate is encouraging and suggests that SMS- and app-based DMSE programs are appealing to patients. However, it is important to note that most studies recruited participants at academic medical centers, hospitals systems, or other institutions focused on clinical care. Previous research has demonstrated that recruitment at clinical settings may offer increased access to patients who are ready to participate in trials because of better information provided by clinicians regarding the study.^34,51–53 As a result, patients with no access to clinical care, compounded by low health literacy, mistrust of the system, or lack of information or awareness of research opportunities, are likely to be excluded.³⁴ Additionally, no study in this review described the target population or indicated representativeness of the study sample to a larger population. Many mobile phone–based interventions in behavioral change fall short on describing the target population, raising concerns of generalizability of these interventions to varying sociodemographic groups.^{21,22,27,36,54} Similarly, the convenience sampling that interventions employed also challenges the understanding of whether the intervention is reaching subgroups of a population and those individuals that could benefit the most.^22,34

Efficacy or effectiveness of SMS- and app-based interventions on diabetes disease or behavioral outcomes was reported in all studies, while information on maintenance of the changes was absent. This finding is consistent with interventions in other areas.³⁶,⁵⁵,⁵⁶,²¹,²⁴ Less than half of the studies performed an intention-to-treat analysis. The rest of the studies presented results of those who completed the follow-up. Whether the SMS- or app-based interventions produce lasting effects is questionable because most studies did not examine maintenance at least 6 months past an intervention. Researchers in a previous review focusing on promoting physical activity through mobile health technology suggested that mobile phone–based interventions are a relatively new area of research that the studies still emphasized to determine whether the interventions can initiate change.²² The researchers also indicated that mobile phone–based interventions may reduce the likelihood of maintaining disease or behavioral outcome changes, as fast advancements in technology could make current interventions obsolete.²² Moreover, the potential of technical problems may reduce motivation and discourage engagement.²² These reasons could in part explain the lack of description on maintenance in SMS- and app-based intervention studies in this review. Additionally, determinants of efficacy or effectiveness remain largely unexplained. Only 5 studies reported using qualitative methods to measure efficacy or effectiveness. The degree to which patients found mobile phone–based interventions to be acceptable, feasible, and sustainable to use are not documented, which hinders the understanding of the potential long-term effects of those interventions.

Organizational or delivery-level aspects of the RE-AIM framework have been historically underreported across behavioral interventions.⁵⁷ This is also the case for SMS- and app-based DSMES research. We found very limited descriptions of the methods used to engage those who delivered the intervention or description of their characteristics, the extent to which the intervention was delivered as intended, and if any adaptions were made to the intervention during the study period. Furthermore, costs across the RE-AIM framework are also important to inform dissemination but are often missed in reporting.⁵⁸ In mobile phone–based interventions specially, previous research has suggested other costs in addition to implementation costs need to be documented. This includes costs associated with recruitment, equipment, technology (eg, mobile phone, service plan, technical maintenance), and even future cost of continuing to use the service.²² Information on cost can help to determine the generatability or replicability of an intervention.

The most obvious omission in reporting is maintenance at the organizational level. Few research projects have the resources to ensure that their interventions can be sustained at the organizational level. Moreover, readiness to adopt and implement mobile phone–based programs are also hindered by financial resources, policies, and workplace culture.^59,60 Indeed, in email conversations with study authors, only one author informed us that their SMS-based diabetes self-management program is being considered for national implementation.³⁷

Mobile phone–based programs have the potential to gather large amounts of health data that can be used to better inform interventions and clinical management plans.⁶¹ These data may also meet the interests of major government stakeholders looking for fiscal efficiencies in healthcare delivery.⁶¹ However, most mobile-based interventions fail to move beyond the pilot or efficacy trial phase, owing to barriers to implementation and sustainability. Previous research discussion indicates that compared with other industries, health care is relatively reluctant to adopt new technologies because of resistance to change within organizations, absence or inadequacy of policies, and funding issues.^60,62–65 Ross et al⁶³ reported that a frequent reason for unsuccessful implementation of a new technology is that it does not fit well with work practices or daily clinical work. They also suggested that unless a technology is adaptable to fit with roles, tasks, and workflows of clinicians and there is access to dedicated technical support staff, resistance to implementing a new technology will remain a challenge.⁶³ More importantly, intervention researchers need to understand the standards and policies regarding data safety and privacy, professional liability, and potential pitfalls of data sharing between systems and organizations.^64,65 Another major implementation barrier is the termination of funding support, as the additional costs of privacy and security testing, ongoing technology support and development, and software maintenance do not fit well with government supported funding cycles for research and development.^60,66 In our review, termination of funding is also one of the most frequently reported implementation barriers by study authors.

Recommendations for future research

Table 3 provides recommendations for future research to improve the assessment and reporting on individual and organizational level factors that will support the internal and external validity of SMS- and app-based intervention in DSMES. We constructed this table using evidence from the current review as well as recommendations from other relevant evaluations on behavioral interventions.^{22,57,67–71} In addition to these recommendations, efforts from other stakeholders including funding organizations and practitioners should also be in place to promote translatability of mobile phone–based solutions in diabetes management. Such efforts include encouraging reporting of negative or unintended outcomes and positive outcomes across RE-AIM dimensions, so that the feasible and efficacious parts of an intervention can be replicated.²³ Funding organizations may consider requiring researchers to develop a plan for sustainability throughout the research. Practitioners should also demand more information on external validity of an intervention with demonstrated efficacy. Information on adoption, implementation, and maintenance will help them clarify whether an efficacious intervention is a good fit for their organization.²³

Table 3.

Recommendations based on gaps identified from the RE-AIM framework evaluation

Dimension and component	Key issues	Overall and specific recommendations
Reach	Representativeness of study sample to target population	Document and understand access, awareness, and appropriateness of intervention to meet target population needs67
		Compare characteristics (eg, sociodemographic, economic, and behavioral) of participants with nonparticipants or the general local population to understand the representativeness of the sample22
		Include inclusion criteria (eg, health/disease conditions, such as hemoglobin A1c ≥8.0%, capable of using mobile phone) and exclusion to Provide explanation on why certain individuals were not eligible for participation22
		Include recruitment setting, methods, and recruitment adaptations22
Efficacy/effectiveness	Suitability and credibility of study design, data collection, and evaluation	Document participants’ satisfaction, negative outcomes, and subgroup effects in addition to reporting primary outcome67
		Report study design (eg, randomized controlled trial) and whether a comparison group is included
		Report use intention-to-treat analysis22 and missing data procedure68
		Report qualitative methods to acceptability and usability of the intervention; report potential negative outcomes of the intervention22
		Report subgroup effects (eg, sex, race, age, or health/disease condition that influence the intervention effect)22
Adoption	Diffusion of intervention program at organizational and delivery level and factors influence the adoption	Document and understand contextual factors related to adoption and developing guides to help users enhance adoption67
		Report characteristics of intervention location and delivery agent and their selection criteria22; if applicable, describe adoption rate of intervention location and delivery agent69
		If applicable, describe the expertise of the delivery staff69
		If applicable, describe participation rate of delivery setting/agent69
		Provide information on the level of human involvement required for an SMS- and app-based intervention compared with the level of human involvement for a routine application70
		Provide information on the prompts/reminders required for SMS- and app-based intervention compared with the level of prompts/reminders for a routine application70
		Provide information on any interventions (including training sessions/support) that are implemented in addition to the targeted SMS- and app-based intervention70
Implementation	Fidelity of intervention program, including intervention uptake, development, monitoring, and adaptation and factors influence the implementation	Document standardized measures for capturing implementation fidelity, multilevel assessment of cost (both monetary and time) and adaption made for implementation67
		Describe intervention content and use parameters (eg, frequency, optimal timing for use, heaviness of use)70, and frequency of inter-person and virtual sessions, if applicable.
		Provide information intervention costs including price for mobile phone, mobile phone data plan, and incentives for program development/participation given to staff or participants68
		Record percent delivered as intended (eg, SMS sent/unsent/received/not received; any application functioning problems)22
		Describe adaptations made to the intervention during implementation (eg, fitting strategies and methods to user culture)68
		Report strategies to monitoring and gathering feedback (from both participants and staffs) during implementation69
		Report time cost to participants and staff in addition to time used for intervention implementation.
Maintenance	Sustainability of intervention program at both the individual and organizational levels	Document and understand dynamic, complex, and multilevel factors leading to sustainment67
		Include an assessment of maintenance of the intervention outcomes (clinical and/or behavioral, such as hemoglobin A1c reduction, healthy eating) 6 mo after the completion of the intervention71
		Report broader outcomes (eg, policy development associated with the intervention)57
		Provide a context in which to evaluate the long-term outcome (eg, ongoing institutional and policy support for program maintenance and ongoing cost)57
		Include a sustainability plan regarding how the intervention could be sustained at both the individual and organizational levels or, if applicable, provide data on the degree to which the intervention is sustained over time22,71

RE-AIM: Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance; SMS: short message service.

Limitations

Several limitations concerning this review need to be considered. First, there was a small number of eligible studies available for this study. However, we used a validated PRISMA approach when we selected literature and extracted results from studies using RCTs or quasi-experimental designs. Authors’ comments were also incorporated into findings of the review. Small sample size and inadequate blinding of some studies in this review may still contribute to risk of bias. Larger and methodologically robust trails are very much needed. Second, it is possible that our search strategy could not identify all relevant articles. For example, we excluded articles that were published in non–English-language or trial registry data. However, we searched 3 main databases including PubMed, Embase, and Web of Science, in addition to 2 topic-specific databases, CINAHL Plus and the IEEE Xplore Digital Library. We used broad inclusion and exclusion criteria to increase the likelihood of capturing relevant studies to minimize publication bias. We also used a manual search of reference lists of eligible articles for relevant systematic reviews and narrative reviews. We addressed a focused area of mobile phone–based solutions, specifically interventions delivered by SMS and apps that are the most common in technology based behavioral interventions due to increased smartphone ownership. The exclusive review on SMS- and app-based interventions, rather than a broad area of mobile health interventions including various wearable devices, improves the transferability of our results to a prime area of smartphone-based interventions. Further, some authors did not report on specific adoption, implementation, and maintenance strategies that were in fact used in their studies, increasing the risk of reporting bias. To address this concern, we contacted the authors directly to inquire missing information regarding external validity.

CONCLUSION

SMS- and app-based interventions may have potential to promote diabetes self-management among patients with diabetes as access to smartphone increase; however, the population impact in the United States and other countries remains to be confirmed. Our systematic review based on the RE-AIM framework synthesized important findings from this emergent body of literature. This review is among the first to address issues in external validity of SMS- and app-based DSMES studies that is lacking from traditional reviews predominately focused on internal validity and intervention efficacy or effectiveness. Our review demonstrated that the comprehensiveness of reporting on RE-AIM criteria across the mobile phone–based DSMES studies was relatively low and with many gaps in internal validity reporting (ie, extent to which the protocol was delivered as intended) and, more so in external validity (across domains in adoption, implementation, and maintenance). Without this information, it is difficult to determine the internal validity and external validity of mobile phone–based interventions in DSMES. We encourage researchers to improve reporting around RE-AIM dimensions, specifically information on intention-to-treat analysis; mixed methods to understand acceptability, usability, and implementation of an intervention; costs; location, delivery agent selection; characteristics; and maintenance planning. Continued efforts in improving quality of research development and reporting will ensure mobile phone–based interventions to address diabetes will be broadly applicable across diverse settings and populations.

FUNDING

This study was supported in part by the National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health (award numbers P30DK092950 and P30DK020579) and the Centers for Disease Control and Prevention (Cooperative Agreement number U48DP006395). SJP’s contribution was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number KL2 TR002346. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

AUTHOR CONTRIBUTIONS

YY designed the study, undertook the literature search and data extraction, contacted individual study authors, and wrote the article. SJP refined the coding book, verified the coding results, and assessed clinical applicability of the included studies. RCB, SAB, MK, VAF, and EJS provided critical appraisals and edits to the article. RD, KW, DAG, AT, AR, and CM provided important information regarding their individual studies.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.