Table 3.
Recommendations based on gaps identified from the RE-AIM framework evaluation
| Dimension and component | Key issues | Overall and specific recommendations |
|---|---|---|
| Reach | Representativeness of study sample to target population | Document and understand access, awareness, and appropriateness of intervention to meet target population needs67 |
| Compare characteristics (eg, sociodemographic, economic, and behavioral) of participants with nonparticipants or the general local population to understand the representativeness of the sample22 | ||
| Include inclusion criteria (eg, health/disease conditions, such as hemoglobin A1c ≥8.0%, capable of using mobile phone) and exclusion to Provide explanation on why certain individuals were not eligible for participation22 | ||
| Include recruitment setting, methods, and recruitment adaptations22 | ||
| Efficacy/effectiveness | Suitability and credibility of study design, data collection, and evaluation | Document participants’ satisfaction, negative outcomes, and subgroup effects in addition to reporting primary outcome67 |
| Report study design (eg, randomized controlled trial) and whether a comparison group is included | ||
| Report use intention-to-treat analysis22 and missing data procedure68 | ||
| Report qualitative methods to acceptability and usability of the intervention; report potential negative outcomes of the intervention22 | ||
| Report subgroup effects (eg, sex, race, age, or health/disease condition that influence the intervention effect)22 | ||
| Adoption | Diffusion of intervention program at organizational and delivery level and factors influence the adoption | Document and understand contextual factors related to adoption and developing guides to help users enhance adoption67 |
| Report characteristics of intervention location and delivery agent and their selection criteria22; if applicable, describe adoption rate of intervention location and delivery agent69 | ||
| If applicable, describe the expertise of the delivery staff69 | ||
| If applicable, describe participation rate of delivery setting/agent69 | ||
| Provide information on the level of human involvement required for an SMS- and app-based intervention compared with the level of human involvement for a routine application70 | ||
| Provide information on the prompts/reminders required for SMS- and app-based intervention compared with the level of prompts/reminders for a routine application70 | ||
| Provide information on any interventions (including training sessions/support) that are implemented in addition to the targeted SMS- and app-based intervention70 | ||
| Implementation | Fidelity of intervention program, including intervention uptake, development, monitoring, and adaptation and factors influence the implementation | Document standardized measures for capturing implementation fidelity, multilevel assessment of cost (both monetary and time) and adaption made for implementation67 |
| Describe intervention content and use parameters (eg, frequency, optimal timing for use, heaviness of use)70, and frequency of inter-person and virtual sessions, if applicable. | ||
| Provide information intervention costs including price for mobile phone, mobile phone data plan, and incentives for program development/participation given to staff or participants68 | ||
| Record percent delivered as intended (eg, SMS sent/unsent/received/not received; any application functioning problems)22 | ||
| Describe adaptations made to the intervention during implementation (eg, fitting strategies and methods to user culture)68 | ||
| Report strategies to monitoring and gathering feedback (from both participants and staffs) during implementation69 | ||
| Report time cost to participants and staff in addition to time used for intervention implementation. | ||
| Maintenance | Sustainability of intervention program at both the individual and organizational levels | Document and understand dynamic, complex, and multilevel factors leading to sustainment67 |
| Include an assessment of maintenance of the intervention outcomes (clinical and/or behavioral, such as hemoglobin A1c reduction, healthy eating) 6 mo after the completion of the intervention71 | ||
| Report broader outcomes (eg, policy development associated with the intervention)57 | ||
| Provide a context in which to evaluate the long-term outcome (eg, ongoing institutional and policy support for program maintenance and ongoing cost)57 | ||
| Include a sustainability plan regarding how the intervention could be sustained at both the individual and organizational levels or, if applicable, provide data on the degree to which the intervention is sustained over time22,71 |
RE-AIM: Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance; SMS: short message service.