mHealth interventions for weight loss: a guide for achieving treatment fidelity

Abstract

mHealth interventions have shown promise for helping people sustain healthy behaviors such as weight loss. However, few have assessed treatment fidelity, that is, the accurate delivery, receipt, and enactment of the intervention. Treatment fidelity is critical because the valid interpretation and translation of intervention studies depend on treatment fidelity assessments. We describe strategies used to assess treatment fidelity in mobile health (mHealth) interventions aimed at sustaining healthy behaviors in weight loss. We reviewed treatment fidelity recommendations for mHealth-based behavioral interventions and described how these recommendations were applied in three recent weight loss studies. We illustrate how treatment fidelity can be supported during study design, training of providers, treatment delivery, receipt of treatment, and enactment of treatment skills. Pre-planned strategies to ensure the treatment fidelity of mHealth interventions will help counter doubts concerning valid conclusions about their effectiveness and allow investigators and clinicians to implement robustly efficacious mobile health programs.

Trial registration number

1F31 NR012599.

Introduction

Mobile health (mHealth) is the use of mobile devices to support continuous health monitoring and the healthy behaviors of individuals across a variety of demographic, socioeconomic, and geographic populations.^1–4 Devices include mobile phones, wireless devices, and sensors intended to be worn, carried, or accessed by people during normal daily activity.⁵ mHealth technologies include short messaging service (SMS) (also known as text messaging), multimedia messaging service (MMS), smartphone applications or ‘apps’, as well as more complex functionalities including global position systems (GPS), Bluetooth technology, and wearable audio/visual components.⁶

The use of mHealth technologies, particularly via mobile phones and wireless consumer health devices, is burgeoning. There are an estimated 6.8 billion mobile phone subscriptions worldwide, of which approximately 1 billion are for smartphones.⁷ Smartphones allow for advanced computing capabilities once available only through traditional computers and provide other functions not obtainable through a computer, such as GPS and on-the-move sensors (eg, accelerometer).

mHealth interventions that used evidence-based content have shown promise for helping people sustain behaviors that lead to improved health outcomes.^8–10 Systematic reviews of mHealth studies of smoking cessation, diabetes self-management, and weight management have shown positive short-term behavioral and clinical outcomes,^{8 11–13} but few have assessed treatment fidelity, that is, the accurate delivery, receipt, and enactment of the intervention. Treatment fidelity is critical because the valid interpretation and translation of intervention studies depends on treatment fidelity assessments.^14–16 In mHealth, information flows both to and from an individual. Because mHealth allows us to use interactive capabilities to receive and send data in real time, it may be particularly useful for weight loss management. Weight loss requires daily, if not hourly, behavior change involving multiple factors that may interact and change in importance over time. We reviewed standards for the assessment of the treatment fidelity of mHealth interventions and strategies used in three randomized control trials (RCTs) targeting the initiation and maintenance of weight loss.

Treatment fidelity

Treatment fidelity utilizes methodological strategies to ensure the integrity, including the reliability and validity, of behavioral interventions and is integral to the interpretation and generalizability of research. Comprehensive assessment of treatment fidelity in pilot studies strengthens large-scale clinical trials¹⁶ and supports appropriate dissemination of effective interventions in clinical practice.^{14 16} For clinicians who lack experience with behavior change research, evidence showing which strategies support treatment fidelity provides guidelines for translating and implementing research-based interventions into clinical practice.¹⁴

The Treatment Fidelity Workgroup of the NIH Behavior Change Consortium (BCC)¹⁴ developed goals for integrating fidelity strategies into five research process phases: study design, training providers, treatment delivery, treatment receipt, and treatment skills enactment. Metrics for assessing the receipt and dosing of mHealth treatments include tracking the length of time spent on study websites and the number of contacts completed.¹⁷ While training providers, treatment delivery, treatment receipt, and treatment skills enactment are first developed in the design phase, they must be continuously assessed and incorporated into the workflow during subsequent phases. It is important to acknowledge that while mHealth interventions can be implemented automatically, the fidelity of treatment delivery, treatment receipt, and treatment skills enactment are still essential to ensure an increased likelihood of obtaining validity and reliable information to ensure appropriate implementation in the future. Three mHealth studies of weight management and strategies that illustrate application of these recommendations are summarized below.

mHealth studies of weight management

Table 1 presents treatment fidelity strategies from three recent mHealth weight loss interventions: Sustaining Weight Loss through Text Messaging (mSustain),¹⁰ Weighing Everyday to Improve and Gain Health (WEIGH),¹⁸ and Shape Plan.¹⁹ mSustain¹⁰ was a 3-month, three-arm mixed methods study of obese adult patients (N=120) who had recently lost ≥5% of their body weight and were randomized to either a promotion- or prevention-framed weight loss message group or a general health message group (table 2). Thirty text messages were queued and delivered daily to participants’ cell phones at 8:00 in their respective time zone²⁰ nationwide, which accommodated extended travel. Outcome data were collected at 1 and 3 months after baseline via an online survey accessible by computer or smartphone.

Table 1.

Treatment fidelity strategies with mHealth examples¹⁴

Goal and criteria	Definition	mSustain10	WEIGH18	Shape plan19	mHealth challenges
Study design planning
Treatment standardization	Plan that all participants within a treatment condition receive the same dose, including frequency and contact length	Fully automated system sent texts at 8:00 daily in the participant's respective time zone and adjusted for extended travel Automated log of delivered text messages generated	Instruction to weigh daily for all subjects Automated system sent weekly emails Equal access to weight graphs and lessons	Fully automated system sent texts at 8:00 daily and weekly emails Automated log of delivered text messages generated	Planning for circumstances that may impede intervention delivery such as software updates from third party applications, rebooting of servers, loss of power
Preventing contamination	Plan that treatments delivered across conditions do not influence each other	Automated message delivery	Automated message delivery	Automated message delivery	Preventing participants in each treatment arm from sharing information with each other
Plan for implementation setbacks	Plan for setbacks (ie, patients or clinicians dropping out)	Daily confirmation of text message delivery to intervention staff	Weekly email also assessed technical issues	Notification sent for unsuccessful text message attempts	Planning for interoperability and technical issues
Tracking provider training
Standardize training and provider skills	Ensure that training for providers is consistent Increases reliability of skills over time	Automated system did not require training Standardized intervention did not introduce provider concerns	Automated system did not require training Standardized intervention did not introduce provider concerns	Automated system did not require training Standardized intervention did not introduce provider concerns	Semi-automated interventions where provider skills may change over time but the automation is consistent
Tracking treatment delivery
Adherence to treatment protocol	Ensure treatment is being delivered as intended over time	Standardized content and auto-delivered consistently Text message program auto-recorded a daily log of delivered content and timing of the message sent	Weekly assessment of self-weighing frequency and reminders to weigh daily Record of log-ins to view weight graph	Automated system ensured text message delivery per protocol Weekly email reminders to adhere to treatment for those with non-adherence	Conditions that hinder intervention delivery such as third party application software updates, server reboots, or power loss
Minimize contamination	Ensure that treatments delivered across conditions do not influence each other and that treatment groups do not influence each other, especially when treatment delivered by the same provider	Standardized content and auto-delivered consistently Program auto-recorded a daily log of delivered content and timing of the message sent All groups received messages and self-weighed for consistency	Participants in both groups received mobile scales to ensure an objective measure of self-weighing frequency Control participants did not receive any other contact	Standardized content and delivered to intervention group only Design controlled for contact and pedometer use to isolate behavioral goals and text messaging components	Tracking when participants in individual treatment arms share information with each other
Tracking treatment receipt
Participant receipt of intervention	Actively assess that participants are receiving the intervention	Message logs Participants completed an online questionnaire asking if they read the messages	Daily use of mobile scale with cellular connectivity Log-ins to weight graphs and clicks to website	Receipt of self-monitoring data Attendance at two group sessions	Loss of cell phone signal or internet connection or dead battery
Participant comprehension	Assess if participants cognitively understand the received intervention	Pilot tested the messages for comprehension Interviewed participants about message content	Orientation session with instructions on how to use the mobile scale	Tested behavioral goal approach in previous trials Orientation session on using text messaging	Technology impeding the understanding of content that is delivered
Participant ability to use skills	Assess that participants have the ability to use the behavioral skills that are taught and encouraged	Skills were verified prior to enrolment Participants gave thoughts on the messages Those with physical limitations discussed inability to perform skills	Orientation session with instructions on how to use the mobile scale	Orientation session on using text messaging Assessment of ease of use of text messaging	Participants’ ability to use new technologies and troubleshoot technical difficulties
Tracking treatment enactment
Participant performed behavioral/ cognitive skills, or used given tools	Actively assess that participants are using the cognitive skills and/or performing the behavioral skills delivered	Participant exercise behaviors were measured through both an online questionnaire and in a phone interview Participants completed an online questionnaire asking if they read the messages	Self-report surveys of eating and exercise associated with weight loss Objective measure of self-weighing frequency Log-ins to weight graphs Number of clicks to web-based lessons	Self-reported questionnaires on behavioral goals Receipt of self-monitoring data Attendance at group sessions Goal attainment	Loss of cell phone signal or internet connection or dead battery

Table 2.

mSustain survey to assess enactment of treatment¹⁰

	Total (N=84)*	Control (N=26)	Promotion (N=28)	Prevention (N=30)
What do you do when you receive a weight loss sustaining text message?	83	25	28	30
Ignore it completely	0 (0%)	0 (0%)	0 (0%)	0 (0%)
Read it occasionally	3 (4%)	1 (4%)	0 (0%)	2 (7%)
Read it after accumulating too many	2 (2%)	2 (8%)	0 (0%)	0 (0%)
Read it when I get time	16 (20%)	2 (8%)	4 (14%)	10 (33%)
Read it right away	60 (74%)	20 (80%)	24 (86%)	18 (60%)
How much do you read messages you received?	82	24	28	30
Not at all	2 (3%)	1 (4%)	0 (0%)	1 (3%)
Read about a quarter of a message	0 (0%)	0 (%)	0 (0%)	2 (7%)
Read about half of a message	0 (0%)	0 (%)	0 (0%)	0 (0%)
Read about three-quarters of a message	2 (3%)	0 (0%)	0 (0%)	0 (0%)
Read the whole message	76 (95%)	23 (96%)	28 (100%)	27 (90%)

*Eighty-four out of 120 participants responded to this survey.

WEIGH¹⁸ was a 6-month RCT of overweight and obese adults (N=91). WEIGH included a cellular-connected mobile scale for daily weighing, web-based weight graphs, and weekly emails with tailored feedback and weight control lessons. The cellular-connected scale transmitted weight data to a computer server that displayed weight graphs on a website and data for each participant on self-weighing frequency and weight loss in a separate researcher interface. In order to maintain adherence to daily weighing, participants took the scale with them to work or when they were out of town or on vacation. A delayed intervention control group was blinded to the focus on daily weighing.

Shape Plan¹⁹ was a 6-month RCT of obese black women (N=50) who were randomized to a text-messaging intervention or education control arm. Both arms received weight loss information, pedometers, a baseline face-to-face group session, and a skills training DVD. The intervention group received individualized behavior change goals (eg, walk 10 000 steps daily) based on an algorithm that accounted for participants’ needs and self-efficacy around changing behaviors, and expected caloric deficit to achieve goals. A fully-automated system for tracking daily goals included one daily text message at 8:00 requesting performance evaluation of the previous day's goals, an automated feedback message on progress, and tips for changing low-scoring goals.

mHealth treatment fidelity strategies

Planning study design

Treatment fidelity strategies used during the study design of weight loss interventions should include checks that the within-group treatment dose/intervention is consistent regarding planned frequency and amount of content (table 1); across groups, the potential for contamination and implementation setbacks or delays (eg, low recruitment or drop-out) should be anticipated.²¹ The use of mHealth can allow for automated standardized digital doses of the intervention and the use of timing algorithms to tailor content to individuals (ie, daily or weekly tailored feedback messages) to ensure receipt of the correct message at the appropriate time.

Tracking provider training

Standardized training and continuous evaluation of content providers reduces provider interaction that might bias treatment implementation.¹⁴ Training should be observed and measured over time to maintain consistency and intervention quality.

mHealth interventions permit automated delivery features that can be fully or partially digitized, making provider training relevant only where the study includes in-person components. For example, an mHealth intervention that automatically tailors diet recommendations based upon a mobile phone diary can reduce dependence upon actual providers to deliver more frequent treatments, allowing training to focus on less frequent in-person encounters. In the three examples of mHealth studies discussed here, automation of outgoing text and email messages and web-based materials reduced the potential for discrepancies in provider performance typical of traditional research models.

Tracking treatment delivery

Treatment delivery strategies include processes that monitor how and whether the treatment was delivered according to the study protocol as well as any contamination across groups.¹⁴ Although first planned for in the design phase, treatment delivery must be actively assessed throughout the study; standardization and protocol checks are common strategies. Implementation setbacks, such as messages not being transmitted or received (eg, loss of cell phone signal, dead battery), can be addressed using programming code that delivers confirmation or notification of errors to the interventionist automatically (ie, confirmation of data transmission). Because mHealth interventions can be automated, programming code can ensure consistent treatment delivery. Furthermore, ongoing logs can track overall delivery of content and delivery to the correct intervention group. In the treatment delivery phase, the interventionist must read and assess the automated notifications. Automated notifications can be delivered via text message to the interventionist's phone, email, or log to be checked.

Tracking treatment receipt

Receipt of treatment refers to whether participants actually received intervention components and understood what they received. In behavioral weight control studies using mHealth, a mobile-specific treatment fidelity check is required due to the potential for technology-related errors (eg, loss of cell phone signal, dead battery) that might prevent participants from receiving the intervention. In addition, tracking understanding of the intervention involves assessing ability to perform cognitive (eg, overcoming the temptation to eat sweets) and physical behaviors (eg, specific exercise techniques).¹⁴ Unlike checks that in-person treatments have been received, mobile-delivered interventions may require pilot testing, ongoing surveys, or follow-up surveys.

Tracking treatment enactment

The fifth treatment fidelity check refers to the actual performance of the skills (eg, the subject takes the prescribed medication or avoids salt) by participants in the planned situation and at the proper time.¹⁴ It is possible, albeit challenging, to monitor and assess the enactment of skills using mobile technology. Participants could take pictures of their meals with their smartphones to demonstrate food choice skills or transmit a count of daily steps via a pedometer attached to a mobile phone and paired with phone-based GPS tracking software to illustrate distance traveled. For mHealth studies of blood glucose, participants could transmit an ongoing log of their blood sugar values via text or kept in a log in their phone, or use a mobile-based monitor that automatically records the values in the phone and transmits them to the researcher or provider.

mHealth challenges

mHealth is not without its treatment fidelity challenges. For example, third party applications can interfere with intervention delivery when their software is being updated. Interoperability with new devices and integration into an electronic health record (EHR) is improving but still limited. Other issues that are difficult to address include participants’ forgetting to charge their phones, losing cell phone signals, or traveling overseas.

Discussion

Assessment and assurance of treatment fidelity is critical for the translation of research findings into real-world settings.¹⁴ As mHealth is increasingly investigated as a health delivery platform and translated into practice for complex behaviors such as weight loss that require frequent behavioral change, the fidelity of intervention delivery requires evaluation to ensure valid findings and identify adherence and technical problems. Statistical validity requires minimization of within-group error for an outcome variable to detect significant change and enhance the likelihood of sound conclusions.²¹

Intervention fidelity should be a primary goal of the initial planning and design of mHealth interventions,¹⁴ both at the onset and consistently throughout the study during process evaluation. As smartphones become increasingly more common²² and scientists learn how to leverage sophisticated features to deliver care and assess behavior (eg, through geospatial tracking and apps that tailor information in real time), successful strategies to ensure intervention fidelity will be developed.²³ A mobile device will be able to monitor and relay information automatically to the healthcare provider or researcher as targeted behaviors are occurring. Whether or not patients monitor their blood sugar will be uploaded automatically to the EHR via mobile phones. Passive monitoring of daily steps through a smartphone-based pedometer or tracking of proximity to an eating location through GPS technology may be employed. Although these technologies are available, their optimal use for patient outcomes and integration into health systems are underdeveloped.

Finally, mHealth interventions must adapt to ensure ‘stickiness’ and participant follow-through. Over time, it can be easy to ignore a text message,²⁴ become disenchanted with the novelty of mobile apps, and forget to charge batteries in your wrist-worn physical activity tracker.²⁵ While treatment needs to be standardized to clarify the intervention impact, content also needs to adapt to and engage participants over time to prevent a loss of interest in longer-term mHealth interventions. Weight loss interventions will need to adapt to participants who relapse, and retarget content to address new needs and barriers and re-energize exercise and healthy eating.

Conclusions

While mHealth is increasingly used as a medium for delivering health promotion interventions such as those encouraging weight loss, accurate assessment of intervention fidelity and treatment receipt are required to ensure reliable intervention delivery and valid outcomes. A specific plan to monitor fidelity will help counter threats to validity and allow investigators to draw more accurate conclusions regarding the effectiveness of their mHealth intervention. This will ultimately enhance the implementation and dissemination of these interventions into practice settings.