User Engagement With mHealth Interventions to Promote Treatment Adherence and Self-Management in People With Chronic Health Conditions: Systematic Review

Abstract

Background

There are numerous mobile health (mHealth) interventions for treatment adherence and self-management; yet, little is known about user engagement or interaction with these technologies.

Objective

This systematic review aimed to answer the following questions: (1) How is user engagement defined and measured in studies of mHealth interventions to promote adherence to prescribed medical or health regimens or self-management among people living with a health condition? (2) To what degree are patients engaging with these mHealth interventions? (3) What is the association between user engagement with mHealth interventions and adherence or self-management outcomes? (4) How often is user engagement a research end point?

Methods

Scientific database (Ovid MEDLINE, Embase, Web of Science, PsycINFO, and CINAHL) search results (2016-2021) were screened for inclusion and exclusion criteria. Data were extracted in a standardized electronic form. No risk-of-bias assessment was conducted because this review aimed to characterize user engagement measurement rather than certainty in primary study results. The results were synthesized descriptively and thematically.

Results

A total of 292 studies were included for data extraction. The median number of participants per study was 77 (IQR 34-164). Most of the mHealth interventions were evaluated in nonrandomized studies (157/292, 53.8%), involved people with diabetes (51/292, 17.5%), targeted medication adherence (98/292, 33.6%), and comprised apps (220/292, 75.3%). The principal findings were as follows: (1) >60 unique terms were used to define user engagement; “use” (102/292, 34.9%) and “engagement” (94/292, 32.2%) were the most common; (2) a total of 11 distinct user engagement measurement approaches were identified; the use of objective user log-in data from an app or web portal (160/292, 54.8%) was the most common; (3) although engagement was inconsistently evaluated, most of the studies (99/195, 50.8%) reported >1 level of engagement due to the use of multiple measurement methods or analyses, decreased engagement across time (76/99, 77%), and results and conclusions suggesting that higher engagement was associated with positive adherence or self-management (60/103, 58.3%); and (4) user engagement was a research end point in only 19.2% (56/292) of the studies.

Conclusions

The results revealed major limitations in the literature reviewed, including significant variability in how user engagement is defined, a tendency to rely on user log-in data over other measurements, and critical gaps in how user engagement is evaluated (infrequently evaluated over time or in relation to adherence or self-management outcomes and rarely considered a research end point). Recommendations are outlined in response to our findings with the goal of improving research rigor in this area.

Trial Registration

PROSPERO International Prospective Register of Systematic Reviews CRD42022289693; https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42022289693

Introduction

Background

As smartphones have become an integral part of modern daily life [1,2], mobile health (mHealth) interventions for treatment adherence and self-management promotion have rapidly developed. These interventions often use existing smartphone features, such as sending SMS text notifications as cues to take prescribed medications [3]; others may integrate external technologies, such as a Bluetooth-enabled glucometer linked to a smartphone app to track blood glucose levels over time with the goal of supporting diabetes management [4]. In general, more frequent engagement, or interaction, with mHealth tools is expected to result in improved treatment adherence or self-management [5]. However, mHealth tools are frequently abandoned by users. Among mobile phone users in the United States, more than half reportedly downloaded an mHealth app but nearly half also stopped using the app due to high data entry burden, low interest, and costs [6].

There is a disconnect between research findings supporting positive correlations between engagement and adherence or self-management and user tendencies to stop using mHealth tools. This discrepancy may reflect an argument put forth by Arigo et al [7] that the mHealth field “lacks a science of engagement.” Specifically, there is (1) a lack of consensus in how mHealth user engagement is measured, defined, and reported; (2) no consensus on the optimal level of mHealth user engagement to facilitate meaningful behavior change; and (3) infrequent consideration of user engagement as a research end point. Not treating user engagement as a research end point suggests that this domain is poorly defined and haphazardly evaluated, particularly in terms of how user engagement might evolve over the course of the intervention or relate to behavioral and health outcomes. While the tendency for mHealth use to decline over time suggests that users will not experience maximum benefit from accessing these tools, poor measurement of user engagement has presented a challenge to researchers’ abilities to characterize exactly how and why engagement may decrease and how these decreases may affect intervention outcomes. These critical gaps in the science of user engagement significantly limit the utility, effectiveness, uptake, and scalability of mHealth interventions for adherence and self-management promotion.

Recent systematic reviews have examined aspects of user engagement with mHealth interventions for specific diagnoses, including hypertension [8], physical activity [9,10], depressive symptoms [11,12], and mental health conditions [13]. The applicability of these reviews to adherence and self-management mHealth interventions is significantly limited by the small number of studies included and a lack of unified focus on adherence and self-management behaviors. These prior reviews have been further limited by using a very broad definition of engagement to include usability, feasibility, user satisfaction, and acceptability [13], limiting the review to studies in which only postassessment retention data were obtained [11] (excludes interventions earlier in the design phase), and focusing on design features associated with user engagement rather than the evaluation of user engagement itself [10]. Another review identified a range of valid and reliable measurement approaches for evaluating user engagement with mHealth interventions for behavior change but used a snowballing method to identify sources rather than a rigorous systematic review of the literature [14]. To encourage the continued use of, and optimal engagement with, mHealth interventions to facilitate adherence and self-management behavior change, it is imperative to comprehensively and systematically evaluate the recent scientific landscape of mHealth user engagement with a clear focus on adherence and self-management behavior.

Objectives

In response to gaps identified in the current scientific literature [7], our registered systematic review aimed to (1) characterize user engagement with interventions promoting mHealth treatment adherence or self-management for adults and youth with health conditions and (2) generate user engagement–focused research recommendations. “User engagement” with mHealth tools can be conceptualized as both behavioral (the extent of use) and experiential (the subjective experience of interacting with the technology) [15]. To enhance the practical application of our review findings, we focused on the behavioral aspects of user engagement to evaluate the degree of use and interaction with the mHealth tool [7] among users with chronic health conditions. Users’ behavioral interaction with mHealth tools, features, and associated behavior change components is known as “Little e” engagement. In theory, increased “Little e” engagement is expected to contribute to increased engagement in the desired health behavior, known as “Big E” [16]. Thus, better precision in how behavioral interaction, or “Little e” engagement, is empirically evaluated could help to facilitate greater changes in “Big E” outcomes, thus improving the overall efficacy of mHealth interventions for adherence and self-management. A systematic review approach was selected due to expected heterogeneity in both the measurement of user engagement and adherence and self-management outcomes, which precludes the use of a meta-analysis [17-19]. We specifically aimed to answer the following research questions: (1) How is user engagement defined and measured in studies of mHealth interventions to promote adherence to prescribed medical or health regimens or self-management among people living with a health condition? (2) To what degree are patients engaging with these mHealth interventions? (3) What is the association between user engagement with mHealth interventions and adherence or self-management outcomes? (4) How often is user engagement a research end point?

We also developed the following exploratory question: are there differences in user engagement measurement approaches and levels between studies that provide monetary compensation and those that do not?

Methods

This systematic review was registered with PROSPERO (CRD42022289693) and prepared in accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. The review team prepared and followed a standard manual of procedures designed for this systematic review.

Ethical Considerations

Institutional review board approval was not required because the study was not considered human participant research.

Information Sources and Search Strategy

The search strategy (Multimedia Appendix 1) was developed by the first and last authors (CE and KR) in collaboration with an informationist at Johns Hopkins Libraries; implemented in Ovid MEDLINE, Embase, Web of Science, PsycINFO, and CINAHL; and restricted to manuscripts published between 2016 and 2021. All citations returned from the search were imported into our Covidence (Veritas Health Innovation Ltd) [20] database for screening and data extraction.

Eligibility Criteria

The inclusion and exclusion criteria are presented in Textbox 1.

Inclusion and exclusion criteria.

Inclusion criteria

• Peer-reviewed manuscripts reporting on original qualitative or quantitative investigations published in English between 2016 and 2021

• Participants followed a medical or health regimen or engaged in adherence or self-management activities for a chronic physical or mental health condition

• A mobile health (mHealth) intervention was used by these participants or their caregivers in a home setting, was at least partially automated (could not only include manual 2-way SMS text messaging or video web conferencing), and was accessible on a mobile device (smartphone or tablet device, including internet browser–based programs)

• The primary intervention target was treatment adherence (eg, taking medication, exercising, or following a diet); or self-management of the health condition, often measured by a health outcome associated with treatment adherence behavior (eg, glycated hemoglobin test, viral load, or BMI)

• User engagement (use and interaction) with the mHealth intervention was a measured study outcome, either by objective (eg, app-recorded log-in data) or subjective (eg, user self-report or qualitative interviews) metrics

Exclusion criteria

• Meta-analyses, systematic reviews, published abstracts, dissertations, and published protocols, as well as studies reporting on usability testing or intervention development only

Selection Process

Citations were imported into Covidence; duplicate citations were removed; and title and abstract screening was conducted, followed by a full-text review. At each level of review, 2 separate review team members evaluated each article against the inclusion and exclusion criteria, with discrepancies resolved by CE and N Muthusamy. Any study meeting the inclusion criteria after the full-text review progressed to the data extraction phase (detailed in the next subsection).

Data Extraction

Overview

Data extraction was performed by 2 separate review team members independently of each other using a standard data extraction form developed by the study team, with discrepancies resolved by CE and N Muthusamy. For each study, the reviewers recorded both quantitative and qualitative data relevant to study design (eg, randomized controlled trial and case-control) and methodology (eg, monetary compensation for participation); demographic characteristics of participants; mHealth intervention targets and characteristics; measurement of user engagement, including whether it was a research end point (a key outcome being measured and potentially impacted by participation in the intervention); and the terminology used to describe the behavioral aspects of user engagement (researcher-evaluated user interactions with the mHealth technology).

Study results were summarized as described in the following subsections.

Level of Engagement With the Intervention

This was categorized as “high,” “medium,” “low,” “>1 level reported due to the use of multiple measurement or analytic approaches,” or “not characterized.” Categorizations were assigned based on the language used by the authors to characterize users’ level of engagement with the mHealth intervention (eg, the authors described user engagement with the mHealth intervention as “high”).

Change in Level of Engagement

This was categorized as “increased,” “no change,” “decreased,” “>1 direction reported due to the use of multiple measurement or analytic approaches,” or “not assessed.” For studies that assessed change in engagement over time, categorizations were assigned based on the data presented by the authors (eg, the authors presented data showing that the user engagement measurement decreased over time).

Association With Treatment Adherence or Self-Management Study Outcomes

This was categorized as “higher engagement, positive treatment adherence or self-management outcomes”; “moderate engagement, positive treatment adherence or self-management outcome)”; “lower engagement, positive treatment adherence or self-management outcomes”; “no association”; “>1 association reported due to the use of multiple measurement or analytic approaches”; or “not assessed.” For studies that assessed this association, categorizations were assigned based on how the authors reported and framed the study results and conclusions (eg, the authors’ reporting and framing of study results and conclusions suggested that higher engagement with the mHealth intervention was associated with positive study outcomes, such as higher treatment adherence or improved self-management outcome or outcomes).

Technology Dosage

This was categorized as “yes, given” (the researchers told participants how often or in what way or ways they should use the mHealth intervention components, such as complete 1 module per week and log medication administration in the app every day) or “no, not given.”

Minimum Engagement Research Benchmark

This was categorized as “yes, selected” (the researchers reported in their manuscript that a minimum cutoff for technology engagement was set as an empirical outcome to denote adequate participant engagement; eg, to be adequately engaged, a participant needed to use the Bluetooth-enabled glucometer at least once a day during the study period) or “no, not selected.”

No Formal Risk-of-Bias Assessment

We decided not to conduct a formal risk-of-bias assessment, given that the primary aim of this review was to characterize the evaluation and measurement of user engagement rather than certainty in the primary study results. Therefore, it was deemed inappropriate to evaluate the studies using standard risk-of-bias assessment tools.

Synthesis Methods

Statistical analyses were conducted using SPSS software (version 28.0; IBM Corp) [21]. The extracted data were summarized using frequencies and percentages. Methods of measuring user engagement with the mHealth intervention were thematically grouped into discrete measurement categories. Subgroup sensitivity analyses were conducted to examine studies involving pediatric samples (participants aged 0-18 y or aged up to 25 y if the sample was characterized as “pediatric” by the authors) separately from those involving adults only (participants aged >18 y). In these age-based subgroup analyses, of the 292 included studies, 5 (1.7%) were excluded due to including both pediatric and adult participants, and 1 (0.3%) was excluded due to not reporting participant ages. Given that this investigation was designed as a systematic review, no effect measures or meta-regressions were used. No missing summary statistics or data conversions were used.

Results

Search Results and Screening Process

Figure 1 presents the PRISMA flow diagram of the screening process. The initial search returned 3955 citations, from which 70 (1.77%) duplicates were removed. During title and abstract screening, the remaining 3885 studies were screened, and 2736 (70.42%) were excluded. During full-text screening, the remaining 1149 studies were evaluated, and 857 (74.59%) were excluded. The primary reasons for exclusion were as follows: ineligible manuscript type (eg, published abstract; 330/857, 38.5%), user engagement with the mHealth intervention was not measured (206/857, 24%), and ineligible participant population (111/857, 12.9%). The final review included 292 studies [22-313] (refer to Multimedia Appendix 2 for all included studies and characteristics).

Figure 1.

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram of the screening process. mHealth: mobile health.

Basic Study and mHealth Intervention Characteristics

Nearly half of the studies (135/292, 46.2%) were conducted in the United States and used a randomized controlled trial design. The median number of participants was 77 (IQR 34-164). Nearly half (135/292, 46.2%) were considered feasibility studies. The median study length was 90 (IQR 60-180) days. Diabetes (51/292, 17.5%) and mental health conditions (35/292, 11.9%) were the most common diagnoses. Study characteristics were similar between adult and pediatric studies, with the exception of health conditions, reflecting expected age-based differences in diagnoses more common among adults than among children (eg, type 2 diabetes and substance use were more commonly studied in adult samples than in pediatric samples).

Table S1 in Multimedia Appendix 3 contains specific details on the mHealth interventions’ adherence or self-management targets, intervention components, and intended users. The most frequently targeted adherence or self-management concern was taking medication (98/292, 33.6%), followed by exercise (93/292, 31.8%) and diet (73/292, 25%). Nearly all mHealth interventions were used by the patient (291/292, 99.7%), but some of them included health care providers (58/292, 19.9%) or caregivers (20/292, 6.8%). The majority of mHealth interventions comprised a mobile app (220/292, 75.3%), SMS text messaging or push notifications (74/292, 25.3%), websites or web portals not within a mobile app (48/292, 16.4%), and nonwearable monitoring devices (47/292, 16.1%). Interventions that did not include a mobile app primarily comprised text messaging, a wearable device, a nonwearable device, video web conferencing or telephone calls, or a website or web portal not within a mobile app. Nearly three-quarters of the interventions (216/292, 74%) prompted users to engage with the mHealth intervention. Less than half of the studies (124/292, 42.5%) provided monetary compensation for participation. Only 10.6% (31/292) allowed users to continue using the mHealth intervention after the formal study period. Intervention characteristics were generally similar between adult and pediatric studies, with the exception of intervention target behavior, reflecting expected age-based differences in health concerns more common among adults than among children (eg, exercise, mental health management, and drug or alcohol use or abuse were more commonly targeted in adult samples than in pediatric samples).

How Is User Engagement Defined and Measured?

Definition

Terminology defining user engagement varied widely (Table S2 in Multimedia Appendix 3). There were 33 unique terms used to define engagement that appeared in at least 2 (0.7%) of the 292 studies. Of the 292 studies, 31 (10.6%) studies each used a unique term that appeared in only that 1 study. “Use” (102/292, 34.9%) and “engagement” (94/292, 32.2%) were the most common terms. Although most terms were synonymous with “use,” “engagement,” or “interaction” with the technology (reflecting our a priori definition of behavioral user engagement and our search strategy), other studies notably used disparate terms, including “acceptability” (26/292, 8.9%), “fidelity” (5/292, 1.7%), “satisfaction” (6/292, 2%), and “perception” (6/292, 2%).

Measurement

Across all studies, 11 distinct user engagement measurement approaches emerged, comprising both objective (n=9, 82%) and subjective (n=2, 18%) methods. User engagement was most frequently evaluated via objective user log-in data from the app or web portal (eg, number of log-ins; 160/292, 54.8%), followed by manually entering data in an app (77/292, 26.4%), qualitative interviews (54/292, 18.5%), and responding to text notifications (49/292, 16.8%). There were “other objective measures” that did not fall into any of the 11 main categories (15/292, 5.1%; eg, notification reading rate or downloading podcasts). These results were similar between adult and pediatric studies (Table 1; Table S3 in Multimedia Appendix 3).

Table 1.

How is user engagement measured (n=292)?

Measurement methods			Examples		Studies, n (%)a
Objective measures
	User log-in data retrieved from app or website	Number of log-ins to app or website Length of time spent in app or website Frequency of accessing specific features within the app or website		160 (54.8)
	Manual user data entry in app-or website-based self-monitoring diaries	User manually enters data in the app, such as blood glucose level, date and time when medicine was taken, or blood pressure values		77 (26.4)
	Response to SMS text messages or push notifications	User types and sends a response to a SMS text message asking if they took their medicine that day		49 (16.8)
	Number or proportion of intervention program modules completed within app or website	User completes 3 out of 6 possible modules on pain management skills		48 (16.4)
	Interacting via chats, phone calls, or social media posts	Number of times user sends a chat message to care team through app		33 (11.3)
	Wearing an electronic monitoring device	Length of time the user wore a Fitbit device to track daily step count		26 (8.9)
	Using a nonwearable electronic monitoring device	Medication adherence is monitored with an electronic pill bottle that tracks when the bottle is opened and closed to administer medicine		26 (8.9)
	Submitting videos via app	Medication adherence is measured using a mobile app designed to directly observe therapy		5 (1.7)
	Other objective measures	Notification message reading rate or downloading podcasts		13 (4.5)
Subjective measures
	Qualitative interview	User completes a qualitative interview about their experience using the mHealthb app		54 (18.5)
	Participant-reported survey	User self-reports frequency of using the app		29 (9.9)

^aPercentages do not add up to 100% because studies could fall into >1 category.

^bmHealth: mobile health.

When examining engagement definitions by measurement approaches, “use” (7/9, 78%) and “engagement” (9/9, 100%) were most commonly used across the nine objective measurement approaches. The exceptions were “wearing an electronic monitoring device” and “submitting videos via app” for which “adherence” (11/26, 42%, 2/5, 40%, respectively) was most commonly used within the measurement approach. Qualitative interviews had the widest range of terminologies used, with “user experience” being the most common (21/54, 39%; Table 2).

Table 2.

Associations between user engagement evaluation methods and definitions.

Evaluation methods	Termsa used to define user engagement, n (%)b
User log-in data retrieved from app or website (n=160)	Use: 78 (49) Engagement: 62 (39) Feasibility: 27 (17) Adherence: 25 (16)
Manual user data entry in app- or website-based self-monitoring diaries (n=77)	Use: 31 (40) Engagement: 24 (31) Adherence: 19 (25) Feasibility: 17 (22) Compliance: 10 (13)
Response to SMS text messages or push notifications (n=49)	Engagement: 21 (43) Response: 19 (39)
Number or proportion of intervention program modules completed within app or website (n=48)	Use: 22 (46) Engagement: 12 (25) Adherence: 11 (23)
Interacting via chats, phone calls, or social media posts (n=33)	Engagement: 14 (42) Use: 13 (39) Feasibility: 5 (15) Compliance: 4 (12)
Wearing an electronic monitoring device (n=26)	Adherence: 11 (42) Engagement: 9 (35) Feasibility: 6 (23)
Using a nonwearable electronic monitoring device (n=26)	Use: 9 (25) Adherence: 8 (31) Engagement: 6 (23)
Submitting videos via app (n=5)	Adherence: 2 (40) Use: 2 (40) Compliance: 1 (20) Engagement: 1 (20) Acceptability: 1 (20)
Other objective measures (n=13)	Engagement: 7 (54) Feasibility: 2 (15) Use: 2 (15) Response: 2 (15)
Qualitative interview (n=54)	User experience: 21 (39) Engagement: 18 (33) Use: 17 (31) Acceptability: 13 (24) Feasibility: 8 (15) Adherence: 8 (15)
Participant-reported survey (n=29)	Use: 13 (45) Feasibility: 10 (24) Engagement: 6 (21) Acceptability: 6 (21) Adherence: 5 (17)

^aWe report terms used to describe user engagement in at least 10% of the studies using a given evaluation method; this cutoff was selected to enhance interpretability due to the wide range of terms used to describe user engagement (refer to Table S2 in Multimedia Appendix 3 for details of user engagement definitions).

^bPercentages within categories do not add up to 100% because studies could fall into >1 category.

The use of user log-in data was the most common measurement method across mHealth intervention components, except for SMS text messaging or push notifications and wearable devices. When SMS text messaging or push notifications was an intervention component, response to SMS text messaging or push notifications was the most common metric (38/74, 51%). When a wearable device was an intervention component, wearing an electronic monitoring device was the most common metric (26/39, 67%; Table S4 in Multimedia Appendix 3).

To What Degree Are Participants Engaging With These mHealth Interventions?

User Engagement Level

User engagement level was characterized in two-thirds of the reviewed studies (195/292, 66.8%), of which a little more than half (99/195, 50.8%) reported >1 level of engagement due to the use of multiple measurement methods or analyses. Only one-third of the studies (99/292, 33.9%) examined change in engagement over time; when it was examined, engagement tended to decrease (76/99, 77%). These results were similar between adult and pediatric studies (Table 3).

Table 3.

Degree of engagement and association with treatment adherence or self-management.

Characteristic		All studies (n=292), n (%)	Adult studies (n=241), n (%)	Pediatric studies (n=45), n (%)
User engagement level
	High	63 (21.6)	55 (22.8)	7 (15.6)
	Medium	6 (2.1)	4 (1.7)	2 (4.4)
	Low	27 (9.2)	19 (7.9)	6 (13.3)
	>1a level reported	99 (33.9)	81 (33.6)	18 (40.0)
	Not characterized	97 (33.2)	82 (34.0)	12 (26.7)
Change in user engagement level
	Increased	3 (1.0)	3 (0.01)	0 (0)
	No change	15 (5.1)	13 (5.4)	1 (2.2)
	Decreased	76 (26.0)	61 (25.3)	13 (28.9)
	>1 direction reported	5 (1.7)	4 (1.7)	1 (2.2)
	Not assessed	193 (66.1)	160 (66.4)	30 (66.7)
Association with adherence or SMb outcomes
	Higher engagement, positive adherence or SM outcomes	60 (20.5)	49 (20.3)	8 (17.8)
	Moderate engagement, positive adherence or SM outcomes	1 (0.3)	1 (0.4)	0 (0)
	Lower engagement, positive adherence or SM outcomes	1 (0.3)	1 (0.4)	0 (0)
	No association	18 (6.2)	15 (6.2)	3 (6.7)
	>1 association reported	23 (7.9)	18 (7.5)	5 (11.1)
	Not assessed	189 (64.7)	157 (65.1)	29 (64.4)

^aCategories with “>1” finding reflect the use of multiple measurements or analyses, leading to multiple results in different directions (eg, for “Change in user engagement level,” engagement is shown to increase and decrease depending on the measurement used).

^bSM: self-management.

Measurement Approach by User Engagement Level

Compared to studies characterized as having high user engagement, studies with low user engagement tended to measure (≥10% difference) engagement with user log-in data (19/27, 70% vs 36/63, 57%) and module completion (7/27, 26% vs 9/63, 14%). Compared to studies characterized as having low user engagement, studies with high user engagement tended to measure engagement with response to text notifications (14/63, 22% vs 1/27, 4%); interacting via chats, phone calls, or social media posts (13/63, 21% vs 1/27, 4%); wearing an electronic monitoring device (6/63, 10% vs 0); using a nonwearable electronic monitoring device (7/63, 11% vs 0); and qualitative interviews (12/63, 19% vs 2/27, 7%; Table 4).

Table 4.

Measurement approach by user engagement level with intervention.

Measurement	Characterization of user engagement level with intervention
	Low (n=27), n (%)a	Medium (n=6), n (%)a	High (n=63), n (%)a	>1 (n=99), n (%)a	Not characterized (n=97), n (%)a
User log-in data retrieved from app or website	19 (70)	5 (83)	36 (57)	57 (57)	43 (44)
Manual user data entry in app-or website-based self-monitoring diaries	8 (29)	2 (33)	15 (24)	29 (29)	23 (24)
Response to SMS text messages or push notifications	1 (4)	0 (0)	14 (22)	14 (14)	20 (21)
Number or proportion of intervention program modules completed within app or website	7 (26)	2 (33)	9 (14)	16 (16)	14 (14)
Interacting via chats, phone calls, or social media posts	1 (4)	0 (0)	13 (21)	10 (10)	9 (33)
Wearing an electronic monitoring device	0 (0)	0 (0)	6 (10)	9 (9)	11 (11)
Using a nonwearable electronic monitoring device	0 (0)	0 (0)	7 (11)	11 (11)	8 (8)
Submitting videos via app	2 (7)	0 (0)	0 (0)	2 (20)	1 (20)
Other objective measures	2 (7)	0 (0)	3 (5)	5 (5)	3 (3)
Qualitative interview	2 (7)	1 (17)	12 (19)	16 (16)	23 (24)
Participant-reported survey	1 (4)	2 (33)	4 (6)	10 (10)	12 (12)

^aPercentages within categories do not add up to 100% because studies could fall into >1 category.

Technology Dosages and Minimum Engagement Research Benchmarks

Technology dosages denote when researchers provided participants with specific recommendations for mHealth intervention use (eg, log in to the app at least 3 times a week). Minimum engagement research benchmarks denoted when researchers set a minimum research cutoff for adequate participant engagement (eg, a participant who responded to ≥75% of SMS text messages during the study period was considered by the researchers to be adequately engaged with the mHealth intervention). A research benchmark could be set without giving a technology dosage and vice versa. Technology dosages were given less than half the time to participants by the researchers across all studies (119/292, 40.8%) and when examined by age group. Whether technology dosages were given or not, researchers characterized engagement level as “low” (14/27, 52% vs 13/27, 48%) and “high” (29/63, 46% vs 34/63, 54%) in relatively equal proportions (<10% difference; Tables 5 and 6).

Table 5.

Study characteristics based on technology dosage and minimum engagement research benchmark.

		All studies (n=292), n (%)	Adult studies (n=241), n (%)	Pediatric studies (n=45), n (%)
Technology dosage given to participants
	Yes	119 (40.8)	96 (39.8)	22 (48.9)
	No	173 (59.2)	145 (60.2)	23 (51.1)
Minimum engagement research benchmark set
	Yes	81 (27.7)	71 (29.4)	10 (22.2)
	No	211 (72.3)	170 (70.5)	35 (77.8)

Table 6.

Engagement levels based on technology dosage and minimum engagement research benchmark.

		Low (n=27), n (%)	Medium (n=6), n (%)	High (n=63), n (%)	>1 (n=99), n (%)	Not characterized (n=97), n (%)
Dosage given to participants
	Yes	14 (52)	2 (33)	29 (46)	43 (43)	31 (32)
	No	13 (48)	4 (67)	34 (54)	56 (57)	66 (68)
Minimum engagement research benchmark set
	Yes	11 (41)	2 (33)	15 (24)	38 (38)	15 (15)
	No	16 (59)	4 (67)	48 (76)	61 (62)	82 (85)

A minimum engagement research benchmark was set as the outcome criterion less than one-third of the time across all studies (81/292, 27.7%) and when examined by age group. When a minimum engagement research benchmark was set, researchers tended to characterize user engagement levels as “low” (11/27, 41%) rather than “high” (15/63, 24%). When no minimum engagement research benchmark was set, researchers tended to characterize user engagement as “high” (48/63, 76%) rather than “low” (16/27, 59%; Table 5).

Among studies that had a minimum engagement research benchmark (81/292, 27.7%), less than half (35/81, 43%) gave a technology dosage to participants. Among the studies that gave a technology dosage to participants (119/292, 40.8%), only 29.4% (35/119) also had a minimum engagement research benchmark.

Of the 292 studies, 35 (12%) gave both a technology dosage and set a minimum engagement research benchmark; these indices matched in 94% (33/35) of the studies. Of these 35 studies, 29 (83%) characterized engagement level, of which the majority (n=13, 45%) reported >1 level of engagement due to the use of multiple measurement methods or analyses.

When a minimum engagement research benchmark was set or a recommended technology dosage was given, engagement tended to be measured with user log-in data (48/81, 59% and 69/119, 58%, respectively), manual data entry (26/81, 32% and 32/119, 26.9%, respectively), or module completion (16/81, 20% and 21/119, 17.6%, respectively; Table 7).

Table 7.

User engagement measurement approach when minimum engagement research benchmarks were set or technology dosages were given.

Measurements	User engagement level set by researchers
	Minimum engagement research benchmark (n=81), n (%)a	Technology dosage given (n=119), n (%)a
User log-in data retrieved from app or website	48 (59.3)	69 (58.0)
Manual user data entry in app- or website-based self-monitoring diaries	26 (32.1)	32 (26.9)
Response to SMS text messages or push notifications	9 (11.1)	15 (12.6)
Number or proportion of intervention program modules completed within app or website	16 (19.8)	21 (17.6)
Interacting via chats, phone calls, or social media posts	12 (14.8)	14 (11.8)
Wearing an electronic monitoring device	10 (12.3)	18 (15.1)
Using a nonwearable electronic monitoring device	10 (12.3)	14 (11.8)
Submitting videos via app	2 (2.5)	5 (4.2)
Other objective measures	4 (4.9)	7 (5.9)
Qualitative interview	9 (11.1)	16 (13.4)
Participant-reported survey	1 (0.01)	12 (10.1)

^aPercentages within categories do not add up to 100% because studies could fall into >1 category.

What Is the Association Between User Engagement With mHealth Interventions and Adherence or Self-Management Outcomes?

The association between engagement and treatment adherence or self-management outcomes was only assessed in a little more than one-third of the studies (103/292, 35.3%). Among these 103 studies, 60 (58.3%) tended to report and frame results and conclusions to suggest that higher engagement was associated with positive adherence or self-management outcomes. These results were similar between adult and pediatric studies (Table 3).

How Often Is User Engagement a Research End Point?

User engagement was a research end point in only 19.2% (56/292) of the reviewed studies, with similar results in adult (44/241, 18.3%) and pediatric (11/45, 24%) studies. Of these 56 studies, 30 (54%) used nonrandomized experimental designs, and 27 (48%) were feasibility studies. User engagement was typically defined as “engagement” (24/56, 43%), “adherence” (16/56, 29%), or “use” (16/56, 29%; Table S2 in Multimedia Appendix 3) and was most frequently measured with user log-in data (31/56, 55%) or manual data entry (22/56, 39%; Table S5 in Multimedia Appendix 3).

Exploratory Question: What Is the Association Between Providing Study Participants With Monetary Compensation and User Engagement Level?

Whether study participants were provided monetary compensation for their participation or not (or if compensation was not reported), user engagement measurement methods were used in similar proportions (<10% difference). Similarly, user engagement levels were observed in similar proportions between studies that provided monetary compensation and those that did not (or did not report compensation; Table S6 in Multimedia Appendix 3).

Of the 292 studies, 10 (3.4%) included as an intervention component financial incentives for using the technology or meeting mHealth intervention goals. Of these 10 studies, 7 (70%) reported user engagement levels. Among these 7 studies, in 1 (14%), the user engagement level was characterized as “high”; in 1 (14%) as “medium”; and in 1 (14%), as “low”; and 4 (57%) reported >1 level due to multiple measurement or analytic approaches.

Discussion

Principal Findings

The principal findings relative to our specific research questions are provided in the following subsections.

How Is User Engagement Defined and Measured in Studies of mHealth Interventions to Promote Adherence to Prescribed Medical or Health Regimens or Self-Management Among People Living With a Health Condition?

Terms used to describe user engagement outcomes were wide ranging, but the most commonly used were “use” (102/292, 34.9%) and “engagement” (94/292, 32.2%). Across all studies reviewed, 11 distinct user engagement measurement approaches were identified, comprising both objective and subjective methods. The most common methods were user log-in data from the app or web portal (160/292, 54.8%), manually entering data in an app (77/292, 26.4%), qualitative interviews (54/292, 18.4%), and responding to SMS text messages or push notifications (49/292, 16.8%).

To What Degree Are Patients Engaging With These mHealth Interventions?

User engagement level was difficult to quantify because it was only characterized in two-thirds of the reviewed studies (195/292, 66.8%), of which a little more than half (99/195, 50.8%) reported >1 level of engagement due to the use of multiple measurement methods or analyses. Only one-third of the studies (99/292, 33.9%) evaluated change in engagement over time, which tended to decrease.

What Is the Association Between User Engagement With mHealth Interventions and Adherence or Self-Management Outcomes?

Only one-third of the studies (103/292, 35.3%) evaluated the association between engagement and treatment adherence or self-management outcomes. When evaluated, the study authors tended to report and frame results and conclusions to suggest that higher engagement was associated with positive adherence or self-management outcomes.

How Often Is User Engagement a Research End Point?

User engagement was rarely considered a research end point (56/292, 19.2%).

Exploratory Question: Are There Differences in User Engagement Measurement Approaches and Levels in Studies That Provide Monetary Compensation Compared to Those That Do Not?

No; whether study participants were provided monetary compensation for their participation or not (or if compensation was not reported), user engagement methods were used in similar proportions, and user engagement levels were observed in similar proportions.

Implications and Future Directions

Despite immense focus in both commercial and research sectors on using mHealth to support chronic illness treatment adherence and self-management [314-316], people do not remain engaged with mHealth interventions in the long term [6]. Our systematic review also found that user engagement tends to decline over time. Consistent with the “Little e, Big E framework” [16], mHealth intervention success hinges on the expectation that people will interact with the technology and thereby experience intended behavior changes that will lead to better overall health. Most importantly, our systematic review revealed critical limitations with how user engagement is defined and evaluated, which significantly impedes our ability to (1) communicate about this topic and (2) draw strong conclusions about how much user engagement is necessary to achieve desired behavior and health changes.

A principal finding of our review was that there is no agreed-upon definition of mHealth user engagement, which is a direct barrier to interdisciplinary communication about this topic. We found >60 terms used to define user engagement, even when limiting inclusion criteria to the behavioral evaluation of this concept [15]. While “use” (102/292, 34.9%) and “engagement” (94/292, 32.2%) were the most common terms, “adherence” (59/292, 20.2%) also appeared frequently. “Adherence” as a term for mHealth user engagement can be problematic because it is closely tied to treatment adherence [317], a key target of many of the mHealth interventions included in this systematic review. “Adherence” can apply to engaging with the mHealth technology as intended, such as logging in to an app [109] or responding to SMS text messages [243] rather than following a prescribed treatment regimen. Engaging with the technology can have a direct connection to treatment adherence (eg, using a Bluetooth-enabled blood pressure cuff reflects adherence to a key part of the hypertension management regimen as well as engaging with the mHealth technology [118,308]). However, such interchangeability and a lack of consistency in terminology will continue to be a barrier to communicating within and outside the field and add to existing challenges with defining and evaluating this domain. The extremely wide range of terms used to define user engagement likely reflects a major critique of mHealth research—that there is a lack of a science of engagement [7]. We encourage standardization in terminology used when the intention is to evaluate users’ behavioral engagement with mHealth technology, rather than treatment regimen, with the most common terms found in this systematic review: “use” or “engagement.” Integrating standard user engagement language in an internationally adopted clinical terminology system, such as the Systematized Nomenclature of Medicine–Clinical Terms, could help facilitate standardization efforts.

Another key finding was that the use of user log-in data was the default mHealth user engagement outcome, except for wearing a device when the intervention involved a wearable device component and responding to SMS text messages and push notifications when the intervention involved SMS text messaging and push notifications. Reliance on user log-in data over the other 10 major measurement approaches likely reflects that these data can be relatively straightforward to extract from a web portal. A strength of the use of user log-in data is that it may indicate more effortful and deliberate interactions with the technology (the user likely needs sufficient motivation to open and navigate an app) compared with, for example, quickly responding to a SMS text message or wearing a pedometer on the wrist. User log-in data may be a more accurate reflection of engagement behavior than self-report, which is potentially subject to social desirability and recall bias. The tendency for engagement assessed using user log-in data to be characterized as “low” may reflect the higher user burden of logging in to, and interacting with an, app, in addition to the overwhelming selection of this method for measuring user engagement. Reliance on user log-in data may bias interpretations of user engagement to favor higher quantities of engagement at the expense of higher-quality engagement (eg, infrequent access of a particularly effective app feature that users perceive as interesting, helpful, or motivating may be higher-quality engagement compared with frequent log-ins to the app homepage). Given the ubiquitous selection of user log-in data and the heterogeneity of adherence and self-management outcomes in this systematic review, it is not possible to evaluate which user engagement metric is best. In general, investigators should select the user engagement metric reflecting essential interactions with the mHealth intervention’s key technology needed to facilitate behavior and health changes.

We found that the exact degree of user engagement with the interventions reviewed in this study was difficult to estimate and that the association between user engagement and adherence or self-management outcomes were rarely evaluated. Both findings may relate to an important secondary finding that researchers rarely set a priori benchmarks for how much user engagement is considered scientifically adequate (no minimum engagement research benchmark set). Concerningly, when no minimum research benchmark was set, researchers tended to characterize user engagement as “high,” suggesting overoptimism about study results in the absence of any hypothesized lower boundary for what constituted minimally acceptable mHealth use. Setting cut points for how much user engagement is hypothesized to be adequate would also help to guide how much and in what ways users should interact with the mHealth intervention components to experience meaningful improvements in target outcomes. Matching technology dosages to minimum engagement research benchmarks, when possible, could inform whether engagement recommendations were met and allow researchers to evaluate whether exceeding expectations is associated with even greater improvement in behavior or health outcomes. Of note, studies with technology dosages or minimum engagement research benchmarks seemed to demand more from participants, as evidenced by the more common use of user log-in data and manual data entry to evaluate user engagement. User engagement burden should be considered when setting mHealth engagement expectations.

Setting minimum engagement research benchmarks could also account for variability in user engagement–level characterization depending on the metric selected and help to define what constitutes effective and meaningful engagement. Specifically, engagement assessed using user log-in data tended to be characterized as “low.” By contrast, responding to text notifications or interacting via chats, phone calls, or social media posts, which may be more socially rewarding and less effortful to complete, tended to be characterized as “high.” A similarly higher pattern of user engagement was observed in studies evaluating the use of wearable (6/63, 10% vs 0) and nonwearable devices (7/63, 11% vs 0), which is often part of the users’ regular self-management routines (eg, Bluetooth-enabled pill bottle or glucometer) or passively worn (eg, pedometer) and thus may be less burdensome to use. A higher SMS text message response rate and comparatively lower app log-in rate may both be minimally acceptable and potentially meaningful levels of engagement to effectively facilitate behavior or health change, particularly considering the level of effort and motivation involved in each type of interaction. Given the inconsistencies in how investigators characterize engagement levels, setting minimum engagement research benchmarks could inform whether user engagement is adequate across metrics and associated with key adherence and self-management outcomes. This approach could also improve precision in how engagement levels are characterized (ie, what level of engagement is considered “low” vs “high”?). Such efforts during the study planning phase could reduce the tendency to characterize user engagement at multiple levels due to the use of multiple metrics and analyses. Rethinking engagement levels as “adequate,” “acceptable,” “effective,” or “meaningful” versus “high” or “low” may better inform how much people should realistically use mHealth technology and evaluate whether engagement level is associated with behavior and health changes.

In the mHealth trials reviewed, an important secondary finding was that participants were rarely told how much they should aim to engage with the technology (no technology dosages). This is in contrast to drug trials in which participants are instructed to take a specific dosage of medication on a strict schedule and trials of in-person behavior change interventions (traditional therapy) in which participants are given therapy regimens and expectations for participation (eg, attend 12 therapy sessions and complete assigned homework each week of treatment) [11]. Although an early-phase research goal may be to see how much people interact with the technology in the absence of recommendations, an important future goal is to understand how much engagement is needed to maximize behavior and health changes. To this end, recommending in what ways and how much users should aim to engage with the technology is necessary. Giving technology dosages would also allow for experimental evaluation of different levels of recommended technology use to see what recommendation helps users achieve the most positive behavior and health changes. Furthermore, improvements in health and wellness can take time, and mHealth users may not observe immediate benefits to using mHealth interventions, which could contribute to premature abandonment of the tool. Providing users with clearer guidance on how much and what type of mHealth intervention use is needed to begin effecting positive behavior and health changes could help encourage sustained use.

Another primary finding was that user engagement was rarely considered a research end point. This is a problem because it is generally expected that users will need to interact with the mHealth technology to experience clinical benefit. Thus, user engagement behavior is a critical aspect of the mHealth intervention itself. Without designating user engagement as a research end point, mHealth intervention trials risk lacking the necessary empirical data and results to help end users understand how to optimally use the technology to see maximal clinical benefit. Lacking these data and results may also be a barrier to informing how these digital tools can be incorporated into regular clinical practice and policy [22]. Thus, in addition to carefully measuring the treatment adherence, self-management behavior, and associated health outcomes directly targeted by the mHealth intervention, equal care should be given to evaluating user engagement, such as how user engagement with the technology may change over time and potentially influence outcomes. Shifting user engagement from afterthought data to an actual research end point would likely improve the quality and rigor of mHealth research and help develop mHealth interventions that motivate users to interact with the technology [318], and, ideally, experience greater clinical benefit.

Our exploratory analysis of studies that provided monetary compensation for study participation versus those that did not (or did not report monetary compensation) showed that engagement-level characterizations were similar between the groups. It is possible that the monetary compensation was provided primarily for completing study procedures (eg, completing study surveys) rather than engaging with the mHealth intervention. In the studies of interventions comprising financial incentives for using the technology or meeting mHealth intervention goals (10/292, 3.4%), no clear pattern emerged for user engagement levels, although this likely reflects the small number of studies incorporating this mHealth component. User engagement evaluation methods were used in similar proportions between the groups, suggesting that monetary compensation was not associated with user engagement measurement selection. An important avenue for future mHealth engagement research is to improve the understanding of how monetary compensation is associated with user engagement levels, particularly when compensation is directed toward mHealth intervention use.

Limitations

This systematic review is among the most comprehensive reviews on this topic, but it has some limitations. First, due to heterogeneity in the study outcomes, we did not conduct a meta-analysis. Thus, this systematic review cannot conclude which user engagement outcome is associated with the most effective interventions. Second, although we found that study authors tended to frame their results and conclusions to suggest that higher user engagement was associated with positive intervention outcomes, this finding should be considered preliminary and hypothesis generating and does not prove that higher engagement leads to better adherence or self-management, particularly given that study authors may have a tendency to report favorable results and frame their findings in a positive light. However, this is an important area to examine in future research to better understand the characteristics of users with higher mHealth engagement as it relates to adherence and self-management outcomes (eg, highly engaged users may represent a patient group with already high adherence and self-management). Third, we did not exclude studies on the basis of their attrition rate; a future direction for improving the science of engagement is to consider how dropout and withdrawal may influence the characterization of user engagement levels. Fourth, a traditional risk-of-bias assessment [319,320] was not conducted due to our interest in user engagement rather than the primary study results targeting adherence and self-management. If user engagement becomes more commonly designated as a research end point, such evaluations of study quality specific to the evaluation of user engagement will be warranted. Fifth, we included a range of medical and mental health conditions but, due to small sample sizes, were unable to conduct subgroup analyses by diagnosis; as the literature on user engagement grows, it may be possible to explore such differences by diagnosis. Sixth, we focused on the behavioral evaluation of user engagement, although this domain likely involves more than behavior alone [15,321], and these other aspects of user engagement may be examined in a future systematic review. Seventh, our interest was in the measurement and evaluation of mHealth user engagement; thus, we did not consider the numerous individual-level factors that could be related to user engagement. Eighth, although study location was not part of the inclusion criteria, nearly half of the reviewed studies (135/292, 46.2%) were conducted in the United States, which may have influenced the types of intervention studies and approaches to evaluating user engagement. Ninth, we conducted our search in specific scientific databases and did not review gray literature or unpublished results; although our search resulted in 292 studies meeting our inclusion criteria, this systematic review may not be generalizable to all mHealth interventions. Finally, it is necessary to select a cutoff date when conducting a systematic review; yet, new studies are constantly being published. An updated review of the mHealth user engagement literature may be warranted in the future. As the science of mHealth user engagement improves, an important next step is to evaluate how factors related to diversity, equity, and inclusion relate to user engagement to promote wider mHealth use and access.

Recommendations for Future Researchers

Our systematic review on mHealth user engagement highlighted critical gaps in mHealth adherence and self-management literature as well as opportunities to improve research in this important area of digital health. mHealth researchers need to prioritize the evaluation of user engagement during the study planning phases. Strengthening mHealth user engagement methodological rigor would likely lead to higher-quality data and more impactful study results to guide practice and policy and ultimately encourage the uptake of interventions promoting mHealth adherence or self-management. Furthermore, improved user engagement evaluation may help researchers to identify effective strategies for supporting meaningful user engagement and sustained interest to help users fully experience the intended behavioral and health benefits of the mHealth intervention [16]. Such research is critical for building the evidence base needed to integrate mHealth interventions into regular clinical care and practice guidelines.

The following recommendations for studies of mHealth user engagement are offered:

• Use consistent terminology. Reflecting the most common trends in the literature reviewed, we recommend referring to the measurement of behavioral mHealth use, interaction, or engagement as “use” or “engagement.” Greater consistency in terminology could help improve rigor and consistency in the measurement itself and facilitate comparisons between studies as well as the evaluation of user engagement in meta-analyses.

• Select mHealth user engagement metrics that reflect interactions with the intervention’s key technology components hypothesized to facilitate behavior and health changes.

• Provide mHealth users with expectations for how much they should aim to interact with the technology (give technology dosages when possible).

• Set minimum engagement research benchmarks to scientifically denote the hypothesized user engagement level for a participant to be considered adequately, acceptably, or meaningfully engaged in the intervention.

• Characterize user engagement levels as adequate or acceptable with minimum engagement research benchmarks to help address issues with engagement being characterized as “high” or “low” depending on the metric selected and consider whether the engagement is adequate, meaningful, or effective across metrics (eg, a higher SMS text message response rate and comparatively lower app log-in rate may both be acceptable and meaningful levels of engagement).

• Designate mHealth user engagement as a research end point to help improve the quality and rigor of the mHealth research and the data collected from these studies. Focusing research efforts on user engagement could lead to mHealth engagement recommendations that could ultimately lead to greater clinical improvement.

Abbreviations

mHealth

mobile health

PRISMA

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Data Availability

Select review materials are available as multimedia appendices in this manuscript. The manual of procedures and the data sets generated and analyzed during this study are available from the corresponding author on reasonable request. The authors did not use generative AI to write any portions of this manuscript.