Understanding the Impact of Perceived App Usability on the Efficacy of Mobile Health Intervention for Traumatic Brain Injury Caregivers

Abstract

Objective:

Caregiving for individuals with traumatic brain injury (TBI) is often highly stressful, and traditional in-person interventions can be inaccessible given the demands of their caregiver role. Mobile health (mHealth) interventions offer a low-burden, scalable alternative by delivering personalized, real-time support. However, how effective these interventions are depending on whether people perceive them as useful and engage with them. The role of perceived usability in mHealth efficacy remains underexplored in the TBI caregiver population. This study primarily evaluates the relationship between self-reported app usability and the efficacy of a fully automated mHealth intervention, delivered via personalized mobile app messages, in reducing caregiver strain, anxiety, and depression in a TBI caregiver population. The secondary aim is to identify variables that moderate the intervention efficacy among caregivers reporting high app usability.

Methods:

We analyzed data from 122 TBI caregivers assigned to receive self-monitoring plus push notifications for self-care as part of a larger randomized controlled trial. Perceived app usability was assessed via caregiver responses to a questionnaire item. We applied multivariate linear models with a weighted and centered least squares estimator to assess the moderating effects of perceived app usability and other variables on message efficacy.

Results:

Messages were more effective in reducing depression among TBI caregivers who reported high app usability. Moreover, shorter caregiving duration, higher Fitbit step count, and lower prior-week anxiety were significantly associated with improved message efficacy among this high usability group.

Conclusions:

Perceived app usability plays a critical role in the mHealth message efficacy among TBI caregivers. Tailoring interventions based on perceived app usability and identified moderators may optimize health outcomes and support more personalized care for this population.

Introduction

Traumatic brain injury (TBI) impacts not only the individual who sustained the injury but also their caregivers (Carlozzi et al., 2015, 2016; Kreutzer et al., 2009; Marsh et al., 2002; Ponsford & Schönberger, 2010). Caregivers of individuals with TBI often experience substantial challenges, including emotional distress, social isolation, and financial strain, which can have a negative impact on the physical and mental health of both the caregiver and care recipient (Sander et al., 2002; Schönberger et al., 2010; Vangel et al., 2011). Providing effective support to this population is crucial for both caregiver well-being and the long-term recovery of the individual with TBI.

Traditional interventions for TBI caregivers, such as psychoeducation and skills training, can help decrease caregiver burden and improve their well-being, but often require time-intensive in-person meetings with trained professionals (Baker et al., 2017; Carnevale et al., 2002). In the context of TBI management, educational approaches generally fall into two groups. Psychoeducation builds a knowledge base, covering topics such as symptom processing (for example, pain mechanisms), realistic recovery expectations, and understanding TBI as a chronic condition. Skills training, often based on self-management models, focuses on actionable competencies such as structured problem solving, goal setting (for example, SMART goals), and communication strategies for medical and social interactions. Unfortunately, such programs are not commonly offered for care partners following TBI, and existing programs can be difficult to access for caregivers who are already stretched by competing responsibilities and limited time. Mobile health (mHealth) technologies offer a promising alternative for delivering timely and tailored support to TBI caregivers with minimal user burden (a few prompts per day). Mobile devices, such as smartphones and wearable devices can collect objective real-time measurements, such as step count and geographic location, as well as self-assessments (e.g., self-report surveys) from caregivers, and the decision agent can use the collected information to provide just-in-time adaptive interventions (JITAIs) when and where they are needed most, potentially mitigating caregiver burden and improving their well-being. Previous studies have shown that JITAIs based on wearable devices and smartphone data are associated with improvements in health outcomes, including improved physical activity (Hardeman et al., 2019; Klasnja et al., 2019; Saponaro et al., 2021; Thomas & Bond, 2015; J. Wang, Fang, et al., 2023), smoking cessation (Riley et al., 2008; Vinci et al., 2025), and reductions in mental health concerns (Teepe et al., 2021; van Genugten et al., 2025; L. Wang & Miller, 2023) and insomnia (Pulantara et al., 2018; Takeuchi et al., 2024).

A critical factor influencing the success of mHealth interventions is app usability (Deniz-Garcia et al., 2023; Georgsson & Staggers, 2016; Z. Li et al., 2024; X. Yan et al., 2023; Zhou et al., 2019). High usability, reflected in participants’ perceptions of the app being easy to use, is often associated with the enhanced efficacy in mHealth interventions (Fu et al., 2017; Overdijkink et al., 2018). Recent works (Deniz-Garcia et al., 2023; Eaton et al., 2024) identify engagement as the key mediator linking usability to efficacy, noting that users are more likely to interact frequently and consistently with highly usable apps. This sustained engagement increases exposure to the intervention’s behavior-change techniques, such as self-monitoring, tailored feedback, reminders, or coaching, which in turn facilitates improvements in health behaviors and clinical outcomes. Other works on app usability evaluations (Inal et al., 2020; Z. Li et al., 2024) similarly emphasize that usability is essential for meaningful and lasting involvement with mHealth interventions.

Conversely, low usability can lead to low engagement and high attrition rates, making the intervention ineffective. This is particularly important for caregivers of people living with TBI, a population often characterized by significant caregiving burden and overwhelming daily tasks, which can potentially impede their engagement and diminish the efficiency of mHealth interventions. Because usability in the context of mHealth intervention is a broad, multidimensional construct that includes factors such as user’s satisfaction, feasibility, acceptability, and actual usage behavior, it is important to clarify that the present study focuses specifically on perceived ease-of-use rather than objective app use. Given this, a better understanding of the relationship between app usability and intervention efficacy is crucial to maximize the benefits of mHealth interventions for TBI caregivers. By elucidating this relationship, researchers can design more user-friendly and effective intervention strategies, thereby improving the health outcomes of TBI caregivers.

In a randomized controlled trial (RCT) of a personalized mHealth intervention for caregivers of individuals with TBI, we compared self-monitoring alone versus self-monitoring plus personalized self-care push notifications and found that approximately 1/3 of the participants showed clinically meaningful improvements regardless of treatment arm. Caregivers who self-reported high app usability during the intervention were more likely to show improvements than those who did not (Carlozzi et al., 2025). As a follow-up to this RCT, the current paper reports on analysis of the relationship between perceived app usability and the efficacy of the personalized self-care push notifications for participants in the treatment arm. Additionally, we examined eleven demographic and pre-treatment variables to assess their potential to modify the efficacy of this personalized mHealth intervention in improving health outcomes.

Methods

Study Design and Settings

This analysis focuses on a subset of participants from a larger two-arm RCT (Carlozzi et al., 2025) that evaluated the efficacy of a low-burden mHealth intervention designed to promote care partner self-awareness and self-care. The primary study was registered on ClinicalTrials.gov (NCT00000000). Of the 257 TBI caregivers initially enrolled, 128 were randomized to receive self-monitoring only, which included passive mobile sensor data feedback and self-reporting of health-related quality of life (HRQOL) through a mobile app developed for the study (“CareQOL”). This app was designed to capture real-time ratings of symptoms as well as sensor data from a Fitbit^®. The remaining 129 caregivers were randomly assigned to the intervention arm, where they also received self-care push notifications informed by the mobile sensor data and daily HRQOL reporting. Each day, caregivers in the intervention arm had a 50/50 probability of receiving a message or not. The intervention arm implemented a micro-randomized trial (MRT) design, which is the focus of this analysis. We have included a flow chart (Figure 1) for the larger primary study to this secondary analysis.

Figure 1:

A flow chart from the primary study to this secondary analysis

Participation in this behavioral trial consisted of a two-hour baseline assessment session, followed by a 10-day run-in period and a six-month home-monitoring period. During the baseline session, participants completed informed consent, several self-report measures (including demographic information and HRQOL measures), and received instructions for the home-monitoring phase. The run-in period allowed time for Fitbit delivery and setup, as well as troubleshooting of the Fitbit device and study app. The home-monitoring phase included continuous monitoring of step count, sleep duration (minutes), and heart rate (beats per minute) using Fitbit devices, as well as daily real-time self-reported HRQOL measures via the CareQOL study app. Participants in the intervention arm had a 50% probability of receiving personalized messages each day during the home-monitoring period. These messages were composed based on contextual information derived from sensor data (accelerometer-based estimates of physical activity and sleep duration) and daily self-reported HRQOL ratings (caregiver strain, anxiety, and depression) collected via the study app. Messages were randomly selected from a pool of over 400 messages, which comprise one or more of the following types: “data feedback”, “facts”, “tips”, and “support” (see the message examples in Table 1 and Carlozzi, Sander, et al., 2022). The control arm did not receive any messages. At the end of the home-monitoring period, participants completed a study-specific feasibility and acceptability questionnaire (Carlozzi, Choi, et al., 2022). Perceived app usability was categorized into two levels based on their response to the item, “The daily questions on the app were easy to answer.” Participants who “Strongly agreed” were classified as the high usability group; all other responses (e.g., “Agree”, “No strong feelings either way”, “Disagree”) were classified as the lower usability group. To test differences in demographic characteristics between the two groups, we used Fisher’s exact test for categorical variables and the Wilcoxon rank-sum test for continuous variables. Full details of the study samples can be found in (Carlozzi et al., 2025).

Table 1.

Examples of personalized push notifications in the just-in-time adaptive intervention (Carlozzi, Sander, et al., 2022)

Feedback domain	Low level (below average performance or problems)	Medium level (average performance or problems)	High level (above average performance or problems)
Mental health (depression)	“Your average sadness rating over the last week was XX. Next time you’re feeling low, watch your favorite funny movie. Laughter is the best medicine!”	“Your average sadness rating over the last week was XX. When you’re feeling low, why not watch your favorite funny movie? Laughter is the best medicine!”	“Your average sadness rating over the last week was XX. If you’re ever feeling low, watch your favorite funny movie. Laughter is the best medicine!”
Mental health (anxiety)	“The next time you feel worried, close your eyes and think of a peaceful, relaxing place. Try to imagine as many different sights, sounds, and smells as you can. Continue until you feel more relaxed, then open your eyes slowly.”	“Are you feeling anxious? Close your eyes and think of a peaceful, relaxing place. Try to imagine as many different sights, sounds, and smells as you can. Continue until you feel more relaxed, then open your eyes slowly.”	“If you ever feel worried, close your eyes and think of a peaceful, relaxing place. Try to imagine as many different sights, sounds, and smells as you can. Continue until you feel more relaxed, then open your eyes slowly.”
Mental health (general)	“Is there a friend you haven’t talked to in a while? When you feel down, try giving them a call. Talking to friends can help boost your spirits!”	“Is there a friend you haven’t talked to in a while? Try giving them a call. Talking to friends can help boost your spirits!”	“Is there a friend you haven’t talked to in a while? If you feel down, try giving them a call. Talking to friends can boost your spirits!”
Mindfulness	“Take a few minutes every day to wind down. Even if you don’t feel stressed all the time, meditating can relieve built-up tension.”	“Take a few minutes every day to wind down. Try meditating to relieve built-up tension.”	“Take a few minutes every day to wind down. Even if you don’t feel stressed right now, meditating can relieve any built-up tension.”
Physical activity	“This past week, your average daily step count has been XX. Try to increase this if you can!”	“This past week, your average daily step count has been XX. Try to maintain this level or even increase it more if you can.”	“This past week, your average daily step count has been XX. Great job! Try to maintain this level.”
Sleep	“You aren’t quite getting the recommended 7–8 hours of sleep per night. Try moving bedtime up by 5–10 minutes each night to get closer to this goal.”	“You’re having a hard time getting the recommended 7–8 hours of sleep per night. We all struggle to get to sleep sometimes. Try moving bedtime up by 5–10 minutes each night.”	“If you ever have a hard time getting the recommended 7–8 hours of sleep per night, try moving bedtime up by 5–10 minutes each night.”

Outcomes and Measures

The primary outcomes of this study were the HRQOL measures of caregiver strain, anxiety, and depression. These constructs were assessed through three daily questions, each drawn from a respective item bank. Caregiver strain was measured using the Neuro-QoL TBI-CareQOL Caregiver Strain item bank (Carlozzi et al., 2019), which was designed to assess feelings of being overwhelmed or burdened by the caregiver role. Anxiety was measured using the PROMIS (Patient-Reported Outcomes Measurement Information System) Anxiety item bank (Cella et al., 2007, 2010), which assesses various aspects of anxiety, including fear, anxious mood, hyperarousal, and somatic symptoms. Depression was assessed using the PROMIS Depression item bank (Cella et al., 2007, 2010), which measures perceived depression across domains such as negative mood, decrease in positive affect, information-processing deficits, negative views of the self, and negative social cognition. All three daily questions were administered on a 5-point scale, with higher scores indicating greater levels of the respective construct.

These real-time assessments were administered through a computer adaptive test (CAT) (Meijer & Nering, 1999), which aims to estimate an examinee’s level of the construct (e.g., caregiver strain) through sequentially administered questions. This method allows for progressively refined estimates by tailoring subsequent items based on prior responses. Each session begins with an initial item and provisional score, which is updated with every subsequent response; the session concludes once a predefined stopping rule (e.g., fixed time limit or achievement of a target precision) is met, yielding a final score. Missing item responses do not trigger any prorating; they simply reduce the accuracy of the final score estimate. In this study, CAT sessions were structured to begin anew each Monday, with a single item presented daily. Given this specific structure, all the analyses in this paper were conducted at the weekly level. The primary outcomes of interest were the HRQOL scores generated at the conclusion of each weekly CAT session (e.g., using Saturday’s score if Sunday’s data were missing). Sessions with fewer than three completed daily responses were excluded due to concerns about data quality (Stopping Rules, n.d.).

Scores on these item banks were standardized to a T-score scale (mean = 50, SD = 10), relative to a reference population. The measure of caregiver strain was standardized to the population of TBI caregivers, while anxiety and depression were standardized to the US general population (Rothrock et al., 2020). Daily step count and sleep hours were recorded through wearable devices, and weekly averages were calculated.

Statistical Analysis

Missing data

Missing data occurred throughout the trial for various reasons: participants forgetting to complete daily surveys, wear the fitness tracker, not wearing the fitness tracker 24 hours a day, as well as technical glitches associated with the CareQOL app or fitness tracker, among others. To address this, we used multiple imputation, a robust method for dealing with missing data, to impute weekly T-scores, average step count, minutes of sleep, and resting heart rate. We imputed the data week-by-week to respect the sequential design of the study. Predictors for the imputation included (i) baseline characteristics (see Table 2) and baseline T-scores, and (ii) participant data from the prior two weeks: average step count, sleep duration, resting heart rate, and mood score, as well as weekly averages for step count and sleep. To avoid look-ahead bias, only information available at baseline or in the two preceding weeks was used for a given week. Imputations were performed in R 4.0.2 using the mice package with predictive mean matching.

Table 2:

A demographic table comparing the high and lower app usability groups. Bolded values indicate statistical significance (p < 0.05).

Variables	Total (n=122)	High (n=49)	Lower (n=73)	p-value
Gender, n (%)				1.
Female	94 (0.77)	38 (0.78)	56 (0.77)	-
Male	28 (0.23)	11 (0.22)	17 (0.23)	-
Race, n (%)				0.01
African American	19 (0.16)	11 (0.22)	8 (0.11)	-
Asian	4 (0.03)	1 (0.02)	3 (0.04)	-
Caucasian	92 (0.75)	31 (0.63)	61 (0.84)	-
Other	7 (0.06)	6 (0.12)	1 (0.01)	-
Ethnicity, n (%)				0.5
Non-Hispanic	101 (0.83)	39 (0.80)	62 (0.85)	-
Hispanic	21 (0.17)	10 (0.20)	11 (0.15)	-
Marital status, n (%)				0.5
Married or having significant other	98 (0.80)	41 (0.84)	57 (0.78)	-
Single	24 (0.20)	8 (0.16)	16 (0.22)	-
Employment status, n (%)				0.3
Employed	72 (0.59)	33 (0.67)	39 (0.53)	-
Unemployed	46 (0.38)	15 (0.31)	31 (0.43)	-
Other	4 (0.03)	1 (0.02)	3 (0.04)	-
Age (years), mean (SD)	51.58 (14.92)	48.6 (13.37)	53.58 (15.65)	0.05
Time in caregiver role (years), mean (SD)	6.09 (5.91)	5.86 (4.58)	6.24 (6.69)	0.7
Age of care recipient (years), mean (SD)	45.11 (17.68)	42.69 (16.51)	46.74 (18.36)	0.2
Relationship to care recipient, n (%)				0.4
Spouse or partner	54 (0.44)	19 (0.39)	35 (0.48)	-
Child	17 (0.14)	9 (0.18)	8 (0.11)	-
Parent	34 (0.28)	13 (0.27)	21 (0.29)	-
Sibling	12 (0.10)	7 (0.14)	5 (0.07)	-
Other	5 (0.04)	1 (0.02)	4 (0.06)	-
Time caregiving, n (%)				0.8
1–2 h/d or less	61 (0.50)	26 (0.53)	35 (0.48)	-
3–4 h/d (half a working day)	24 0.20)	9 (0.18)	15 (0.21)	-
5–8 h/d (full working day)	9 (0.07)	5 (0.10)	4 (0.06)	-
9–12 h/d	9 (0.07)	3 (0.06)	6 (0.08)	-
> 12 h/d or round the clock care	19 (0.16)	6 (0.12)	13 (0.18)	-

Statistical Method

In this study, we focused on analyzing the intervention arm of a randomized controlled trial that employed a MRT design. A key advantage of the MRT design is its capacity to assess both time-invariant (e.g., age, gender) and time-varying effect moderators (e.g., study week, prior step count), whose values can vary across time, reflecting the varying circumstances of each individual and potentially informing the optimal intervention options.

We adopted a weighted and centered least squares (WCLS) estimator (Boruvka et al., 2018) for both primary and secondary analyses. This method is specifically designed for causal inference using data collected under an MRT design, providing asymptotically unbiased estimates of the main causal effect and effect moderation. It is also robust to potential misspecification of non-interacting terms with the treatment variable.

The WCLS estimator was implemented using generalized estimating equations (GEEs) with an independent working correlation structure. Specifically, we fitted linear models with weekly T-scores of caregiver strain, anxiety, and depression as the dependent variable and a treatment indicator, representing low, medium, or high dosage of mobile messages received. We also included an interaction term between the treatment indicator and hypothesized effect moderators. The linear term of week in the study (e.g., a value of 5 for week 5), caregiver’s sex, baseline age, ethnicity, race, duration of caregiving role, daily time spent caregiving, and baseline mental health status (caregiver strain, anxiety, and depression) were included as control variables for improving the precision of the causal estimates (Boruvka et al., 2018). Additionally, prior week’s step count, sleep duration, and the three T-scores were also included to improve statistical efficiency. To address the highly right-skewed distribution of step count, a cubic root transformation was employed to enhance symmetry.

Primary Aim – Moderation effect of perceived app usability on message efficacy

The primary objective of this study is to examine whether self-reported app usability impacts the caregivers’ responses to personalized messages. In this context, perceived app usability is conceptualized as a potential effect moderator that may influence the message efficacy. The primary outcomes of interest are HRQOL T-scores measuring caregiver strain, anxiety, and depression. Given the weekly design of the CAT event, the total number of messages received prior to each week’s final survey response was used as the primary treatment variable. To ensure that outcomes reflected exposure to the intervention, only messages received between Monday and the day before final survey were counted. For instance, if a participant completed the survey on Friday, only messages received between Monday and Thursday were included. Weekly number of messages a participant in the intervention arm received ranged from 0 to 6. Because each day a caregiver had a 50–50 probability of receiving a message, the chance of receiving none across six days is 0.5⁶, meaning some participants will naturally have weeks with zero messages. This likelihood increases further when participants have fewer than six days of available T-scores. To facilitate analysis, we categorized weekly message frequency into four groups: no messages, low frequency (1–2 messages), medium frequency (3–4 messages), and high frequency (5–6 messages). This categorization strategy mirrors that used in (J. Wang, Wu, et al., 2023). To formally assess the significance of the moderation effect of perceived app usability, we employed Wald tests on the interaction terms (Rotnitzky & Jewell, 1990).

Secondary Aim – Moderators of message efficacy among high usability group

To further optimize message efficacy among the high usability group, the secondary aim of this study is to identify the effect moderators that can influence message efficacy. Five time-invariant variables were included in the analysis: age, gender, ethnicity, race, caregiving role duration. These variables were collected via survey during the baseline session. For time-varying moderators, we considered six variables: 1) weeks in the study, 2) prior week’s daily average step count, 3) prior week’s daily average sleep duration, and prior week’s HRQOL T-scores of 4) caregiver strain, 5) anxiety, and 6) depression. The inclusion of weeks in the study was due to the commonly observed decline in treatment efficacy over time in mHealth interventions (Klasnja et al., 2019; Shcherbina et al., 2019; J. Wang et al., 2022). We hypothesized that physical activity and sleep levels would influence the message efficacy (NeCamp et al., 2020). We also explored whether prior week’s HRQOL T-scores could moderate message efficacy, as individuals with poorer HRQOL might be less responsive. We conducted the analyses by including interaction terms between treatment variables and moderator variables in the linear model described in the primary aim. A significant nonzero coefficient of the interaction term would suggest that the corresponding moderator can moderate message efficacy.

Transparency and Openness

Analysis code is available at https://github.com/xxx. Deidentified data supporting the results and figures in this manuscript are available upon request from the corresponding author. Data were analyzed using R (R Foundation for Statistical Computing), version 4.4.2 (R Core Team, 2021), with packages tidyverse, version 2.0.0 (Wickham et al., 2019), ggplot2, version 3.5.1 (Wickham, 2016), ggpubr, version 0.6.0, and geepack, version 1.3.12 (Halekoh et al., 2006; J. Yan, 2002), glmtoolbox, version 0.1.12 (Vanegas et al., 2021).

Results

Study Participants

A total of 122 participants were included in the analysis, with 49 in the high app usability group and 73 in the low app usability group. Demographic details are presented in Table 2. Significant group differences were observed in race proportion and caregiver age. Specifically, the high app usability group had a higher proportion of African American and other racial backgrounds (including Asian, Hawaiian, American Indian, and multiracial) and younger caregivers. These variables were adjusted for in subsequent analyses.

Of the total 122 participants in the analysis, 118 used the study-provided Fitbit Inspire 2, while the remaining 4 participants used their own personal Fitbit devices (1 Fitbit Versa2, 1 Fitbit Inspire, 1 Fitbit Sense, and 1 unknown model). Missing data occurred throughout the study, with an average missing rate of 18.0% (3,813/21,238 person-days) for daily HRQOL surveys (caregiver strain, anxiety, and depression), 10.9% (2,323/21,238 person-days) for daily step counts, and 25.4% (5,395/21,238 person-days) for daily sleep data collected via Fitbit. The participants’ average daily step count was 7,149 steps, and the average daily sleep duration was 6.7 hours.

Primary Aim – Moderation effect of perceived app usability on message efficacy

Table 3 presents the results of the primary analysis, which assessed whether the efficacy of mHealth messages varied by self-reported app usability across three HRQOL outcomes: caregiver strain, anxiety, and depression T-scores.

Table 3.

Estimated effect of delivery of mobile messages on health-related quality of life (HRQOL) T-scores (caregiver strain, anxiety, depression), stratified by group (high and low app usability), along with 95% confidence intervals (CI).

	High App Usability		Low App Usability
	Estimate	95% CI	Estimate	95% CI
Caregiver strain
No message	-a	-	-	-
Low frequency	−0.76	−1.95, 0.43	0.04	−0.97, 1.06
Medium frequency	−0.61	−1.84, 0.62	−0.09	−1.02, 0.84
High frequency	−0.74	−2.28, 0.80	0.04	−1.08, 1.16
Anxiety
No message	-	-	-	-
Low frequency	−0.07	−1.94, 1.80	0.04	−1.21, 1.30
Medium frequency	0.24	−1.58, 2.06	0.38	−0.87, 1.63
High frequency	0.37	−1.79, 2.53	0.72	−0.70, 2.14
Depression
No message	-	-	-	-
Low frequency	−0.94	−2.57, 0.68	0.19	−0.59, 0.96
Medium frequency	−0.83	−2.37, 0.71	0.24	−0.53, 1.01
High frequency	−0.45	−2.33, 1.44	0.35	−0.53, 1.23

^aNot available.

The analysis revealed a significant interaction between perceived app usability and message efficacy in reducing depression T-scores (p = 0.004). Among caregivers reporting high app usability, receiving messages consistently reduced depression T-score (lower score indicate reduced depression) across all message frequency levels. Specifically, relative to no messages, caregivers in this group exhibited mean reductions in depression T-score of 0.94 (95% CI: −0.68, 2.57), 0.83 (95% CI: −0.71, 2.37), and 0.45 (95% CI: −0.44, 2.33) for low-, medium-, and high-frequency messages, respectively. In contrast, caregivers reporting lower app usability showed slight increases in depression scores across all message frequencies, with mean changes of 0.19 (95% CI: −0.59, 0.96), 0.24 (95% CI: −0.53, 1.01), and 0.35 (95% CI: −0.53, 1.23).

For caregiver strain T-score, the interaction between message frequencies and perceived app usability was not statistically significant (p = 0.58); however, a similar trend to that observed for depression T-score was observed. Caregivers reporting high app usability experienced modest reductions in caregiver strain T scores over all message frequency levels, whereas those reporting low app usability did not. Specifically, high usability group who received low-, medium-, and high-frequency messages showed mean reductions in caregiver strain T-scores of 0.76 (95% CI: −0.43, 1.95), 0.61 (95% CI: −0.62, 1.84), and 0.74 (95% CI: −0.80, 2.28), respectively, relative to no messages, while the low usability group showed minimal changes, with mean changes of 0.04 (95% CI: −0.97, 1.06), −0.09 (95% CI: −1.02, 0.84), and 0.04 (95% CI: −1.08, 1.16) across the respective message frequencies.

For anxiety T-score, the interaction between message frequencies and perceived app usability was not statistically significant (p = 0.89), and the message was not effective in reducing T-score for both app usability groups. Among caregivers in the high usability group, those receiving low-frequency messages showed a mean change in anxiety T scores of −0.07 (95% CI: −1.80, 1.94), 0.24 (95% CI: −1.58, 2.06), and 0.37 (95% CI: −1.79, 2.53) for low, medium, and high message frequencies, relative to no messages. In the low usability group, the messages appeared even less effective, with changes in T-scores of 0.04 (95%CI: −1.21, 1.30), 0.38 (95%CI: −0.87, 1.63) and 0.72 (95%CI: −0.70, 2.14) across low, medium, and high message frequencies, respectively.

Secondary Aim - Moderation Analysis in High Usability Group

Our secondary aim was to identify potential variables that moderate the message efficacy among TBI caregivers reporting high app usability. Results for time-invariant and time-varying moderators are shown in Tables 4 and 5, respectively.

Table 4.

Estimated moderation effects of delivery of mobile messages on health-related quality of life (HRQOL) T-scores (caregiver strain, anxiety, depression) for time-invariant variables among caregivers reporting high app usability. Standard errors are shown in parentheses. Bolded values indicate statistical significance (p < 0.05).

HRQOL outcome	Caregiver strain			Anxiety			Depression
	Low	Medium	High	Low	Medium	High	Low	Medium	High
Age (years)	−0.06 (0.05)	−0.01 (0.06)	−0.10 (0.08)	−0.09 (0.06)	−0.03 (0.06)	−0.06 (0.08)	−0.04 (0.06)	−0.01 (0.06)	−0.05 (0.07)
Sex
Female	-	-	-	-	-	-	-	-	-
Male	0.46 (1.15)	1.08 (1.23)	−1.16 (1.79)	1.72 (2.28)	3.01 (2.29)	1.24 (2.55)	−1.54 (1.58)	0.77 (1.50)	−2.54 (1.79)
Ethnicity
Hispanic	-	-	-	-	-	-	-	-	-
Non-Hispanic	2.09 (1.49)	2.21 (1.66)	2.82 (1.67)	3.41 (3.04)	3.67 (2.84)	2.24 (2.98)	2.18 (2.52)	2.30 (2.32)	3.31 (2.43)
Race
White	-	-	-	-	-	-	-	-	-
Others	1.51 (1.29)	0.19 (1.34)	1.70 (1.67)	2.17 (1.90)	1.02 (1.97)	0.08 (2.16)	2.02 (1.68)	0.47 (1.75)	0.37 (1.89)
Years in caregiver role	0.18 (0.10)	0.25 (0.08)	0.34 (0.15)	0.05 (0.19)	0.08 (0.14)	0.18 (0.21)	−0.01 (0.13)	0.03 (0.11)	−0.01 (0.17)

Table 5.

Estimated moderation effects of delivery of mobile messages on health-related quality of life (HRQOL) T-scores (caregiver strain, anxiety, depression) for time-varying variables among caregivers reporting high app usability. Standard errors are shown in parentheses. Bolded values indicate statistical significance (p < 0.05).

HRQOL outcome	Caregiver strain			Anxiety			Depression
	Low	Medium	High	Low	Medium	High	Low	Medium	High
Week	0.09 (0.07)	0.12 (0.07)	0.06 (0.11)	−0.09 (0.13)	−0.04 (0.12)	−0.11 (0.15)	−0.13 (0.08)	−0.07 (0.08)	−0.10 (0.10)
Step count (1,000 steps)	−0.13 (0.12)	−0.09 (0.14)	−0.32 (0.19)	0.01 (0.21)	0.12 (0.20)	−0.08 (0.25)	−0.44 (0.16)	−0.18 (0.13)	−0.33 (0.21)
Sleep duration (1 hour)	0.93 (0.62)	1.04 (0.63)	1.03 (0.69)	1.02 (0.81)	0.86 (0.85)	1.38 (0.85)	0.73 (0.61)	0.47 (0.59)	0.81 (0.64)
Caregiver strain score (1 T-score)	0.03 (0.07)	−0.00 (0.07)	0.04 (0.09)	−0.02 (0.12)	−0.05 (0.10)	−0.06 (0.12)	0.02 (0.09)	−0.06 (0.08)	−0.03 (0.09)
Anxiety score (1 T-score)	0.13 (0.04)	0.09 (0.05)	0.12 (0.06)	0.05 (0.06)	0.02 (0.06)	−0.05 (0.08)	−0.01 (0.07)	−0.04 (0.07)	−0.05 (0.08)
Depression score (1 T-score)	0.08 (0.05)	0.05 (0.06)	0.09 (0.08)	0.02 (0.07)	−0.00 (0.07)	−0.05 (0.09)	0.02 (0.06)	−0.03 (0.06)	−0.02 (0.07)

Among caregivers with high perceived app usability, caregiving duration significantly moderated the message efficacy in reducing caregiver strain T-scores. Specifically, each additional year in the caregiving role reduced the efficacy of medium- and high-frequency messages (relative to no messages) by 0.25 T-scores (95% CI: 0.09, 0.41) and 0.34 T-scores (95% CI: 0.05, 0.63), respectively. No significant time-invariant moderators were identified for anxiety or depression T-scores.

For time-varying variables, higher anxiety levels in the prior week significantly diminished the message efficacy in reducing caregiver strain. For each one-point increase in anxiety T-score, the effect of low- and medium-frequency messages (relative to no messages) decreased by 0.13 T-scores (95%CI: 0.05, 0.21) and 0.09 T-scores (95%CI: 0.02, 0.18), respectively. Additionally, prior-week step count positively moderated the effect of low-frequency messages in reducing depression. Specifically, an increase of 1,000 steps was associated with a 0.44-score (95% CI: −0.13, 0.75) improvement of effect of messages on depression T-score.

To visualize how message effects vary by these identified moderators, we estimated and plotted the effects across the observed range of each moderator. These visualizations are presented in Figures 2 to 4.

Figure 2:

The estimated message’s effect at medium and high frequencies on weekly caregiver strain T score for different years in caregiving role, along with 95% confidence intervals.

Figure 4:

The estimated message’s effect at low frequency on weekly depression T score over previous week’s average step count, along with 95% confidence intervals.

Discussion

This paper examines the impact of perceived app usability on the efficacy of a personalized mHealth intervention, delivered via personalized mobile app messages, on HRQOL outcomes, along with a secondary analysis to identify potential effect moderators among TBI caregivers with high perceived app usability.

Primary Aim:

Our analysis demonstrated that mHealth messages were more effective in reducing depression T-score among TBI caregivers who reported high app usability, compared to those with low usability. Notably, among caregivers in the high app usability group, messages led to reductions in mean depression T-scores across all message frequencies (–0.94, −0.83, −0.45), while caregivers with lower perceived usability showed slight increases (0.19, 0.24, 0.35). This interaction between perceived app usability and message efficacy was statistically significant (p = 0.004). A similar pattern was observed for caregiver strain T-scores, although the interaction was not significant. Caregivers in the high usability group experienced mean reductions in caregiver strain T-score (–0.76, −0.61, −0.74), whereas those in the low usability group showed minimal changes (0.04, −0.09, 0.04). For anxiety T-scores, mHealth messages did not produce meaningful reductions in either usability group, and no significant interaction was detected (p = 0.89).

These findings suggest that perceived app usability is a key factor in the efficacy of mHealth messages for reducing depression and, potentially, caregiver strain T-scores. The most likely mechanism for this is greater engagement with the app and intervention when the app is perceived as more useable; however, lack of usage statistics in this trial precludes testing of this mechanism, which should be investigated in future studies, as it has been shown to moderate the relationship of usability to intervention effects in other populations (Alkhaldi et al., 2016; Amagai et al., 2022; Druce et al., 2019). Our results also suggested that there may be a negative impact of app use on depression for those caregivers who perceive the app as less usable. This could potentially be due to increased feelings of helplessness associated with difficulty figuring out the app. Prior research in mobile health shows that when users perceive an app as difficult or effortful to use, this can reduce engagement and increase frustration, which in some cases is linked to poorer mood or diminished intervention benefit (Inal et al., 2020; Lattie et al., 2019). Future JITAI-based mHealth interventions should focus on increasing perceived utility and engagement for users to maximize efficacy and avoid negative impact for some users.

Secondary Aim:

Our moderation analysis of time-invariant variables revealed that a shorter duration in the caregiving role was associated with increased message efficacy in reducing caregiver strain among caregivers with high perceived app usability. This may reflect the impact of caregiver fatigue and burnout associated with longer caregiving duration, which can potentially diminish the positive impact of messages (Ramdurg et al., 2021; Schulz & Sherwood, 2008). This finding may also be due to the potential for caregivers who have been in the role for a longer time having developed their own strategies and supports for coping with problems and perhaps being less receptive to suggestions from push notifications.

Among time-varying variables, we identified two significant effect moderators. Among caregivers reporting high app usability, lower anxiety levels in the prior week were associated with increased message efficacy in reducing caregiver strain. This suggests that caregivers with better mental health status may be more receptive to and capable of acting on intervention recommendations, to improve their mental well-being (Schmitz et al., 2024). Additionally, higher step counts in the prior week were associated with increased message efficacy in reducing depression. One possible explanation is that caregivers who are more physically active may be more health-conscious and/or have more time for extra activities, and therefore may be more likely to engage with and act on behavioral suggestions delivered through the app. Furthermore, increased physical activity (step count) is associated with better mental health status (e.g. anxiety), further enhancing message benefits (Mahindru et al., 2023; Tamminen et al., 2020).

In summary, we identified caregiving duration as a key time-invariant moderator, and prior-week anxiety and physical activity (step count) as significant time-varying moderators of message efficacy. These findings can inform the development of personalized JITAIs for TBI caregivers. For instance, mHealth messages could be more effective when delivered to caregivers with lower levels of anxiety, thereby maximizing their impact on reducing depression. Beyond human-designed tailoring rules, modern machine learning algorithms, such as contextual bandits (Guo & Murphy, 2023; L. Li et al., 2010) and reinforcement learning (Liao et al., 2020; Sutton & Barto, 2018), can be used to automate this decision-making procedure by learning from the data. By taking a caregiver state as input, informed by the identified effect moderators in this paper, these algorithms can personalize the actions that maximize the caregiver’s received reward (e.g., reduced stress).

Limitations and Future Research

First, our analysis is limited by a small sample size. The original study was not designed for conducting this analysis; therefore, the results are underpowered. However, we did observe small, but significant effects for many of the analyses we conducted. Furthermore, the study’s design, which used a 50/50 probability of daily message delivery, led to an imbalance in the categorization of weekly message frequency (none, low, medium and high), further reducing statistical power. Therefore, the replication of these findings in a large-scale trial is warranted to validate the conclusions from these analyses.

Second, our analysis relied on a single, post-intervention questionnaire item to assess perceived app usability. While efficient, single-item measures may lack comprehensiveness and limit generalizability. Retrospective ratings also introduce potential recall bias, where participants who felt the intervention effective may be more likely to report higher usability. Future research could consider using well-validated, multidimensional app usability questionnaire specifically designed for mHealth application, such as the one developed by (Zhou et al., 2019), to more thoroughly capture users’ experiences during the test-run period. Objective app usability measures during study, such as app usage time or app open rates should be examined in future studies.

Third, although not considered a limitation, the relatively modest message effects observed in our analysis, compared to those typically reported in traditional interventions, warrant discussion. In the PROMIS scoring framework, changes on the order of 2–3 T-score points are often considered minimally important or clinically meaningful at the group level, particularly in the context of longer term or more intensive intervention programs (Kroenke et al., 2020; Terwee et al., 2021). In our analysis, however, estimated effects across message-frequency levels are all less than 1 T-score point. There are two main reasons that likely contribute. First, the inherent nature of a low-touch mobile-app-based message intervention generally has smaller effect sizes than higher-intensity approaches such as in-person therapy. Second, our analysis focuses on short-term weekly changes in HRQOL outcomes, which is likely to have smaller effect, compared to long-term studies evaluating the cumulative effect of interventions, where effects can accumulate over time (Yang & Stee, 2019).

Conclusions

This study demonstrates that perceived app usability influences the efficacy of a personalized mHealth intervention delivered via personalized mobile app messages in reducing depression T-score among TBI caregivers. Higher perceived usability was associated with increased message efficacy. Furthermore, we found that shorter caregiving duration and lower anxiety levels were associated with increased message efficacy in reducing caregiver strain, while higher step counts were associated with increased message efficacy in reducing depression among caregivers reporting high app usability. These findings highlight the dual importance of improving app usability and tailoring interventions based on individual contextual factors. Future research could focus on developing effective tailoring rules using these identified moderators and designing usability-enhancing mechanism to optimize health outcomes for caregivers of people with TBI.

Figure 3:

The estimated message’s effects at low, medium, and high frequencies on weekly caregiver strain T score over previous week’s anxiety T score, along with 95% confidence intervals.

Impact and Implications Statement.

• TBI caregivers who perceived the mHealth app as more usable showed greater reductions in depression and caregiver strain, highlighting the importance of perceived app usability in mHealth message efficacy.

• Tailoring interventions based on perceived app usability, caregiving duration, anxiety levels and physical activity may enhance outcomes and enable more responsive, personalized support for TBI caregivers.

List of Abbreviations

CAT

Computer Adaptive Test

GEE

Generalized Estimating Equation

HRQOL

Health-Related Quality of Life

JITAI

Just-in-time Adaptive Intervention

MRT

Micro-randomized Trial

PROMIS

Patient-Reported Outcomes Measurement Information System

RCT

Randomized Controlled Trial

TBI

Traumatic Brain Injury

WCLS

weighted and centered least squares