Psychometric Evaluation of the Homework Rating Scale—Mobile Application: Reliability and Predictive Validity for Assessing Engagement in Smartphone‐Delivered Cognitive Behavioural Therapy

ABSTRACT

Self‐guided digital mental health apps often show high attrition, highlighting the need to understand engagement beyond behavioural metrics. The Homework Rating Scale—Mobile Application (HRS‐MA) was developed to assess cognitive, emotional and attitudinal aspects of engagement. This study evaluated its psychometric properties. The HRS‐MA was integrated into the MoodMission app and administered after 30 days of use, alongside additional self‐report measures. Data from 2100 users were analysed for reliability, construct, convergent, divergent and predictive validity. The HRS‐MA showed excellent internal consistency (Cronbach's α = 0.90; McDonald's ω = 0.94). Factor analyses supported a two‐factor structure—engagement and difficulty and obstacles. The measure demonstrated strong convergent and divergent validity and modest predictive validity. A derived coefficient of psychological effort, representing the interaction between engagement and perceived barriers, significantly predicted symptom reduction, particularly in depression. The HRS‐MA is a psychometrically sound tool for capturing cognitive and attitudinal dimensions of engagement in digital interventions. It complements behavioural indicators by offering a more comprehensive view of user experience and effort, providing researchers and developers with a valuable framework for optimizing mental health app design and evaluation.

Summary

• The study has revealed that the HRS‐MA has sound psychometric properties and can be used to assess engagement with digital interventions and mental health apps.

• The Psychological Effort Index can be conceived as an indicator of striving with CBT activities to achieve significant symptom reduction.

1. Introduction

Mental health applications (MHApps) have emerged as a promising avenue for delivering accessible, effective support for psychological distress, depression and anxiety. Meta‐analyses confirm their efficacy in reducing symptoms (e.g., Linardon, Torous, et al. ²⁰²⁴; Luo et al. ²⁰²⁵). However, outside of controlled trials, user engagement with these apps tends to be very low (e.g., only ~3.9% of users remain active after about 15 days in Wu et al. ²⁰²¹, and see also Fleming et al. ²⁰¹⁸ and Lipschitz et al. ²⁰²³). This engagement gap between proven efficacy and real‐world impact highlights the need to understand and improve how users interact with MHApps.

To study engagement and its determinants, it is essential to clearly define and operationalize the concept, enabling development of reliable measures. In the context of MHApps, engagement can be conceptualized as the process of attracting and sustaining users' attention, interest, time, emotional involvement and cognitive participation in an intervention (O'Brien et al. ²⁰²⁰). Common metrics of behavioural engagement include app uptake (e.g., downloading or trying the app at least once), sustained use over time and adherence or completion of intended activities. MHApps facilitate easy tracking of such behaviours (e.g., number of logins or exercises completed) compared to other treatments like medication compliance (Lipschitz et al. 2023). Critically, though, behavioural metrics alone cannot capture users' cognitive, emotional and attitudinal engagement. Passive usage data tell us what users did, but not how they felt or thought about those activities. Therefore, self‐report instruments are required to assess these subjective dimensions of engagement.

The present study addresses this need by examining the Homework Rating Scale—Mobile Application (HRS‐MA; Bakker and Kazantzis ²⁰²⁰), a self‐report measure designed to capture the cognitive and attitudinal factors underlying user engagement with MHApps. The HRS‐MA builds on the concept of ‘homework’ in cognitive‐behavioural therapy (CBT): the exercises or activities that clients are assigned between sessions to reinforce learning. Most MHApps are grounded in CBT principles, techniques and interventions (e.g., behavioural activation and cognitive restructuring; Denecke et al. ²⁰²²; Rathbone et al. ²⁰¹⁷). Accordingly, the recommendations and exercises provided by these apps can be viewed as the analogue of therapy homework assignments, intended to promote generalization and maintenance of behavioural and cognitive changes (Tang and Kreindler 2017). Difficulties with engagement in traditional CBT homework often arise because patients must perform therapeutic tasks on their own, without immediate support (Kazantzis 2024). In such unsupervised contexts, negative appraisals (e.g., low self‐efficacy or doubts about the task's value) can be triggered, and practical obstacles (e.g., lack of time or resources) can further undermine homework completion (Kazantzis and Miller 2021). These same challenges likely contribute to poor MHApp engagement, suggesting that lessons from CBT homework compliance can inform our understanding of digital engagement.

1.1. The Comprehensive Model of Homework in CBT

Kazantzis and Miller (2021) proposed a comprehensive model of homework in CBT that places engagement at its core. Unlike ‘compliance’ or ‘adherence’, which primarily ask whether the patient completed a homework task, engagement in this model emphasizes how the person becomes involved and invested in the task. Engagement is determined by a range of theoretically meaningful factors, including practical obstacles in the patient's context, the inherent difficulty of the task, activation of the patient's belief system (e.g., maladaptive thoughts triggered by the task), the patient's perceptions of those obstacles and the quantity and quality of task performance. In essence, the model considers the entire process by which a person connects with a homework assignment, rather than just whether it was completed.

The model delineates three key phases of homework engagement, each supported by specific therapist and patient behaviours. First, there are the phases of selecting and planning the task. The therapist and patient collaboratively choose or design a homework assignment, guided by the case formulation and tailored to the patient's preferences, beliefs and needs. The task is introduced as a joint ‘practice or action plan’, ensuring the patient understands the rationale behind it, and that the assignment is specific and relevant. Establishing a clear rationale and specific plan at this stage is thought to foster buy‐in and preparedness for the task.

The patient then carries out the homework in their own environment between sessions. During this execution phase, real‐world circumstances come into play. The patient's experience of the task's difficulty, the quantity and quality of practice and any practical obstacles encountered (e.g., time constraints and distractions) all influence engagement. The model posits that these factors affect how fully the patient invests effort in the homework.

Finally, in the phase of reviewing the task, both the patient, on their own, and later together with the therapist in the next session, evaluates the outcomes of the homework. This review includes reflecting on any immediate benefits, such as feelings of pleasure or mastery gained from the task, assessing how well the task's results align with therapy goals, and synthesizing what was learned. The therapist helps reinforce progress, draw connections to therapy goals and adjust future assignments based on this review.

Overall, the comprehensive model defines a set of cognitive and attitudinal determinants of engagement on the part of both patient and therapist. The model predicts that engagement will be highest when these facilitating factors are strong (e.g., the patient sees a clear rationale, finds the task relevant and experiences some enjoyment or mastery), and when perceived task difficulty is neither too low nor too high. Both extremes of difficulty can lead to disengagement: Tasks perceived as too easy may feel trivial or unimportant, whereas tasks seen as too difficult can evoke frustration or hopelessness. By contrast, persisting with a challenging task despite obstacles is viewed as a sign of psychological effort and resilience. According to CBT theory, patients who exert greater effort in homework tend to achieve better therapeutic outcomes (e.g., symptom reduction). This notion of sustained effort in the face of difficulty will later be operationalized in our study as the Psychological Effort (PsE) metric.

1.2. The Comprehensive Model of Homework in CBT Applied to MHApps

The above model was developed in the context of face‐to‐face CBT, where a therapist guides the patient through selecting, planning, doing and reviewing homework. In self‐guided MHApps, however, many of these supportive elements are absent. Users typically download and start using apps independently, with no clinician involvement or external monitoring of their homework adherence. Consequently, any components of the comprehensive model that require therapist participation do not directly apply in MHApps. For example, during the planning phase, users must select and organize therapeutic exercises on their own, relying only on the app's interface and prompts for guidance. There is no therapist to collaboratively tailor the task or personally ensure the user understands its rationale. Similarly, in the review phase, users evaluate the task outcomes by themselves, judging their own feelings of pleasure or mastery, and gauging progress toward personal goals without external feedback. This fully self‐directed context introduces unique challenges for engagement. Users may not accurately assess a task's difficulty or anticipate obstacles when choosing activities on their own. They might also bring pre‐existing maladaptive beliefs (e.g., ‘I won't be able to do this’) that colour their expectations and interpretations of the task's results.

During the engagement phase, such negative appraisals can undermine the quality and quantity of practice; for instance, a user who believes a coping exercise will not help, may not put much effort into it or might abandon it at the first setback. Critically, because there is no therapist involved, users do not receive real‐time encouragement or cognitive intervention to counter these negative thoughts. The absence of external feedback means users might continue with maladaptive interpretations of any difficulties they encounter. This likely contributes to the low adherence rates observed for many MHApps, which are substantially lower than homework adherence in traditional therapist‐guided therapy. In summary, while MHApp users go through the same nominal ‘phases’ of engagement (selecting, planning and reviewing), they do so without the safeguards of personalization, support and feedback that the comprehensive CBT model assumes. This context may collapse some distinctions between phases, as discussed next, and places greater importance on the user's own motivational and cognitive resources to stay engaged.

1.3. The HRS‐MA

Based on the comprehensive CBT homework model, the HRS‐MA (Bakker and Kazantzis 2020) is a 12‐item self‐report scale designed to assess how users engage with therapeutic tasks in MHApps. It is adapted from the original Homework Rating Scale (HRS; Kazantzis et al. ²⁰⁰⁴), which was developed to evaluate homework engagement in face‐to‐face clinical practice. The HRS has been widely used and adapted across diverse clinical populations and settings, consistently showing that higher homework engagement (and positive homework‐related beliefs) predict better treatment outcomes (e.g., Alfonsson et al. ²⁰¹⁷; McDonald and Morgan ²⁰¹³; Sachsenweger et al. ²⁰¹⁵; Wilansky et al. ²⁰¹⁶). For example, the HRS and its revisions have been applied in individual and group CBT, in both in‐person and online interventions and with both adults and youth, demonstrating robust psychometric properties in these contexts (e.g., Hildebrand‐Burke et al. ²⁰²⁴; McEvoy et al., ²⁰²⁴; Yew et al. ²⁰²¹; Zelencich et al. ²⁰²⁰), and represent a more comprehensive assessment of engagement than focused measures of quantity and quality of adherence (see systematic review in Kazantzis et al. ²⁰¹⁷). The HRS‐MA extends this line of work into the digital self‐help domain.

Both the original HRS and the HRS‐MA cover the key phases of the comprehensive homework model, with four items devoted to each phase. Specifically, four items assess the selecting and planning phases (covering the user's understanding of the task comprehension, their grasp of the task rationale, the sense of collaboration or guidance in planning and the specificity of the task assignment). Another four items measure engaging with the task (evaluating the quantity and quality of the user's practice, the practical obstacles encountered and the perceived difficulty of the task). The final four items pertain to reviewing the task (assessing the task's match with the user's goals, the pleasure or enjoyment derived, the sense of mastery or control gained and the perceived progress made toward improvement). Each item is rated on a 5‐point Likert scale from 0 (not at all) to 4 (options labelled as completely, extremely or extensively, depending on the item). Item scores can be summed or averaged to produce a total HRS‐MA score, with higher scores indicating a greater overall engagement with the app's therapeutic exercises. This total score reflects the user's perceived engagement with both in‐app activities and any related outside‐of‐app practices, analogous to how a therapy client uses skills in session and as homework between sessions. In other words, the HRS‐MA provides a snapshot of the user's own appraisal of how well they are engaging with the self‐guided ‘homework’ tasks offered by the app. For the present study, the HRS‐MA was integrated into a popular mental health app, MoodMission (Bakker et al. ^2018a), to evaluate its psychometric properties in a real‐world user sample. Below, we briefly introduce the app and prior findings related to engagement, as groundwork for the current investigation.

1.4. MoodMission Mental Health App

MoodMission is a self‐guided CBT‐based MHApp designed to be engaging and accessible to a broad audience, including both clinical and nonclinical users from adolescence through older adulthood (Bakker et al. ^2018b). Since its launch in 2016, MoodMission has been downloaded over 100,000 times on iOS and Android devices. The app delivers a range of brief interventions or ‘missions’ grounded in evidence‐based techniques such as behavioural activation and cognitive reappraisal, and also incorporates strategies like relaxation exercises, mindfulness and graded exposure tasks. Users are prompted to choose missions (e.g., ‘Go for a 10‐minute walk’ or ‘Write down a thought and challenge it’) based on how they feel, aiming to improve mood and coping skills.

MoodMission's usability, user engagement and efficacy have been evaluated in initial studies (Moggia et al. 2024). Notably, the app received above‐average usability scores on the Mobile Application Rating Scale (MARS; Hides et al. ²⁰¹⁴) compared to general health apps (Lee and Bakker 2025). Higher user engagement with MoodMission (as measured by frequency and depth of usage) was associated with greater improvements in well‐being. Furthermore, a randomized controlled trial demonstrated that using MoodMission led to significant reductions in depressive symptoms and increases in mental well‐being compared to a waitlist control (Bakker et al. ^2018a). These findings indicate that MoodMission can produce tangible mental health benefits, especially when users remain engaged with its content.

Importantly, MoodMission was the first app to incorporate the HRS‐MA as a built‐in measure of engagement. A preliminary analysis using the HRS‐MA provided encouraging evidence for its utility (Aizenstros et al. 2021). In that study, users who engaged more consistently and frequently with MoodMission not only showed better clinical outcomes but also reported more positive beliefs and attitudes toward the app. For example, higher engagement users felt a stronger sense of collaboration with the app (analogous to feeling ‘supported’ despite no human coach), perceived greater progress and mastery over their problems and had a clearer understanding of why the app's activities were useful. These cognitive‐attitudinal factors are precisely what the HRS‐MA is designed to measure, suggesting it captures meaningful variance in user experience that relates to outcomes.

A more recent profiling study of MoodMission users further underscored the relevance of engagement as measured by the HRS‐MA. Moggia et al. (2024) found that younger users and those with higher baseline symptom levels tended to report higher HRS‐MA engagement scores, and these users showed a greater likelihood of symptom improvement over 30 days. In contrast, older users generally had lower engagement and correspondingly less improvement. This points to the HRS‐MA's potential for identifying which users might benefit most from app‐based interventions (e.g., younger, more distressed individuals who actively engage). It also suggests that tailoring strategies to boost engagement among lower engagement groups (such as older users) could be important for maximizing an app's impact.

Given that the HRS‐MA's psychometric properties had only been explored in such preliminary studies, the present research aimed to provide a comprehensive evaluation of the scale's reliability and validity. In a sample of 2100 MoodMission users, we examined the HRS‐MA's internal consistency, factor structure, convergent and divergent validity and its ability to predict mental health outcomes. By doing so, we sought to establish the HRS‐MA as a rigorous tool for assessing the cognitive‐behavioural dimensions of engagement, facets of user experience that are not captured by passive app usage metrics alone. Ultimately, our goal is to support the use of the HRS‐MA in both research and practice to better understand and enhance engagement in the growing field of digital mental health interventions.

2. Method

2.1. Procedure

After downloading MoodMission, the users are requested to answer an ad hoc sociodemographic questionnaire and a battery of instruments to assess depression and anxiety symptoms and mental health literacy. After using the app for 30 days, the users are requested to answer the same battery of instruments that were administered at baseline, along with the HRS‐MA and an ad hoc questionnaire to provide feedback on the app's helpfulness (see Section 2.3 below).

2.2. Participants

We drew our sample from the MoodMission archival dataset. This dataset comprised all users (N = 82,941) who downloaded the app between its launch and the end of 2023, which marked the beginning of the current study. For obtaining our sample, we selected all users who have completed the HRS‐MA at 30 days of app use, resulting in a sample of 2100 users. Notably, the HRS‐MA was incorporated into the app in 2019, which excluded a substantial number of users who had downloaded MoodMission prior to that year. The sample characteristics are reported in the first part of the results section.

2.3. Instruments and Measures

Beyond the HRS‐MA, the following instruments and measures were administered:

2.3.1. Patient Health Questionnaire—9 (PHQ‐9)

The PHQ‐9 (Kroenke et al. 2001) is a nine‐item self‐report scale designed to assess depressive symptoms based on the diagnostic criteria for major depressive disorder outlined in the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM‐IV). Respondents indicate how frequently they have experienced each symptom over the past 2 weeks using a 4‐point Likert scale ranging from 0 (not at all) to 3 (nearly every day). Total scores are interpreted as follows: 0–4 (minimal or no symptoms), 5–9 (mild), 10–14 (moderate), 15–19 (moderately severe) and 20–27 (severe depressive symptoms). The PHQ‐9 demonstrates high internal consistency, with a reported Cronbach's α = 0.89 (α = 0.88 in this study). The PHQ‐9 was administered at baseline and at 30 days of app use.

2.3.2. Generalized Anxiety Disorder—7 (GAD‐7)

The GAD‐7 (Spitzer et al. 2006) is a seven‐item self‐report scale designed to assess anxiety symptoms based on the diagnostic criteria for generalized anxiety disorder outlined in the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM‐IV). Respondents indicate how frequently they have experienced each symptom over the past 2 weeks using a 4‐point Likert scale ranging from 0 (not at all) to 3 (nearly every day). Total scores are interpreted as follows: 0–4 (minimal or no symptoms), 5–9 (mild), 10–14 (moderate) and 15–21 (severe anxiety symptoms). The GAD‐7 demonstrates high internal consistency, with a reported Cronbach's α = 0.92 (α = 0.89 in this study). The GAD‐7 was administered at baseline and at 30 days of app use.

2.3.3. Mental Health Literacy Questionnaire (MHLQ)

The MHLQ was developed by Bakker and Rickard (2019) and included two brief vignettes, one describing a male experiencing symptoms consistent with social phobia, and the other depicted a female with symptoms of depression. Participants were asked to identify the mental health condition portrayed and evaluate eight potential support options (e.g., ‘Vitamins and minerals or herbal tonics’, ‘Contacting a psychologist’) for each vignette, ranking them from ‘Most Harmful’ to ‘Most Helpful’. In addition to the vignettes, the MHLQ included seven multiple‐choice items assessing general mental health knowledge (e.g., ‘What is the most common mental condition in Australia?’ and ‘Is anger a normal human emotion?’). To inform scoring, the survey was distributed to a reference group of 32 mental health clinicians, including eight clinical psychologists, who were considered to have high mental health literacy. Their responses were reviewed to establish appropriate scoring criteria.

2.3.4. Feedback on MoodMission

MoodMission administered a feedback questionnaire, based on items and themes from the MARS (Stoyanov et al., 2015). Users rated items on a 5‐point‐scale from Strongly disagree to Strongly agree (e.g., ‘I enjoyed using the app’ and ‘Using it motivated me’). This scale was found to have good internal reliability (Cronbach's α = 0.84; Bakker and Rickard ²⁰¹⁹).

All scores (except MHLQ) were computed as mean scores.

2.4. Data Analyses

Explorative analyses using descriptive statistics were performed to study the sample and variable characteristics. A missing completely at random analysis (MCAR) with Little's test was performed to determine if it was possible to assume that missing values were produced randomly. Once verified, data imputation was performed with the R package missForest (Stekhoven 2013; Stekhoven and Buhlmann ²⁰¹²).

To describe users' characteristics in terms of symptoms change, differences in PHQ‐9 and GAD‐7 scores between baseline and 30 days of app use were analysed using paired‐samples Student's t‐tests, with effect sizes computed as Cohen's d.

Reliability of the HRS‐MA was assessed using both Cronbach's alpha and McDonald's omega coefficients. To evaluate construct validity, the total sample was randomly split into two subsamples using a binomial function. The two subsamples resulted in n ₁ = 1029 and n ₂ = 1071. Shapiro–Wilk's and Mardia's multivariate normality tests were performed in both subsamples considering the HRS‐MA items distributions.

An exploratory factor analysis (EFA) using principal axis factoring (PAF) as extraction method and varimax rotation was conducted on the first subsample. A confirmatory factor analysis (CFA) using weighted least squares with mean and variance adjustment (WLSMV) was performed on the second subsample to compare the fit of three models: one‐factor model (Model 1), the theoretical structure proposed for the HRS‐MA (Model 2) and the structure previously identified with EFA (Model 3). Subscale scores and reliability coefficients were computed based on the best solution. The relationship between subscale scores was explored with visual inspection of a scatter plot with locally estimated scatterplot smoothing (LOESS curve estimation).

Convergent and divergent validity were examined through correlations between the HRS‐MA, users Feedback on MoodMission and the number of missions completed during the 30 days of app use (convergent), as well as with the MHLQ (divergent). Pearson's r with bootstrapping and bias correction accelerated (BCa) was implemented in this regard.

Predictive validity was assessed using linear regression models, testing whether higher engagement (i.e., higher HRS‐MA scores) predicted lower PHQ‐9 and GAD‐7 scores at 30 days of app use, controlling for their respective initial scores.

Finally, a coefficient was computed to estimate the psychological effort (PsE) involved in engaging with the activities. This coefficient aimed to capture the relationship between engagement (E) and perceived difficulty and obstacles (DOr), and was defined as follows:

$$\mathit{PsE}=\frac{E+1}{\mathit{DOr}+1}$$ (1)

In Equation (1), E refers to the mean of all HRS‐MA items except those assessing task difficulty and practical obstacles. DOr represents the mean of these two latter items, reversed in scoring (denoted by r). A constant of 1 is added to both the numerator and denominator to prevent division by zero and to ensure computational stability. This coefficient assumes that higher reported difficulty and obstacles (i.e., lower reversed scores) decrease the denominator, thus increasing the overall PsE value—reflecting greater psychological effort. Conversely, when perceived difficulty and obstacles are low, the denominator increases, reducing the PsE value and suggesting that less effort was required to maintain the same level of engagement. PsE was regressed on PHQ‐9 and GAD‐7 scores at 30 days to examine its association with mental health app outcomes.

Descriptive statistics, missing values analysis, normality tests, pre‐to‐post change, the EFA and the PsE coefficient were computed on IBM SPSS 30. The CFA and reliability coefficients were computed on JASP 0.95.4.

3. Results

3.1. Sample Characteristics

The sample comprised 2100 users, the majority of whom were women (n = 1632; 77.7%), followed by men (n = 380; 18.1%), users who did not report their gender (n = 48; 2.3%) and non‐binary individuals (n = 40; 1.9%). The mean age was 29.29 years (SD = 11.72), with ages ranging from 12 to 77. In terms of educational attainment, most participants had completed tertiary education (n = 684; 32.6%), followed by postgraduate education (n = 612; 29.1%), secondary education (n = 529; 25.2%) and primary education (n = 275; 13.1%). Regarding employment status, the largest group was employed full‐time (n = 805; 38.3%), followed by part‐time workers (n = 608; 29.0%), unemployed individuals (n = 606; 28.9%) and those on sick leave (n = 80; 3.8%). The users completed 7.10 missions on average (SD = 35.5) at 30 days, with 80.2% of them (n = 1684) completing at least seven missions in this period.

Missing data rates were low across variables: 0.2% for age, 0.1% for employment status, 0.2% for PHQ‐9 and GAD‐7 at baseline, 1.4% for both instruments at 30 days of app use and 2.2% for the Feedback on MoodMission questionnaire. The exception to these low rates was the MHLQ with 35.23% of missingness. No missing values were found for the HRS‐MA responses. Little's MCAR test revealed that it was possible to assume that missing values were produced randomly, χ²(437) = 482.14, p = 0.07. Subsequently, missing values were imputed using the missForest R package. However, since missForest provides reliable imputations only up to approximately 30% missingness (Stekhoven and Buhlmann 2012), the MHLQ scores were excluded from imputation.

Regarding symptomatic distress, the baseline mean score for the PHQ‐9 was 2.39 (SD = 0.70), and for the GAD‐7, it was 2.53 (SD = 0.75), indicating severe symptoms of depression and anxiety. At 30 days of app use, the mean of PHQ‐9 score decreased to 1.33 (SD = 0.73), reflecting moderate depressive symptoms, while the mean of GAD‐7 score was 2.46 (SD = 0.79), indicating persistent severe anxiety symptoms. Analysis of symptom change between the two assessment points revealed statistically significant reductions in both measures: PHQ‐9 scores decreased with a large effect size, t(2,099) = 78.32, p < 0.001, Cohen's d = 1.71 [1.64, 1.78], whereas GAD‐7 scores showed a smaller but significant decrease, t(2,099) = 4.67, p < 0.001, Cohen's d = 0.10 [0.06, 0.15].

In terms of mental health literacy (MHLQ), users (n = 1360) demonstrated a mean score of 5.22 (SD = 0.97) at the start of app use, reflecting generally high levels of mental health literacy. On average, the users gave positive feedback on MoodMission, with a mean total score of 3.66 (SD = 0.54). The engagement subscale received a similar mean score of 3.69 (SD = 0.62), and the hinders subscale received a mean of 2.44 (SD = 0.67), indicating fewer perceived barriers to app use.

Further details on descriptive statistics of the different instruments and measures beyond the HRS‐MA are presented in the Supporting Information S3.

3.2. HRS‐MA Item Characteristics

Table 1 shows the mean, standard deviation, skewness and kurtosis for the HRS‐MA items and total score. The total score was computed as a mean score. Histograms and Q‐Q plots of these variables are presented in the Supporting Information S1.

TABLE 1.

Descriptive statistics of HRS‐MA items and total score.

Variable	Mean (SD)	Median	Skewness	Kurtosis
1. Quantity	2.09 (0.99)	2.00	−0.15	−0.03
2. Quality	1.98 (0.98)	2.00	−0.23	−0.14
3. Difficulty	1.10 (0.99)	1.00	0.57	−0.21
4. Obstacles	1.33 (1.01)	1.00	0.38	−0.30
5. Comprehension	2.62 (1.13)	3.00	−0.72	−0.02
6. Rationale	2.44 (1.10)	3.00	−0.55	−0.18
7. Collaboration	1.88 (1.10)	2.00	−0.16	−0.59
8. Specificity	2.31 (1.04)	2.00	−0.53	−0.08
9. Match with goals	2.18 (1.04)	2.00	−0.38	−0.13
10. Pleasure	2.16 (1.01)	2.00	−0.42	0.00
11. Mastery	1.83 (1.07)	2.00	−0.14	−0.52
12. Progress	1.99 (1.04)	2.00	−0.28	−0.31
HRS‐MA total	2.00 (0.72)	2.00	−0.71	1.31

Abbreviation: SD = standard deviation.

Notably, all items and the total score exhibited acceptable skewness and kurtosis values, with the median for most items centred at 2.00—the exact midpoint of the 0–5 Likert scale. Items three and four tended to have lower scores compared to the others, suggesting that users generally did not encounter significant difficulties or obstacles while completing the activities. Conversely, items five and six showed higher scores, indicating that users demonstrated a strong understanding of the missions and the rationale behind them.

Multivariate normality tests resulted statistically significant (p < 0.001) in both subsamples for the HRS‐MA items (see Supporting Information S2). Therefore, non‐multivariate normality should be assumed, which is consistent with the patterns shown in histograms and Q‐Q plots (see Supporting Information S1).

3.3. Reliability

Considering all 12 HRS‐MA items in the optimal CFA solution (see below), Cronbach's alpha was 0.91, and McDonald's omega was 0.94. These two values show high internal consistency and indicate that the items consistently assess a shared underlying construct.

3.4. Validity

3.4.1. Construct Validity

The EFA performed with the first subsample (n ₁ = 1029) revealed a two‐factor solution. Table 2 shows factor loadings after rotation.

TABLE 2.

Rotated factor loadings from exploratory factor analysis.

Items	Factor 1	Factor 2
10. Pleasure	0.87
9. Match with goals	0.87
8. Specificity	0.81
12. Progress	0.81
6. Rationale	0.78
11. Mastery	0.75
2. Quality	0.70
5. Comprehension	0.70
7. Collaboration	0.68
1. Quantity	0.66
4. Obstacles		0.81
3. Difficulty		0.76

The items that clustered on Factor 1 represent ‘Engagement’, while those grouped on Factor 2 represent ‘Difficulty and Obstacles’.

In the second subsample (n ₂ = 1071), three CFA were conducted to identify the best fitting model for the data: a single‐factor model (Model 1), the theoretically proposed structure of the HRS‐MA (Model 2) and the two‐factor structure identified through the earlier EFA (Model 3). Figure 1 depicts the diagram for the models tested. Table 3 shows the goodness of fit indices of the models.

FIGURE 1.

Models tested with confirmatory factor analysis. Note: Representation of measurement errors has been omitted for visual simplicity.

TABLE 3.

Goodness of fit indices for the models tested with confirmatory factor analysis.

Model	Free param	Test of model fit	p	RMSEA	CFI	TLI
1	60	χ2 (54) = 339.68	< 0.001	0.07	0.98	0.97
2	66	χ2 (48) = 281.70	< 0.001	0.07	0.98	0.97
3	61	χ2 (53) = 91.85	< 0.001	0.03	0.99	0.99

Note: Estimation Method: Weighted Least Squares with Mean and Variance Adjustment (WLSMV).

Abbreviations: CFI = comparative fit index, Param = Parameters, RMSEA = root mean square error approximation, TLI = Tucker–Lewis index.

All models yielded significant χ² statistics (p < 0.001), which is common in large samples and does not necessarily indicate poor fit. Furthermore, all models showed good fit (RMSEA < 0.08, CFI ≥ 0.90 and TLI ≥ 0.95; Hu and Bentler ¹⁹⁹⁹). Model 3 exhibited the best fit. These results are consistent with the ones obtained using MLR (see Supporting Information S4). Taken together, Model 3 demonstrated better performance than Models 1 and 2. The standardized factor loadings for Model 3 (estimated with WLSMV) are presented in the Supporting Information S5.

Considering that Model 3 confirmed the structure previously obtained with EFA, we computed subscales scores for the two‐factor solution in the whole sample. The mean of Engagement was 2.15 (SD = 0.84, Mdn = 2.22, Skewness = −0.59, Kurtosis = 0.59) with Cronbach's α = 0.94 and McDonald's ω = 0.94, representing excellent reliability. The mean of Difficulty and Obstacles was 1.21 (SD = 0.90, Mdn = 1.00, Skewness = 0.48, Kurtosis = 0.02), with Cronbach's α = 0.77, indicating acceptable reliability (McDonald's omega cannot be computed with less than three items).

Furthermore, we explore the association between the two subscale scores with visual inspection of a scatter plot with LOESS curve estimation, revealing a cubic relationship (see Supporting Information S6). This relationship tends to go in the direction of what is stated by the theory that both low and high perceived task difficulty can contribute to disengagement.

3.4.2. Convergent and Divergent Validity

As shown in Table 4, the HRS‐MA total score and its subscales obtained statistically significant correlations with the Feedback on MoodMission questionnaire (total score and subscales), supporting convergent validity. Similarly, the HRS‐MA scores showed statistically significant correlations with the number of missions completed at 30 days, further confirming convergence with behavioural indicators of app use. In contrast, the HRS‐MA showed negligible correlations with the MHLQ, indicating that it measures a construct distinct from mental health literacy, thus supporting divergent validity.

TABLE 4.

Correlations between HRS‐MA, feedback, number of mission at 30 days and MHLQ.

	HRS‐MA total	HRS‐MA eng	HRS‐MA diff and obs
Feedback total	0.56** [0.52, 0.60]	0.62** [0.58, 0.65]	−0.21** [−0.27, −0.15]
Feedback engagement	0.58** [0.54, 0.62]	0.62** [0.58, 0.66]	−0.14** [−0.20, −0.08]
Feedback hinders	−0.25** [−0.32, −0.18]	−0.32** [−0.38, −0.25]	0.29** [0.23, 0.36]
Number of missions at 30 days	0.16** [0.12, 0.22]	0.18** [0.13, 0.23]	−0.07* [−0.12, −0.01]
MHLQ	0.03 [−0.03, 0.09]	0.03 [−0.03, 0.09]	0.01 [−0.05, 0.07]

Note: In brackets: 95% confidence interval estimated with bootstrapping and BCa.

^* p < 0.05.

^** p < 0.01.

3.4.3. Predictive Validity

First, two baseline models were computed in which PHQ‐9 and GAD‐7 scores at 30 days of app use were regressed on their respective baseline scores. Next, the HRS‐MA total score was added as a covariate to these models to assess the additional variance explained by engagement beyond baseline symptom levels. Finally, in a third step, the HRS‐MA total score was replaced by its subscales to evaluate whether they provided a better predictive fit than the total score. Model parameters are presented in Table 5.

TABLE 5.

Parameters of linear regression models testing predictive validity of the HRS‐MA.

Dependent variable	Predictor	β	SE	R 2
PHQ‐9 at 30 days	Intercept	−0.23***	0.05	0.38
	PHQ‐9 baseline	0.65***	0.02
	Intercept	−0.26***	0.06	0.38
	PHQ‐9 baseline	0.65***	0.02
	HRS‐MA total score	0.01	0.02
	Intercept	−0.25***	0.06	0.41
	PHQ‐9 baseline	0.62***	0.02
	HRS‐MA engagement	−0.003	0.001
	HRS‐MA diff and obs	0.13***	0.01
GAD‐7 at 30 days	Intercept	0.97***	0.05	0.31
	GAD‐7 baseline	0.59***	0.02
	Intercept	0.86***	0.06	0.32
	GAD‐7 baseline	0.59***	0.02
	HRS‐MA total score	0.06**	0.02
	Intercept	0.84***	0.06	0.34
	GAD‐7 baseline	0.56***	0.02
	HRS‐MA engagement	0.01	0.02
	HRS‐MA diff and obs	0.15***	0.02

Abbreviation: SE = standard error.

^** p < 0.01.

^*** p < 0.001.

Baseline symptom scores were strong predictors of symptom scores at 30 days of app use. When the HRS‐MA total score was added to the models, it did not significantly predict PHQ‐9 scores, and only modestly predicted GAD‐7 scores, with minimal improvements in explained variance. However, when the HRS‐MA subscales were included, the Difficulty and Obstacles subscale emerged as a significant positive predictor for both PHQ‐9 and GAD‐7, suggesting that users who perceived greater barriers to completing tasks were more likely to report higher symptom severity at 30 days. These models also showed improved fit in terms of explained variance. Overall, these results provide modest evidence for the predictive validity of the HRS‐MA, particularly for the Difficulty and Obstacles subscale.

3.4.4. Psychological Effort

The mean value of the psychological effort coefficient (PsE) across the sample was 1.70 (SD = 0.97, Mdn = 1.35), with moderate positive skewness (1.45) and kurtosis (1.44). To examine its predictive value, two linear regression models were conducted with PHQ‐9 and GAD‐7 scores at 30 days of app use as dependent variables. For PHQ‐9, the model yielded the following parameters: β₀ = −0.03 (SE = 0.05, p = 0.62), β for baseline PHQ‐9 = 0.63 (SE = 0.02, p < 0.001) and β for PsE = −0.90 (SE = 0.01, p < 0.001), with an R ² of 0.40. For GAD‐7, results were as follows: β₀ = 1.14 (SE = 0.06, p < 0.001), β for baseline GAD‐7 = 0.58 (SE = 0.02, p < 0.001) and β for PsE = −0.08 (SE = 0.02, p < 0.001), with an R ² of 0.32. These findings suggest that higher psychological effort is associated with lower symptom severity at 30 days of app use, after controlling for baseline scores, providing evidence for the predictive validity of the HRS‐MA.

Furthermore, a visual inspection of three‐dimensional scatter plots illustrating the relationships among PsE, Engagement and symptom levels at 30 days (i.e., outcomes) revealed a clear linear pattern (in contrast to the non‐linear relationship observed between Engagement and Difficulty and Obstacles). Specifically, higher PsE values were associated with increased Engagement and decreased symptom severity, suggesting that greater psychological effort corresponds to both more sustained engagement and better clinical outcomes (see Supporting Information S7).

4. Discussion

This study offers the first comprehensive psychometric evaluation of the HRS‐MA within a large, naturalistic sample of users of a smartphone‐delivered CBT programme. The HRS‐MA demonstrated sound internal consistency and a coherent two‐factor structure—Engagement and Difficulty and Obstacles—with the two‐factor model showing the best comparative fit in confirmatory analyses. These findings support the HRS‐MA as a reliable, theory‐consistent index of the cognitive and attitudinal processes implicated in engagement with digital CBT tasks. We also proposed a derived index of Psychological Effort (PsE) that captures the degree to which users sustain engagement despite perceived barriers; PsE showed meaningful associations with symptom change, especially for depression. Together, the results position the HRS‐MA, and PsE, as useful complements to passive usage metrics for understanding and optimizing engagement in mental health apps.

4.1. Main Findings

The emergence of two factors aligns with contemporary models of CBT homework that place engagement—not mere completion—at the centre of behaviour change (Kazantzis and Miller 2021; Kazantzis ²⁰²⁴). In our data, items theoretically associated with planning and reviewing (e.g., comprehension, rationale and specificity; match with goals, pleasure, mastery and progress) loaded with practice‐proximal items on a single Engagement factor, whereas task difficulty and practical obstacles clustered separately. This pattern suggests that, in self‐guided contexts, users experience planning, doing and reviewing as an integrated, ‘all‐at‐once’ cognitive‐attitudinal process rather than as discrete phases that are scaffolded by a therapist. Such phase ‘compression’ is consistent with the immediacy of mobile interfaces, where selection, execution and evaluation are co‐located temporally and contextually (Alexopoulou et al. 2020; Lundquist et al. ²⁰¹⁴).

Within the Engagement factor, items with the highest loadings were progress, match with goals and specificity, indicating that perceived improvement, goal alignment and clear instructions are the most salient features of ‘felt engagement’ during digital homework. By contrast, quantity and collaboration were less influential—plausible given that (a) users may value quality over frequency of practice in unsupervised settings, and (b) collaboration is inherently constrained in self‐guided apps relative to therapist‐guided CBT. These item‐level findings refine the comprehensive CBT homework model for digital delivery by highlighting design elements likely to enhance engagement (clear rationale/instructions, explicit goal‐links and visible progress).

The Difficulty and Obstacles factor functioned as a perceived barrier rather than as a component of engagement per se. Convergent patterns with app‐level ‘hindrance’ ratings in the feedback questionnaire reinforce this interpretation. Visual inspection indicated a cubic relationship between Difficulty/Obstacles and Engagement—consistent with the theoretical expectation that tasks that are too easy or too hard undermine involvement, whereas moderate, attainable challenge supports persistence. This aligns with CBT accounts in which maladaptive appraisals (e.g., low self‐efficacy) and contextual impediments (e.g., time constraints) can derail homework, particularly in unsupervised settings (Kazantzis 2024; Kazantzis and Miller ²⁰²¹).

Convergent validity was supported by significant associations between HRS‐MA scores and both app feedback indices, including engagement and hindrance subscales, and the number of missions completed at 30 days, where associations were all in the expected directions, magnitudes small‐to‐moderate. Divergent validity was evident in negligible relations with mental health literacy (MHLQ), consistent with the distinctness of knowledge versus engagement constructs. These patterns align with the HRS‐MA's purpose: It targets users' cognitive‐attitudinal stance toward therapeutic tasks, not global app satisfaction or declarative knowledge.

Baseline symptoms were strong predictors of outcomes at 30 days for both depression and anxiety, as expected. Beyond baseline, the HRS‐MA total did not significantly predict PHQ‐9, and it showed a very small, statistically significant positive association with GAD‐7 (β = 0.06), a counterintuitive but negligible effect in practical terms. In contrast, the Difficulty and Obstacles subscale was a positive predictor of both PHQ‐9 (β = 0.13) and GAD‐7 (β = 0.15), modestly improving explained variance, indicating that greater perceived barriers forecast higher symptom severity at follow‐up. These results underscore the clinical relevance of targeting barriers to digital homework within app design and user support.

The PsE coefficient operationalizes the theorized role of sustained effort in the face of challenge, indexing engagement level relative to perceived task difficulty and obstacles. PsE showed a strong negative association with PHQ‐9 at 30 days (β = −0.90; R ²  = 0.40) and a small negative association with GAD‐7 (β = −0.08; R ²  = 0.32), controlling for baseline symptoms. Three‐dimensional visualizations indicated a clear linear pattern: Higher PsE co‐occurred with higher Engagement and lower symptoms. These findings are consistent with CBT theory and prior work on homework: Persisting with effortful therapeutic actions, despite barriers, is linked to better outcomes (e.g., Kazantzis et al. ²⁰⁰¹; Sachsenweger et al. ²⁰¹⁵). As a derived indicator, PsE deepens the HRS‐MA's clinical utility by capturing how hard users are working to remain engaged, not just how much they engage.

4.2. Strengths, Limitations and Future Directions

A key strength is the large, ecologically valid sample, enabling robust psychometric testing and evaluation of convergent, divergent and predictive validity. Still, several limitations temper inference. First, users engaged independently without therapist scaffolding, introducing heterogeneity in how missions were selected and used, potentially diluting engagement–outcome links. Second, outcomes relied exclusively on self‐report, which is sensitive to cognitive biases (e.g., negative appraisals among more symptomatic users) that may also influence HRS‐MA responses (Kazantzis et al. 2001). Third, the comprehensive CBT homework model's phase structure was not recovered, reflecting contextual differences between therapist‐guided and self‐guided delivery rather than a failure of the underlying theory. Finally, convergent validity was assessed against related but not identical constructs; future work should examine the HRS‐MA alongside validated MHApp engagement instruments.

Future psychometric refinements should leverage item response theory to calibrate item discrimination and difficulty and network analysis to map inter‐item dependencies and identify central ‘leverage points’ within the engagement network. Conceptually, revising or replacing lower‐performing items (e.g., collaboration) and enhancing sensitivity to personalized guidance may improve fit to self‐guided settings. On the intervention side, embedding adaptive difficulty algorithms and just‐in‐time rationale/progress cues may enhance the very constructs (specificity, goal alignment and perceived progress) that anchor the Engagement factor.

5. Conclusion

The HRS‐MA provides a psychometrically sound, theory‐informed window into the cognitive and attitudinal determinants of digital homework engagement, with a two‐factor structure that reflects how users experience self‐guided CBT tasks. While predictive effects of the total score were modest, perceived barriers and the derived PsE index were meaningfully associated with outcomes, particularly depressive symptoms. As MHApps scale, integrating the HRS‐MA, and PsE, with behavioural telemetry offers a more complete account of why users engage—and how to support effortful, goal‐directed practice that translates to symptom improvement.

Funding

The authors have nothing to report.

Ethics Statement

The research project associated with MoodMission was approved by the ethics committee of Monash University.

Conflicts of Interest

Kazantzis disclosed his royalties from the Guilford Press (Therapeutic Relationship in Cognitive Behaviour Therapy; Cognitive and Behaviour Theories in Clinical Practice), Routledge (Using Homework Assignments in Cognitive Behaviour Therapy); Springer Nature publishers (CBT: Science into Practice Book Series; Handbook of Homework Assignments in Psychotherapy: Research, Practice and Prevention).

Supporting information

Data S1: Supporting information.

Moggia, D. , Lutz W., Bakker D., Schwartz B., and Kazantzis N.. 2026. “Psychometric Evaluation of the Homework Rating Scale—Mobile Application: Reliability and Predictive Validity for Assessing Engagement in Smartphone‐Delivered Cognitive Behavioural Therapy.” Clinical Psychology & Psychotherapy 33, no. 1: e70239. 10.1002/cpp.70239.

Data Availability Statement

The archived data from MoodMission for this study are available upon request from David Bakker (david@moodmissionapp.com).