Skip to main content
NCBI home page
Internet Interventions logoLink to Internet Interventions
. 2024 Oct 28;38:100783. doi: 10.1016/j.invent.2024.100783

Are there unique facets of therapeutic alliance for users of digital mental health interventions? An examination with the eHealth Therapeutic Alliance Inventory

Ohad Ashur 1, Chen R Saar 1, Or Brandes 1, Amit Baumel 1,⁎
PMCID: PMC11565397  PMID: 39552938

Abstract

Therapeutic alliance (TA) is a well-established predictor of clinical outcomes in traditional psychotherapy. However, its association with outcomes in eHealth interventions has been inconsistent, which might be due to the absence of measurements specifically designed to capture TA in eHealth settings. The eHealth Therapeutic Alliance Inventory (ETAI) incorporates conventional as well as unique eHealth TA subscales, enabling to examine the contribution of new facets of TA beyond traditional concepts. This study investigates the predictive contribution of eHealth TA subscales compared to conventional TA subscales on clinical outcomes and evaluates the concurrent criterion validity of the ETAI. The study was conducted within the framework of a randomized controlled trial involving a 10-week digital parent training program aimed at addressing child disruptive behaviors. Parents were randomly assigned to either an enhanced-quality or a standard-quality program. Parents from 68 families completed the ETAI at five weeks' post-program initiation and at the post-intervention phase. The primary outcome was the improvement in child behavior, measured by the Eyberg Child Behavior Inventory. Positive Pearson correlations were found between all ETAI subscales covering unique eHealth TA aspects, measured at the 5-week time-point, and improvement in child behavior at post-intervention (rs ≥ 0.23, ps < 0.03). The conventional TA subscale showed no significant Pearson correlation with improvement in child behavior. When examining the unique contributions of ETAI-subscales to explain the improvement in child behavior, only ETAI-Perceived Emotional Investment subscale was found to have a unique contribution (β = 0.29, p = 0.019). In addition, scores on most ETAI subscales were significantly higher among parents using the enhanced-quality program compared with the standard program (Cohen's ds > 0.48), reinforcing ETAI's criterion validity. The development of TA scales that incorporates unique eHealth TA subscales show initial promise in predicting outcomes. Further research is needed to better understand how different factors of eHealth TA relate to clinical outcomes across diverse clinical targets and programs.

Keywords: Therapeutic alliance, Measurements, Digital health, eHealth, Digital parent training

Highlights

  • •

    Unique facets of therapeutic alliance emerge in eHealth settings, differing from traditional face-to-face therapy.

  • •

    Findings indicate that therapeutic alliance scales, incorporating unique eHealth subscales, may provide improved predictions of clinical outcomes.

  • •

    Results show that user's emotional investment in the eHealth program may serve as a key factor in predicting outcomes.

1. Introduction

1.1. Background

Numerous studies and systematic reviews have recognized the importance of therapeutic alliance (TA) in digital health interventions (Baumel et al., 2017; Berger, 2017; Cavanagh and Millings, 2013; Wehmann et al., 2020). Studies suggest the feasibility of establishing TA, even within the context of fully automated, unguided interventions (Berger, 2017; Lopez et al., 2019; D'Alfonso et al., 2020; Hollis et al., 2018). Notably, TA with conversational agents has been shown to resemble the alliances that humans report with their therapists (Darcy et al., 2021; Gupta et al., 2022). However, while the association between TA and clinical outcomes has been well-established in traditional psychotherapy (Flückiger et al., 2018; Horvath et al., 2011; Lambert and Barley, 2001; Raue and Goldfried, 1994), documentation of such a relationship in the digital space has been infrequent and inconsistent (Clarke et al., 2016; Meyer et al., 2015; Ormrod et al., 2010). One of the main concerns in this regard is that existing eHealth TA measures are insufficient for fully capturing the unique nature of TA in the digital environment (Danaher et al., 2015; Hekler et al., 2016; Riley et al., 2011; Webb et al., 2010). Researchers have argued that most eHealth TA measures have been merely adapted from existing face-to-face scales without fully accounting for key factors that are unique to eHealth TA. Accordingly, researchers have emphasized the need to develop a new measurement that encompasses the particular nature of eHealth TA (Berry et al., 2018; Goldberg et al., 2021).

To date, the literature lacks a comprehensive theoretical framework for the various facets of eHealth TA. In response to this gap, our previous research (Ashur et al., 2023) utilized an empirical approach to investigate whether factors identified in the broader literature as significant for eHealth TA correlate more strongly with outcomes than traditional TA factors. This led to the development of the eHealth Therapeutic Alliance Inventory (ETAI), a measure that incorporates different subscales developed to capture TA facets unique to the digital space (Ashur et al., 2023). This measure was built to enable an ongoing examination in different research settings of how different facets of eHealth TA impact usage and outcomes. One new subscale, Application Induced Accountability (ETAI-Accountability), assesses the program's ability to instill in users a sense of accountability toward the therapeutic process. The second, Sense of Relatedness (ETAI-Relatedness), evaluates how easily users can personify the program and feel as if they are interacting with “someone” who is attentive to their needs. In addition, for research purposes, the ETAI incorporates items taken from conventional TA (ETAI-Conventional), representing an adapted digitized version of TA as measured in face-to-face settings (see Ashur et al., 2023 for a detailed description of the ETAI's development process).

In a preliminary study, the ETAI demonstrated adequate internal consistency and promising preliminary results regarding the association of the subscales with clinical outcomes. The study involved 273 adults participating in a 6-month mobile alcohol reduction intervention who completed the ETAI at the end of the intervention. Findings indicated that the new ETAI subscale (ETAI-Accountability) had a greater contribution than the conventional subscale (ETAI-Conventional) to explaining two clinical outcomes: users experiencing positive change in drinking habits and users' commitment to reducing alcohol consumption in the future (Ashur et al., 2023).

However, the previous study of the ETAI examined the factor structure and internal consistency without thoroughly examining its predictive validity. That is, the study lacked an examination of ETAI scorings during the intervention and their relationship to program usage and clinical outcomes at subsequent time points post-treatment. In addition, the previous study included a single study group, which prevented an investigation of the ETAI's concurrent criterion validity, for example by validating the intrinsic association between the program's quality and the extent to which eHealth TA may be established.

1.2. Current study

The primary objective of this study was to investigate the relationship between the ETAI, program usage, and outcomes, and to examine whether a digital program that is aimed at fostering a better user experience also results in better ETAI metrics. To achieve this goal, we conducted an examination within the context of digital parent training programs (DPTs), utilizing the ETAI to assess the TA users developed with the programs. We hypothesized that the new eHealth TA subscales of the ETAI (Application Induced Accountability, Perceived Emotional Investment and Sense of Relatedness) would demonstrate stronger associations with usage and outcomes, particularly the primary outcome of reducing children's nonadaptive behavior, compared to the conventional TA subscale (ETAI-Conventional). Furthermore, as part of the concurrent criterion validity assessment, we hypothesized that the degree of eHealth TA would be influenced by the quality of the intervention, anticipating higher eHealth TA in the DPT with enhanced quality.

2. Materials and methods

This study was incorporated into a randomized controlled trial focused on a self-guided DPT aimed at treating child disruptive behaviors. A comprehensive description of the trial can be found in the primary study paper (Baumel et al., 2023). The primary study examined whether programs that incorporate features specifically designed to enhance positive changes in users' lives (i.e., programs that are high in “therapeutic persuasiveness”) outperform equivalent programs lacking therapeutic persuasiveness features, in terms of their association with clinical outcomes and usage metrics. Ethical approval of the trial was obtained from University of Haifa institutional review board (approval number: 058/22).

2.1. Participant and recruitment procedure

Parents were recruited through a targeted Facebook advertising campaign between May and July 2022. Parents who provided their contact information were directed to complete a concise eligibility screening questionnaire, encompassing inclusion and exclusion criteria and inquiries pertaining to their child's behaviors. The eligibility criteria were: (1) having a child between the ages of 3 and 7 with (2) elevated levels of behavior problems, as determined by the Eyberg Child Behavior Inventory (ECBI) subscales (ECBI-problem ≥15 or ECBI-intensity ≥132), and (3) access to a smartphone device with cellular and Internet connection. Exclusion criteria were as follows: (1) the child was undergoing treatment for behavior or emotional problems, or the parent was participating in another parent training program; and (2) the child had been diagnosed with intellectual disability or developmental delay. Eligible parents underwent a verification process via telephone to confirm screening criteria and receive further details about the study. Those choosing to proceed were guided to electronically sign a consent form and complete a baseline assessment battery. Enrolled parents were randomly assigned to either the improved-DPT or standard-DPT using a computerized random sequence generation procedure.

2.2. Interventions

The study comprised two study groups: Improved-DPT and Standard-DPT. While both programs were the same in terms of content, they differed in terms of quality, specifically regarding their level of therapeutic persuasiveness – that is, the extent to which the program is designed to foster positive changes in users' lives. The Improved-DPT study group incorporated additional features aligned with the therapeutic persuasiveness conceptual model (Baumel et al., 2023). These extra elements included a focused phase after each module to encourage engagement in therapeutic activities, timely triggers through text messages, and the adaptation of subsequent modules based on the user's current state, among other enhancements. Treatment fidelity was evaluated using the Enlight quality expert rating scale (Baumel et al., 2017), revealing a distinct differentiation in the quality of the interventions in the anticipated direction. Compared to Standard-DPT, participants allocated to Improved-DPT used the program significantly more (ps < 0.001; Cohen's ds = 0.91–2.22) and reported greater improvement in child behavior problems (ps < 0.05; Cohen's ds = 0.43–0.54) (Baumel et al., 2023).

2.3. Measures

2.3.1. ETAI (Ashur et al., 2023)

TA was measured using the ETAI. Participants were instructed to complete the ETAI at Time 2 (T2), which was 5-weeks post-baseline. To investigate the difference in alliance development between the two study groups, we also measured the ETAI at Time 3 (T3) – post-treatment, at 10-weeks post-baseline. The latter measurement was aimed at evaluating how the experience of each program version by the end of the intervention might impact alliance development. The original ETAI has 14 items, using a 7-point Likert scale to record respondents' level of agreement with each statement (1 = strongly disagree; 7 = strongly agree). The original ETAI has three subscales that are calculated based on averaging their corresponding items. Sense of Relatedness (ETAI- Relatedness) comprises four items reflecting how easily users can personify the program and feel like they are relating to an attentive “someone” in their interaction with the program's therapeutic representative (e.g., a therapeutic bot) and the platform as a whole, or experience a sense of belonging to the program's community collaborating for positive change. The conventional subscale – ETAI-Conventional, comprises six items that represent a digitized version of TA measured in face-to-face settings. Application Induced Accountability (ETAI-Accountability) measures the extent to which the program instills in users a sense of accountability toward engaging with the therapeutic process. This subscale originally consisted of four items; however, we chose to separate two items measuring ETAI-Perceived Emotional Investment (ETAI-Emotional) from the remaining two items. ETAI-Emotional reflects the extent to which users perceive the therapeutic goal as emotionally important, suggesting that not achieving it would instill a sense of disappointment. This decision stemmed from inconsistencies found in the four items of the former subscale; while two items indeed represented the accountability induced by the application to take action in a practical sense, the other two reflect users' emotional connection to their goals, suggesting that failure to achieve them could lead to a sense of disappointment. As a result, we decided to cluster and test these items separately (see the ETAI items and subscales in Appendix A – under the theme of Emotional). In this study, Cronbach's alpha scores indicated adequate internal consistency for the ETAI-total (all 14 items), ETAI-Accountability, ETAI-Relatedness, and ETAI-Conventional (Cronbach's alphas of 0.747, 0.877, 0.761, and 0.897, respectively). The only exception was ETAI-Emotional, for which the Cronbach's alpha for internal consistency was 0.625.

2.3.2. Eyberg Child Behavior Inventory-Intensity (ECBI-I) (Eyberg and Pincus, 1999)

The Eyberg Child Behavior Inventory-Intensity (ECBI-I) serves as our primary outcome variable. The ECBI-I encompasses 36 behaviors commonly reported by parents of children with behavior problems. Parents rate the intensity of each behavior on a Likert scale ranging from 1 to 7 (1 = never; 7 = always; Burns and Patterson, 1990). The ECBI-I has proven to be useful in discriminating between children's problem and non-problem behaviors for the purposes of evaluation and in reflecting change in behavior problem symptoms following treatment interventions (Eyberg et al., 1993; Eyberg et al., 2008). In this study, the Cronbach's alpha for the ECBI-I was 0.80.

2.3.3. Parenting Scale (PS) (Arnold et al., 1993)

The PS comprises statements that capture parental responses to their child's misbehavior. Each statement illustrates both effective and ineffective reactions in a hypothetical child-related scenario. Parents rate their typical responses on a 7-point Likert scale, with 7 indicating the effective end of the scale and 1 representing the ineffective end (Arnold et al., 1993). In this study, two subscales of the PS were utilized: over-reactivity (PS-OV), consisting of 11 items, and laxness (PS-LA), consisting of 10 items. Both subscales exhibited adequate internal consistency, with Cronbach's alpha coefficients of 0.75 for over-reactivity and 0.83 for laxness.

2.3.4. Me as a Parent Scale (MaaPs) (Hamilton et al., 2015)

Overall parental self-efficacy was assessed using the self-efficacy subscale of MaaPs. This subscale consists of four items, such as “I have confidence in myself as a parent.” Parents rate each item on a Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). In this study, the self-efficacy subscale of MaaPs demonstrated good internal consistency, with a Cronbach's alpha coefficient of 0.79.

2.3.5. Data on user engagement with the digital intervention

The utilization patterns of users, including login activities, date and time of use, time spent in the program, and accessed features, were automatically recorded within the digital platform. Two usage variables were employed in this study: (1) program completion rate, which indicates the portion of program modules completed by the user within a specific timeframe (calculated at T2 or T3), with 1 denoting completion of all modules and 0 indicating completion of no modules; and (2) number of login days, which represents the total number of distinct days on which the user logged into the application (measured at T2 or T3).

2.4. Patient and public involvement

The current study focuses on the evaluation of a digitized parent training program delivered via the web, without direct therapist guided support. As such, the research design and implementation involved service users who were parents of children with behavior problems, searching for guidance. The interventions were developed based on user-centered-design approach, which leaned on users' feedback during its development and optimization phases to make sure it meets their needs. The participants in the current study were individuals voluntarily participating in online training, and their involvement was limited to the completion of the program and related assessments. Therefore, the nature of the current study did not lean on direct patient involvement in the study design, conduct, analysis, or interpretation of data.

2.5. Data analysis

Pearson correlations were computed to examine the relationships between each subscale of the ETAI measured at T2 and the dependent variables. Since T3 is not suitable for predicting outcomes due to the lack of a time gap and likely reflects user satisfaction more than alliance (as it is measured at the program's end), T3 correlations are included in Appendix B for data completeness. The assessment of all dependent variables was based on the difference in cumulative scores between T1 (pre-treatment) and T3, with a positive value indicating beneficial change. We evaluated normality using the Shapiro-Wilk test, which revealed that the ECBI-I, Number of Login Days, and all ETAI variables, except for ETAI-Emotional, were non-normally distributed. To address non-normality in distributed data, we applied the two-step approach outlined by Templeton (2011). A sensitivity analysis indicated no significant differences between the original data and the transformed values. To assess the relationship between the ETAI and usage metrics, Pearson correlations were calculated between each subscale of the ETAI and the two usage variables. Hierarchical linear regression (stepwise) was then used to assess the unique contribution of each of the ETAI subscales and the two usage variables, measured at T2, to predict the clinical improvement in child behavior at T3 (post-intervention). To assess the concurrent criterion validity of the scale, independent samples t-tests were performed, evaluating the potential divergence between the two study groups in the development of TA. Cohen's d was used to evaluate effect sizes. All statistical analyses were performed using SPSS 28.0 (IBM Corp).

3. Results

The study involved a total of 88 participants, of which 68 completed the ETAI at T2 and all measurements (including the ETAI) at T3. Among these participants, 34 individuals received the Improved-DPT intervention and 34 received the Standard-DPT intervention. At the beginning of the intervention, the leading parent's average age was 36.5 (SD = 3.61); in 84 (95.5 %) families, the leading parent of the intervention was a mother; the mean age of the children was 4.9 years (SD = 1.32); and 50 (56.8 %) were men. There were no significant differences in demographic characteristics between the two study groups (ps ≥ 0.11).

3.1. Relationship between the ETAI and clinical outcomes

Table 1 presents the associations between the ETAI subscales measured at T2, the ETAI complete scale (all 14 items), and the outcome variables. Pearson correlations between the ETAI complete scale and outcome variables were positive and statistically significant (rs ≥ 0.25, ps ≤ 0.022). Pearson correlations between the ETAI-Accountability and the ECBI-I and PS-OV measurements were found to be positive and significant (r = 0.28, p = 0.011; r = 0.26, p = 0.017, respectively), but not significant with MaaPs and PS-LA (ps ≥ 0.07). With respect to the ETAI-Emotional, Pearson correlations were positive and significant only with the ECBI-I (r = 0.25, p = 0.02), but not with parental outcomes (ps ≥ 0.06). Pearson correlations for the ETAI-Relatedness were positive and significant with all outcome measures (rs ≥ 0.21, ps ≤ 0.045), except for PS-LA (p = 0.07). Pearson correlations for the ETAI-Conventional were significant only for the parental outcome variables (rs ≥ 0.24, p ≤ 0.028), but not for the ECBI-I (p = 0.202).

Table 1.

Pearson correlations between ETAI factors at T2, improvement in child's behavior symptoms over time (T1–T3), and parenting variables.

 ECBI-I
 MaaPs
 PS over-reactivity
 PS laxness
r p r p r p r p
ETAI-total 0.255 0.019 0.323 0.004 0.247 0.022 0.246 0.022
ETAI- Accountability 0.280 0.011 0.182 0.070 0.259 0.017 0.103 0.203
ETAI- Emotional 0.252 0.020 0.121 0.164 0.012 0.462 0.187 0.064
ETAI- Relatedness 0.233 0.029 0.257 0.018 0.209 0.045 0.182 0.070
ETAI- Conventional 0.103 0.202 0.346 0.002 0.235 0.028 0.246 0.022
Open in a new tab

Note. ECBI-I = Eyberg Child Behavior Inventory Intensity level; MaaPs = Me as a Parent scale; PS over-reactivity = the over-reactivity subscale of the Parenting Scale; PS laxness = the laxness subscale of the Parenting Scale; ETAI-total = the complete ETAI comprising 14 items. Significant results are in bold.

3.2. Relationship between the ETAI and program usage

Table 2 shows the associations between the ETAI metrics measured at T2 and the two program usage metrics (program completion rate and number of login days) measured post-intervention. There were no significant Pearson correlations found between program completion rate and any of the ETAI subscales or the ETAI complete scale (ETAI-total). Significant positive Pearson correlations with number of login days were found only for the ETAI-Accountability (r = 0.24, p = 0.022) and ETAI-Conventional (r = 0.27, p = 0.014).

Table 2.

Pearson correlations between ETAI Factors at T2 and usage variables post-intervention.

 Program completion rate
 Number of login days
r p r p
ETAI-total −0.053 0.335 0.108 0.193
ETAI-Accountability 0.080 0.257 0.245 0.022
ETAI- Emotional −0.065 0.301 −0.078 0.266
ETAI-Relatedness −0.074 0.277 0.135 0.137
ETAI-Conventional 0.050 0.343 0.267 0.014
Open in a new tab

Note. Program Completion Rate = the portion of program modules completed by the user by post-intervention; Number of Login Days = the total count of distinct days on which the user logged into the application, measured at post-intervention; ETAI-total = the complete ETAI comprising 14 items. Significant results are in bold.

3.3. Assessing the ETAI's unique contribution to predicting improvement in child behavior

Hierarchical linear regression analysis predicting change over time in the ECBI-I (child behavior problems) was conducted using a stepwise approach (see Table 3). The analysis included all the ETAI subscales and the two usage variables (program completion rate and number of login days) all measured at T2. The hierarchical linear regression analysis yielded two models. In the first model, only the ETAI-Emotional was retained, explaining 8.9 % of the variance of improvement in child behavior problems. The second model included two variables – ETAI-Emotional and program completion rate – which explained 17 % of the variance. In the second model, the ETAI-Emotional exhibited only descriptively larger contribution to explaining the dependent variable compared to program completion rate (ETAI-SC: β = 0.313, p = 0.011; program completion rate: β = 0.286, p = 0.019).

Table 3.

Hierarchical linear regression analysis (Stepwise) predicting change over time in ECBI-I, with ETAI factors and usage measured at T2 (number of login days and program completion rate).

Independent variables β t p
MODEL 1
1. ETAI-Emotional 0.298 2.417 0.019
R2 0.089 n/a 0.019
MODEL 2
1. ETAI-Emotional 0.313 2.634 0.011
2. PRC T2 0.286 2.408 0.019
R2 0.170 n/a 0.004
Open in a new tab

Note. ETAI-Emotional represents the perceived emotional investment subscale of the ETAI; PRC T2 = program completion rate at T2. Significant results are in bold.

3.4. Assessing the concurrent criterion validity of the ETAI

To assess the ETAI's concurrent criterion validity, independent samples t-tests were conducted, examining the difference between Improved-DPT and Standard-DPT in terms of the development of TA (ETAI metrics). The first set of tests was measured at T2 and the second at T3 (see Table 4). At T2, a significant difference in the development of TA was observed solely for the ETAI-Relatedness, favoring the Improved-DPT study group (p = 0.046, Cohen's d = 0.413). At T3, all the ETAI subscales, except for the ETAI-Emotional, exhibited a significant divergence (ps ≤ 0.03; Cohen's ds ≥ 0.481), pointing to more favorable TA development in the Improved-DPT study group.

Table 4.

Independent samples T-Tests comparing study conditions (Improved-DPT vs Standard-DPT) using the ETAI subscales.

DPT-IMP (n = 34) DPT-STD (n = 34)
Mean (SD) Mean (SD) t (df) p d
T2
ETAI-total 5.214 (0.845) 4.932 (1.006) 1.248 (66) 0.108 0.303
ETAI-Accountability 5.720 (1.194) 5.352 (1.530) 1.104 (66) 0.136 0.268
ETAI-Emotional 3.838 (1.485) 4.397 (1.402) −1.595 (66) 0.059 −0.387
ETAI-Relatedness 4.941 (1.220) 4.411 (1.341) 1.703 (66) 0.046 0.413
ETAI-Conventional 5.686 (0.899) 5.318 (1.033) 1.565 (66) 0.061 0.379
T3
ETAI-total 5.114 (0.903) 4.488 (0.999) 2.615 (61) 0.005 0.660
ETAI-Accountability 5.727 (1.305) 4.816 (1.583) 2.499 (61) 0.007 0.630
ETAI-Emotional 3.575 (1.269) 3.700 (1.087) −0.415 (61) 0.340 −0.105
ETAI-Relatedness 4.780 (1.224) 3.613 (1.549) 3.354 (62) <0.001 0.839
ETAI-Conventional 5.646 (1.071) 5.122 (1.108) 1.909 (61) 0.030 0.481
Open in a new tab

Note. DPT-IMP = the improved DPT study group; DPT-STD = the standard DPT study group; d represents Cohen's d; ETAI-total = the complete ETAI comprising 14 items; Significant results are in bold. The mean values presented in the table are the raw scores and do not reflect the normalized scores, which were rated on a scale from 0 to 1.

4. Discussion

To the best of our knowledge, this study is the first to assess within a randomized control trial setting whether subscales developed uniquely to capture digital TA outperform conventional TA subscales in explaining clinical outcomes, and the first study to investigate quantitatively whether the quality of the intervention influences the extent to which TA might develop, thereby reinforcing the scale's validity.

Concerning the main clinical outcome in this study – improvement in child behavior – the new ETAI subscales (ETAI-Accountability, −Emotional and -Relatability) and the overall scale (ETAI-total) measured at T2 were positively correlated with improvement in child behavior post-intervention. By contrast, the conventional TA scale (ETAI-Conventional) did not demonstrate significant correlation with improvement in child behavior. These preliminary findings align with the limited documentation of a relationship between TA development and outcomes in the digital space (Danaher et al., 2015; Hekler et al., 2016; Riley et al., 2011; Webb et al., 2010). Furthermore, the fact that subscales designed specifically to capture digital TA show significant correlations with clinical outcomes reinforces the recommendations of previous studies to design scales that are tailored to capture digital TA (Berry et al., 2018; Goldberg et al., 2021).

Nevertheless, when assessing the unique contribution of usage and ETAI metrics to predicting improvement in child behavior, only one ETAI subscale – ETAI-Emotional included in the model, alongside the program completion rate. While both ETAI-Emotional and ETAI-Accountability were correlated with outcomes, only ETAI-Emotional contributed significantly to the hierarchical linear regression model. This suggests that emotional investment and motivation may play a more pivotal impact on therapeutic success than enhancing practical accountability, particularly in unguided digital programs where users must take personal responsibility. Theoretically, these findings encourage exploration of the intersection between Ryan and Deci's Self-determination theory (Ryan, 2017) of emotional ambitions, and Mohr et al.'s (2011) conceptualization of supportive accountability, which emphasizes the importance of enhancing clients' accountability in eHealth interventions. Additionally, empirical studies (Kocielnik et al., 2018; Ashur et al., 2023) have preliminary demonstrated a link between accountability and clinical outcomes, highlighting the need for further investigation into the interplay between emotional investment, accountability, and outcomes in eHealth settings.

In terms of the ETAI's associations with usage metrics, contrary to expectations, no significant correlation was found between any of the ETAI subscales (measured at T2) and program completion rate (measured at T3). However, notable associations were observed with an alternative usage metric – number of login days (measured at T3) – but only with the ETAI-Accountability and ETAI-Conventional (both measured at T2). The link between ETAI-Accountability and number of login days seems straightforward, reflecting the intrinsic nature of this subscale, whereby higher accountability aligns with increased program engagement. The lack of such an association with other subscales remains unclear and will be discussed in the Limitations and Future Directions section.

Regarding the impact of program quality on the development of TA, significant differences were identified between the two study groups only at T3, revealing higher TA levels in the enhanced-quality intervention (the intervention with therapeutic persuasiveness features). An exception is the ETAI-Relatedness, which exhibited a significant difference at T2 as well. This time-dependent discrepancy may reflect the gradual nature of TA development and thus the greater potential for detecting differences at a later stage. Given the identical content in both interventions, it is plausible that the effects of therapeutic persuasiveness features require time to influence factors such as the development of TA. In simpler terms, the 5-week period (T2) may have been insufficient for users to fully experience the benefits of the improved program in a way that would have a significant impact on the development of TA. The significant difference in the establishment of a sense of relatedness to the program (ETAI-Relatedness) already at T2 highlights the pivotal role that therapeutic persuasiveness may play in eHealth interventions, especially considering the extensive literature emphasizing the central significance of the therapeutic bond in psychotherapy (Martin et al., 2000).

4.1. Limitations

This study has several limitations that should be acknowledged. First, the reliability of the ETAI-Emotional subscale requires a re-evaluation of its items and overall structure due to the questionable internal consistency identified in this study. Second, the findings indicate that the ETAI emerges as a more robust metric at T3, particularly in its associations with program quality. It might be beneficial for future research to explore whether the measurement of the ETAI at T3 (representing the intervention's conclusion) holds superior predictive value for follow-up outcomes compared to the ETAI measured at T2. Regarding the ETAI's association with usage metrics, no correlations were found with program completion rate. However, it is noteworthy that program completion rate entered the second model during the stepwise regression procedure, indicating a unique contribution of this usage metric to predicting the main clinical outcome (improvement in child's behavior). Additionally, the findings related to secondary parental outcomes were mixed and varied across different ETAI subscales in an unclear manner, warranting further exploration. This implies that the relationship between usage, TA, and clinical outcomes may be more complicated than initially assumed, highlighting the need for more targeted and comprehensive investigations in future studies.

It is important to note that while the new subscales of the ETAI demonstrated strength in their associations with clinical outcomes compared to the conventional TA scale, these represent just three possibilities among many potential facets of eHealth TA. A more comprehensive theoretical exploration of the eHealth TA construct is necessary to fully encompass aspects of eHealth TA that might not have been addressed or investigated in this current study. However, the decision to employ a quantitative empirical evaluation in this study compensates for this limitation. Furthermore, this study focused on a unique study group: parents participating in a DPT. To enhance the external validity of ETAI, future studies should examine ETAI across different intervention modalities (e.g. web-based versus mobile applications, guided versus unguided intervention) and various clinical aims.

4.2. Conclusions

The results of this study suggest that developing new subscales tailored uniquely to capture eHealth TA holds potential value in predicting clinical outcomes and in improving our understanding of how TA evolves in eHealth settings. Moreover, the findings point out the importance of a program's quality as a possible factor in the development of eHealth TA. Further exploration is required to better understand how application induced accountability, perceived emotional investment, sense of relatedness, and other unique eHealth TA factors contribute to positive clinical outcomes.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. eHealth Therapeutic Alliance Inventory

Instructions:

Below is a list of statements that describe some of the different ways people might have thought or felt about the eHealth program they were using.

Please carefully consider your experience with the eHealth program you use (or have used in the past), and then for each statement circle the number indicating how strongly you agree or disagree, on a seven-point scale:

If the statement strongly describes the way you feel (or think) circle the number 7; if the statement strongly contradicts the way you feel (or think) circle the number 1.

Use the numbers in between to describe the variations between these extremes.

Please rate every statement.

Reminder: This questionnaire is confidential. Only the research team will see your answers.

Thank you for your cooperation.

Item number Item content Item theme
1 {{the program / bot}} strengthens my commitment to {{name goal here}} Accountability
2 {{the program / bot}} makes me feel responsible for {{name goal here}} Accountability
3 If I fail to {{name goal here}}, it will feel as if I am disappointing {{the program / bot}} itself. Perceived Emotional Investment
4 If I fail to {{name goal here}}, I will be very disappointed with myself. Perceived Emotional Investment
5 Even though I know that {{the program / bot}} is only a digital application, while using it - it feels as if I am communicating with someone who understands me and supports me. Relatedness
6 I feel like {{the program / bot}}’s content/messages incorporate relatable figure/s I can easily relate to. Relatedness
7R I feel like {{the program/bot}}‘s is not attentive to my needs. Relatedness
8R While using {{the program / bot}} I feel the absence of meeting face-to-face with a real coach/therapist. Relatedness
9 {{The program/bot}} is in tune with my personal goals and helps me achieve them. Conventional
10 The contents/tasks incorporated within {{the program / bot}} focus on helping me achieve goal(s) that I find most important. Conventional
11 I am highly satisfied with the way {{the program / bot}} tries to help me. Conventional
12 {{The program/bot}} offers good information/advice on how to improve my situation. Conventional
13 I feel {{the program / bot}} is dedicated to helping me. Conventional
14 I feel {{the program / bot}} supports me and has my best interest at heart. Conventional
Open in a new tab

Notes: Accountability, Perceived Emotional Investment, Relatedness, and Conventional represent the different subscales of the ETAI (R) = reverse-scored item.

Appendix B. Pearson correlations between ETAI factors at T2 and at T3, with improvement in child's behavior symptoms over time (T1–T3), comparison between the two study groups (Standard vs Improved)

 T2
 T3
DPT-STANDARD DPT-
IMPROVED		 DPT-
STANDARD		 DPT-
IMPROVED
r p r p r p r p
ETAI-total 0.120 0.250 0.374 0.016 0.332 0.036 0.357 0.022
ETAI- Accountability 0.157 0.188 0.395 0.011 0.046 0.405 0.329 0.033
ETAI- Commitment 0.181 0.152 0.455 0.004 0.127 0.251 0.279 0.061
ETAI- Relatedness 0.133 0.227 0.274 0.061 0.399 0.014 0.264 0.072
ETAI- Conventional −0.002 0.496 0.143 0.214 0.319 0.043 0.257 0.078
Open in a new tab

Note. DPT-STANDARD = the standard DPT study group; DPT-IMPROVED = the improved DPT study group; ETAI-total = the complete ETAI comprising 14 items; Significant results are in bold.

References

  1. Arnold D.S., O’leary S.G., Wolff L.S., Acker M.M. The parenting scale: a measure of dysfunctional parenting in discipline situations. Psychol. Assess. 1993;5(2):137. [Google Scholar]
  2. Ashur O., Muench F., Baumel A. Evaluating the contribution of new factors in the assessment of eHealth therapeutic Alliance. Journal of Technology in Behavioral Science. 2023:1–12. [Google Scholar]
  3. Baumel A., Faber K., Mathur N., Kane J.M., Muench F. Enlight: a comprehensive quality and therapeutic potential evaluation tool for mobile and web-based eHealth interventions. J. Med. Internet Res. 2017;19(3) doi: 10.2196/jmir.7270. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Baumel A., Brandes O., Brendryen H., Muench F., Kane J.M., Saar C. The impact of therapeutic persuasiveness on engagement and outcomes in unguided interventions: a randomized pilot trial of a digital parent training program for child behavior problems. Internet Interv. 2023;34 doi: 10.1016/j.invent.2023.100680. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Berger T. The therapeutic alliance in internet interventions: a narrative review and suggestions for future research. Psychother. Res. 2017;27(5):511–524. doi: 10.1080/10503307.2015.1119908. [DOI] [PubMed] [Google Scholar]
  6. Berry, K., Salter, A., Morris, R., James, S., & Bucci, S. (2018). Assessing Therapeutic Alliance in the Context of mHealth Interventions for Mental Health Problems: Development of the Mobile Agnew Relationship Measure (mARM) Questionnaire. J Med Internet Res 2018;20(4):E90Https://Www.Jmir.Org/2018/4/E90, 20(4), e8252. doi: 10.2196/JMIR.8252. [DOI] [PMC free article] [PubMed]
  7. Burns G.L., Patterson D.R. Conduct problem behaviors in a stratified random sample of children and adolescents: new standardization data on the Eyberg child behavior inventory. Psychological Assessment: A Journal of Consulting and Clinical Psychology. 1990;2(4):391. [Google Scholar]
  8. Cavanagh K., Millings A. (inter)personal computing: the role of the therapeutic relationship in E-mental health. J. Contemp. Psychother. 2013;43(4):197–206. doi: 10.1007/s10879-013-9242-z. [DOI] [Google Scholar]
  9. Clarke J., Proudfoot J., Whitton A., Birch M.R., Boyd M., Parker G., Fogarty A. Therapeutic alliance with a fully automated mobile phone and web-based intervention: secondary analysis of a randomized controlled trial. JMIR mental health. 2016;3(1) doi: 10.2196/mental.4656. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. D'Alfonso, S., Lederman, R., Bucci, S., & Berry, K. (2020). The Digital Therapeutic Alliance and Human-Computer Interaction. JMIR Ment Health 2020;7(12):E21895Https://Mental.Jmir.Org/2020/12/E21895, 7(12), e21895. doi: 10.2196/21895. [DOI] [PMC free article] [PubMed]
  11. Danaher, B. G., Brendryen, H., Seeley, J. R., Tyler, M. S., & Woolley, T. (2015). From black box to toolbox: Outlining device functionality, engagement activities, and the pervasive information architecture of mHealth interventions. In Internet Interventions (Vol. 2, Issue 1, pp. 91–101). Elsevier. doi: 10.1016/j.invent.2015.01.002. [DOI] [PMC free article] [PubMed]
  12. Darcy A., Daniels J., Salinger D., Wicks P., Robinson A. Evidence of human-level bonds established with a digital conversational agent: cross-sectional, retrospective observational study. JMIR Formative Research. 2021;5(5) doi: 10.2196/27868. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Eyberg S.M., Boggs S.R., Rodriguez C.M. Relationships between maternal parenting stress and child disruptive behavior. Child Family Behav. Ther. 1993;14(4):1–9. [Google Scholar]
  14. Eyberg S.M., Nelson M.M., Boggs S.R. Evidence-based psychosocial treatments for children and adolescents with disruptive behavior. J. Clin. Child Adolesc. Psychol. 2008;37(1):215–237. doi: 10.1080/15374410701820117. [DOI] [PubMed] [Google Scholar]
  15. Eyberg S.M., Pincus D. Eyberg Child Behavior Inventory and Sutter-Eyberg Behavior Inventory-Revised: Professional Manual. Psychological Assessment Resources. 1999 Odessa, FL. [Google Scholar]
  16. Flückiger C., Del A.C., Wampold B.E., Horvath A.O. The Alliance in adult psychotherapy: a Meta-analytic synthesis. Psychotherapy. 2018;55(4):316–340. doi: 10.1037/pst0000172. [DOI] [PubMed] [Google Scholar]
  17. Goldberg S.B., Baldwin S.A., Riordan K.M., Torous J., Dahl C.J., Davidson R.J., Hirshberg M.J. Alliance with an unguided smartphone app: validation of the digital working Alliance inventory. Assessment. 2021 doi: 10.1177/10731911211015310. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Gupta M., Malik T., Sinha C. Delivery of a mental health intervention for chronic pain through an artificial intelligence–enabled app (Wysa): protocol for a prospective pilot study. JMIR Research Protocols. 2022;11(3) doi: 10.2196/36910. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Hamilton V.E., Matthews J.M., Crawford S.B. Development and preliminary validation of a parenting self-regulation scale:“me as a parent”. J. Child Fam. Stud. 2015;24(10):2853–2864. [Google Scholar]
  20. Hekler E.B., Michie S., Pavel M., Rivera D.E., Collins L.M., Jimison H.B., Garnett C., Parral S., Spruijt-Metz D. Advancing models and theories for digital behavior change interventions. Am. J. Prev. Med. 2016;51(5):825–832. doi: 10.1016/j.amepre.2016.06.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Hollis C., Sampson S., Simons L., Davies E.B., Churchill R., Betton V., Butler D., Chapman K., Easton K., Gronlund T.A., Kabir T., Rawsthorne M., Rye E., Tomlin A. Identifying research priorities for digital technology in mental health care: results of the James Lind Alliance priority setting partnership. Lancet Psychiatry. 2018;5(10):845–854. doi: 10.1016/S2215-0366(18)30296-7. [DOI] [PubMed] [Google Scholar]
  22. Horvath, A. O., Re, A. C. del, Flückiger, C., & Symonds, D. (2011). Alliance in Individual Psychotherapy doi: 10.1037/a0022186. [DOI] [PubMed]
  23. Kocielnik R., Xiao L., Avrahami D., Gary H. Reflection companion: a conversational system for engaging users in reflection on physical activity. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2. 2018;2:1–26. [Google Scholar]
  24. Lambert M.J., Barley D.E. Research summary on the therapeutic relationship and psychotherapy outcome. Psychotherapy. 2001;38(4):357–361. doi: 10.1037/0033-3204.38.4.357. [DOI] [Google Scholar]
  25. Lopez, A., Schwenk, S., Schneck, C. D., Griffin, R. J., & Mishkind, M. C. (2019). Technology-Based Mental Health Treatment and the Impact on the Therapeutic Alliance. Current Psychiatry Reports 2019 21:8, 21(8), 1–7. doi: 10.1007/S11920-019-1055-7. [DOI] [PubMed]
  26. Martin D.J., Garske J.P., Davis M.K. Relation of the therapeutic alliance with outcome and other variables: a meta-analytic review. J. Consult. Clin. Psychol. 2000;68(3):438. [PubMed] [Google Scholar]
  27. Meyer B., Bierbrodt J., Schröder J., Berger T., Beevers C.G., Weiss M., Jacob G., Späth C., Andersson G., Lutz W., Hautzinger M., Löwe B., Rose M., Hohagen F., Caspar F., Greiner W., Moritz S., Klein J.P. Effects of an internet intervention (Deprexis) on severe depression symptoms: randomized controlled trial. Internet Interv. 2015;2(1):48–59. doi: 10.1016/j.invent.2014.12.003. [DOI] [Google Scholar]
  28. Mohr, D. C., Cuijpers, P., & Lehman, K. (2011). Supportive accountability: A model for providing human support to enhance adherence to eHealth interventions. Journal of Medical Internet Research (Vol. 13, Issue 1, p. e30). Journal of Medical Internet Research. doi: 10.2196/jmir.1602. [DOI] [PMC free article] [PubMed]
  29. Ormrod J.A., Kennedy L., Scott J., Cavanagh K. Computerised cognitive behavioural therapy in an adult mental health service: a pilot study of outcomes and alliance. Cogn. Behav. Ther. 2010;39(3):188–192. doi: 10.1080/16506071003675614. [DOI] [PubMed] [Google Scholar]
  30. Raue P.J., Goldfried M.R. Cognitive-behavior therapy. The working alliance: Theory, research, and practice. 1994;173:131. [Google Scholar]
  31. Riley W.T., Rivera D.E., Atienza A.A., Nilsen W., Allison S.M., Mermelstein R. Health behavior models in the age of mobile interventions: are our theories up to the task? Transl. Behav. Med. 2011;1(1):53–71. doi: 10.1007/s13142-011-0021-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Ryan R.M. Guilford Press; 2017. Self-Determination Theory: Basic Psychological Needs in Motivation, Development, and Wellness. [Google Scholar]
  33. Templeton G.F. A two-step approach for transforming continuous variables to Normal: implications and recommendations for IS research. Commun. Assoc. Inf. Syst. 2011;28(1):4. doi: 10.17705/1CAIS.02804. [DOI] [Google Scholar]
  34. Webb T.L., Joseph J., Yardley L., Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J. Med. Internet Res. 2010;12(1) doi: 10.2196/jmir.1376. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Wehmann, E., Köhnen, M., Härter, M., & Liebherz, S. (2020). Therapeutic Alliance in Technology-Based Interventions for the Treatment of Depression: Systematic Review. J Med Internet Res 2020;22(6):E17195Https://Www.Jmir.Org/2020/6/E17195, 22(6), e17195. doi: 10.2196/17195. [DOI] [PMC free article] [PubMed]

Articles from Internet Interventions are provided here courtesy of Elsevier

RESOURCES