Acceptance towards digital health interventions – Model validation and further development of the Unified Theory of Acceptance and Use of Technology

Abstract

Internet- and mobile-based interventions (IMI) offer an effective way to complement health care. Acceptance of IMI, a key facilitator of their implementation in routine care, is often low. Based on the Unified Theory of Acceptance and Use of Technology (UTAUT), this study validates and adapts the UTAUT to digital health care.

Following a systematic literature search, 10 UTAUT-grounded original studies (N = 1588) assessing patients' and health professionals' acceptance of IMI for different somatic and mental health conditions were included. All included studies assessed Performance Expectancy, Effort Expectancy, Social Influence, Facilitating Conditions and acceptance as well as age, gender, internet experience, and internet anxiety via self-report questionnaires. For the model validation primary data was obtained and analyzed using structural equation modeling.

The best fitting model (RMSEA = 0.035, SRMR = 0.029) replicated the basic structure of UTAUT's core predictors of acceptance. Performance Expectancy was the strongest predictor (γ = 0.68, p < .001). Internet anxiety was identified as an additional determinant of acceptance (γ = −0.07, p < .05) and moderated the effects of Social Influence (γ = 0.07, p < .05) and Effort Expectancy (γ = −0.05, p < .05). Age, gender and experience had no moderating effects.

Acceptance is a fundamental prerequisite for harnessing the full potential of IMI. The adapted UTAUT provides a powerful model identifying important factors – primarily Performance Expectancy - to increase the acceptance across patient populations and health professionals.

Highlights

• •Acceptance of digital health interventions, a key determinant of use, is low.
• •The Unified Theory of Acceptance and Use of Technology was successfully adapted to digital interventions.
• •Performance Expectancy is the most important determinant of acceptance.
• •Internet anxiety moderates the effects of Social Influence and Effort Expectancy.

1. Introduction

Technology has become more and more present in society and offers the opportunity to digitalize and thereby complement and improve health care, for example through internet- and mobile-based interventions (IMI) (Andersson, 2018; Ebert et al., 2018). IMI are characterized by their flexibility in terms of location and time of use and can thereby overcome structural barriers (Ebert et al., 2018). Furthermore, IMI offer a low-threshold option for treatment, which may reduce or even eliminate the fear of stigmatization, discrimination, or embarrassment (Andrade et al., 2014; Baumeister et al., 2017). The outbreak of the Covid-19 pandemic and the associated restrictions on social and professional life, as well as on individual mobility, have made the need for viable, scalable, and flexible alternatives and additions to traditional health treatments even more apparent.

The effectiveness of IMI is well documented for a broad range of somatic and mental health conditions (Ebert et al., 2018; Moshe et al., 2021b). However, patients' and health professionals' acceptance of IMI is a fundamental requirement before they can significantly impact health care (Cranen et al., 2012). Current evidence consistently shows that the acceptance towards IMI is low to moderate among patients (Baumeister et al., 2014, Baumeister et al., 2015; Ebert et al., 2015) as well as health professionals (Baumeister et al., 2020; Hennemann et al., 2017). This lack of acceptance might explain the overall low uptake and adherence rates (Karyotaki et al., 2017; Lillevoll et al., 2014; Lin et al., 2018; Mitchell and Gordon, 2007). Hence, it is of utmost importance to understand the factors that contribute to accepting IMI.

1.1. Theoretical background and determinants of acceptance

In the field of technology and its acceptance determinants, the Unified Theory of Acceptance and Use of Technology (UTAUT) (Venkatesh et al., 2003) is the most frequently used model providing a theoretical framework for potential factors influencing acceptance (Blut et al., 2021). UTAUT was developed based on conceptual similarities between eight previously existing user acceptance models: Theory of Reasoned Action (TRA; Fishbein and Ajzen, 1975), Technology Acceptance Model (Davis, 1989), Motivational Model (Davis et al., 1992), Theory of Planned Behavior (Ajzen, 1985), Combined TAM and TPB (Taylor and Todd, 1995), Model of PC Utilization (Thompson et al., 1991), Innovation Diffusion Theory (Moore and Benbasat, 1991) and Social Cognitive Theory (Bandura, 1986). Venkatesh et al. (2003) reviewed and analyzed those models, resulting in the identification of relevant constructs as determinants of technology acceptance and use behavior. The basic underlying concept of UTAUT is that the intention to use information technology is a direct predictor of the actual usage. Behavioral intention is conceptualized as the acceptance towards technology (Holden and Karsh, 2010). UTAUT postulates four core determinants of acceptance and use of technology: 1) The expected benefit the individual will receive from using the technology (Performance Expectancy), 2) the expected ease of using the technology (Effort Expectancy), 3) the perception that significant others believe that the technology should be used (Social Influence) and 4) the expected technical or organizational support while using the technology (Facilitating Conditions) as well as four moderators: age, gender, experience and voluntariness of use (Venkatesh et al., 2003). Although the UTAUT model has originally been developed and formulated in a workplace context (Venkatesh et al., 2003), it has been successfully applied to various other areas, such as internet banking (Sok Foon and Chan Yin Fah, 2011), digitalization of education (Chao, 2019; Marchewka et al., 2007) or online gaming (Chen et al., 2011; Dwivedi et al., 2020), as well as in the medical sector e.g., for the adoption of electronic medical records (Hennington and Janz, 2007; Wills et al., 2008), clinical decision support systems (Heselmans et al., 2012) or disease monitoring and managing applications (Rho et al., 2015; Seethamraju et al., 2018). Given this generalizability of the UTAUT, the determinants of acceptance in the UTAUT may also be promising predictors for acceptance towards IMI.

However, in digital health care use internet/technology anxiety is another frequently discussed factor inhibiting the uptake and acceptance (Celik and Yesilyurt, 2013; Jimison et al., 2008). Internet anxiety is defined as a fear or mistrust experienced while using the internet and is influenced by the user's personality as well as individual beliefs such as resources or facilitating conditions and social support (Thatcher et al., 2007). Hence, for extending the UTAUT, internet anxiety may be a relevant determinant for acceptance and use of IMI.

1.2. Study objectives

Despite frequent use, the UTAUT model has not been validated in the context of digital health interventions. The aim of the present study is to conduct a model validation and further development of UTAUT (e.g., investigating the influence of internet anxiety) in the context of IMI including patients with various conditions, somatic as well as mental health, and health professionals in order to better understand predictors of acceptance and to identify ways to facilitate acceptance towards IMI. The following research questions were investigated:

• 1.Do Effort Expectancy, Performance Expectancy, and Social Influence have an influence on acceptance as proposed by the UTAUT?
• 2.What are the direct and moderating effects of gender, age, and experience?
• 3.What is the effect of internet anxiety on the acceptance and its predictors?

2. Methods

2.1. Identification and selection of studies

This is a secondary analysis based on the primary data derived from multiple original studies assessing the acceptance towards IMI. To identify relevant studies a systematic literature search was conducted. The databases MEDLINE, PsycINFO and Embase were searched on November 12, 2020, using the terms “mobile”, “internet”, “online”, “smart*”, “web”, “blended”, “acceptance” and “intention”. After the removal of duplicates, titles and abstracts were screened by two independent researchers (PP, YT; agreement: 97%, κ_n = 0.96). Full-texts of the remaining studies were obtained and screened for eligibility (PP, YT; agreement: 100%, κ_n = 1) (Brennan and Prediger, 1981).

Eligible studies had to 1) measure acceptance with the UTAUT model and 2) focus on acceptance towards IMI in stand-alone or blended care settings. 3) Patients as well as health professionals (e.g., therapists) were eligible target groups. Studies focusing on acceptance towards a specific product (i.e. retrospective satisfaction with an IMI) and not on the general acceptance towards digital treatments were excluded. In addition, studies only reporting qualitative data were excluded.

The corresponding authors of the identified studies were contacted and asked to share their original data. In case of non-response studies were excluded. Subsequently, all provided data sets were homogenized and merged into a single data set. 14 eligible studies were identified of which 10 provided primary data (for more detailed results see Section 3.1).

2.2. Measures

2.2.1. Acceptance and primary predictors

All included studies assessed acceptance towards digital interventions based on UTAUT. In the included studies the individual adaptations of the UTAUT questionnaire were not always identical and the number of items per study could differ. Items were matched across included studies if the formulation was sufficiently similar. Similarity was determined by two independent researchers (PP, YT). Conflicts were resolved in discussion. The final items included in the analysis were the following: Acceptance was operationalized as Behavioral Intention (Venkatesh et al., 2003) with four items, Performance Expectancy and Effort Expectancy were included with three items each, Social Influence and Facilitating Conditions with two items each. All items were self-reported and rated on a 5-point scale ranging from “does not apply at all” (1) to “applies completely” (5), see Supplementary Table 1 for all items. Confirmatory factor analysis yielded an excellent model fit of the questionnaire (RMSEA = 0.04, SRMR = 0.03, Supplementary Table 2 and Supplementary Fig. 1) highlighting the construct validity of the used questions.

2.2.2. Moderators

The original UTAUT defines four moderators of the relationship between the predictors and acceptance: age, gender, experience, and voluntariness of use (Venkatesh et al., 2003). Experience was operationalized as the average usage of the internet and the frequency of reading and writing e-mails, self-reported, and rated on a 5-point scale ranging from “seldom or never” (1) to “multiple times per day” (5). Voluntariness of use was not assessed in any study and, therefore, not included as a moderator. Internet anxiety, an IMI-specific variable, was included as a potential moderator and direct predictor of acceptance. Three items of internet anxiety (e.g., “The Internet is something threatening to me”), also self-reported and rated on a 5-point scale, ranging from “does not apply at all” (1) to “applies completely” (5), were included in the analysis. For all items see Supplementary Table 1.

2.3. Statistical analysis

Structural equation modeling was applied. To evaluate the model fit and to compare different models, various fit indices were considered. The χ²-test tends to reject mis-specified models too sensitively (Browne and Cudeck, 1992; Moshagen, 2012; Moshagen and Erdfelder, 2016), therefore the root mean square error of approximation (RMSEA) as a non-centrality parameter and the standardized root mean square residual (SRMR) as a residual index were used. Cut-off values for an acceptable goodness of fit were based on standard modeling criteria: RMSEA <0.06, SRMR <0.08 (Hu and Bentler, 1999). Akaike information criterion (AIC) and Bayesian information criterion (BIC) were used for model comparisons. Nested models were compared via model deviance tests. For further investigation of the structure and potential sources of misfit, modification indices were evaluated (MacCallum et al., 1992). Parameters were estimated using the full information maximum likelihood estimator, which enables an accurate parameter estimation even in case of missing data, which e.g. arose in the matching process (Enders, 2010).

Model 1. To assess the proposed structure of UTAUT, Model 1 consisted of three latent factors (Performance Expectancy, Effort Expectancy, and Social Influence) as predictors of acceptance (Behavioral Intention). Consistent with the UTAUT, Facilitating Conditions were not included as a predictor of acceptance. Correlations between the main factors (i.e. Performance Expectancy, Effort Expectancy, Social Influence, and Facilitating Conditions) were allowed. Model 1 is presented in Fig. 1.

Fig. 1.

Model 1.

2.3.1. Moderator analysis

In order to evaluate whether effects of gender (categorical variable) were present, invariance between the two gender groups was tested using a modeling approach. At first, an unconstrained configural invariance model was estimated, then all loadings were constrained to equality between the two groups (metric invariance), then all item intercepts were constrained to equality (scalar invariance). Furthermore, invariance of the latent structure was investigated by constraining the factor variances, covariances and regressions to equality (Millsap, 2007). Stepwise model comparisons were conducted. In case of an equivalent model fit, the more restrictive model was chosen due to its parsimony.

All other investigated potential moderators (age, experience, internet anxiety) were continuous. Due to its superiority to alternative approaches like median split, interaction variables were calculated by multiplying each z-standardized moderator with each of the z-standardized main predictors (MacCallum et al., 2002). For evaluating the moderation effect, both the main effects of the moderators and the interaction variables were included as predictors of acceptance. The moderators were allowed to correlate with the original main factors (i.e. Performance Expectancy, Effort Expectancy, Social Influence, and Facilitating Conditions) and with each other. Non-significant parameters were excluded in a stepwise fashion and model comparisons conducted by comparing the model with freely estimated parameters with a model in which the parameters of the interaction variables were stepwise fixed to zero. In the case of equal model fit, the more restrictive model was chosen due to its parsimony.

2.3.2. Analysis software

The software R was used for all analyses (R Development Core Team, 2016). For confirmatory factor analysis and structural equation modeling, the R package “lavaan” (version 0.6-8) was used (Rosseel, 2009).

3. Results

3.1. Descriptive statistics

The systematic literature search yielded 14 relevant studies that assessed the participants' acceptance towards IMI based on UTAUT. All corresponding authors were contacted. In total, 10 of the 14 contacted research groups responded and agreed to share their primary data (Apolinário-Hagen et al., 2018; Baumeister et al., 2014, Baumeister et al., 2015, Baumeister et al., 2020; Ebert et al., 2015; Hennemann et al., 2016; Kott et al., 2020; Lin et al., 2018; Messner and Baumeister, 2020a, Messner and Baumeister, 2020b). The flow diagram can be found in Fig. 2.

Fig. 2.

Study flow diagram.

The ten included studies yielded an overall sample of N = 1588 individuals. Two studies focused on an IMI for depression (n = 294; 18%), one targeted diabetes and comorbid depression (n = 141; 9%), n = 219 (14%) participants were included from two trials focusing on chronic pain, another trial targeted well-being and health in elderly (n = 47; 3%), one assessed the acceptance towards an IMI for gastrointestinal problems (n = 152; 13%), one focused on aftercare for inpatients (n = 287; 18%), another one targeted multiple sclerosis (n = 175; 11%), and lastly one study assessing psychotherapists' acceptance towards blended therapy was included (n = 273; 17%). The majority of participants were female (57%). The mean age in the sample was M = 44.1 years (SD = 17.0), ranging from 18 to 93 years. The average acceptance was low to moderate (M = 2.82, SD = 1.12, scale range 1-5). An overview of the descriptive statistics of all included variables can be found in Table 1.

Table 1.

Descriptive statistics of predictors, moderators and acceptance.

	Female M (SD)	Male M (SD)	Overall M (SD)
Gender n (%)1	906 (57%)	585 (37%)
Age	41.1 (17.0)	48.9 (15.9)	44.1 (17.0)
Behavioral Intention	2.93 (1.11)	2.68 (1.12)	2.82 (1.12)
Performance Expectancy	2.89 (1.14)	2.69 (1.12)	2.80 (1.14)
Effort Expectancy	3.41 (1.10)	3.26 (1.18)	3.34 (1.14)
Social Influence	2.57 (1.11)	2.56 (1.09)	2.55 (1.10)
Facilitating Conditions	3.53 (1.18)	3.62 (1.18)	3.56 (1.18)
Experience	3.24 (1.40)	2.71 (1.40)	2.96 (1.42)
Internet Anxiety	2.01 (0.96)	1.77 (0.87)	1.92 (0.94)

Note. 1) n = 97 did not state their gender. % of gender refer to the total N.

3.2. Model validation

To assess the proposed structure of the original UTAUT model, structural equation modeling was applied. Model 1 consisted of three predictors of acceptance. All direct effects postulated by the UTAUT were significant. Notably, Performance Expectancy (γ = 0.67) was the strongest predictor explaining 45.4% variance of acceptance alone (R² = 0.45).

Interestingly, there was a strong correlation between the latent factors Facilitating Conditions and Effort Expectancy (r = 0.95). Therefore, a one factor solution for Facilitating Conditions and Effort Expectancy was tested but resulted in a worse fit (see Supplementary Table 3). Modification indices suggested a correlation between the residual variances of items FC01 and EE02, which significantly improved the model fit (see Table 2 for model comparison) and hence was introduced to the model (Model 1b). The final model (1b) including parameter estimates is presented in Fig. 3.

Table 2.

Model fit comparison of Model 1 and Model 1b.

	df	χ2	p	AIC	BIC	RMSEA	SRMR
Model 1	68	214		44,470	44,741	0.043	0.027
Model 1b	67	184	<0.001	44,433	44,709	0.039	0.023

Note. AIC: Akaike information criterion; BIC: Bayesian information criterion; RMSEA: root mean square error of approximation; SRMR: root mean square residual.

Fig. 3.

Estimates of Model 1b.

Note. Estimates are standardized. Error terms are omitted for means of readability. All paths are significant.

3.3. Moderator analysis

Based on Model 1b, the proposed moderators were evaluated. For analyzing potential effects of gender, Model 1b was divided into two groups (male and female), allowing for group separate parameter estimation. Invariance was established by stepwise constraining parameters to equality across the two groups and comparing with the previous, less restrictive, model. Results indicated that loadings, variances, covariances, and regressions were equal in men and women. Only the intercepts of items EE03, BI02, and BI04 differed. Hence, the structural and regressive relationships of the model were independent of gender (Millsap, 2007).

Next, a model including all remaining potential moderators (i.e. age, experience, and internet anxiety) as main and interaction effects was estimated. Non-significant parameters were excluded in a stepwise fashion and model comparisons conducted. There were no significant effects of experience or age (neither main nor interaction effects). The final best fitting model (RMSEA = 0.035, SRMR = 0.029) included a main effect of internet anxiety as well as interaction effects of internet anxiety with Social Influence and Effort Expectancy next to the main effects of the original UTAUT predictors Performance Expectancy, Effort Expectancy, and Social Influence. Internet anxiety had a significant negative effect on acceptance (γ = −0.07), moderated the relationship between Social Influence and acceptance (γ = 0.07) and moderated the relationship between Effort Expectancy and acceptance (γ = −0.05). The model including these directed paths of internet anxiety was significantly better than the model without directed paths. The final adapted model including the found moderation effects can be found in Fig. 4. The regression paths of the final model were also invariant to indication areas (affective conditions vs other: ∆RMSEA = 0.001; somatic conditions vs other: ∆RMSEA = −0.001).

Fig. 4.

Estimates of the adapted Model.

Note. Estimates are standardized. Error terms are omitted for means of readability. All paths are significant.

4. Discussion

To our knowledge, the present study is the first to validate UTAUT in the context of IMI with a large and broad sample covering multiple somatic and mental health conditions with patients as well as health professionals. First of all, the excellent model fit showed that the basic structure of UTAUT with the three main predictors of acceptance, Performance Expectancy, Effort Expectancy, and Social Influence, also holds true in the context of IMI (Venkatesh et al., 2003).

However, moderator analysis revealed that none of the moderating variables of the original UTAUT had a moderating effect on acceptance of IMI. Instead, internet anxiety was identified as a relevant moderator and also a meaningful direct predictor. Hence, we propose an adapted UTAUT model that attributes acceptance to the original main factors Performance Expectancy, Effort Expectancy, and Social Influence, and also formulates a direct effect of internet anxiety and moderating effects of internet anxiety for Social Influence and Effort Expectancy on acceptance.

The fact that internet anxiety moderates the relationship between Social Influence and acceptance indicates that social support is especially important for more anxious people. This is in accordance with the conception of internet anxiety as it includes person-related aspects as well as social support aspects (Thatcher et al., 2007). Evidence suggests that providing adequate support, promoting trust in the technology, and assuring that important social persons encourage using the technology are important for decreasing internet anxiety (Thatcher et al., 2007). In the context of IMI, this might be achieved by providing enough information about the procedure and the data security of the IMI and by offering technical help to ensure that the users always feel supported in case of problems or questions.

Besides the importance of these findings on moderators, the most striking finding in the UTAUT model for IMI was the strong influence of Performance Expectancy on acceptance (γ = 0.68). Performance Expectancy alone explained about 46% of the variance in acceptance. This effect was also independent of the potential moderators (gender, age, internet experience, and internet anxiety). Hence, the performance expectancy of using an IMI should be the core target when it comes to increasing the acceptance (e.g., in patient healthcare professional interaction) and thus, potential use of digital health treatments. This finding is in line with the literature on health behavior change models such as the Health Action Process Approach, which describes Outcome Expectancy as an important predisposing factor for the formation of health-related intentions (Schwarzer, 2008; Schwarzer et al., 2011). Besides, the importance of Performance Expectancy becomes evident not only for increasing the acceptance and use, but also for increasing the effectiveness of an intervention: Positive expectancy appears to be a primary mechanism for effecting change in treatments for mental disorders (Rutherford et al., 2010) and for predicting surgery outcomes (Auer et al., 2015; Rief et al., 2017). Hence, it is likely that Performance Expectancy is also an important determinant of the effectiveness of IMI, which further increases the need for Performance Expectancy management (e.g., interventions to promote Performance Expectancy).

One promising way to improve Performance Expectancy and other UTAUT factors could be through acceptance (e.g., Baumeister et al., 2014) or engagement (e.g., Batterham et al., 2019) facilitating interventions. To date, there are several studies on acceptance facilitating interventions in the context of IMI targeting Performance Expectancy as well as other factors of the UTAUT model (Baumeister et al., 2014, Baumeister et al., 2015, Baumeister et al., 2020; Ebert et al., 2015; Lin et al., 2018). However, both the format of the deployed interventions (e.g., short-videos, case-examples, presentations) as well as their effects are heterogeneous ranging from non-significant effects (Baumeister et al., 2014) to large effect sizes of d = 0.81 (Baumeister et al., 2015). Hence, future studies are highly needed to identify in which way acceptance can be increased in the most effective way.

4.1. Limitations and implications for future research

While this work contributes to understanding the most important determinants of acceptance of IMI, it also has some limitations that should be considered. First, it needs to be highlighted that the present study focused solely on acceptance and fell short in the question of how acceptance of technology (i.e., behavioral intention) and actual use behavior are connected. The UTAUT formulates the behavioral intention as a direct predictor of the actual use behavior (Venkatesh et al., 2003). However, people do not always act upon their intentions, a phenomenon often described as the “intention-behavior gap” (Orbell and Sheeran, 1998). This phenomenon could also be present in the context of digital health. Therefore, there is a need for future research to address the lack of evidence supporting the relationship between the intention to use an IMI and the actual uptake and adherence (Baumel et al., 2019; Lin et al., 2018).

Second, the broad range of patients with different somatic and mental conditions and therapists in the present sample represents a unique contribution to the field, as conclusions can be drawn about a broad population. However, health professionals are underrepresented in the overall sample and future studies are needed to shed more light on this specific population. For this reason, results related to health professionals should be considered with caution.

Lastly, the present study focused exclusively on IMI with a treatment focus and not on applications with other objectives, such as reporting systems, monitoring apps or digital diagnostics (e.g., smart sensing) (Baumeister and Montag, 2019; Messner et al., 2019; Moshe et al., 2021a; Opoku Asare et al., 2021). In those applications other models or adaptations of the here proposed model might be needed (Salgado et al., 2020). In addition, the factors compatibility, education, personal innovativeness and costs were recently identified as a relevant extension of the UTAUT model in a different context (i.e., workplace) in a meta-analysis of k = 1935 independent samples (Blut et al., 2021). Whether these extensions found in other contexts also apply to the field of IMI is currently unclear and should be explored in future studies. However, for the here focused context of digital treatment (i.e. IMI) the proposed model provides a powerful framework of factors influencing acceptance explaining 81.5% of the variance in acceptance.

4.2. Conclusion

The UTAUT was successfully adapted to the context of acceptance towards IMI. The general structure with the three core predictors of acceptance, Performance Expectancy, Effort Expectancy, and Social Influence was confirmed, with the addition of internet anxiety as a direct determinant as well as a moderator of the effects of Social Influence and Effort Expectancy on acceptance. The results showed that the questionnaire and model used are appropriate for assessing the acceptance of IMI and invariantly across genders and indication areas. With regard to practical implications, the strong direct effect of Performance Expectancy on acceptance is of particular importance, specifically for the design of acceptance facilitating interventions and should, for instance, be focused in the patient healthcare professional interaction to foster the actual uptake and usage of IMI. Furthermore, providing adequate support and information as well as promoting trust in technology could contribute to reducing internet anxiety. However, future research is needed to gain more insight into how strongly acceptance is connected to the actual usage of digital treatments (e.g., the uptake and adherence of IMI) and whether some of the recently applied extensions to UTAUT (e.g., compatibility, personal innovativeness, costs, education) from other contexts transfer to IMI.

Funding details

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Availability of data and materials

Agreements for data sharing with the corresponding authors of the included projects covered only the access for the present project. Thus, the data set will not be made public. Data can be requested by the corresponding authors of the respective projects. Providing of data may be subject to data sharing agreements and support of the corresponding authors depends on available resources. The R-script of the present analysis can be requested by YT.

CRediT authorship contribution statement

YT, HB, and PP designed the study. YT and PP conducted the systematic literature search and study selection process. HB, JAH, DDE, SH, JL, LK und EMM provided the primary data of the original studies. PP conducted the data analysis and drafted the manuscript. YT provided methodological and statistical support and supervised the data analysis. All authors have read, revised and approved the final manuscript.

Declaration of competing interest

HB and EMM received consultancy fees, reimbursement of congress attendance and travel costs as well as payments for lectures from Psychotherapy and Psychiatry Associations as well as Psychotherapy Training Institutes in the context of E-Mental-Health topics. SH received payments from psychotherapy training institutes in the context of E-Mental-Health topics. DDE possess shares in the GET.On Institut GmbH, which works to transfer research findings on internet- and mobile-phone-based health interventions into routine care. DDE has received payments from several companies and health insurance providers for advice on the use of Internet-based interventions. He has received payments for lectures from Psychotherapy and Psychiatry Associations and has been the beneficiary of third-party funding from health insurance providers. All other authors declare no conflicts of interest.

Appendix A. Supplementary data

Supplementary material