Skip to main content
PMC home page
Internet Interventions logoLink to Internet Interventions
. 2017 Jun 1;9:38–45. doi: 10.1016/j.invent.2017.05.004

Adherence to a web-based pre-treatment for phobias in outpatient clinics

Robin N Kok a,b,c,⁎, Aartjan TF Beekman d, Pim Cuijpers a, Annemieke van Straten a
PMCID: PMC6096310  PMID: 30135835

Abstract

Background

Non-adherence in Internet interventions is a persistent and multifaceted issue and potentially limits the applicability and effectiveness of these interventions. Factors that influence non-adherence are poorly understood, especially in outpatient samples with more complex symptoms.

Objective

The current study is a secondary analysis of data from a randomised controlled trial that examined the cost-effectiveness of offering an Internet-based exposure treatment to phobic outpatients on a wait-list to receive face-to-face psychotherapy.

Methods

We collected baseline demographic and clinical information, and adherence data of the Internet-based intervention and conducted regression analyses to predict non-adherence to the intervention.

Results

The adherence to the intervention was low, with only 13.3% of 105 patients completing all five lessons of the intervention. The median number of exercises completed (out of a possible 8) was 3. In a multi-predictor model, a higher baseline score of anxiety (OR = 0.94, 95% CI 0.90–0.99) was a risk factor for low adherence. Higher age (OR = 1.05, 95% CI 1.00–1.09) was a protective factor against non-adherence. Participants who adhered to the intervention were more likely to complete post-test assessments, further biasing results. However, overall participant attrition was high. The results are based on a small subset of participants and should be interpreted with caution.

Conclusions

Poor baseline clinical status and age are factors to consider when deciding whether to offer an Internet-based intervention to outpatients. Low adherence among those patients might be related to intrinsic motivation and might even be lower in outpatient settings where participants expect to receive face-to-face treatment. It might be worthwhile to develop a concise instrument to assess intrinsic motivation and treatment expectations for using Internet-based interventions, and for the therapist to review the range of possible (Internet-based) intervention options to suit personal preferences and expectations.

Trial registration

Netherlands Trial Register, NTR2233.

Keywords: Internet Intervention, Non-adherence, Dropout, Adherence, Predictors

Highlights

  • •

    Non-adherence to Internet-based interventions is still a poorly understood topic.

  • •

    In this outpatient clinic trial, initial uptake was good but few participants finished all five weeks of the intervention.

  • •

    Worse baseline clinical status was associated with higher non-adherence in this outpatient sample.

  • •

    Adherent patients were also more likely to complete post-treatment assessments.

  • •

    Assessment of intrinsic motivation, treatment expectations and patient preferences are suggestions to reduce non-adherence.

1. Introduction

The use of the Internet to deliver mental health interventions has been increasing almost exponentially in the past decade. The flexibility and easy implementation of Internet-based mental health interventions makes it possible to deliver therapies to a large number of people in their own environment, at a convenient moment. Furthermore, these interventions have the potential to reach populations hitherto unreachable, due factors such as distance to healthcare providers or stigma. Internet interventions in the form of Internet-based cognitive behavioural therapy (iCBT) have been shown to be as effective as face-to-face (FtF) psychotherapy for a number of common mental disorders (Andersson et al., 2014). Additionally, preliminary evidence suggests that these interventions may be cost-effective when compared to alternative treatments or wait-list control groups (Donker et al., 2015).

One of the main challenges of offering Internet-based interventions is premature discontinuation of the intervention (Christensen and Mackinnon, 2006, Eysenbach, 2005). Participants who discontinue prematurely are non-adherent to the intervention, as they do not experience the content of the intervention as it was intended. Non-adherence estimates in Internet interventions range from 2% to 83%, with an average of approximately 31% (Melville et al., 2010). It has been found that adherence to an intervention tends to be higher when the intervention has some of guidance by a professional (Christensen et al., 2009, Richards and Richardson, 2012). The non-adherence to guided Internet-based interventions is comparable to discontinuation of FtF psychotherapy (van Ballegooijen et al., 2014), as a recent large meta-analysis found that approximately 26.2% of patients discontinued FtF psychotherapy prematurely (Fernandez et al., 2015). Although the non-adherence to guided Internet-based interventions seems comparable to FtF treatments, the effects of these psychological interventions might be increased when the problem of non-adherence can be reduced. This makes it important to examine factors that influence non-adherence to psychological treatments.

Previously found factors associated with treatment non-adherence in FtF psychotherapy were a lower educational level (Keijsers et al., 2001), higher baseline depression scores, or being in specialised (outpatient) rather than in primary care, although the latter may be indicative of worse clinical status. Research on the influence of gender on adherence has produced equivocal results, with some indicating higher adherence in men (Issakidis and Andrews, 2004), some in women (Herbert et al., 2005, McEvoy, 2007), and others reporting no differences (Erwin et al., 2002, Hofmann and Suvak, 2006, Santana and Fontenelle, 2011).

With regard to Internet interventions, several categories of adherence predictors can be identified. Broadly, these are divided into socio-demographic variables, psychological problems (e.g., duration and severity of problems, comorbid anxiety and depression) and treatment-related variables such as e.g., treatment credibility, expectations and presentation and motivation to participate (Melville et al., 2010). In Internet-based interventions, sociodemographic variables that were associated with non-adherence in unguided Internet-based therapy included, as in FtF treatment, a low educational level. Furthermore, male gender, and younger age were also associated with non-adherence (Karyotaki et al., 2015). Psychological problems associated with lower online adherence are also similar to FtF psychotherapy, including higher pre-treatment symptoms and comorbidity (Christensen et al., 2009). Treatment-related variables include expectations of whether the intervention will work (Boettcher et al., 2013), feeling accountability to the therapist (Mohr et al., 2011) or a sense of duty to researchers carrying out the intervention and being able to identify with the programme (Donkin and Glozier, 2012). Additionally, persuasive design features – which explicitly motivate a user to keep using the intervention – also play a part. For example, more frequent intended usage, more frequent updates and more extensive use of dialogue support (e.g., an online coach), significantly predicted better adherence (Kelders et al., 2012).

However, one shortcoming of previous research on predictors of adherence is that it was mostly based on either non-clinical samples, self-selected samples who were offered no other treatment other than an Internet-based treatment (i.e., studies with wait-list control conditions), or samples from unguided Internet-based interventions (e.g., Karyotaki et al., 2015). Currently, we are not aware of any research on predictive variables of adherence to an Internet-based intervention in patients that are actively awaiting outpatient psychological treatment. The upcoming face-to-face psychological treatment may influence the attitudes of patients towards Internet-based treatment, or may influence attitudes or behaviours. For example, outpatients might have different clinical profiles (e.g., worse symptom severity) which influence adherence. Additionally, knowing that one is guaranteed to receive face-to-face psychotherapy could lower the threshold to discontinue the Internet-based intervention. Information on predictors of adherence can help therapists and healthcare workers to identify different patient profiles, e.g. those likely to complete an Internet-based treatment relatively independently, patients for whom an Internet-based intervention should be more closely monitored e.g. in form of extra guidance, and perhaps patients best directly referred to FtF treatment.

In short, treatment dropout and its causes remain a research priority. To add to currently available information, the current study uses data from a previously conducted RCT (Kok et al., 2012, Kok et al., 2014), which investigated the potential cost-effectiveness of offering an Internet-based guided pre-treatment to outpatients during wait-list for FtF psychotherapy. In short, the rationale was that offering an Internet-based pre-treatment might reduce the number of FtF sessions by relegating routine tasks to guided self-help.

In the current study, we investigate predictors of adherence in this clinical sample of phobic outpatients. We will examine prognostic variables related to intervention adherence in an outpatient sample of phobic patients. As Internet-based interventions increasingly find their way into clinical practice, it is necessary to identify predictive variables for adherence to these interventions in clinical samples.

2. Method

2.1. Design

This study is a secondary analysis from a randomised controlled trial examining the cost-effectiveness of guided Internet-based treatment for phobic patients waiting for face-to-face treatment (Kok et al., 2012, Kok et al., 2014). In this trial, 212 outpatients from eight Dutch outpatient clinics were randomised to either 1) a 5-week wait-list control group (n = 107) or 2) a 5-week Internet-based guided intervention (N = 105) based on exposure homework exercises. After wait-list control group or the Internet-based intervention, all participants were scheduled to receive face-to-face-psychotherapy; all but 4 participants (2 in either condition) did continue to face-to-face psychotherapy.

For this secondary analysis on intervention adherence, data from the 105 participants receiving the Internet-based intervention were used.

The study from which these data originated has been approved by the Medical Ethical Committee of the VU University Medical Centre (registration number 2010/77).

2.2. Participants and recruitment

Phobic patients were referred to the research team by outpatient clinic staff. If eligible for inclusion, the mood and anxiety sections of a diagnostic interview (CIDI; World Health Organization, 1990) were administered by telephone, informed consent was sought and baseline measurements were recorded. We included computer-literate, Dutch-speaking patients, 18 years or older, with a DSM-IV-TR diagnosis of any phobia (social phobia, agoraphobia or specific phobia). Stable psychotropic medication use was allowed. Patients with psychotic disorders or at elevated risk for suicide were excluded and received care as usual at the outpatient clinic.

2.3. Intervention

Intervention patients were offered a five-week Internet-based intervention, based on exposure therapy. This intervention comprised 8 exercises, which were to be completed in 5 weeks; the average time spent waiting for face-to-face psychotherapy. In the first week, patients were offered psychoeducational information on phobias and reviewed the treatment rationale for exposure therapy. Additionally, patients had to assess motivation for treatment and were asked to affirm they were ready to start confronting their fear by self-exposure. In the second week, patients categorised their fears and avoidance behaviours, constructed a fear hierarchy (ranking fear-inducing situation from ‘no fear’ to ‘complete terror’) and started planning to complete this fear hierarchy in the coming weeks by planning gradual exposure homework exercises. During week 3, 4 and 5 patients were asked to complete their planned exposure exercises. This means that patients needed to expose themselves to situations or objects that elicited fear. They had made a fear hierarchy and were supposed to expose themselves to fearful situations in a gradual way starting with the least fearful situation and increasing levels of intensity each week. After each lesson, patients reported back to their coaches. These trained coaches were Master's level clinical psychology students who were instructed to provide procedural (i.e., not therapeutic) guidance (approx. 15 min per week), which served to guide patients through the self-exposure exercises, to explain things more clearly if necessary and to motivate them to carry on. After these 5 weeks, all patients were to receive FtF psychotherapy. An overview of exercises per week can be found in Table 1. The intervention is described in more detail elsewhere (Kok et al., 2012, Kok et al., 2014). The intervention was tunnelled, i.e., participants had to complete a week's exercise before they could progress to the next week. However, they had unlimited access to previous weeks.

Table 1.

Intervention planning and adherence percentages.

Week Exercise Content of the exercise Percentage adherence
1 Exercise 1 Select fear inducing situations 20
Exercise 2 Assess motivation for treatment
2 Exercise 3 Assess avoidance behaviour 20
Exercise 4 Construct fear hierarchy
Exercise 5 Construct exercise plan
3 Exercise 6 Complete exposure exercises 20
4 Exercise 7 Complete exposure exercises 20
5 Exercise 8 Complete exposure exercises, relapse prevention, prepare FtF therapy 20
Total 8 exercises Intended use 100
Open in a new tab

2.4. Definition of adherence

The Internet intervention consisted of five lessons. Therefore, the completion of one week, including all exercises for that particular week, contributed 20% to the total adherence. During the first two weeks, patients were asked to perform more than one exercise. The 20% of adherence for those weeks is broken down by the number of exercises (week 1: 10% for completing each exercise; week 2: 6.7% for completing each exercise). Patients were considered to be adherent to the intervention when they had completed 50% of more of the intervention. The rationale for this cut-off point is that at 50%, participant had crossed from planning exposure exercises to actually performing in-vivo exposure, which we had anticipated to be the largest obstacle for participants. The remainder of the intervention after this is repetition of exposure exercises that do not necessarily need guidance.

2.5. Assessments

In the primary study from which data for the current analyses were used, anxiety, depression and quality of life were assessed using Internet-based questionnaires at pre-randomisation and at post-intervention (5 weeks). A short description of these instruments is provided below. Apart from these instruments, we recorded demographic variables (age, sex, use of psychotropic medication such as anxiolytics or antidepressants; also whether the participant was highly educated, this being defined as having finished a degree comparable to at least bachelor's level). Before randomisation, the prospective participants were screened using the CIDI (World Health Organization, 1990), but using only the sections screening for anxiety and mood disorders to minimise participant burden. As such, other comorbidities were not assessed as this would have results in a large burden for the participant in terms of assessment time.

2.5.1. Primary Outcome Measure

The Fear Questionnaire (FQ) (Marks and Mathews, 1979) was the primary outcome measure, which measures phobic fear severity and avoidance. Its psychometric validity has been established for the Dutch version (Van Zuuren, 1988).

2.5.2. Secondary Outcome Measures

Secondary outcome measures were the Beck Anxiety Inventory (BAI) (Beck and Steer, 1993), a 21-item self-report questionnaire measuring physiological anxiety symptoms and the Center for Epidemiological Studies-Depression Scale (CES-D) (Radloff, 1977). The BAI has been validated for patients with different anxiety disorders (de Beurs et al., 1997, Leyfer et al., 2006). Similarly, the Dutch version of the CES-D shows good psychometric validity (Donker et al., 2010). Internal consistency in this sample was excellent for the BAI (Cronbach's α = 0.92) and the CES-D (Cronbach's α = 0.92). Quality of Life was assessed using the EQ-5D (EQ-5D), which has previously been assessed for validity (Brooks, 1996, van Agt et al., 1994). These scores result in utility scores which range from death (0) to perfect health (1).

2.6. Analyses

Logistic regression models using adherence as a dependent variable were constructed using independent baseline demographic and clinical predictor variables. First, based on previous findings into the factors influencing non-adherence, predictor variables age and baseline measurements of the FQ, BAI, CES-D and EQ-5D were entered separately as univariate predictors with adherence status (adherent/not adherent) as a dependent variable. Secondly, variables significantly associated (p < 0.20) in the univariate analyses were entered in a backward conditional logistic multi-predictor regression model with adherence status as dependent variable. Reported confidence intervals (CI) are 95% and analyses were performed with SPSS 21 for Windows (SPSS, Chicago IL, USA).

2.6.1. Missing data

Out of 105 participants, adherence status for 4 participants (3.8%) could not be retrieved. Therefore, all analyses using adherence as a dependent variable were performed on the remaining 101 participants. Furthermore, 46 out of 101 participants (45.5%) did not complete the post-test assessment, non-completion of which was used to calculate the association between adherence and non-completion of post-test questionnaires.

3. Results

3.1. Adherence

Of 101 participants from the intervention group available for analyses; 20 participants (19.8%) never started using the intervention (pre-treatment dropout) and were coded as being 0% adherent; 50 participants (49.5%) used 1–50% of the intervention and 31 (30.7%) used at least 50% of the intervention and were considered adherent.

Table 2 reports demographic and clinical variables for the full sample and for the three usage groups. On average, participants were 36 years old; 57.4% were female and 57.4% were highly educated. Alongside one or more phobias, almost half of participants (44.6%) were diagnosed as suffering from a DSM-IV diagnosis of major depressive disorder (MDD) at baseline, and 78.1% of all participants was diagnosed with 2 or more disorders. Almost half of the sample (48.6%) was suffering from 3 or more disorders. The descriptive statistics suggest that those of older age, less severe baseline clinical scores and higher quality of life were more likely to be in a more adherent group. Moreover, the 20 participants that did not start using the intervention appear to have the least favourable baseline scores of all three groups.

Table 2.

Descriptive statistics of baseline demographic and clinical scores in relation to adherence.

% of adherence1
All (N = 101) Did not start (n = 20) 1 to 50% (N = 50) ≥ 50% (N = 31)
Demographic variables
Age 35.9 (11.9) 32.4 (9.1) 34.1 (11.5) 40.3 (11.9)
Female sex 58 (57.4%) 10 (50.0%) 27 (54.0%) 19 (61.3%)
Higher education 58 (57.4%) 11 (55.0%) 31 (62.0%) 20 (64.5%)



Baseline diagnoses
Dysthymia present 8 (7.5%) 3 (15.0%) 5 (10.0%) 0 (0%)
MDD present 45 (44.6%) 3 (15.0%) 24 (48.0%) 9 (29.0%)
SAD present 52 (49.5%) 5 (25.0%) 30 (60.0%) 15 (48.4%)
GAD present 36 (34.3%) 7 (35.0%) 17 (34.0%) 10 (32.3%)
Panic disorder present 64 (61.0%) 13 (65.0%) 34 (68.0%) 14 (45.2%)
Agoraphobia present 19 (18.1%) 5 (25.0%) 8 (16.0%) 6 (19.4%)
Specific phobia present 61 (58.1%) 12 (60.0%) 28 (56.0%) 18 (58.1%)
2 or more specific phobias 24 (22.9%) 4 (20.0%) 12 (24.0%) 7 (22.6%)
2 or more diagnoses 82 (78.1%) 16 (80.0%) 41 (82.0%) 21 (67.7%)
3 or more diagnoses 51 (48.6%) 8 (40.0%) 29 (58.0%) 12 (38.7%)
Comorbid anxiety/depression 46 (43.5%) 10 (50.0%) 25 (50.0%) 9 (29.0%)
Using psychotropic medication 14 (13.3%) 0 (0%) 5 (7.1%) 8 (25.8%)



Baseline clinical scores
FQ score (M/SD) 42.0 (23.4) 50.4 (21.9) 45.8 (23.7) 32.8 (21.7)
BAI score (M/SD) 45.0 (13.8) 50.8 (15.2) 48.2 (13.4) 36.9 (11.0)
CES-D score (M/SD) 25.0 (8.6) 27.7 (9.5) 27.1 (8.7) 20.2 (6.3)
EQ-5D score (M/SD) 0.60 (0.28) 0.51 (0.27) 0.53 (0.28) 0.74 (0.20)
Open in a new tab
1

Adherence is defined as % of intended usage.

Table 3 shows the flow of participants through the interventions. As can be seen, each week shows a marked drop in participants. The relative largest drops in adherence can be seen in the weeks in which actual in-vivo exposure was to be exercised; weeks 3–5.

Table 3.

Frequencies and percentages of participants who completed the exercises.

Week Exercise Content of the exercise Completed by (N/%) Relative attrition (N/%)
0 – Start of intervention 101 (100%) –
1 Exercise 1 Select fear inducing situations 79 (78.2%) 22 (21.8%)
Exercise 2 Assess motivation for treatment 77 (76.2%) 2 (2.5%)
2 Exercise 3 Assess avoidance behaviour 59 (58.4%) 18 (23.4%)
Exercise 4 Construct fear hierarchy 49 (48.57%) 10 (16.9%)
Exercise 5 Construct exercise plan 46 (45.5%) 3 (6.1%)
3 Exercise 6 Complete exposure exercises 33 (32.7%) 13 (28.3%)
4 Exercise 7 Complete exposure exercises 18 (17.8%) 15 (45.5%)
5 Exercise 8 Complete exposure exercises, relapse prevention, prepare FtF therapy 14 (13.9%) 4 (22.2%)
Open in a new tab

3.2. Association of baseline demographic, clinical variables and adherence

Univariate analyses show that younger age and not using psychotropic medication were significantly associated with higher adherence. Furthermore, higher baseline severity of phobic fears, higher scores of general anxiety, depression, and quality of life were all significantly associated with non-adherence in univariate analyses. Thus, patients with more severe symptoms at baseline, and patients with lower quality of life were less likely to adhere to the intervention (Table 4).

Table 4.

Univariate association between baseline demographic and clinical scores and intervention adherence (logistic regression analysis).

At least 50% adherent (N = 31)
OR (95% CI) p
Higher age 1.05 (1.01–1.08) 0.02
Female sex 0.75 (0.32–1.78) 0.51
Higher education 0.57 (0.24–1.36) 0.20
Using psychotropic medication 4.52 (1.34–15.23) 0.02
Baseline clinical scores
FQ 0.97 (0.95–1.00) 0.01
BAI 0.93 (0.89–0.97) 0.00
CES-D 0.89 (0.83–0.95) 0.00
EQ-5D 30.34 (3.73–245.42) 0.00
Open in a new tab

Next, we entered the variables of interest into a backward multi-predictor analysis (Table 5). The results show that higher baseline anxiety, lower quality of life and psychotropic medication use are significant predictors of a worse adherence status. Most predictors that were significant as univariate predictors were no longer significant in this model; age (p = 0.156), sex (p = 0.157), having a higher education (p = 0.438), baseline FQ score (p = 0.213) and baseline depression score (p = 0.288).

Table 5.

Multi-predictor association between baseline demographic and clinical scores and dichotomous adherence status.

At least 50% adherent (N = 31)
OR (95% CI) p
Age 1.05 (1.004–1.093) 0.03
Baseline BAI score 0.944 (0.901–0.990) 0.02
Open in a new tab

Note: R2Nagelkerke = 0.333. Model X2 = 26.96 (p = 0.000).

3.3. Association between adherence and post-test assessment completion

Finally, we explored if adherence was associated with completing post-test assessments (dropout attrition). Of those who were non-adherent to the intervention (< 50% adherent, N = 70), only 27 (38.5%) completed the post-test assessment, of those who were adherent to the intervention (N = 31), 26 (86.7%) completed the post-test assessment. Analyses show that in this sample, when adherence was defined at completing ≥ 50% of the intervention, adherence to the intervention was indeed associated with a higher probability of completing the post-test assessment (χ2 = 17.68, p = 0.000).

4. Discussion

4.1. General discussion

Identifying factors for non-adherence to Internet-based interventions in routine clinical practice is a key issue to improve acceptability of those interventions to therapists, researchers and patients. Moreover, improving adherence may serve to improve clinical effectiveness, dissemination and adoption and cost-effectiveness of Internet-based interventions. In the current study, we attempted to identify prognostic variables of adherence to a guided Internet-based intervention for outpatient phobic patients.

The clinical status was unfavourable for the sample, with almost 50% of the sample suffering from three or more comorbid diagnoses at baseline. The average adherence to the intervention was low, with only 13.3% of 101 patients completing all 5 weeks of the intervention. The median number of exercises completed (out of a possible 8) was 3. In a multi-predictor model, adherence was associated with baseline characteristics of more severe anxiety symptoms at baseline and higher age. Contrary to findings on unguided Internet-based interventions (Karyotaki et al., 2015), demographic variables such as sex and educational level were not significantly associated with adherence in this sample. Additionally, participants who did not adhere to the intervention were less likely to complete the post-test assessment.

The adherence to guided Internet interventions among patients recruited from the general population varies between 50% and 80% (Christensen et al., 2009, Richards and Richardson, 2012). However, research data on adherence to Internet-based psychological interventions from patients actively seeking psychological treatment in outpatient care is scarce. The current study, being one of the first in this outpatient setting, shows low adherence rates. This is corroborated by two recent studies in comparable outpatient settings (Kolovos et al., 2016, Mathiasen et al., 2016). The results suggest that patients with better clinical status at baseline are more likely to keep using the intervention, which is in line with previous research in non-clinical populations and unguided Internet interventions (Karyotaki et al., 2015).

Apart from severity of the primary disorder, it has been shown that comorbidity has profound implications for the severity, chronicity and impairment of patients (Klein Hofmeijer-Sevink et al., 2012) and is associated with a worse clinical prognosis (Penninx et al., 2011). Depression is a common comorbidity for patients with a current anxiety disorder (Lamers et al., 2011) and depressed patients are more likely to be non-adherent to both psychological (Ledley et al., 2005) and pharmacological treatments (DiMatteo et al., 2000). In Internet interventions, it is also associated with lower adherence and worse treatment outcomes (Boettcher et al., 2013, Ebert et al., 2013, Karyotaki et al., 2015). In the current sample, patients suffered from either anxiety-anxiety comorbidity, anxiety-depression comorbidity, or both (Kok et al., 2014), with almost 50% of patients suffering from three or more diagnoses at baseline. Additionally, most patients had baseline CES-D depression scores well over the commonly used cut-off scores of 16 and 20 (Lewinsohn et al., 1997). The high co-morbidity rates with other anxiety disorders as well as with depression may be an explanation of the relatively low adherence rates in this trial. This association has also been demonstrated in other trials (Johnston et al., 2013, Kolovos et al., 2016, Mathiasen et al., 2016).

With worse clinical status, one would perhaps expect less clinical improvement. However, a recent individual patient data meta-analysis demonstrated that both patients with severe and mild symptoms benefitted equally from guided Internet based interventions (Bower et al., 2013). As this meta-analysis was performed on depressed patients, the results may not be fully generalizable to our sample; although depression scores were also relatively high in the current trial (Kok et al., 2014) and comorbid depression was present in almost half of all patients. However, in the current sample too few patients were non-comorbid to perform meaningful analyses on subgroups. Despite this, based on the literature, comorbid depression may be an important factor to consider when offering Internet-based interventions to anxious outpatients, especially since comorbidity can be high in these patients.

Previously, no association between adherence to the intervention and improvement in terms of clinical scores was found in an unguided web-based intervention for depression (Donkin et al., 2013). A similar result was recently demonstrated in a small RCT on a guided Internet intervention for patients with social phobia or panic disorder (Mathiasen et al., 2016) and confirmed in a recent, broader meta-analysis on web-based interventions for a variety of conditions (Wildeboer et al., 2016). Given that the definition of adherence is open to interpretation, the drop-out at post-test was relatively high and the short-term effect sizes small (Kok et al., 2014), analyses on whether adherence was associated with therapeutic benefit would have been underpowered and inconclusive.

One explanation for the previously found lack of association between adherence and improvement is, as mentioned above, that some participants may have felt that they had derived sufficient benefit and no longer needed the intervention (Hilvert-Bruce et al., 2012). However, we lacked the short-term, repeated measurements to investigate this effect in a way that is psychometrically valid and not too burdensome for the patients. It should be noted that all participants in the current sample were scheduled to receive – and nearly all did receive – FtF treatment. Had they derived sufficient benefit, they would have opted out of FtF treatment. Indeed, it may be argued that participants experienced a low threshold to dropping out as they were assured of receiving FtF treatment regardless of their progress in the Internet-based intervention. These findings question the assumption of a dose-response relationship between exposure to the intervention and clinical benefit (Donkin et al., 2013).

Another reason for a lack of association might be caused by inconsistencies in defining and measuring the concept of (non)adherence (Kelders et al., 2012). In some studies, it is defined as e.g., whether a web page was accessed or content was displayed on a user's screen. This approach, however, implicitly assumes that the presented content was also read, understood and acted upon in the case of homework exercises. Unlike pharmacological interventions that require taking a pill, psychological interventions that require metacognitive techniques or behavioural techniques need practice and elaboration in real life before they can have an effect. As such, one possible explanation for the weak association between adherence and effect is that adherence is an inherently poor metric for assessing whether the intervention content was internalised as intended. Another possible explanation is that adherence and clinical effect have a reciprocal relationship, where participants need to experience improvement or success rapidly in the beginning of the intervention to ‘kick-start’ as an incentive to create adherence. In exposure therapy, although this is partly addressed by setting attainable exposure goals in the fear hierarchy, in the current intervention the first two of five weeks are theory-based and offer little in the way of such successful experiences. In the current study, however, non-adherence throughout the weeks with in-vivo exposure was steady.

4.2. Limitations

A number of limitations reduce the generalisability of the findings of this study. Firstly, we defined adherence as having performed at least 50% of the intervention. This might be considered a low threshold as some reviews have advocated higher thresholds (e.g., 75% or 80% (Karyotaki et al., 2015, van Ballegooijen et al., 2014). However, not only would this have resulted in too few participants in the adherent group to perform subgroup analyses, FtF treatment was scheduled to start after five weeks and finishing all of the intervention in this comparatively short time would have posed a rather large burden on the participants. Secondly, participants in this trial were scheduled to receive FtF treatment regardless of whether they participated in the Internet intervention, as the primary goal of this trial was to assess whether FtF psychotherapy sessions could be reduced (Kok et al., 2012, Kok et al., 2014). This limits the generalisability of the results as different motivational factors may play a role in samples that receive only an Internet intervention – for these samples, an Internet intervention is the only treatment they will receive and motivation to persist with the intervention may be greater. Knowing that one will receive face-to-face treatment regardless of whether one completes on Internet-based intervention may lower the threshold of discontinuing such an intervention.

The current analyses into the factors of non-adherence were grounded on previous literature and were explorative in nature. Therefore, we may not have measured important other determinants of non-adherence. Although our assumptions included demographic factors and symptomatology, these are probably not the sole causative factors of non-adherence. Alternative factors we did not assess are e.g., that some participants may have felt that they had derived sufficient benefit and decided to discontinue the intervention (Hilvert-Bruce et al., 2012). Other factors not explicitly assessed were low treatment motivation (Donkin and Glozier, 2012), side effects (e.g., worsening of symptoms), treatment expectations (Alfonsson et al., 2016) and practical barriers as possible reasons to be non-adherent to FtF and Internet-based treatments for anxiety (Taylor et al., 2012). Additionally, a worse clinical status such as in the currently described clinical population may aggravate or mediate generic factors, and further research should investigate whether a combination of these factors actually predict worse adherence in clinical populations.

Lastly, we were not able to analyse any influence of adherence status on short-term treatment outcome. Given the large drop-out and the small treatment effects at post-test (Kok et al., 2014) we would not be able to reliably detect any effect of adherence on outcomes, should they be present.

4.3. Clinical implications

Many clinical trials have now established the effectiveness of guided Internet-based interventions. As a result, a number of countries in the world now adopt those interventions for reasons of patient preference (therapy can be performed where and whenever the patient chooses) and initial results suggest possible cost-effectiveness of these interventions (Donker et al., 2015). However, most effectiveness studies have been performed among patients recruited in the general population.

The current trial was performed among patients recruited in routine outpatient mental health care settings. Our findings indicate that the sustained uptake among these patients is limited. This might indicate that guided Internet interventions are less suitable in outpatient settings. However, since about a third of the patients did use the intervention, we feel that it is too soon to disregard guided Internet interventions in routine care. Although our results suggest that clinicians might have to be more cautious with more severely ill patients, ultimately we think that better integration in mental health care will produce better effects.

Some recommendations can be given based on the current study and previous research. When offering an Internet-based intervention to patients, baseline severity and age are important factors to consider. Given that baseline anxiety, as measured through the BAI, but not more specific fear (as measured through the FQ) was a significant predictor of adherence, there is a possibility that non-adherence may be influenced more by general anxiety than specific anxiety. It is quite possible that the challenge of having to perform in-vivo exposure was not the decisive reason for patients to discontinue this intervention. However, little is currently known about how and why anxiety may influence non-adherence to interventions.

Although it is often assumed that younger patients would have a natural affinity for Internet-based programmes, the current study shows that it is older patients that are more likely to actually keep using the programme long enough to probably be beneficial. Again, it is unclear why older patients may persist in using an intervention, and this finding may be confounded with variables not measure in the current study, such as cultural factors, duration of illness or individual attitudes to psychotherapy.

Other factors to consider when offering these interventions to waitlisted patients, are that patients should ideally be guided by the therapist who performed the intake interview and carried out the treatment (rather than being coached by researchers outside of routine care) to improve a sense of continuity in treatment. This would likely increase the accountability to the therapist (Mohr et al., 2011), Alternatively, guided Internet interventions might be blended with face-to-face treatments, alternating sessions between FtF sessions and Internet-based sessions, which might increase motivation. Although number of such ‘blended’ interventions are currently being trialled (e.g., Kooistra et al., 2016, Romijn et al., 2015), clients and clinicians at present seem to prefer different ratios of online versus online content (van der Vaart et al., 2014). Furthermore, assessment of patient expectations (Greenberg et al., 2006), preferences (Beattie et al., 2009, Boettcher et al., 2013) and intrinsic motivation (Mohr et al., 2011) should be assessed carefully. Finally, treatment credibility can be discussed with the patient (El Alaoui et al., 2015).

4.4. Future research

A number of issues put forward in this discussion suggest areas of future research. Firstly, there is the myriad ways to define and measure adherence limit the cross-study generalisability of results (Kelders et al., 2012). Even if there were a perfect measure of adherence, it would be difficult to interpret these data as there is very probably no linear relationship between adherence and outcome. Notwithstanding, consensus or guidelines on the standardisation of adherence measurement would greatly improve further research. Additionally, reporting of adherence metrics should also be standardised, preferably following the proposed eHealth extension of the CONSORT reporting guidelines (Eysenbach, 2011). Analogous to this, there is currently little agreement on how to reliably assess patient expectancy of the effect of psychological interventions (Constantino et al., 2012). Moreover, to our knowledge, none of the proposed measures have been adapted and validated to be used for Internet-based intervention. Future research should aim to fill this gap.

Additionally, more data on the acceptability, feasibility and effectiveness of guided Internet interventions in outpatient mental health care settings is needed. Preferably, future research should incorporate at least the demographic and clinical predictors of adherence as suggested earlier, including treatment expectancy if a suitable instrument can be adapted or developed. In the future, greater access to full datasets from studies will allow us to perform individual patient data analyses (e.g., Karyotaki et al., 2015) or network meta-analyses (Leucht et al., 2016) that allow for greater statistical power to pinpoint predictors of non-adherence. Moreover, these large datasets may help to model the proposed type of relationship between adherence and treatment outcome (linear, quadratic, etc.).

Finally, the association between non-adherence to the intervention and not completing post-test measurements again highlights the issues with high drop-out, as previously noted (Cunningham, 2014).

5. Conclusions

In short, outpatient clinics may do well to consider baseline severity, age, but also intrinsic motivation, expectations when assessing outpatients for eligibility to use an Internet intervention. In all, we would like to stress that our results need not necessarily mean that it is unfeasible or ineffectual to offer the current intervention to patients in outpatient settings. After all, initial uptake of the intervention was good, and about a third of the patients adhered. In a shared-decision model, patient and therapist should take into account patient preferences and possible personal and circumstantial barriers to (continue) using Internet-based intervention.

Acknowledgements

We would like to thank ms. Julia van Oenen for data collection and adherence calculations, and all patients and outpatient staff involved. Furthermore, we would like to thank Marleen Bakker and Kim Setkowski for data collection and Koen Nijenhuis for statistical input. This study was funded by ZonMw (project number 80-82310-97-10015). The funding body had no influence in the design, execution, analysis, or interpretation of the results of this study.

References

  1. Alfonsson S., Olsson E., Hursti T. Motivation and treatment credibility predicts dropout, treatment adherence, and clinical outcomes in an internet-based cognitive behavioral relaxation program: a randomized controlled trial. J. Med. Internet Res. 2016;18(3) doi: 10.2196/jmir.5352. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Andersson G., Cuijpers P., Carlbring P., Riper H., Hedman E. Guided Internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: a systematic review and meta-analysis. World Psychiatry. 2014;13(3):288–295. doi: 10.1002/wps.20151. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Beattie A., Shaw A., Kaur S., Kessler D. Primary-care patients' expectations and experiences of online cognitive behavioural therapy for depression: a qualitative study. Health Expect. 2009;12(1):45–59. doi: 10.1111/j.1369-7625.2008.00531.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Beck A.T., Steer R.A. Psychological Corperation; San Antonio, TX: 1993. Beck Anxiety Inventory Manual. [Google Scholar]
  5. Boettcher J., Renneberg B., Berger T. Patient expectations in internet-based self-help for social anxiety. Cogn. Behav. Ther. 2013;42(3):37–41. doi: 10.1080/16506073.2012.759615. [DOI] [PubMed] [Google Scholar]
  6. Bower P., Kontopantelis E., Sutton A., Kendrick T., Richards D.A., Gilbody S.…Liu E.T.-H. Influence of initial severity of depression on effectiveness of low intensity interventions: meta-analysis of individual patient data. BMJ. 2013;346(feb26 2):f540. doi: 10.1136/bmj.f540. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Brooks R. EuroQol: the current state of play. Health Policy (Amsterdam, Netherlands) 1996;37(1):53–72. doi: 10.1016/0168-8510(96)00822-6. (Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/10158943) [DOI] [PubMed] [Google Scholar]
  8. Christensen H., Mackinnon A. The law of attrition revisited. J. Med. Internet Res. 2006;8(3) doi: 10.2196/jmir.8.3.e20. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Christensen H., Griffiths K.M., Farrer L.M. Adherence in internet interventions for anxiety and depression. J. Med. Internet Res. 2009;11(2) doi: 10.2196/jmir.1194. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Constantino M.J., Ametrano R.M., Greenberg R.P. Clinician interventions and participant characteristics that foster adaptive patient expectations for psychotherapy and psychotherapeutic change. Psychotherapy. 2012;49(4):557–569. doi: 10.1037/a0029440. [DOI] [PubMed] [Google Scholar]
  11. Cunningham J.A. Intervention adherence is related to participant retention: implications for research. J. Med. Internet Res. 2014;16(5) doi: 10.2196/jmir.2948. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. de Beurs E., Wilson K.A., Chambless D.L., Goldstein A.J., Feske U. Convergent and divergent validity of the Beck Anxiety Inventory for patients with panic disorder and agoraphobia. Depression Anxiety. 1997;6(4):140–146. doi: 10.1002/(sici)1520-6394(1997)6:4<140::aid-da2>3.0.co;2-g. (Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/9559283) [DOI] [PubMed] [Google Scholar]
  13. DiMatteo M.R., Lepper H.S., Croghan T.W. Depression is a risk factor for noncompliance with medical treatment: meta-analysis of the effects of anxiety and depression on patient adherence. Arch. Intern. Med. 2000;160(14):2101–2107. doi: 10.1001/archinte.160.14.2101. [DOI] [PubMed] [Google Scholar]
  14. Donker T., van Straten A., Marks I.M., Cuijpers P. Brief self-rated screening for depression on the Internet. J. Affect. Disord. 2010;122(3):253–259. doi: 10.1016/j.jad.2009.07.013. [DOI] [PubMed] [Google Scholar]
  15. Donker T., Blankers M., Hedman E., Ljótsson B., Petrie K., Christensen H. Economic evaluations of Internet interventions for mental health: a systematic review. Psychol. Med. 2015;45(16):3357–3376. doi: 10.1017/S0033291715001427. [DOI] [PubMed] [Google Scholar]
  16. Donkin L., Glozier N. Motivators and motivations to persist with online psychological interventions: a qualitative study of treatment completers. J. Med. Internet Res. 2012;14(3) doi: 10.2196/jmir.2100. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Donkin L., Hickie I.B., Christensen H., Naismith S.L., Neal B., Cockayne N.L., Glozier N. Rethinking the dose-response relationship between usage and outcome in an online intervention for depression: randomized controlled trial. J. Med. Internet Res. 2013;15(10) doi: 10.2196/jmir.2771. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Ebert D.D., Gollwitzer M., Riper H., Cuijpers P., Baumeister H., Berking M. For whom does it work? Moderators of outcome on the effect of a transdiagnostic internet-based maintenance treatment after inpatient psychotherapy: randomized controlled trial. J. Med. Internet Res. 2013;15(10) doi: 10.2196/jmir.2511. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. El Alaoui S., Ljótsson B., Hedman E., Kaldo V., Andersson E., Rück C.…Lindefors N. Predictors of symptomatic change and adherence in internet-based cognitive behaviour therapy for social anxiety disorder in routine psychiatric care. PLoS One. 2015;10(4) doi: 10.1371/journal.pone.0124258. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Erwin B.A., Heimberg R.G., Juster H., Mindlin M. Comorbid anxiety and mood disorders among persons with social anxiety disorder. Behav. Res. Ther. 2002;40(1):19–35. doi: 10.1016/s0005-7967(00)00114-5. [DOI] [PubMed] [Google Scholar]
  21. Eysenbach G. The law of attrition. J. Med. Internet Res. 2005;7(1) doi: 10.2196/jmir.7.1.e11. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Eysenbach G. CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J. Med. Internet Res. 2011;13(4) doi: 10.2196/jmir.1923. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Fernandez E., Salem D., Swift J.K., Ramtahal N., Fernandez E., Salem D.…Ramtahal N. Meta-analysis of dropout from cognitive behavioral therapy. J. Consult. Clin. Psychol. 2015;83(6):1108–1122. doi: 10.1037/ccp0000044. [DOI] [PubMed] [Google Scholar]
  24. Greenberg R.P., Constantino M.J., Bruce N. Are patient expectations still relevant for psychotherapy process and outcome? Clin. Psychol. Rev. 2006;26(6):657–678. doi: 10.1016/j.cpr.2005.03.002. [DOI] [PubMed] [Google Scholar]
  25. Herbert J.D., Gaudiano B.A., Rheingold A.A., Myers V.H., Dalrymple K., Nolan E.M. Social skills training augments the effectiveness of cognitive behavioral group therapy for social anxiety disorder. Behav. Ther. 2005;36:125. [Google Scholar]
  26. Hilvert-Bruce Z., Rossouw P.J., Wong N., Sunderland M., Andrews G. Adherence as a determinant of effectiveness of internet cognitive behavioural therapy for anxiety and depressive disorders. Behav. Res. Ther. 2012;50(7–8):463–468. doi: 10.1016/j.brat.2012.04.001. [DOI] [PubMed] [Google Scholar]
  27. Hofmann S.G., Suvak M. Treatment attrition during group therapy for social phobia. J. Anxiety Disord. 2006;20(7):961–972. doi: 10.1016/j.janxdis.2006.03.005. [DOI] [PubMed] [Google Scholar]
  28. Issakidis C., Andrews G. Pretreatment attrition and dropout in an outpatient clinic for anxiety disorders. Acta Psychiatr. Scand. 2004;109(6):426–433. doi: 10.1111/j.1600-0047.2004.00264.x. [DOI] [PubMed] [Google Scholar]
  29. Johnston L., Titov N., Andrews G., Dear B.F., Spence J. Comorbidity and internet-delivered transdiagnostic cognitive behavioural therapy for anxiety disorders. Cogn. Behav. Ther. 2013:37–41. doi: 10.1080/16506073.2012.753108. (March 2013) [DOI] [PubMed] [Google Scholar]
  30. Karyotaki E., Kleiboer A., Smit F., Turner D.T., Pastor A.M., Andersson G.…Cuijpers P. Predictors of treatment dropout in self-guided web-based interventions for depression: an “individual patient data” meta-analysis. Psychol. Med. 2015;45(13):2717–2726. doi: 10.1017/S0033291715000665. [DOI] [PubMed] [Google Scholar]
  31. Keijsers G.P.J., Kampman M., Hoogduin C.A.L. Dropout prediction in cognitive behavior therapy for panic disorder. Behav. Ther. 2001;32(4):739–749. [Google Scholar]
  32. Kelders S.M., Kok R.N., Ossebaard H.C., Van Gemert-Pijnen J.E.W.C. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J. Med. Internet Res. 2012;14(6) doi: 10.2196/jmir.2104. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Klein Hofmeijer-Sevink M., Batelaan N.M., Van Megen H.J.G.M.G.M., Penninx B.W., Cath D.C., Van Den Hout M.A., van Balkom A.J.L.M.L.M. Clinical relevance of comorbidity in anxiety disorders: a report from the Netherlands Study of Depression and Anxiety (NESDA) J. Affect. Disord. 2012;137(1–3):106–112. doi: 10.1016/j.jad.2011.12.008. [DOI] [PubMed] [Google Scholar]
  34. Kok R.N., van Straten A., Beekman A.T.F., Bosmans J.E., de Neef M., Cuijpers P. Effectiveness and cost-effectiveness of web-based treatment for phobic outpatients on a waiting list for psychotherapy: protocol of a randomised controlled trial. BMC Psychiatry. 2012;12(1):131. doi: 10.1186/1471-244X-12-131. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Kok R.N., van Straten A., Beekman A.T.F., Cuijpers P. Short-term effectiveness of web-based guided self-help for phobic outpatients: randomized controlled trial. J. Med. Internet Res. 2014;16(9) doi: 10.2196/jmir.3429. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Kolovos S., Kenter R.M.F., Bosmans J.E., Beekman A.T.F., Cuijpers P., Kok R.N., van Straten A. Economic evaluation of Internet-based problem-solving guided self-help treatment in comparison with enhanced usual care for depressed outpatients waiting for face-to-face treatment: a randomized controlled trial. J. Affect. Disord. 2016;200:284–292. doi: 10.1016/j.jad.2016.04.025. [DOI] [PubMed] [Google Scholar]
  37. Kooistra L.C., Ruwaard J., Wiersma J.E., van Oppen P., van der Vaart R., van Gemert-Pijnen L.J.W.E.C., Riper H. Development and initial evaluation of blended cognitive behavioural treatment for major depression in routine specialized mental health care. Internet Interv. 2016 doi: 10.1016/j.invent.2016.01.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Lamers F., van Oppen P., Comijs H.C., Smit J.H., Spinhoven P., van Balkom A.J.L.M.…Penninx B.W.J.H. Comorbidity patterns of anxiety and depressive disorders in a large cohort study. J. Clin. Psychiatry. 2011;72(3):341–348. doi: 10.4088/JCP.10m06176blu. [DOI] [PubMed] [Google Scholar]
  39. Ledley D.R., Huppert J.D., Foa E.B., Davidson J.R.T., Keefe F.J., Potts N.L.S. Impact of depressive symptoms on the treatment of generalized social anxiety disorder. Depression Anxiety. 2005;22(4):161–167. doi: 10.1002/da.20121. [DOI] [PubMed] [Google Scholar]
  40. Leucht S., Chaimani A., Cipriani A.S., Davis J.M., Furukawa T.A., Salanti G. Network meta-analyses should be the highest level of evidence in treatment guidelines. Eur. Arch. Psychiatry Clin. Neurosci. 2016:10–13. doi: 10.1007/s00406-016-0715-4. [DOI] [PubMed] [Google Scholar]
  41. Lewinsohn P.M., Seeley J.R., Roberts R.E., Allen N.B. Center for Epidemiologic Studies Depression Scale (CES-D) as a screening instrument for depression among community-residing older adults. Psychol. Aging. 1997;12(2):277–287. doi: 10.1037//0882-7974.12.2.277. [DOI] [PubMed] [Google Scholar]
  42. Leyfer O.T., Ruberg J.L., Woodruff-Borden J. Examination of the utility of the Beck Anxiety Inventory and its factors as a screener for anxiety disorders. J. Anxiety Disord. 2006;20(4):444–458. doi: 10.1016/j.janxdis.2005.05.004. [DOI] [PubMed] [Google Scholar]
  43. Marks I.M., Mathews A.M. Brief standard self-rating for phobic patients. Behav. Res. Ther. 1979;17(3):263–267. doi: 10.1016/0005-7967(79)90041-x. [DOI] [PubMed] [Google Scholar]
  44. Mathiasen K., Riper H., Ehlers L., Valentin J.B., Rosenberg N.K. Internet-based CBT for social phobia and panic disorder in a specialised anxiety clinic in routine care: Results of a pilot randomised controlled trial. Internet Interv. 2016 doi: 10.1016/j.invent.2016.03.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. McEvoy P.M. Effectiveness of cognitive behavioural group therapy for social phobia in a community clinic: A benchmarking study. Behav. Res. Ther. 2015;45(12):3030–3040. doi: 10.1016/j.brat.2007.08.002. [DOI] [PubMed] [Google Scholar]
  46. Melville K.M., Casey L.M., Kavanagh D.J. Dropout from Internet-based treatment for psychological disorders. Br. J. Clin. Psychol./Br. Psychol. Soc. 2010;49(Pt 4):455–471. doi: 10.1348/014466509X472138. [DOI] [PubMed] [Google Scholar]
  47. Mohr D.C., Cuijpers P., Lehman K. Supportive accountability: a model for providing human support to enhance adherence to eHealth interventions. J. Med. Internet Res. 2011;13(1):11. doi: 10.2196/jmir.1602. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Penninx B.W.J.H.J.H., Nolen W.A., Lamers F., Zitman F.G., Smit J.H., Spinhoven P.…Beekman A.T.F. Two-year course of depressive and anxiety disorders: results from the Netherlands Study of Depression and Anxiety (NESDA) J. Affect. Disord. 2011;133(1–2):76–85. doi: 10.1016/j.jad.2011.03.027. [DOI] [PubMed] [Google Scholar]
  49. Radloff L.S. The CES-D Scale: a self-report depression scale for research in the general population. Appl. Psychol. Meas. 1977;1(3):385–401. [Google Scholar]
  50. Richards D., Richardson T. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin. Psychol. Rev. 2012;32(4):329–342. doi: 10.1016/j.cpr.2012.02.004. [DOI] [PubMed] [Google Scholar]
  51. Romijn G., Riper H., Kok R.N., Donker T., Goorden M., van Roijen L.H.…Koning J. Cost-effectiveness of blended vs. face-to-face cognitive behavioural therapy for severe anxiety disorders: study protocol of a randomized controlled trial. BMC Psychiatry. 2015;15(1):311. doi: 10.1186/s12888-015-0697-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Santana L., Fontenelle L.F. A review of studies concerning treatment adherence of patients with anxiety disorders. Patient Prefer. Adherence. 2011;5:427–439. doi: 10.2147/PPA.S23439. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Taylor S., Abramowitz J.S., McKay D. Non-adherence and non-response in the treatment of anxiety disorders. J. Anxiety Disord. 2012;26(5):583–589. doi: 10.1016/j.janxdis.2012.02.010. [DOI] [PubMed] [Google Scholar]
  54. van Agt H.M., Essink-Bot M.L., Krabbe P.F., Bonsel G.J. Test-retest reliability of health state valuations collected with the EuroQol questionnaire. Soc. Sci. Med. 1994;39(11):1537–1544. doi: 10.1016/0277-9536(94)90005-1. (Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/7817218) [DOI] [PubMed] [Google Scholar]
  55. van Ballegooijen W., Cuijpers P., van Straten A., Karyotaki E., Andersson G., Smit J.H., Riper H. Adherence to Internet-based and face-to-face cognitive behavioural therapy for depression: a meta-analysis. PLoS One. 2014;9(7) doi: 10.1371/journal.pone.0100674. [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. van der Vaart R., Witting M., Riper H., Kooistra L.C., Bohlmeijer E.T., van Gemert-Pijnen L.J.E.W.C. Blending online therapy into regular face-to-face therapy for depression: content, ratio and preconditions according to patients and therapists using a Delphi study. BMC Psychiatry. 2014;14:355. doi: 10.1186/s12888-014-0355-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  57. Van Zuuren F.J. The fear questionnaire. Some data on validity, reliability and layout. Br. J. Psychiatry. 1988;153:659–662. doi: 10.1192/bjp.153.5.659. (Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/3255455) [DOI] [PubMed] [Google Scholar]
  58. Wildeboer G., Kelders S.M., van Gemert-Pijnen J.E.W.C. The relationship between persuasive technology principles, adherence and effect of web-Based interventions for mental health: a meta-analysis. Int. J. Med. Inform. 2016 doi: 10.1016/j.ijmedinf.2016.04.005. [DOI] [PubMed] [Google Scholar]
  59. World Health Organization . World Health Organization; Geneva: 1990. Composite International Diagnostic Interview (CIDI) [Google Scholar]

Articles from Internet Interventions are provided here courtesy of Elsevier

RESOURCES