Skip to main content
NCBI home page
Internet Interventions logoLink to Internet Interventions
. 2019 Sep 4;18:100278. doi: 10.1016/j.invent.2019.100278

Negative effects associated with internet-delivered cognitive behaviour therapy: An analysis of client emails

Kirsten M Gullickson a, Heather D Hadjistavropoulos a,⁎, Blake F Dear b, Nickolai Titov c
PMCID: PMC6926340  PMID: 31890627

Abstract

Internet-delivered cognitive behaviour therapy (ICBT) is an efficacious form of treatment for anxiety and depression, yet it is still possible for clients to experience negative effects associated with treatment. In the ICBT literature, the term negative effects is broadly used to refer to all potentially adverse or unwanted events or experiences that are perceived as undesirable by the client and may or may not be associated with long-term symptoms or distress. Previous ICBT studies have asked clients to retrospectively describe negative effects at post-treatment; however, no research has examined the content of clients' emails to their therapist to see whether clients are reporting negative effects as they arise. In the current study, 96 clients (80 completers; 16 non-completers) were randomly selected from a published ICBT trial and directed content analysis was used to examine client emails for mention of negative effects. In addition, correlational analyses were used to examine the relationship between negative effects and: 1) demographic characteristics; 2) treatment engagement; 3) treatment satisfaction; 4) working alliance; and 5) symptom outcomes among completers. The results indicated that 61.5% of clients experienced at least one negative effect during treatment, although total number of negative effects was not significantly correlated with client demographic characteristics, lessons completed, working alliance, treatment satisfaction, or symptom outcomes. Among completers, technical difficulties, implementation problems, and negative emotional states were the most commonly reported negative effects, whereas dropout was the most commonly reported negative effect by non-completers. Negative effects that have been identified in previous research, such as symptom deterioration, novel symptoms, and severe adverse events, were not identified in client emails. The high incidence of negative effects in the current study suggests there may be value in systematically monitoring client emails for negative effects throughout treatment as a supplement to retrospective post-treatment reports. This will give therapists the opportunity to intervene as negative effects occur and potentially mitigate any impact they have on treatment outcomes. Future research, both qualitative and quantitative, is needed to gain a more nuanced understanding of negative effects associated with ICBT.

Keywords: Internet-delivered cognitive behaviour therapy, Negative effects, Adverse effects

Highlights

  • •

    61.5% of ICBT clients in the sample (n = 59/80) reported experiencing at least one negative effect during treatment.

  • •

    Negative effects were not correlated with lessons completed, working alliance, satisfaction, or symptom outcomes.

  • •

    Among treatment completers, technical difficulties, implementation problems, and negative emotional states were most common.

  • •

    Among treatment non-completers, dropout was most common.

  • •

    ICBT therapists should monitor for negative effects in client emails to supplement retrospective post-treatment reports.

1. Introduction

Internet-delivered cognitive behaviour therapy (ICBT) is becoming increasingly popular due to its numerous advantages over face-to-face treatment, such as greater accessibility, increased privacy, and cost effectiveness (Andersson and Titov, 2014). ICBT typically consists of online lessons presented over the course of several weeks, combined with regular symptom monitoring and homework assignments (Titov et al., 2018). In therapist-assisted ICBT programs, clients correspond with their therapist as they work through the course content (Andersson, 2016). It is well established that therapist-assisted ICBT produces clinically significant symptom improvements that are comparable to the effects found with face-to-face cognitive behaviour therapy (CBT; Andersson et al., 2019).

Despite the effectiveness of ICBT, it is possible for clients to experience negative effects associated with treatment (Andersson et al., 2019; Ebert et al., 2018; Rozental et al., 2014). Several recent articles have noted the lack of research in this area and emphasized the need for further exploration of negative effects associated with ICBT (Andersson et al., 2019; Ebert et al., 2018; Holmes et al., 2018; Rozental et al., 2018). Such articles attribute the lack of negative effects research to the absence of a coherent framework for defining and measuring negative effects (Parry et al., 2016; Rozental et al., 2018; Scott and Young, 2016). For example, in the broader psychotherapy literature, the terms adverse events, unwanted events, deterioration, harmful effects, negative therapeutic reaction, and many more have been used synonymously with the term negative effects (Parry et al., 2016). For the purpose of this study, negative effects will be used as an umbrella term to refer to all potentially adverse or unwanted events or experiences that have an undesirable effect on the client and may or may not be associated with long-term symptoms or distress (Rozental et al., 2018).

Negative effects of ICBT can be measured using a variety of methods (Rozental et al., 2018). ICBT clinical trials sometimes report rates of deterioration (i.e., percentage of the sample that experienced a significant increase in symptoms) or non-response (i.e., percentage of the sample that reported a non-significant improvement in symptoms) by comparing clients' symptom scores at pre- and post-treatment (Barak et al., 2008; Boettcher et al., 2014; Ebert et al., 2016; Rozental et al., 2019a; Rozental et al., 2017). Researchers have also used self-report measures, such as the Negative Effects Questionnaire, at post-treatment as a means of exploring negative effects associated with ICBT, although such measures have not yet been widely adopted (Rozental et al., 2016; Rozental et al., 2019b).

Qualitative research methods are increasingly being used to explore the nature and prevalence of negative effects in ICBT (Dimidjian and Hollon, 2010; Rozental et al., 2018). Several studies have interviewed clinical trial participants after treatment to ask about a number of topics including negative effects (Bendelin et al., 2011; Fernández-Álvarez et al., 2017; Olsson Halmetoja et al., 2014). The most insight, however, has come from studies that explicitly asked clients to describe negative effects they experienced over the course of treatment using open-ended comment boxes at post-treatment (Boettcher et al., 2014; Rozental et al., 2015). The two studies that have assessed negative effects in this way will be reviewed below as they are most relevant to the current study.

In the largest study of its kind, Rozental et al. (2015) analyzed negative effects across 4 large ICBT clinical trials that addressed social anxiety, panic disorder, depression, and severe procrastination (N = 558). The authors defined negative effects as any unwanted events or experiences that the client believed to be related to treatment. At post-treatment, they found, on average, 9.3% of patients reported one negative effect that might be related to treatment. The authors grouped the negative effects into two main categories: Patient-related negative effects consisted of insight (i.e., greater understanding and awareness of an ongoing condition which lead to greater distress) and symptoms (i.e., increase in symptom severity), and treatment-related negative effects consisted of implementation (i.e., difficulty performing tasks assigned during treatment or adhering to treatment) and format (i.e., issues with the online treatment environment).

Another notable study by Boettcher et al. (2014) asked 133 clients participating in an ICBT program for social anxiety disorder about negative effects at post-treatment. The authors defined negative effects as any unwanted effects or events that the client associated with treatment. In total, 14% of participants (n = 19) described experiencing unwanted negative events. The authors identified seven types of negative effects: emergence of new symptoms, deterioration of symptoms, negative well-being, lack of clear treatment result, non-compliance, changes in work situation, and stigmatization. Notably, their results indicated that experiencing negative effects was not related to symptom deterioration at post-treatment or 4-month follow-up.

Although asking clients to report on negative effects at post-treatment is valuable, there are limitations to this retrospective reporting style. In particular, asking clients to think back about negative effects increases the risk of memory bias (i.e., clients providing less or inaccurate detail, clients forgetting about negative effects experienced early in treatment; Rozental et al., 2018). Therefore, there is value in exploring complementary ways of assessing negative effects of ICBT, such as inquiring about negative effects at different points throughout treatment (Rozental et al., 2018). In therapist-assisted ICBT programs, emails sent to therapists provide insight into clients' perceptions of treatment on a weekly basis. Thus, qualitatively examining client emails represents a potentially effective supplemental approach to learning about negative effects associated with ICBT. Understanding negative effects is essential because it allows clients to make informed decisions about the risks and benefits of ICBT, helps researchers continuously improve the design and delivery of ICBT, and ensures clients do not experience unnecessary harm or distress (Crawford et al., 2016; Rozental et al., 2018; Scott and Young, 2016). Moreover, from the clinician perspective, understanding negative effects puts therapists in a position to respond to and address negative effects in treatment, which could ultimately serve to bolster outcomes and the therapeutic alliance, similar to how past researchers have found that therapist response to an alliance rupture can be beneficial to outcomes in therapy (Eubanks et al., 2018).

The purpose of the current study was to explore negative effects reported by clients in their emails to their therapist over the course of an 8-week, therapist-assisted ICBT intervention for anxiety and depression. Specifically, the current study aimed to determine the percentage of clients who report negative effects in their correspondence during treatment and which negative effects are most commonly identified during treatment. We also conducted a preliminary investigation of correlates of negative effects (i.e., relationship between negative effects and a variety of demographic characteristics, engagement variables, and treatment outcomes). Given that this was the first study to explore negative effects in client emails, no formal hypotheses were made; however, we generally expected clients to report similar negative effects to what has been found in previous ICBT research, such as negative emotional states, difficulties with implementation, and format problems (Boettcher et al., 2014; Rozental et al., 2015).

2. Method

2.1. Participants

The present study utilized data from a previously published ICBT trial for depression and anxiety that examined results of ICBT in routine care (ISRCTN42729166; Hadjistavropoulos et al., 2016). Clients learned about the government-funded ICBT program via providers working in community mental health clinics, family physicians, family and friends, media, online searches, email announcements, and printed posters/cards. Interested clients then self-referred to ICBT by completing an online screening and subsequent telephone interview to assess inclusion/exclusion criteria including: 1) being at least 18 years of age and residing in Saskatchewan, Canada; 2) having access to a computer with Internet connection; 3) consenting to participate and for physician notification; 4) exhibiting symptoms of depression or anxiety; 5) the absence of severe or unmanaged mental health symptoms (e.g., high suicide risk, schizophrenia); and 6) not receiving regular psychotherapy at the time of enrollment. More information about the participants, recruitment, and screening can be found elsewhere (Hadjistavropoulos et al., 2016).

2.2. Intervention

Clients in the current study participated in an 8-week, transdiagnostic, therapist-assisted ICBT program for anxiety and depression. The transdiagnostic approach is based on the premise that anxiety and depression are often comorbid and share common symptoms, such as maladaptive thinking patterns and avoidance. As such, across the 5 lessons, the program taught clients a variety of symptom management strategies that are applicable to both anxiety and depression: 1) the cognitive-behavioural model; 2) thought monitoring and challenging; 3) de-arousal strategies and pleasant activity scheduling; 4) graduated exposure; and 5) relapse prevention (Dear et al., 2015; Titov et al., 2015). Each lesson was comprised of psychoeducation, instruction on cognitive behavioural strategies used for symptom reduction, case stories, lesson summaries, and homework exercises. Clients emailed their therapist as many times as needed each week, while therapists spent 15 to 20 min a week responding to client emails on a predetermined day. Clients provided demographic information during the online screening and completed online symptom measures at pre-treatment, post-treatment, and three-month follow-up, as well as at the start of each lesson. For the purposes of the present study, clients who completed at least one lesson and post-treatment measures were considered “completers” and clients who completed at least one lesson but did not complete post-treatment measures were considered “non-completers”.

2.3. Measures

Depressive symptoms were assessed using the Patient Health Questionnaire – 9 items (PHQ-9; range 0–27; ≥10 = clinical depression; Kroenke et al., 2001) and anxiety symptoms were measured using the Generalized Anxiety Disorder – 7 items (GAD-7; range 0–21; ≥10 = clinical anxiety; Spitzer et al., 2006). At post-treatment, clients completed the 12-item Working Alliance Inventory-Short Revised (WAI-SR; range 12–24; Hatcher and Gillaspy, 2006) and responded to questions about treatment satisfaction (e.g., Overall, how satisfied were you with treatment?; range 1–5).

2.4. Analyses

Client emails to their therapist were the primary source of data for the present study. The emails were imported to NVivo 12 and the first author used directed content analysis to examine client emails for mention of negative effects. Directed content analysis involves using pre-existing themes to code the data, but allows for creation of new themes if applicable (Elo and Kyngäs, 2008; Hsieh and Shannon, 2005). In this case, the pre-existing themes were the negative effects categories identified in previous ICBT research: deterioration, non-response, implementation problems, format problems, novel symptoms, negative emotional states, severe adverse events, non-compliance/dropout, stigmatization, increased awareness (Boettcher et al., 2014; Fernández-Álvarez et al., 2017; Rozental et al., 2014; Rozental et al., 2015). After all emails were coded, the identified negative effects were discussed by the first and second authors and the coding guide was finalized. Subsequently, the first author re-analyzed the data to ensure client emails were coded accurately and comprehensively in accordance with the guide. Finally, a third researcher who was not involved in the initial analysis reviewed all coded material to ensure fit with the definitions provided in the coding guide. The second author was responsible for resolving any coding discrepancies that were identified using her knowledge of ICBT and the existing negative effects literature, although this rarely occurred.

Descriptive statistics were calculated to describe the sample's demographic characteristics and treatment outcomes, and determine the frequency of each negative effect. Completers and non-completers are presented separately for comparison purposes, but it was not possible to conduct statistical comparisons across the groups due to the small number of non-completers. For completers, correlational analyses were used to examine the relationship between total number of negative effects and: 1) demographic variables; 2) treatment engagement; 3) treatment satisfaction; 4) working alliance; and 5) symptom outcomes. Spearman's rank-order correlation coefficient (Spearman's Rho) was used because it is a robust test that guards against violations of statistical assumptions (e.g., normality) and is appropriate with mixed variable types (e.g., ordinal and continuous). Given the exploratory nature of the correlation analyses, statistical corrections were not used to account for the number of correlations conducted. A priori G*Power analysis using a two-tailed correlation point-biserial model indicated that a minimum sample of 80 participants (alpha = 0.05; power = 0.80) was suitable to detect medium-to-large effects (Faul et al., 2007).

3. Results

3.1. Sample characteristics

The sample consisted of 96 randomly selected trial clients. To ensure statistical power, 80 clients were selected from the 378 who were classified as completers in the previous trial (21%). Subsequently, 16 clients were selected from the 80 clients who were classified as non-completers in the previous trial (20%). Our goal was to select approximately the same proportion of completers and non-completers from the original trial. Demographic characteristics and treatment outcomes for the sample are presented in Table 1. Completers and non-completers were similar in terms of demographic characteristics and pre-treatment depression and anxiety scores, but completers were more engaged in treatment than non-completers (e.g., number of log-ins, lessons started, and emails to their therapists). Completers reported good working alliance with their therapist, were satisfied with treatment, and reported significant reductions in anxiety and depression symptoms from pre- to post-treatment.

Table 1.

Sample demographics and treatment outcomes.

Demographics
 Completers (n = 80)
 Non-Completers (n = 16)
M (SD) M (SD)
Age 39.28 (11.78) 37.44 (10.72)
n (%) n (%)
Female 53 (66.0%) 14 (87.5%)
Caucasian 73 (91.3%) 15 (83.8%)
Married or common-law 55 (68.8%) 9 (56.3%)
Urbana 44 (55.0%) 6 (37.5%)
University educated 43 (53.8%) 5 (31.3%)
Employed 55 (68.8%) 7 (43.8%)
Treatment Outcomes Completers (n = 80) Non-Completers (n = 16)
Engagement M (SD) M (SD)
 Log-ins 27.30 (14.25) 8.56 (3.81)
 Lessons started 4.74 (0.78) 2.31 (0.95)
 Emails to therapist 5.69 (3.64) 1.44 (1.03)
 Emails from therapist 9.86 (1.88) 7.19 (2.88)
Working Allian M (SD) M (SD)
 WAI-SR-Total (12-84) 67.68 (13.41) -
Treatment Satisfaction M (SD) M (SD)
 Overall satisfaction (1-5) 4.03 (0.98) -
Symptom Measures M (SD) Descriptor M (SD) Descriptor
Depression (PHQ-9; 0-27)
 Pre-treatment 12.53 (5.68) Moderate range 13.00 (5.91) Moderate range
 Post-treatment 5.65 (4.87) Minimal range - -
Anxiety (GAD-7; 0-21)
 Pre-treatment 12.35 (4.70) Moderate range 10.31 (5.45) Moderate range
 Post-treatment 5.33 (4.27) Minimal range - -
Open in a new tab

Note: Completers = completed post treatment questionnaires; Non-Completers = did not complete post treatment questionnaires; M = mean; SD = standard deviation; WAI-SR = Working Alliance Inventory-Short Revised PHQ-9 = Patient Health Questionnaire-9; GAD-7 = Generalized Anxiety Disorder-7.

a

Urban = residing in a city with a population over 200,000.

3.2. Negative effects

Qualitative content analysis of 478 client emails sent to 39 therapists resulted in the identification of two categories. Patient-related negative effects consisted of three subcategories: negative emotional states, dropout, and non-response. Treatment-related negative effects consisted of five subcategories: therapist support concerns, content/format problems, implementation problems, technical difficulties, and questionnaire concerns. The majority of these negative effects were similar to categories described in past research, however therapist support concerns, technical difficulties, and questionnaire concerns represented newly identified categories. A description of each type of negative effect is provided in Table 2 along with example quotes and frequencies for completers and non-completers. Among completers, technical difficulties was the most commonly reported negative effect, followed by implementation problems and negative emotional states. Among non-completers, dropout was the most commonly reported negative effect.

Table 2.

Negative effect categories, descriptions, quotes, and frequencies.

Categories and subcategories Definition Example quotes Completers (n = 80) Non-completers (n = 16)
Patient-related negative effects
Negative emotional states Experiencing a negative emotional state as a result of the program (e.g., anxiety, frustration, hopelessness, stress, discomfort) “[ICBT] has only been one more thing on my to do list that is adding, rather than alleviating, stress.” (client 66) 13 (16.3%) –
Dropout Premature treatment termination “I have decided to withdraw from the program.” (client 117) 2 (2.5%) 3 (18.8%)
Non-response Lack of symptom improvement or lack of progress “I'm still feeling that I have made no progress.” (client 152) 1 (1.25%) –



Treatment-related negative effects
Therapist support concernsa Concerns about the frequency/quality of therapist support or the nature of email contact (e.g., not enough therapist contact, confused about what to write in emails to therapist) “I have a really hard time conveying thoughts to email.” (client 94) 4 (5.0%) 1 (6.3%)
Content/format problems Problems with the way the program content is presented (e.g., too much text, examples not relevant/helpful, already encountered skills in previous therapy) “It is easy to read the resources on the site, but I just feel like there has been too much at once for me.” (client 167) 11 (13.75%) –
Implementation problems Problems implementing the skills taught in the program (e.g., challenging negative thoughts, creating exposure ladder) “I am having a tough time with constructing ladders beyond the point of one or two steps.” (client 94) 16 (20.0%) –
Technical difficultiesa Difficulty using the program interface (e.g., delivery of email to therapist, accessing lessons, locating supplemental materials) “Not sure if it's the website or my iPad, but my response got frozen and then kicked me out after a fairly lengthy response, so this one will be a bit shorter.” (client 225) 18 (22.5%) 1 (6.3%)
Questionnaire concernsa Concerns about the questionnaire response options (e.g., difficulty choosing a response option) “I would like to note, for the depression and anxiety measurement questions, the options several, more than half, and every day, sometimes don't apply.” (client 320) 3 (3.75%) –
Open in a new tab

Note: Completers = completed post treatment questionnaires; Non-Completers = did not complete post treatment questionnaires.

a

Novel negative effects that have not been identified in previous research.

Out of all 96 clients, 59 (61.5%) reported experiencing at least one negative effect and 17 (17.7%) reported experiencing more than one negative effect. Follow-up analysis revealed that 43 completers (53.8%) and 16 non-completers (25.0%) reported at least one negative effect, while more than one negative effect was reported by 16 completers (20.0%) and 1 non-completer (6.3%).

3.3. Correlations between negative effects and treatment outcomes

Correlation values are presented in Table 3. No significant correlations were observed between the total number of negative effects and completer demographic characteristics (e.g., age, gender), working alliance (e.g., WAI-SR total score), treatment satisfaction (e.g., overall satisfaction rating), or symptom outcomes (e.g., PHQ-9 and GAD-7 change scores). Significant correlations were observed, however, between total negative effects and engagement variables. That is, total number of negative effects was weakly correlated with number of client logins and number of client emails sent, suggesting that clients who reported a greater number of negative effects logged in to the ICBT program more frequently and sent more emails to their therapist. Negative effects were not associated with number of lessons completed. When each type of negative effect was examined independently, the only significant correlations were for content/format problems. Specifically, weak negative correlations with the WAI-SR total score and overall ratings of treatment satisfaction were observed, suggesting that clients who reported issues relating to content/format reported slightly poorer working alliance and treatment satisfaction.

Table 3.

Negative effect total and domain correlations with demographics, engagement, working alliance, treatment satisfaction, and patient outcomes (n = 80).

Total count Negative emotional states Content/format problems Implementation problems Technical difficulties
Demographics
Age 0.07 0.16 −0.10 0.11 0.03
Female (0)/male (1) −0.03 −0.04 0.06 −0.17 −0.12
Not married (0)/married (1) −0.08 0.15 −0.04 0.00 −0.09
No university (1)/university (1) −0.03 −0.14 0.08 0.03 0.02



Engagement
Log-ins 0.24⁎ 0.20 0.12 0.22 0.19
Lessons started 0.20 0.06 0.05 0.19 0.11
Emails to therapist 0.26⁎ 0.18 0.16 0.19 0.13



Working alliance
WAI-SR-total −0.20 −0.11 −0.24⁎ −0.12 0.03



Treatment satisfaction
Overall satisfaction rating −0.11 −0.04 −0.28⁎ −0.12 0.22



Patient outcomes
GAD-7 change score −0.16 0.01 −0.17 0.09 −0.18
PHQ-9 change score −0.13 0.07 0.09 −0.21 −0.06
Open in a new tab

Note: Correlations for dropout, non-response, therapist support concerns, and questionnaire concerns are not reported because of their low incidence among completers (i.e., < 10% of clients). WAI-SR = Working Alliance Inventory-Short Revised; GAD-7 = Generalized Anxiety Disorder-7; PHQ-9 = Patient Health Questionnaire-9.

⁎

p < .05.

4. Discussion

The aim of the current study was to explore negative effects reported by 96 clients in their emails to their therapist over the course of an 8-week, therapist-assisted, transdiagnostic ICBT intervention for anxiety and depression. This is the first study to examine negative effects in client emails during treatment as opposed to the usual method of relying on retrospective post-treatment questionnaires or interviews. We found that 61.5% of clients in our sample reported at least one negative effect during treatment. The overall incidence of negative effects was significantly higher in the current study than has been found in previous research that has examined negative effects retrospectively at post-treatment (i.e., 9.3–12.9%; Boettcher et al., 2014; Rozental et al., 2015). There are several potential explanations for this discrepancy. First, it is possible that different definitions of negative effects and coding guides influence the nature and frequency of negative effects. In this study, we coded quite liberally, broadly including all adverse or unwanted events or experiences. Second, it is possible that clients underreported negative effects in previous studies due to memory bias. For example, it is possible they forgot about or overlooked negative effects that occurred earlier in treatment. Third, it is conceivable that negative effects of ICBT differ based on the condition being treated, the content/format of the program, or the website quality. Most likely, a combination of all three factors accounts for the higher incidence of negative effects in the current study.

Informally comparing negative effects reported by treatment completers and non-completers allowed us to gain preliminary insight into differences across groups. Completers reported a greater total number of negative effects than non-completers, as well as negative effects in more categories, most commonly technical difficulties, negative emotional states, and implementation problems. This is likely the product of completers finishing more of the course content and sending more emails to their therapist than non-completers. The most common negative effect reported by non-completers was dropout, which most clients attributed to competing demands for their time or a desire for more intensive therapist support. Importantly, negative effects did not appear to be the impetus to clients terminating treatment prematurely.

Noteworthy is that we found negative effects were not related to client demographic characteristics, lessons completed, working alliance, treatment satisfaction, or symptom outcomes among completers, despite the high incidence of negative effects in the current study. This result compliments the work of Boettcher et al. (2014) who also found no relationship between negative effects and symptom deterioration at post-treatment or 4-month follow-up. Together, these findings raise interesting questions about why negative effects do not appear to be related to several important treatment outcomes. One potential explanation is that the negative effects reported by clients resulted in distress that was short-lived and did not persist long enough to influence treatment outcomes. For example, clients who reported technical difficulties, such as loss of a composed but unsent email, might have experienced only temporary frustration that did not have a serious, long-term negative impact. If this were the case, it could be argued that some negative effects are not really “negative” at all, unless they are detrimental to clients functioning at post-treatment. The lack of significant correlations could also be attributed to statistical factors, such as low power for identifying small effects, insufficient variability, or the dichotomous nature of some of the variables. Alternatively, it may be that the inclusion of therapist support allowed ICBT clinicians to respond to and address negative effects as they were reported by their clients, thus preventing negative effects from influencing treatment outcomes, or that strong working alliance acted as a buffer against negative effects. The latter possibility is bolstered by the observed non-significant relationship between negative effects and working alliance, which suggests that negative effects are not necessarily detrimental to the development or maintenance of the client-therapist relationship and do not appear to cause alliance ruptures.

We did, however, find that total number of negative effects was significantly correlated with number of client logins and number of client emails sent. Given that correlational analyses do not specify causal relationships, one interpretation of the data is that clients who logged in to the program more frequently were exposed to a greater number of negative effects and clients who sent more emails had more opportunity to report negative effects. When each type of negative effect was examined independently, only content/format problems was significantly related to outcomes. That is, clients who reported content/format problems scored lower on a measure of working alliance and reported lower levels of satisfaction with treatment. If this finding can be replicated in future research, it could suggest that certain negative effects that occur during treatment may be detrimental to particular outcomes, while others are not. Additional research is needed to know more about these relationships.

The negative effects identified in the present study share some commonalities with those reported in the existing literature. We retained the two overarching categories first described by Rozental et al. (2015): patient-related negative effects and treatment-related negative effects. With regard to patient-related negative effects, the subcategory negative emotional states is similar to the negative well-being category described by Boettcher et al. (2014) and includes the insight theme identified by Rozental et al. (2015). We also identified a few examples of dropout and non-response, negative effects that have been traditionally measured using quantitative approaches (e.g., Rozental et al., 2017). In terms of treatment-related negative effects, the subcategories implementation problems and content/format problems were previously reported by Rozental et al. (2015).

Several negative effects that have not been reported in previous studies were also identified: questionnaire concerns, technical difficulties, and therapist support concerns. It is possible previous research included these negative effects in a larger category (e.g., technical difficulties might have been captured under the format category by Rozental et al., 2015), or alternatively, these may be novel negative effects associated with the ICBT program used by clients in the current study. It is plausible that issues related to questionnaires, technical problems, and the presence or absence of therapist assistance might differ across ICBT programs. Finally, there were several negative effects that have been recognized in previous research that were not found in the current study. Specifically, clients did not discuss symptom deterioration, novel symptoms, or severe adverse events such as suicidality or hospitalization in their emails to their therapist (Rozental et al., 2015).

The findings from the current study (and the study of negative effects more generally) have important implications for research and clinical practice. From the perspective of ICBT program developers, information about negative effects can be used to improve the design and delivery of programs offered. For example, changes can be made to the program to reduce the likelihood of negative effects such as technical difficulties, questionnaire concerns, and content/format problems. Program developers can also monitor and publish rates of dropout, deterioration, and non-response, which may provide an indication that a program needs to be revised. From the ICBT clinician perspective, research on negative effects generates ideas about how to reduce harms to ICBT clients. In particular, it highlights the value in systematically monitoring client emails for negative effects to give therapists the opportunity to intervene as negative effects occur (e.g., normalize certain negative emotional states, provide ideas related to implementation) and potentially mitigate any impact they have on treatment outcomes. Accordingly, it is essential to train ICBT therapists on how to identify and respond to negative effects reported by clients during treatment (Bystedt et al., 2014). Suggested therapist responses to client-reported negative effects are provided in Table 4. If negative effects are systematically monitored during treatment and at post-treatment using retrospective open-ended reports, it will provide researchers and clinicians with a comprehensive picture of the negative effects associated with ICBT. Lastly, from the ICBT client perspective, negative effects are relevant to informed consent. That is, ICBT clients should be informed about the possible negative effects associated with treatment prior to formal enrolment so they can make an informed decision about the risks and benefits of participating.

Table 4.

Suggested therapist responses to client-reported negative effects.

Negative effect Suggested therapist response
Patient-related negative effects
Negative emotional states Provide psychoeducation (e.g., normalize); encourage practice (e.g., reassure client that practicing skills will lessen negative emotional states)
Dropout Phone client to discuss; build rapport (e.g., address client concerns); provide feedback (e.g., on symptom progress); facilitate understanding (e.g., ask questions or provide suggestions to help client)
Non-response Phone client to discuss; build rapport (e.g., address client concerns); provide feedback (e.g., on symptom progress); facilitate understanding (e.g., ask questions or provide suggestions to help client)



Treatment-related negative effects
Therapist support concerns Build rapport (e.g., address client concerns); facilitate understanding (e.g., ask questions or provide suggestions to help client); clarify administrative procedures (e.g., provide clear instruction about therapist support)
Content/format problems Build rapport (e.g., address client concerns); facilitate understanding (e.g., ask questions or provide suggestions to help client); clarify administrative procedures (e.g., explain program format)
Implementation problems Build rapport (e.g., address client difficulties); provide psychoeducation (e.g., normalize challenges); facilitate understanding (e.g., ask questions or provide suggestions to help client succeed at implementation); encourage practice (e.g., reinforce the value of practice)
Technical difficulties Build rapport (e.g., address client difficulties); clarify administrative procedures (e.g., provide instructions on how to overcome technical difficulties)
Questionnaire concerns Build rapport (e.g., address client concerns); facilitate understanding (e.g., ask questions or provide suggestions to help clients complete questionnaires)
Open in a new tab

Note: Based on the recommended ICBT therapist behaviours described in Hadjistavropoulos et al. (2018).

4.1. Limitations

The current study had several notable limitations. First, we were not able to explore how negative effects reported in client emails correspond with negative effects reported by the same clients at post-treatment, because the clinical trial design did not include open-ended negative effects questions at post-treatment. Such questions have been added to subsequent trials providing a future research direction. Second, we did not statistically compare negative effects for completers and non-completers due to the fact that the random sample of non-completers was not large enough to ensure adequate power. Future research should include a much larger sample size to ensure sub-analyses can be conducted and statistical corrections can be made to account for the larger number of correlations being run. Third, clients in the current study were not explicitly instructed to report negative effects in emails to their therapist, and it is possible that additional negative effects could have been reported if clients were prompted to do so. Finally, we cannot say definitively that the negative effects identified in the current study are the direct result of the ICBT program. As indicated by Rozental et al. (2016), it is difficult to establish a cause-effect relationship, given natural fluctuations in symptoms, unique client stressors, and variation in clients' attribution of negative effects. Given the exploratory nature of the current study, we chose to err on the side of inclusivity in our definition of negative effects, but future researchers may need to carefully delineate whether they choose to include only negative effects that are clearly a result of treatment.

4.2. Future directions

The exploration of negative effects associated with ICBT is in its infancy (Andersson et al., 2019; Holmes et al., 2018; Rozental et al., 2018). The findings of the current study make it clear that future research should focus on refining the terminology related to negative effects. For example, should a negative effect really be labelled a negative effect if it is not detrimental to treatment outcomes? Perhaps researchers should differentiate between negative effects (events that have an impact on outcomes) and negative experiences (events that are unrelated to outcomes yet still important to recognize). Additionally, more research that includes diverse ICBT programs, sophisticated statistical analyses, and larger sample sizes is needed to learn more about the nuances of negative effects.

The results of the current study also provide several specific avenues for future research pursuits. As mentioned previously, a valuable next step would be exploring how the negative effects reported in client emails corresponds with negative effects reported retrospectively at post-treatment using open-ended questions or the Negative Effects Questionnaire (Rozental et al., 2016; Rozental et al., 2019b). Additionally, comparison studies that examine differences in negative effects across groups would be useful. For example, researchers could compare completers to non-completers, therapist-assisted ICBT programs to self-guided ones, or face-to-face therapy to ICBT. With regard to treatment outcomes, future research is needed to elucidate which negative effects are most critical to outcomes, as well as determine if certain clients are more affected by negative effects (e.g., clients with low motivation or more severe symptoms) than others. Lastly, additional studies that examine therapist responsiveness to client-reported negative effects would be beneficial. Specifically, a measure could be created to train ICBT therapists and systematically monitor client emails for negative effects in routine practice.

5. Conclusions

In order to accurately inform clients about the risks and benefits of treatment, minimize harm, and improve the design and delivery of ICBT, negative effects cannot be ignored. Although total number of negative effects was not correlated with lesson completion, working alliance, treatment satisfaction, or symptom outcomes, over half the clients included in the current study reported experiencing at least one negative effect during treatment. This highlight the importance of training ICBT therapists to systematically monitor for and respond to negative effects as they arise as a supplement to retrospective post-treatment reports. Overall, the findings of the current study raise interesting questions about the nature and impact of negative effects. Future research, both qualitative and quantitative, is needed to gain a more nuanced understanding of negative effects associated with ICBT.

Acknowledgments

Acknowledgements

The authors would like to thank Dr. Swati Mehta for her assistance with data analysis, as well as the research staff, therapists, and clients associated with the Online Therapy Unit at the University of Regina.

Funding

This work was supported by funding provided by the Canadian Institutes of Health Research (reference # 293379 & 152917), Saskatchewan Health Research Foundation, and Rx & D Health Research Foundation. Funders had no involvement in the study design, collection, analysis, or interpretation of the data.

Declaration of competing interest

None.

Contributor Information

Kirsten M. Gullickson, Email: kgullick@unb.ca.

Heather D. Hadjistavropoulos, Email: hadjista@uregina.ca.

Blake F. Dear, Email: blake.dear@mq.edu.au.

Nickolai Titov, Email: nick.titov@mq.edu.au.

References

  1. Andersson G. Internet-delivered psychological treatments. Annu. Rev. Clin. Psychol. 2016;12:157–179. doi: 10.1146/annurev-clinpsy-021815-093006. [DOI] [PubMed] [Google Scholar]
  2. Andersson G., Titov N. Advantages and limitations of internet-based interventions for common mental disorders. World Psychiatry. 2014;13:4–11. doi: 10.1002/wps.20083. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Andersson G., Titov N., Dear B.F., Rozental A., Carlbring P. Internet-delivered psychological treatments: from innovation to implementation. World Psychiatry. 2019;18:20–28. doi: 10.1002/wps.20610. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Barak A., Hen L., Boniel-Nissim M., Shapira N. A comprehensive review and meta-analysis of the effectiveness of internet-based psychotherapeutic interventions. J. Technol. Hum. Serv. 2008;26:2–4. [Google Scholar]
  5. Bendelin N., Hesser H., Dahl J., Carlbring P., Nelson K.Z., Andersson G. Experiences of guided internet-based cognitive-behavioural treatment for depression: a qualitative study. BMC Psychiatry. 2011;11(1):107. doi: 10.1186/1471-244X-11-107. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Boettcher J., Rozental A., Andersson G., Carlbring P. Side effects in internet-based interventions for social anxiety disorder. Internet Interv. 2014;1(1):3–11. [Google Scholar]
  7. Bystedt S., Rozental A., Andersson G., Boettcher J., Carlbring P. Clinicians' perspectives on negative effects of psychological treatments. Cogn. Behav. Ther. 2014;43(4):319–331. doi: 10.1080/16506073.2014.939593. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Crawford M.J., Thana L., Farquharson L., Palmer L., Hancock E., Bassett P.…Parry G.D. Patient experience of negative effects of psychological treatment: results of a national survey. Br. J. Psychiatry. 2016;208:260–265. doi: 10.1192/bjp.bp.114.162628. [DOI] [PubMed] [Google Scholar]
  9. Dear B.F., Staples L.G., Terides M.D., Karin E., Zou J., Johnston L.…Titov N. Transdiagnostic versus disorder-specific and clinician-guided versus self-guided internet-delivered treatment for generalized anxiety disorder and comorbid disorders: a randomized controlled trial. J. Anxiety Disord. 2015;36:63–77. doi: 10.1016/j.janxdis.2015.09.003. [DOI] [PubMed] [Google Scholar]
  10. Dimidjian S., Hollon S.D. How would we know if psychotherapy were harmful? Am. Psychol. 2010;65(1):21–33. doi: 10.1037/a0017299. [DOI] [PubMed] [Google Scholar]
  11. Ebert D.D., Donkin L., Andersson G., Andrews G., Berger T., Carlbring P.…Cuijpers P. Does internet-based guided-self-help for depression cause harm? An individual participant data meta-analysis on deterioration rates and its moderators in randomized controlled trials. Psychol. Med. 2016;46(13):2679–2693. doi: 10.1017/S0033291716001562. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Ebert D.D., Van Daele T., Nordgreen T., Karekla M., Compare A., Zarbo C.…Baumeister H. Internet- and mobile-based psychological interventions: applications, efficacy, and potential for improving mental health. Eur. Psychol. 2018;23(2):167–187. [Google Scholar]
  13. Elo S., Kyngäs H. The qualitative content analysis process. J. Adv. Nurs. 2008;62(1):107–115. doi: 10.1111/j.1365-2648.2007.04569.x. [DOI] [PubMed] [Google Scholar]
  14. Eubanks C.F., Burckell L.A., Goldfried M.R. Clinical consensus strategies to repair ruptures in the therapeutic alliance. J. Psychother. Integr. 2018;28(1):60–76. doi: 10.1037/int0000097. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Faul F., Erdfelder E., Lang A.-G., Buchner A. G*power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav. Res. Methods. 2007;39(2):175–191. doi: 10.3758/bf03193146. [DOI] [PubMed] [Google Scholar]
  16. Fernández-Álvarez J., Díaz-García A., González-Robles A., Baños R., García-Palacios A., Botella C. Dropping out of a transdiagnostic online intervention: a qualitative analysis of client's experiences. Internet Interv. 2017;10:29–38. doi: 10.1016/j.invent.2017.09.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Hadjistavropoulos H.D., Nugent M.M., Alberts N.M., Staples L., Dear B.F., Titov N. Transdiagnostic internet-delivered cognitive behaviour therapy in Canada: an open trial comparing results of a specialized online clinic and nonspecialized community clinics. J. Anxiety Disord. 2016;42:19–29. doi: 10.1016/j.janxdis.2016.05.006. [DOI] [PubMed] [Google Scholar]
  18. Hadjistavropoulos H.D., Schneider L.H., Klassen K., Dear B.F., Titov N. Development and evaluation of a scale assessing therapist fidelity to guidelines for delivering therapist-assisted internet-delivered cognitive behaviour therapy. Cogn. Behav. Ther. 2018;47(6):447–461. doi: 10.1080/16506073.2018.1457079. [DOI] [PubMed] [Google Scholar]
  19. Hatcher R.L., Gillaspy A. Development and validation of a revised short version of the Working Alliance Inventory. Psychother. Res. 2006;16(1):12–25. [Google Scholar]
  20. Holmes E.A., Ghaderi A., Harmer C.J., Ramchandani P.G., Cuijpers P., Morrison A.P.…Craske M.G. The Lancet Psychiatry Commission on psychological treatments research in tomorrow's science. Lancet Psychiatry. 2018;5(3):237–286. doi: 10.1016/S2215-0366(17)30513-8. [DOI] [PubMed] [Google Scholar]
  21. Hsieh H.-F., Shannon S.E. Three approaches to qualitative content analysis. Qual. Health Res. 2005;15(9):1277–1288. doi: 10.1177/1049732305276687. [DOI] [PubMed] [Google Scholar]
  22. Kroenke K., Spitzer R.L., Williams J.B.W. The PHQ-9: validity of a brief depression severity measure. J. Gen. Intern. Med. 2001;16:606–613. doi: 10.1046/j.1525-1497.2001.016009606.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Olsson Halmetoja C., Malmquist A., Carlbring P., Andersson G. Experiences of internet-delivered cognitive behavior therapy for social anxiety disorder four years later: a qualitative study. Internet Interv. 2014;1(3):158–163. [Google Scholar]
  24. Parry G.D., Crawford M.J., Duggan C. Iatrogenic harm from psychological therapist - time to move on. Br. J. Psychiatry. 2016;208:210–212. doi: 10.1192/bjp.bp.115.163618. [DOI] [PubMed] [Google Scholar]
  25. Rozental A., Andersson G., Boettcher J., Ebert D.D., Cuijpers P., Knaevelsrud C.…Carlbring P. Consensus statement on defining and measuring negative effects of internet interventions. Internet Interv. 2014;1(1):12–19. [Google Scholar]
  26. Rozental A., Boettcher J., Andersson G., Schmidt B., Carlbring P. Negative effects of internet interventions: a qualitative content analysis of patients' experiences with treatments delivered online. Cogn. Behav. Ther. 2015;44(3):223–236. doi: 10.1080/16506073.2015.1008033. [DOI] [PubMed] [Google Scholar]
  27. Rozental A., Kottorp A., Boettcher J., Andersson G., Carlbring P. Negative effects of psychological treatments: an exploratory factor analysis of the Negative Effects Questionnaire for monitoring and reporting adverse and unwanted events. PLoS One. 2016;11(6) doi: 10.1371/journal.pone.0157503. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Rozental A., Magnusson K., Boettcher J., Andersson G., Carlbring P. For better or worse: an individual patient data meta-analysis of deterioration among participants receiving internet-based cognitive behavior therapy. J. Consult. Clin. Psychol. 2017;85(2):160–177. doi: 10.1037/ccp0000158. [DOI] [PubMed] [Google Scholar]
  29. Rozental A., Castonguay L., Dimidjian S., Lambert M., Shafran R., Andersson G., Background P.C. Negative effects in psychotherapy: commentary and recommendations for future research and clinical practice. BJPsych Open. 2018;4:307–312. doi: 10.1192/bjo.2018.42. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Rozental A., Andersson G., Carlbring P. In the absence of effects: an individual patient data meta-analysis of non-response and its predictors in Internet-based cognitive behavior therapy. Front. Psychol. 2019:10. doi: 10.3389/fpsyg.2019.00589. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Rozental A., Kottorp A., Forsstrom D., Mansson K., Boettcher J., Andersson G.…Carlbring P. The Negative Effects Questionnaire: psychometric properties of an instrument for assessing negative effects in psychological treatments. Behav. Cogn. Psychother. 2019:1–14. doi: 10.1017/S1352465819000018. [DOI] [PubMed] [Google Scholar]
  32. Scott J., Young A.H. Psychotherapies should be assessed for both benefit and harm. Br. J. Psychiatry. 2016;208(208–209) doi: 10.1192/bjp.bp.115.169060. [DOI] [PubMed] [Google Scholar]
  33. Spitzer R.L., Kroenke K., Williams J.B.W., Lowe B. A brief measure for assessing generalized anxiety disorder. Arch. Intern. Med. 2006;166:1092–1097. doi: 10.1001/archinte.166.10.1092. [DOI] [PubMed] [Google Scholar]
  34. Titov N., Dear B.F., Staples L.G., Terides M.D., Karin E., Sheehan J.…Mcevoy P.M. Disorder-specific versus transdiagnostic and clinician-guided versus self-guided treatment for major depressive disorder and comorbid anxiety disorders: a randomized controlled trial. J. Anxiety Disord. 2015;35:88–102. doi: 10.1016/j.janxdis.2015.08.002. [DOI] [PubMed] [Google Scholar]
  35. Titov N., Dear B., Nielssen O., Staples L., Hadjistavropoulos H., Nugent M.…Kaldo V. ICBT in routine care: a descriptive analysis of successful clinics in five countries. Internet Interv. 2018;13:108–115. doi: 10.1016/j.invent.2018.07.006. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from Internet Interventions are provided here courtesy of Elsevier

RESOURCES