Skip to main content
NCBI home page
Internet Interventions logoLink to Internet Interventions
. 2021 Oct 14;26:100469. doi: 10.1016/j.invent.2021.100469

Negative effects in internet-based interventions for depression: A qualitative content analysis

Friederike Fenski a,⁎, Alexander Rozental b, Manuel Heinrich a, Christine Knaevelsrud a, Pavle Zagorscak a, Johanna Boettcher c,d
PMCID: PMC8564554  PMID: 34754755

Abstract

An increasing number of studies is proving the efficacy of Internet-based interventions (IBI) for treating depression. While the focus of most studies is thereby lying on the potential of IBI to alleviate emotional distress and enhance well-being, few studies are investigating possible negative effects that might be encountered by participants. The current study was therefore exploring self-reported negative effects of participants undergoing a cognitive-behavioral IBI targeting mild to moderate depression over 6 weeks. Data from the client pool of a German insurance company (n = 814, 68% female) revealed that 8.6% of the participants reported the experience of negative effects. Qualitative content analysis yielded two broad categories and five subcategories for the nature of participants' experiences of negative effects: participant-related negative effects (insight and symptom) and program-related negative effects (online format, contact, and implementation). By using both, qualitative and quantitative methods, results did not only shed light on the characteristics of negative effects but analyses also found that working alliance was a predictor for the experience of negative effects. Monitoring the occurrences of negative effects as well as working alliance throughout treatment was considered essential to help prevent negative effects and attrition among participants undergoing IBI for depression.

Abbreviation: IBI, Internet-based interventions

Keywords: Internet-based interventions, Cognitive-behavioral therapy, Depression, Qualitative research, Negative effects

Highlights

  • •

    Negative effects in an Internet-based intervention for depression were assessed.

  • •

    8.6% of the participants reported the experience of negative effects.

  • •

    Negative effects were not necessarily related to depressive symptoms reduction.

  • •

    Working alliance was a predictor for the experience of negative effects.

  • •

    Monitoring negative effects throughout treatment might prevent drop-out rates.

1. Theoretical background

Research on Internet-based interventions (IBI) has demonstrated their efficacy in treating various mental health conditions, including depression (Andrews et al., 2018; Richards and Richardson, 2012). Meta-analyses have proven IBI to be equally effective as face-to-face treatments (Andrews et al., 2018; Carlbring et al., 2018), with several potential advantages, such as increased access to evidence-based treatments, the possibility of higher cost-effectiveness, and greater opportunity to reach patients in remote locations (Andersson and Titov, 2014).

Despite many promising results, IBI might not be suitable for all participants and carry the possibility to encounter negative effects (Rozental et al., 2014). For years, research on psychological treatments has focused almost exclusively on positive outcomes, while paying nearly no attention to the possible existence of negative effects (Barlow, 2010). Besides limited research in this field, there is still no consensus in the literature on how to define and measure negative effects (Peterson et al., 2013). Only in recent years have researchers begun to formulate a consensus to guide future research in this area (Linden and Schermuly-Haupt, 2014; Peterson et al., 2013; Rozental et al., 2014). Several questionnaires and checklists have been developed to distinguish negative effects occurring in face-to-face treatments (Linden and Schermuly-Haupt, 2014), including measures of the deterioration of symptoms, the emergence of new symptoms, and stigmatization (Linden, 2013) as well as ineffective treatment and the dislike of the therapist (Parker et al., 2013). Besides these, strains in family relations (Schermuly-Haupt et al., 2018), social stigmatization, nonresponse to the treatment, and becoming dependent on the treatment or the therapist are negative effects that might occur during psychological treatments (Dimidjian and Hollon, 2010). Most often mentioned in the literature of negative effects is the deterioration of targeted symptoms (Barlow, 2010; Rozental et al., 2014). Several studies suggest that between 5 and 10% of all participants undergoing psychological treatments deteriorate (Hansen et al., 2002; Rozental et al., 2017; Vittengl et al., 2016). However, this does not necessarily have to be the result of the treatment but may be related to other causes (Dimidjian and Hollon, 2010), illustrating the importance to compare these rates with those in control conditions. The results of a recent meta-analysis on the effects of psychotherapy for depression indicate that deterioration rates are below 5% in the treatment condition and between 7% and 13% in control conditions (care-as-usual, waitlist, or pill placebo; Cuijpers et al., 2021). Being aware of various definitions existing in the literature today, the current article uses the term negative effect as a description of every possible event a participant might encounter during the course of treatment that is experienced as negative and perceived as related to the treatment by the participant.

Lately, clinical trials of IBI have started to examine the occurrence and characteristics of negative effects, mostly focusing on deterioration rates. Results of recent meta-analyses have shown clinically significant deterioration rates of IBI for different psychiatric disorders of 3.6% (Ebert et al., 2016) to 5.8% in treatment conditions (Karyotaki et al., 2018; Rozental et al., 2017) compared to 9.1% (Karyotaki et al., 2018) to 17.4% in control conditions (Rozental et al., 2017). Besides the very relevant question of how many participants deteriorate when undergoing online treatment, it is also important to investigate whether there are specific negative effects associated with this new treatment format. Qualitative studies using open-ended questions have the potential to identify negative effects that may be specifically related to the online format. A study assessing negative effects in cognitive-behavioral IBI for social anxiety disorder (SAD) found that 14% of participants experience different forms of negative effects, including the deterioration of targeted symptoms, the emergence of new symptoms, and not understanding the treatment rationale (Boettcher et al., 2014). Further, in an evaluation study of IBI for SAD participants lamented that the intervention was difficult to understand at times and that participation was emotionally demanding and associated with hard work (Halmetoja et al., 2014). Rozental et al. (2015) used a qualitative content analysis to evaluate self-reported negative effects that were described by participants from four clinical trials investigating the efficacy of IBI for different mental disorders and for procrastination. In the four clinical trials, participants answered open-ended questions concerning the occurrence and characteristics of negative effects at post-treatment assessment (Rozental et al., 2015). Results showed that 9.3% of all participants reported the experience of at least one negative effect that might be related to treatment (Rozental et al., 2015). A qualitative content analysis of the participants' responses yielded both patient- and treatment-related negative effects. Each of these contained two subcategories: insight and symptoms, and implementation and format. While the subcategory insight included a greater understanding and awareness of the ongoing condition, the subcategory symptoms derived from themes such as the increase of symptom severity as well as participants' stress levels. Finally, the subcategory implementation involved difficulties because of technical problems or troubles understanding the treatment rationale, while the subcategory format resulted from themes such as frustration or time pressure. These studies offer important insights into the nature of negative effects in online treatments. However, sample sizes of the individual studies were rather small and, as negative effects only occur in a small minority of participants, larger sample sizes are needed to get a more robust picture of negative effects in online settings.

In addition to the question of what kind of negative effects may be induced by online treatment, it is also important to investigate potential predictors of negative effects (Rozental et al., 2014). When examined in an IBI for SAD, relationship status, age, and gender did not show any significant effects on the occurrence of negative effects (Boettcher et al., 2014). However, a meta-analysis of 29 clinical trials of Internet-based CBT focusing on deterioration rates as an indicator for negative effects found higher educational level, older age, and being in a relationship to be associated with lower odds of deterioration (Rozental et al., 2017). The present article aims at shedding light on the question of which subgroups of participants may be more prone to experience negative effects by analyzing correlates of negative effects in a large sample of individuals with mild to moderate depression symptom load.

Beyond sociodemographic variables, the amount of therapeutic guidance might be another plausible predictor for the experience of negative effects. While results of several meta-analyses support the idea that greater amount of therapist's support in IBI targeting depression and anxiety relates positively to outcome (Baumeister et al., 2014; Andersson and Cuijpers, 2009; Spek et al., 2007; Richards and Richardson, 2012), the relationship between the amount of therapist support and the experience of negative effects remains insufficiently researched. A recent study comparing standard weekly versus optional weekly therapist support in an IBI targeting depression and anxiety did not find a significant difference between the groups regarding self-reported negative effects (Hadjistavropoulos et al., 2017). However, lack of therapeutic support could potentially jeopardize compliance as the progress of participants is not monitored by the therapist (Newman et al., 2011). Since research on this particular topic is scarce, the current study aims at shedding further light on the influence of guidance on the occurrence of negative effects.

Not only the mere presence of therapist support, but especially the quality of the therapeutic relationship, or working alliance, is a factor that is widely regarded as important in IBI research (Andersson and Titov, 2014; Berger, 2017). A study of Parker et al. (2013) clearly emphasized negative therapeutic alliance as a key risk factor for encountering negative effects in face-to-face treatments. Similar to that, participating psychotherapists in a study by Bystedt et al. (2014) presumed the working alliance to be related with the occurrence of negative effects in psychological treatments. In detail, therapists stressed the participant's need to understand the treatment rationale and the explicit goals of the treatment what suggests that not only guidance per se, but also the quality of the therapeutic relationship is associated with the experience of negative effects (Bystedt et al., 2014).

In line with the suggestion of Dimidjian and Hollon (2010), the present study is going to apply both qualitative as well as quantitative methods to assess if and how participants undergoing cognitive-behavioral IBI for mild to moderate depression experience self-reported negative effects in order to understand their own comprehension of these effects, how these effects might influence their well-being and treatment outcome. To help future research in this field, the current study also investigates if and how negative effects are addressed with the online counsellor and how this is portrayed by the participants. Finally, this study will examine whether various sociodemographic and clinical variables, the amount of contact with the online counsellor, and working alliance are associated with the experience of negative effects.

2. Methods

2.1. Participants

The data was collected as part of an RCT conducted in Germany to determine the benefit of individualizing feedback messages after completion of the treatment modules of a cognitive-behavior-based IBI (Zagorscak et al., 2018). The Research Ethics Committee of the Freie Universität Berlin approved the protocol. The RCT was preregistered at URL https://www.anzctr.org.au (ID: ACTRN12614000312640). Individuals were recruited among clients of a German health insurance company.

Participants had to meet criteria for mild to moderate depression (scores between 14 and 28) according to the Beck Depression Inventory II (BDI-II; Hautzinger et al., 2006) and not be at risk for suicide (score ≤ 1 on BDI-II item 9). Additionally, participants with current mania/hypomania or psychotic symptoms (lifetime) were excluded after a structured clinical interview by phone (SCID-I). A detailed flow-chart is published in Zagorscak et al. (2018). A total of N = 1089 participants took part in the trial. While participants of treatment Arm 1 received individualized feedback (IF group), individuals of Arm 2 received a fully standardized and generic feedback after completing a treatment module (SF group). At any time during the intervention, participants of both treatment-conditions could receive individual contact on demand in case of technical problems or specific questions or problems concerning the intervention content. Additional individual contact in both conditions was provided in case of symptom exacerbation (i.e., higher PHQ-9 score compared to last screening in 2 consecutive measurements) or if participants rated item 9 of PHQ-9 ≥ 1 (“Thoughts that you would be better off dead or hurting yourself in some way”) in weekly symptom screenings. In that case, the responsible counsellor was immediately notified via e-mail to contact the participant and take appropriate actions.

The current study only uses data of 814 (74.7%) participants that answered the question about experiences with negative effects at post-treatment assessment. Sociodemographic variables were assessed pre-treatment within a screening and are summarized in Table 1. These participants did not differ significantly from the participants who did not answer the question about negative effects at post-treatment assessment regarding their pre-treatment depression, t(452) = −1.79, p = .07 and anxiety levels, t(468) = −0.19, p = .85. However, participants answering the question about negative effects were older, t(445) = 3.96, p < .001, d = 0.29, more often female than male, χ2(1, N = 1089) = 6.08, p = .014, OR = 1.44 (CI: 1.09–1.91), and showed differences within their work situation, Fisher's Exact Test, p = .005. There were no differences regarding group affiliation (SF or IF; χ2(1, N = 1089) = 1.46, p = .23).

Table 1.

Sociodemographic sample characteristics.

Variable All (N = 814) No negative effects (n = 744) Negative effects (n = 70) Statistical
Test p
Mean agea (SD) 45.9 (11) 45.8 (11) 47.1 (11) −0.98b 0.33
Female gender, n (%) 551 (68) 498 (67) 53 (76) 1.87c 0.17
Education, n (%) 0.40d
 No certificate 2 (0.3) 2 (0.2) 0
 Secondary school 249 (30.6) 229 (30.8) 20 (28.6)
 High school 179 (21.9) 168 (22.6) 11 (15.7)
 College/University 384 (47.2) 345 (46.4) 39 (55.7)
Work Situation, n (%) 0.37d
 Working 653 (80.2) 598 (80.4) 55 (78.6)
 Studying/training 22 (2.7) 21 (2.8) 1 (1.4)
 Unemployed 56 (6.9) 53 (7.1) 3 (4.3)
 Other 83 (10.2) 72 (9.7) 11 (15.7)
Living Situation, n (%) 2.2c 0.14
 Living alone 202 (24.8) 179 (24.1) 23 (32.8)
 Living with family/roommate 612 (75.2) 565 (75.9) 47 (67.2)
Open in a new tab
a

Five cases providing implausible values for age were excluded.

b

t-test for independent samples.

c

χ2-test.

d

Fisher's Exact Test.

2.2. Intervention and procedure

Participants completed the cognitive-behavioral IBI consisting of seven consecutive modules over a period of six to eight weeks. Each module was based on well-established cognitive-behavioral techniques (e.g. psychoeducation, thought records, and behavioral experiments) and consisted of four parts: (1) psychoeducation, (2) online tools to complete the tasks at hand (3) a feedback letter, and (4) introduction of homework (see Zagorscak et al., 2018 for a more detailed description). In the IF group, each participant was assigned to a personal counsellor who provided semi-standardized written feedback on the platform after each completed treatment module. In the SF group, feedback was provided automatically in a general, standardized, and non-individualized form. Treatment material and psychoeducation did not differ between groups. Both groups could receive additional individual contact on demand.

2.3. Measures

Negative effects. The experience of negative effects occurring during and due to the intervention was measured post-treatment by a set of nine questions consisting of a mix of open-ended and closed questions (all items can be found in Table 2). Participants were asked whether they had experienced any form of negative effect or event that they attributed to treatment. They were asked to describe the effect and to rate the negative impact of the effect. Participants also described at which point in the course of treatment the negative effect occurred and whether they discussed the effect with their online counsellor/the study team.

Table 2.

Items measuring the experience of negative effects.

Item Response format
1. During the period of the intervention, did you experience any negative effect that you would attribute to the treatment? Yes/no (if “no” end of questionnaire)
2. Please describe this negative effect. Open answer
3. When did this negative effect occur during the treatment? Open answer
4. Would you relate this negative effect to a specific module or exercise of the treatment? Yes/no (if “no” continue with item 6)
5. If yes, with which one? Open answer
6. How much did this negative effect affect your well-being at the time it occurred? 4-point Likert scale (0 = not at all – 3 = strongly)
7. How much does the negative effect affect your well-being today? 4-point Likert scale (0 = not at all – 3 = strongly)
8. Did you mention the occurrence of the negative effect to the team? Yes/no
9. If no, why not? If yes, how did you experience that? Open answer
Open in a new tab

Outcome measures. This article concentrates on the main outcome measures of the original study (change in depression and anxiety) that were assessed online at pre-treatment, post-treatment and 3-months-follow-up (see Zagorscak et al., 2018 for an overview of outcome measures and assessment points). Depressive symptoms were assessed with the German version of the PHQ-9 (Löwe et al., 2002). The PHQ-9 measures frequency of nine depressive symptoms over the past 2 weeks according to DSM-IV diagnostic criteria. The PHQ-9 score ranges from 0 to 27, as each of the 9 items is scored from 0 (not at all) to 3 (nearly every day). Anxiety symptoms were measured using the German version of the GAD-7 (Spitzer et al., 2006). The GAD-7 measures frequency of the seven core symptoms of generalized anxiety disorder over the past 2 weeks according to DSM-IV diagnostic criteria. Response options range from 0 (not at all) to 3 (nearly every day), thus, resulting in GAD-7 scores ranging from 0 to 21.

Working alliance was measured mid-treatment with the German version of the 12-item WAI-SR by Wilmers et al. (2008) that was adapted for online self-help programs (Berger et al., 2014). The WAI-SR measures the three core dimensions of Bordin's theory of working alliance (bond, task, goal; Bordin, 1979). In this study, items of the bond subscale of the WAI-SR for the SF group were changed into asking for the degree of mutual trust, acceptance, and confidence with the research team responsible for the intervention, and not with the counsellor. Participants were asked to rate each statement on a 5-point rating scale ranging from 1 (rarely) to 5 (always).

Adherence. The platform hosting the intervention automatically tracked several quantitative indicators of the participants' intensity of treatment use. The frequency of logins and the time spent on the platform (login-duration) was counted for each treatment module. Participants could access the treatment module they currently work with as well as the content of completed modules at any time. There was no upper bound regarding the number of logins and minutes participants were allowed to spend working on treatment tasks. In addition, every participant's number of modules started was tracked.

2.4. Statistical analyses

2.4.1. Qualitative analysis

To explore participants' responses to the open-ended questions regarding negative effects, qualitative content analysis was applied (Krippendorff, 2012). We used an inductive approach because of the lack of research in this field, as to our knowledge there is no study today focusing on negative effects in IBI specifically targeting depression, nor one including the examination of participants' decisions to address negative effects with their therapist. Krippendorff (2012) mentions four steps that are usually undertaken during qualitative content analysis: (1) the sampling of the data; (2) the unitizing of the data in terms of words or propositions; (3) bringing the data in context with the researcher's own comprehension of the context in which the data has been taken; and (4) relating the findings to the research question.

Qualitative content analysis was conducted using Excel Version 16.10. Within the process, open-ended questions were read and unitized into units that captured the meaning of the entire response of a participant. Units then were coded according to themes which in turn lead to categories and subcategories. To ensure the retainment of the original meaning conveyed by the participants, individual quotations were reread and compared to the wording of subsequent themes, subcategories, and categories throughout the analysis. Because participants provided statements in German, all themes, subcategories, and categories first emerged in German and were only translated into English after the completion of analysis. Translations were discussed in a peer-group setting to ensure that the original meaning was retained.

2.4.2. Quantitative analysis

Quantitative data analyses were carried out using R Version 3.3.2 (R Core Team, 2016), making use of the following packages: car (Fox and Weisberg, 2011) and lme4 (Bates et al., 2015).

Sociodemographic characteristics. Descriptive statistics were used to summarize sociodemographic data. Baseline group differences in sociodemographic characteristics were investigated using χ2-tests, t-tests and Fisher's Exact Test.

Negative effects and outcome. Linear mixed-effects models were used to evaluate the effect of the experience of negative effects on the change of symptoms of depression and anxiety. The experience of negative effects (yes/no), time (pre-, post-treatment, and 3-month follow-up), and the interaction of negative effects × time were entered as fixed effects, whereby the intercept was allowed to vary across individuals. Within this analysis, missing data on outcome measure were handled using FIML.

Adherence. First analyses revealed that some participants displayed irregular patterns of uptake. For example, 95% of the participants did not log in more than five times at module 1; however, some participants logged in >30 times (maybe due to technical problems). Consequently, the upper 5% of login-frequencies and login-durations were replaced with less extreme values while maintaining the ordering of all individuals within the sample (Winsorizing; Wilcox, 2003).

Negative effects and adherence. To analyze the association of negative effects with the uptake of specific treatment components, three independent t-tests were calculated. Thereby, the experience of negative effects (yes/no) was entered as independent variable. In the first analysis, the number of modules started by the participants was entered as a dependent variable; the number and frequencies of logins were respectively entered as dependent variables in the second and third analyses.

Predictors of negative effects. To examine whether group affiliation or working alliance ratings predicted negative effects, two logistic regression analyses were carried out. Experiencing negative effects (yes/no) was entered as dependent variable. Group affiliation (SF/IF) was entered as independent variable in the first logistic regression and WAI total scores of N = 683 participants providing data at mid-treatment were entered as independent variable in the second logistic regression.

3. Results

3.1. Frequency and ratings of negative effects

Of the 814 participants who were included in the current article, 70 (8.6%) reported that they experienced at least one negative effect that might be related to the treatment. In terms of the average degree to which the negative effects affected the participants' well-being, when it occurred during treatment, the results yielded a rating of M = 2.24 (SD = 0.65). Additionally, participants were asked how the experience of negative effects was still affecting the participants' well-being at post-treatment assessment. The rating of M = 1.17 (SD = 0.95) indicates a possible decrease between the time the negative effect occurred and the end of treatment, which was also confirmed by a dependent t-test, t(69) = 9.89, p < .001, d = 1.18.

3.2. Qualitative content analysis

3.2.1. Description of negative effects

In total, 70 participants indicated that they had experienced some kind of negative effect during treatment. Hence, only a small minority of participants (8.6%) experienced negative effects. One statement could not be categorized as it was describing a positive event (gaining the courage to talk about eating behaviors). A qualitative content analysis of the remaining 69 descriptions yielded two categories: program-related negative effects and participant-related negative effects. While program-related negative effects were linked to the structure of the online-intervention and its effects on participants, participant-related negative effects emerged from participants' feelings due to the natural course of psychotherapy treatment, illustrating a process in which deterioration, novel symptoms and gaining awareness about one's situation are relatively common. Program-related negative effects were derived from three subcategories (online format, contact, and implementation), whereas participant-related negative effects emerged from two subcategories (symptom and insight). Related to the subcategory online format were themes such as problems with logging in the program or time pressure due to tight time schedules. Themes such as automated generated answers and impersonal interactions were connected to the subcategory contact, while self-doubts, stress and frustration were themes associated with the implementation of the program. Different types of distress such as sadness or sweating were themes connected to the subcategory symptom, while memories and a greater awareness of the condition were related to the subcategory insight.

3.2.1.1. Program-related negative effects

A majority of participants (n = 43) who reported the experience of negative effects described these occurrences as directly linked to the program itself. For instance, feelings of frustration and anger were mentioned due to problems with technical features, difficulties understanding the treatment rationale, or troubles implementing exercises. Several participants of those reporting the experience of negative effects from the SF group experienced the lack of personal contact within the IBI as negative.

Online format. The fact that the current intervention was performed in a linear fashion (i.e., one module after another in a pre-defined order) within a pre-defined time schedule seemed to displease eighteen participants resulting in negative feelings towards the online format. Participants criticized the experience of time pressure and stress resulting from the schedule of the intervention and the reminders they received via e-mail: ‘Stress when filling out the day planner. Stress when receiving e-mails with the request to log in the online program, when I would have liked to deal with the exercise at a later time point’ (participant 292013). Additionally, some participants mentioned technical problems such as lost data or a broken computer that resulted in sadness or anger. There were also complications mentioned when logging into the program that made it impossible for some participants to continue treatment.

Contact. Eleven participants from the SF group declared the absence of personal contact as negative: ‘The answers were automatic, impersonal, and sometimes even incongruous, that bothered me’ (participant 289636). In some cases when participants received standardized feedbacks to letters that were challenging to write, statements indicated a perceived imbalance of the feelings they put into words and the short answer they received that caused the participants to feel left alone with their condition or resulting in anger towards the program. Additionally, there were three participants from the IF group criticizing the insufficient support as ‘too impersonal’ (participant 293145).

Implementation. A third subcategory included statements of eleven participants who experienced negative effects because they could not apply the program in the way they would have wanted to. Some of these participants described difficulties to perform certain exercises within the program as they were perceived as too demanding resulting in stress and frustration. In some cases, these complications generated self-doubts and sadness as the participants thought that they were failing the treatment, which was often attributed to personal characteristics: ‘This feeling that I couldn't fill out the day planner as demanded and how others would do. This feeling that I couldn't follow the program.’ (participant 289838).

3.2.1.2. Participant-related negative effects

Not all negative effects were directly linked to the program's features; 26 participants related negative effects rather to themselves, like the worsening of symptoms or the emergence of new symptoms.

Symptom. Fifteen participants described how they experienced symptoms of depression or anxiety during the treatment. Most symptoms were thereby described as psychological feelings of discomfort (‘I was ashamed and felt worthless’, participant 2505), while few others were explained as physical (‘Crying, strong inner unrest, sweating’, participant 291361). Participants also mentioned a deterioration of initial symptoms: ‘Worsening of the mood and the negative feelings’ (participant 1510).

Insight. Eleven participants described negative feelings emerging when dealing with their situation and gaining awareness of underlying reasons behind some of their problems. For some participants, addressing their condition and facing negative feelings and memories often resulted in great sadness and depression, while other participants mentioned disappointment and inner unrest. One participant described ‘[t]hat negative things (thoughts, situations) were brought up again. Insights that weren't there before and that bothered me. Thoughts go around in circles. Not being able to sleep because of the thoughts that go around in circles. Inner unrest and being agitated’ (participant 289567).

3.2.2. Addressing negative effects

3.2.2.1. Reasons for not addressing negative effects

Of the 70 participants who reported the experience of negative effects, 47 (67%) did not address the occurrences to an online counsellor. Of these 47 participants, 25 (53%) belonged to the SF group. A total of 42 participants provided explanations that could be categorized into two categories: program-related reasons and participant-related reasons. In turn, each of these categories contained two subcategories: means and support, as well as coping and discounting.

Program-related reasons. Means. Twelve participants stated that they did not see any possibility to contact a member of the research team. As main reasons participants indicated problems with technical features of the online tool and the absence of visible means to reach a member of the research team: ‘At this stage of the program, there was no apparent possibility for me to get in contact with someone from the program’ (participant 2723).

Support. When describing their experiences with the occurrence of negative effects, eleven participants emphasized the lack of individual contact. Consequently, this feeling of insufficient support was not only declared as a negative effect per se but was also a reason for participants not to address their experiences of negative effects: “During the whole period, I did not feel any individual ‘support’” (participant 291361). Participants' explanations included feelings of disappointment and resignation due to the lack of personal interactions within the program: ‘Because I only received standardized responses written by a machine. I didn't have any expectations of someone actually reading my texts or even answering them (…)’ (participant 292261). Interestingly, the subcategory support did only emerge for participants from the SF group.

Participant-related reasons. Coping. When facing negative effects, twelve participants decided to handle the situation without considering the help of a member of the research team. In their statements, participants described how they managed the negative effects by themselves, either because they thought that these experiences were common when undergoing treatment: ‘I think that these experiences go with the treatment. I perceived my reaction as reasonable, although unpleasant.’ (participant 292541), or when they were feeling that they could cope with the situation without external help: ‘I didn't address it because I could manage the situation without help’ (participant 1489). One participant connected the unpleasant experience at that particular moment with potential benefits for the future: ‘Because I think that it is normal for the body to react in that way and I imagine that it will be positive in the long run’ (participant 4161). Discounting. The last subcategory includes statements of seven participants who dismissed the negative effects at the moment they occurred without providing any specific underlying reasons: ‘In that moment, I didn't consider it as noteworthy’ (participant 4059).

3.2.2.2. Experiences when addressing negative effects

Of 23 participants who did address these occurrences with their online counsellor, eight participants (35%) belonged to the SF group. One participant did not provide any description and the statements of two participants could not be categorized because they did not provide an explanation of their experiences but rather explained how they addressed these negative effects within the program (addressed in letters, described broadly). The analysis of the remaining 20 statements generated two categories: helpful and unhelpful.

Helpful. Eight participants mentioned that they experienced feelings of reassurance and relief. One participant from the SF group explained how decent it felt to receive individual support: ‘It felt good not to receive another general advice, but to get back an answer from a person who actual considered my message, my situation, me’ (participant 884800).

Unhelpful. The remaining twelve participants, however, regarded the interactions with the counsellor as rather unhelpful. While one participant directly stated the feeling of not being helped and another participant described the experience as ‘relatively indifferent’ (participant 289659), other participants experienced feelings of negative mood and emerging symptoms after turning to an online counsellor: ‘What's remaining is an uncomfortable aftertaste’ (participant 291717).

3.3. Quantitative results

Sociodemographic characteristics. Participants who reported negative effects and participants who did not report negative effects did not differ on any sociodemographic characteristics (all p > .14; see Table 1).

Depressive Symptoms. Individuals who reported negative effects did not differ significantly from participants who did not experience of negative effects with regard to depression severity at pre-treatment assessment, b = −0.26, t(1726.4) = −0.559, p = .576. For participants who reported negative effects and participants who did not report negative effects, the symptom severity declined from pre-treatment to post-treatment assessment, b = −4.92, t(1433.8) = −32.70, p < .001, as well as from pre- to 3-month follow-up assessment, b = −4.93, t(1505.0) = −29.82, p < .001. However, the amount of improvement from pre- to post-treatment, b = 2.21, t(1433.8) = 4.30, p < .001, as well as from pre-treatment to 3-month follow-up assessment, b = 1.84, t(1518.3) = 3.21, p = .001, was significantly lower for those who reported negative effects.

Anxiety Symptoms. At pre-treatment assessment, individuals who reported negative effects did not differ significantly from participants who did not experience of negative effects with regard to anxiety severity, b = 0.60, t(1724.6) = 1.34, p = .182. For participants who reported negative effects and participants who did not report negative effects, the symptom severity declined from pre-treatment to post-treatment assessment, b = −3.57, t(1429.8) = −25.07, p < .001, as well as from pre- to 3-month follow-up assessment, b = −3.63, t(1501.3) = −23.23, p < .001. However, the amount of improvement from pre- to post-treatment, b = 1.12, t(1429.8) = 2.29, p = .022, was significantly lower for those who reported negative effects. No difference in change emerged with regard to change from pre-treatment to 3-month follow-up assessment, b = 0.79, t(1514.6) = 1.45, p = .149.

Adherence. Analyses revealed that participants who did report the experience of negative effects were starting a significantly lower number of modules (M = 6.24, SD = 1.85) than participants who did not report the experience of negative effects (M = 6.78, SD = 0.97), t(73) = 2.39, p = .009, d = 0.5. However, additional independent t-tests showed no significant differences between participants who experienced negative effects and those who did not regarding login frequencies or login durations (all p-values > .10).

Group affiliation. From the 70 participants who reported the experience of negative effects, 30 (42.9%) participants belonged to the IF group whereas 40 (57.1%) participants belonged to the SF group. Analysis revealed that the chance to report negative effects was not significant higher for individuals from the SF condition, z = 1.61, p = .108, OR = 1.5 (CI: 0.92–2.48).

Working Alliance. Results showed that working alliance was a predictor for the experience of negative effects, z = −4.63, p < .001, OR = 0.40 (CI: 0.27–0.59). Participants with the experience of negative effects reported a significantly lower WAI score (M = 2.65, SD = 0.87) than participants without that experience (M = 3.17, SD = 0.72).

4. Discussion

The present study is the first to examine negative effects of an online treatment in a large sample of depressed participants. While the results of the qualitative content analysis provide information about the characteristics of negative effects, the results of the quantitative analyses show how experiencing negative effects affects depressive and anxiety symptoms as well as adherence. At post-treatment assessment, 8.6% of all participants reported the experience of negative effects that they connected to treatment, which is in line with rates found in comparable studies investigating negative effects in IBI for various disorders (4.9–12.6%; Rozental et al., 2015). Replicating findings of the study by Rozental et al. (2015), both participant- and program-related negative effects emerged. Similar to several studies examining negative effects in face-to-face-treatment and IBI (Rozental et al., 2015; Schermuly-Haupt et al., 2018), participants in the present sample reported the emergence of new symptoms as well as symptom deterioration. Also in line with previous research in IBI were negative effects associated with the online treatment format such as contact modalities and technical difficulties (Rozental et al., 2015).

Program-related negative effects connected to the online format such as technical problems or stress and frustration due to difficulties with the implementation of the tasks might possibly be prevented by developing more intuitive and user-friendly programs and by providing greater flexibility in terms of deadlines and session contents. As some of the participants of the current study were not aware of the possibility to get in contact with a member of the research team at any time, future programs should provide sufficient information on contact modalities. In the literature, increased contact to a therapist was found to be valuable for improving adherence and reducing dropout (Christensen et al., 2009) and could possibly prevent some of the frustration participants reported in the present study.

However, the amount of therapist support as indicated by group affiliation (IF/SF) was not a significant predictor for the experience of negative effects in the current sample. This is similar to a study by Hadjistavropoulos et al. (2017) which found no differences between participants of a standard support and an optional support group regarding unwanted negative effects. Although the current study did not find effects of therapeutic support on the frequency of self-reported negative effects, the qualitative content analysis of participants' descriptions of self-reported negative effects revealed the category contact, summarizing statements of participants who did not feel supported and valued enough due to the contact modalities. This is similar to a qualitative analysis investigating participants' experiences of a guided cognitive-behavioral IBI for depression, where themes as lack of support and insufficient support emerged from the data (Bendelin et al., 2011). Some participants in a qualitative study examining IBI for SAD directly expressed the wish for more support and feedback (Halmetoja et al., 2014); requests that also came up in the current analysis. Moreover, only participants from the SF group described that they did not get in contact with a member of the research team after experiencing negative effects because of the lack of support they received prior to these occurrences.

The current analysis found that the experience of negative effects was negatively associated with adherence. It therefore seems likely that some participants experiencing negative effects discontinue treatment and do not provide post-treatment data. Bendelin et al. (2011) suggested that support should be adjusted on the basis of individual preferences and specific motivational levels, which could potentially counteract frustration and dropout but requires further exploration in future studies.

Although the amount of therapeutic guidance per se was not a significant predictor for the experience of negative effects, the quality of the therapeutic relationship was associated with the experience of negative effects, as participants with lower ratings of the working alliance were more likely to report the occurrence of negative effects. This is in line with prior research that emphasized the association of a negative therapeutic relationship and negative effects (Parker et al., 2013). Moreover, in a content analysis of client e-mails in guided IBI for depression Svartvatten et al. (2015) found significant positive correlations between participants' statements highlighting a warm and trusting relationship and a change in outcome. It could be beneficial for future research to examine working alliance and the experience of negative effects at various stages of the treatment in order to ascertain the causal connection between the two factors.

While it would seem as if some of the program-related negative effects could possibly be prevented by adjusting technical structures of the online program or adding more flexibility to contact modalities and timelines, the role of the online counsellor in targeting participant-related negative effects is more complex. As some participants might be more inclined to react negatively to some part of the treatment due to participants' characteristics or specific features of the targeted condition, the role of the online counsellor to help participants deal with this in a constructive way becomes even more important. In a qualitative study by Bystedt et al. (2014), clinicians were not able to distinguish any specific group of participants that they thought would be of higher risk of suffering from negative effects. Similar, no sociodemographic characteristics explained the occurrence of self-reported negative effects in the present study. Although this is in line with a study by Boettcher et al. (2014) using similar open-ended questions, a recent meta-analysis did find sociodemographic characteristics to predict the experience of negative effects (Rozental et al., 2017). This discrepancy could possibly be explained by the differential operationalizations of the term negative effects in both studies. While Boettcher et al. (2014) assessed the experience of negative effects using both self-reports and frequencies of deterioration, Rozental et al. (2017) solely used deterioration rates as an indicator of the experience of negative effects. Interestingly, Boettcher et al. (2014) showed that participants who experienced self-reported negative effects in their study were not necessarily inclined to report deterioration on outcome measures. Although in the present study participants of both groups showed a decline in depressive and anxiety symptoms over time, results indicated that participants experiencing negative effects during treatment had a lower decrease of depressive and anxiety symptoms than participants not experiencing negative effects. Thus, it seems important to differentiate between open descriptions of negative effects that participants experience during treatment and negative outcome assessed by standard questionnaires as well as following up on participants over the course of treatment.

It seems possible that the experience of negative effects during treatment might not necessarily have long term consequences. Several participants mentioned that they did not address the occurrence of negative effects with the online counsellor as they already found their own solution or that they thought that their experiences were elemental when undergoing treatment. This is in line with research showing that facing challenges and redefining obstacles in treatment can prove valuable for participants (Bendelin et al., 2011). Moreover, some of the negative effects experienced during treatment might even hold beneficial effects for participants in the long run (Dimidjian and Hollon, 2010; Rozental et al., 2015). For example, deepening emotional processing is a core change factor in experiential therapies (Pos et al., 2009) and activating anxiety is considered a necessary step towards fear reduction in exposure therapies (Craske et al., 2008). This is also reflected by the decrease of participants' ratings of the impact negative effects had on their well-being while they occurred and at post-treatment assessment. However, for some participants the experience of negative effects might have resulted in dropout or deterioration. Thus, it is important to monitor their emergence continuously during treatment for the purpose of immediate reaction and help (Rozental et al., 2018).

The present study has several limitations that need to be considered. First, participants who did not complete the post-treatment assessment, including the questions regarding the occurrence of negative effects could be those who experienced negative effects to a greater degree. When comparing participants who answered the question about negative effects at post-assessment with those who did not, the present study found differences in age, sex, and employment status but not in pre-treatment symptom severity, living situation or work situation. This pattern of sociodemographic variables is commonly linked to compliance in online treatments (Karyotaki et al., 2015). This could indicate that drop-out was not necessarily a consequence of experiencing negative effects. At the same time, in a study using qualitative content analysis, experiences of negative effects were found to be reasons of non-adhering in an IBI for GAD (Johansson et al., 2015). Thus, findings in the current article may potentially have missed negative effects that were experienced by the participants, but not reported. Future studies might therefore address the occurrence of negative effects continuously throughout treatment and follow-up on participants who chose not to go through with the intervention. This also pertains to the assessment of the average degree to which the experience of negative effects affected the participants' well-being, as it might prove useful to assess it closer to its occurrence as a retrospective assessment might bias participants' ratings (Schwarz, 2007). It could also be possible that participants who were satisfied with their outcome of the IBI chose not to report any negative effect, whereas those participants displeased with the intervention might have exaggerated their response or chose not to answer in the first place. It would certainly be very informative to consider different perspectives on the experiences of negative effects, asking participants as well as therapists and significant others.

Results of the present article should be considered within the context of qualitative research as they cannot be regarded as representative for all people receiving IBI for depression. As the statements of participants were analyzed out of context, it is possible that some of the negative effects were not directly related to the intervention but were rather a result of independent events such as conflicts within relationships, employment issues or financial difficulties (Rozental et al., 2014). Since the study was specifically targeting depression, different types of negative effects are possible for populations with other disorders. However, as current findings are similar to prior research in this field (Halmetoja et al., 2014; Boettcher et al., 2014; Rozental et al., 2015), categories and subcategories found in the present study might represent recurrent negative effects participants potentially encounter during IBI independent of specific mental health conditions. More research is warranted to confirm these findings, but the current article has contributed to the knowledge of negative effects in IBIs, thus enabling participants as well as clinicians to make more fully informed treatment decisions.

5. Conclusions

By using both qualitative and quantitative methods, this article thoroughly analyzed negative effects of an IBI for depression. Results of the qualitative analysis revealed both program- as well as participant-related negative effects, including the deterioration of the targeted condition as well as novel symptoms. While the quantitative results show that depressive and anxiety symptoms decreased over time for all participants, they also indicate that participants who experienced negative effects showed a smaller decrease than participants who did not. Several additional negative effects connected to the specific online format could potentially be eliminated in future studies including more extensive descriptions of the program's features and providing more flexibility in terms of non-linear treatment or time-flexible treatment. Independent of the extent of therapeutic guidance, a perceived lack of therapist support was experienced as problematic by some participants, which could also serve as a potential reason not to address the occurrence of negative effects within the program. While participant characteristics as well as the amount of feedback-individualization do not seem to influence the experience of negative effects per se, the quality of the therapeutic relationship might play a significant role, with participants who rate the working alliance as more positive reporting fewer negative effects. Monitoring the therapeutic alliance, as well as the occurrence of negative effects throughout treatment using standard outcome measures as well as open format questions might help prevent drop-out among participants undergoing IBI for depression.

Funding

The original RCT the study is based on was funded by the German public health insurance company “Techniker Krankenkasse”. The funding body was not involved in the study design, collection, analysis, and interpretation of the data, or in the decision to submit this article for publication.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References

  1. Andersson G., Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cogn. Behav. Ther. 2009;38(4):196–205. doi: 10.1080/16506070903318960. [DOI] [PubMed] [Google Scholar]
  2. Andersson G., Titov N. Advantages and limitations of internet-based interventions for common mental disorders. World Psychiatry. 2014;13(1):4–11. doi: 10.1002/wps.20083. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Andrews G., Basu A., Cuijpers P., Craske M.G., McEvoy P., English C.L., Newby J.M. Computer therapy for the anxiety and depression disorders is effective, acceptable and practical health care: an updated meta-analysis. J. Anxiety Disord. 2018 doi: 10.1016/j.janxdis.2018.01.001. [DOI] [PubMed] [Google Scholar]
  4. Barlow D.H. Negative effects from psychological treatments: a perspective. Am. Psychol. 2010;65(1):13. doi: 10.1037/a0015643. [DOI] [PubMed] [Google Scholar]
  5. Bates D., Maechler M., Bolker B., Walker S. Fitting linear mixed-effects models using lme4. J. Stat. Softw. 2015;67(1):1–48. [Google Scholar]
  6. Baumeister H., Reichler L., Munzinger M., Lin J. The impact of guidance on internet-based mental health interventions—a systematic review. Internet Interv. 2014;1(4):205–215. doi: 10.1016/j.invent.2014.08.003. [DOI] [Google Scholar]
  7. Bendelin N., Hesser H., Dahl J., Carlbring P., Nelson K.Z., Andersson G. Experiences of guided internet-based cognitive-behavioral treatment for depression: a qualitative study. BMC Psychiatry. 2011;11(1):107. doi: 10.1186/1471-244X-11-107. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Berger T. The therapeutic alliance in internet interventions: a narrative review and suggestions for future research. Psychother. Res. 2017;27(5):511–524. doi: 10.1080/10503307.2015.1119908. [DOI] [PubMed] [Google Scholar]
  9. Berger T., Boettcher J., Caspar F. Internet-based guided self-help for several anxiety disorders: a randomized controlled trial comparing a tailored with a standardized disorder-specific approach. Psychotherapy. 2014;51(2):207. doi: 10.1037/a0032527. [DOI] [PubMed] [Google Scholar]
  10. Boettcher J., Rozental A., Andersson G., Carlbring P. Side effects in internet-based interventions for social anxiety disorder. Internet Interv. 2014;1(1):3–11. doi: 10.1016/j.invent.2014.02.002. [DOI] [Google Scholar]
  11. Bordin E.S. The generalizability of the psychoanalytic concept of the working alliance. Psychother. Theory Res. Pract. 1979;16(3):252. doi: 10.1037/h0085885. [DOI] [Google Scholar]
  12. Bystedt S., Rozental A., Andersson G., Boettcher J., Carlbring P. Clinicians' perspectives on negative effects of psychological treatments. Cogn. Behav. Ther. 2014;43(4):319–331. doi: 10.1080/16506073.2014.939593. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Carlbring P., Andersson G., Cuijpers P., Riper H., Hedman-Lagerlöf E. Internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: an updated systematic review and meta-analysis. Cogn. Behav. Ther. 2018;47(1):1–18. doi: 10.1080/16506073.2017.1401115. [DOI] [PubMed] [Google Scholar]
  14. Christensen H., Griffiths K.M., Farrer L. Adherence in internet interventions for anxiety and depression: systematic review. J. Med. Internet Res. 2009;11(2) doi: 10.2196/jmir.1194. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Craske M.G., Kircanski K., Zelikowsky M., Mystkowski J., Chowdhury N., Baker A. Optimizing inhibitory learning during exposure therapy. Behav. Res. Ther. 2008;46(1):5–27. doi: 10.1016/j.brat.2007.10.003. [DOI] [PubMed] [Google Scholar]
  16. Cuijpers P., Karyotaki E., Ciharova M., Miguel C., Noma H., Furukawa T.A. The effects of psychotherapies for depression on response, remission, reliable change, and deterioration: a meta-analysis. Acta Psychiatr. Scand. 2021 doi: 10.1111/acps.13335. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Dimidjian S., Hollon S.D. How would we know if psychotherapy were harmful? Am. Psychol. 2010;65(1):21. doi: 10.1037/a0017299. [DOI] [PubMed] [Google Scholar]
  18. Ebert D.D., Donkin L., Andersson G., Andrews G., Berger T., Carlbring P.…Kleiboer A. Does internet-based guided-self-help for depression cause harm? An individual participant data meta-analysis on deterioration rates and its moderators in randomized controlled trials. Psychol. Med. 2016;46(13):2679–2693. doi: 10.1017/S0033291716001562. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Fox J., Weisberg S. Second edition. Sage; Thousand Oaks CA: 2011. An {R} Companion to Applied Regression. [Google Scholar]
  20. Hadjistavropoulos H.D., Schneider L.H., Edmonds M., Karin E., Nugent M.N., Dirkse D.…Titov N. Randomized controlled trial of internet-delivered cognitive behavior therapy comparing standard weekly versus optional weekly therapist support. J. Anxiety Disord. 2017;52:15–24. doi: 10.1016/j.janxdis.2017.09.006. [DOI] [PubMed] [Google Scholar]
  21. Halmetoja C.O., Malmquist A., Carlbring P., Andersson G. Experiences of internet-delivered cognitive behavior therapy for social anxiety disorder four years later: a qualitative study. Internet Interv. 2014;1(3):158–163. doi: 10.1016/j.invent.2014.08.001. [DOI] [Google Scholar]
  22. Hansen N.B., Lambert M.J., Forman E.M. The psychotherapy dose-response effect and its implications for treatment delivery services. Clin. Psychol. Sci. Pract. 2002;9(3):329–343. doi: 10.1093/clipsy.9.3.329. [DOI] [Google Scholar]
  23. Hautzinger M., Keller F., Kühner C. Auflage Harcourt Test Services; Frankfurt/Main: 2006. BDI-II Beck-depressions-inventar. [Google Scholar]
  24. Johansson O., Michel T., Andersson G., Paxling B. Experiences of non-adherence to internet-delivered cognitive behavior therapy: a qualitative study. Internet Interv. 2015;2(2):137–142. doi: 10.1016/j.invent.2015.02.006. [DOI] [Google Scholar]
  25. Karyotaki E., Kleiboer A., Smit F., Turner D.T., Pastor A.M., Andersson G.…Christensen H. Predictors of treatment dropout in self-guided web-based interventions for depression: an ‘individual patient data'meta-analysis. Psychol. Med. 2015;45(13):2717–2726. doi: 10.1017/S0033291715000665. [DOI] [PubMed] [Google Scholar]
  26. Karyotaki E., Kemmeren L., Riper H., Twisk J., Hoogendoorn A., Kleiboer A.…Littlewood E. Is self-guided internet-based cognitive behavioral therapy (iCBT) harmful? An individual participant data meta-analysis. Psychol. Med. 2018:1–11. doi: 10.1017/S0033291718000648. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Krippendorff K. Sage; 2012. Content Analysis: An Introduction to its Methodology. [Google Scholar]
  28. Linden M. How to define, find and classify side effects in psychotherapy: from unwanted events to adverse treatment reactions. Clin. Psychol. Psychother. 2013;20(4):286–296. doi: 10.1002/cpp.1765. [DOI] [PubMed] [Google Scholar]
  29. Linden M., Schermuly-Haupt M.L. Definition, assessment and rate of psychotherapy side effects. World Psychiatry. 2014;13(3):306–309. doi: 10.1002/wps.20153. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Löwe B., Spitzer R.L., Zipfel S., Herzog W. 2002. Gesundheitsfragebogen für Patienten (PHQ). Komplettversion und Kurzform, ed 2 (Patient Health Questionnaire [PHQ-D]. Complete and Shortened Version, ed 2). Karlsruhe, Pfizer. [Google Scholar]
  31. Newman M.G., Szkodny L.E., Llera S.J., Przeworski A. A review of technology-assisted self-help and minimal contact therapies for anxiety and depression: is human contact necessary for therapeutic efficacy? Clin. Psychol. Rev. 2011;31(1):89–103. doi: 10.1016/j.cpr.2010.09.008. [DOI] [PubMed] [Google Scholar]
  32. Parker G., Fletcher K., Berk M., Paterson A. Development of a measure quantifying adverse psychotherapeutic ingredients: the experiences of therapy questionnaire (ETQ) Psychiatry Res. 2013;206(2):293–301. doi: 10.1016/j.psychres.2012.11.026. [DOI] [PubMed] [Google Scholar]
  33. Peterson A.L., Roache J.D., Raj J., Young-McCaughan S. The need for expanded monitoring of adverse events in behavioral health clinical trials. Contemp. Clin. Trials. 2013;34(1):152–154. doi: 10.1016/j.cct.2012.10.009. [DOI] [PubMed] [Google Scholar]
  34. Pos A.E., Greenberg L.S., Warwar S.H. Testing a model of change in the experiential treatment of depression. J. Consult. Clin. Psychol. 2009;77(6):1055. doi: 10.1037/a0017059. [DOI] [PubMed] [Google Scholar]
  35. R Core Team . R Foundation for Statistical Computing. Austria. URL; Vienna: 2016. R: A language and environment for statistical computing.https://www.R-project.org/ [Google Scholar]
  36. Richards D., Richardson T. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin. Psychol. Rev. 2012;32(4):329–342. doi: 10.1016/j.cpr.2012.02.004. [DOI] [PubMed] [Google Scholar]
  37. Rozental A., Andersson G., Boettcher J., Ebert D.D., Cuijpers P., Knaevelsrud C.…Carlbring P. Consensus statement on defining and measuring negative effects of internet interventions. Internet Interv. 2014;1(1):12–19. doi: 10.1016/j.invent.2014.02.001. [DOI] [Google Scholar]
  38. Rozental A., Boettcher J., Andersson G., Schmidt B., Carlbring P. Negative effects of internet interventions: a qualitative content analysis of participants' experiences with treatments delivered online. Cogn. Behav. Ther. 2015;44(3):223–236. doi: 10.1080/16506073.2015.1008033. [DOI] [PubMed] [Google Scholar]
  39. Rozental A., Magnusson K., Boettcher J., Andersson G., Carlbring P. For better or worse: an individual participant data meta-analysis of deterioration among participants receiving internet-based cognitive behavior therapy. J. Consult. Clin. Psychol. 2017;85(2):160. doi: 10.1037/ccp0000158. [DOI] [PubMed] [Google Scholar]
  40. Rozental A., Castonguay L., Dimidjian S., Lambert M., Shafran R., Andersson G., Carlbring P. Negative effects in psychotherapy: commentary and recommendations for future research and clinical practice. BJPsych Open. 2018;4(4):307–312. doi: 10.1192/bjo.2018.42. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Schermuly-Haupt M.L., Linden M., Rush A.J. Unwanted events and side effects in cognitive behavior therapy. Cogn. Ther. Res. 2018:1–11. doi: 10.1007/s10608-018-9904-y. [DOI] [Google Scholar]
  42. Schwarz N. Retrospective and concurrent self-reports: the rationale for real-time data capture. The science of real-time data capture: Self-reports in health research. 2007;11:26. [Google Scholar]
  43. Spek V., Cuijpers P.I.M., Nyklíček I., Riper H., Keyzer J., Pop V. Internet-based cognitive behavior therapy for symptoms of depression and anxiety: a meta-analysis. Psychol. Med. 2007;37(3):319–328. doi: 10.1017/S0033291706008944. [DOI] [PubMed] [Google Scholar]
  44. Spitzer R.L., Kroenke K., Williams J.B., Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch. Intern. Med. 2006;166(10):1092–1097. doi: 10.1001/archinte.166.10.1092. [DOI] [PubMed] [Google Scholar]
  45. Svartvatten N., Segerlund M., Dennhag I., Andersson G., Carlbring P. A content analysis of client e-mails in guided internet-based cognitive behavior therapy for depression. Internet Interv. 2015;2(2):121–127. doi: 10.1016/j.invent.2015.02.004. [DOI] [Google Scholar]
  46. Vittengl J.R., Jarrett R.B., Weitz E., Hollon S.D., Twisk J., Cristea I.…Faramarzi M. Divergent outcomes in cognitive-behavioral therapy and pharmacotherapy for adult depression. Am. J. Psychiatr. 2016;173(5):481–490. doi: 10.1176/appi.ajp.2015.15040492. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Wilcox R.R. Elsevier; San Diego, CA, United States of America: 2003. Applying Contemporary Statistical Techniques. [Google Scholar]
  48. Wilmers F., Munder T., Leonhart R., Herzog T., Plassmann R., Barth J., Linster H.W. Die deutschsprachige Version des Working Alliance Inventory-short revised (WAI-SR)-Ein schulenübergreifendes, ökonomisches und empirisch validiertes Instrument zur Erfassung der therapeutischen Allianz. Klinische Diagnostik und Evaluation. 2008;1(3):343–358. [Google Scholar]
  49. Zagorscak P., Heinrich M., Sommer D., Wagner B., Knaevelsrud C. Benefits of individualized feedback in internet-based interventions for depression: a randomized controlled trial. Psychother. Psychosom. 2018;87(1):32–45. doi: 10.1159/000481515. [DOI] [PubMed] [Google Scholar]

Articles from Internet Interventions are provided here courtesy of Elsevier

RESOURCES