Skip to main content
PMC home page
Trials logoLink to Trials
. 2014 Jan 30;15:39. doi: 10.1186/1745-6215-15-39

GET.ON Mood Enhancer: efficacy of Internet-based guided self-help compared to psychoeducation for depression: an investigator-blinded randomised controlled trial

David Daniel Ebert 1,2,✉, Dirk Lehr 1, Harald Baumeister 3,4, Leif Boß 1, Heleen Riper 1,5,6, Pim Cuijpers 1,6, Jo Annika Reins 1, Claudia Buntrock 1, Matthias Berking 1,2
PMCID: PMC3917536  PMID: 24476555

Abstract

Background

Major depressive disorder (MDD) imposes a considerable disease burden on individuals and societies. A large number of randomised controlled trials (RCTs) have shown the efficacy of Internet-based guided self-help interventions in reducing symptoms of depression. However, study quality varies considerably. The aim of this study is to evaluate the efficacy of a new Internet-based guided self-help intervention (GET.ON Mood Enhancer) compared to online-based psychoeducation in an investigator-blinded RCT.

Methods/design

A RCT will be conducted to compare the efficacy of GET.ON Mood Enhancer with an active control condition receiving online psychoeducation on depression (OPD). Both treatment groups will have full access to treatment as usual. Adults with MDD (n = 128) will be recruited and randomised to one of the two conditions. Primary outcome will be observer-rated depressive symptoms (HRSD-24) by independent assessors blind to treatment conditions. Secondary outcomes include changes in self-reported depressive symptom severity, anxiety and quality of life. Additionally, potential negative effects of the treatments will systematically be evaluated on several dimensions (for example, symptom deteriorations, attitudes toward seeking psychological help, relationships and stigmatisation). Assessments will take place at baseline, 6 and 12 weeks after randomisation.

Discussion

This study evaluates a new Internet-based guided self-help intervention for depression using an active control condition (psychoeducation-control) and an independent, blinded outcome evaluation. This study will further enhance the evidence for Internet-based guided self-help interventions for MDD.

Trial registration

German Clinical Trial Registration (DRKS): DRKS00005025

Keywords: Guided self-help, Internet-based, Major depressive disorder, Randomised controlled trial, Negative effects of psychotherapy, Active control

Background

Major depressive disorder (MDD) is one of the most prevalent psychiatric disorders with a lifetime prevalence of more than 16% [1-3]. Moreover, MDD is related to a considerable quality of life decrement [4,5], increased mortality rates [6], and substantial economic costs [7-9]. Currently, MDD ranks as the fourth disorder with the highest disease burden and is projected to be the leading cause of disability in high-income countries by 2030 [10].

Although there is ample evidence for the effectiveness of psychotherapy in the treatment of depression [11,12], many individuals remain untreated [13]. People who could particularly benefit from treatment disregard treatment for several reasons including a lack of knowledge of what to do, prohibitive costs, anticipated negative (social) consequences or preference for self-help [14]. Moreover, those seeking help hardly receive immediate access to evidence-based treatment because of long waiting lists for psychotherapeutic treatment [14,15]. Limited availability of clinicians, geographical inaccessibility and difficulties to ‘attend therapy during usual business hours’ are further barriers.

Using the Internet to provide guided self-help interventions may help to overcome some of the limitations of traditional treatment services. Internet-based guided self-help strategies have several advantages over face-to-face approaches. These include: (1) interventions are more easily accessible at any time and place; (2) anonymity is assured when patients want to avoid stigmatisation; (3) a greater potential for the integration of acquired skills in daily life exists because of an emphasis on the participants’ active role in (guided) self-help interventions [16]; (4) participants can work at their own pace and go through materials as often as they want; (5) travel time and costs for both participants and clinicians are eliminated; (6) Internet-based interventions may attract people who do not (want to) make use of traditional mental health services [17]; and (7), Internet-based interventions are easily scalable, implying that only a small increase of therapeutic resources is required for reaching a greater proportion of the eligible population using these interventions.

Accumulating empirical evidence suggests that Internet-based interventions are well-accepted by participants [18] and effective in the treatment of MDD [19], subthreshold depression [20] and as maintenance treatment [21]. A systematic review of 19 randomised controlled trials (RCTs) evaluating Internet-based interventions for symptoms of depression in 2,996 individuals [19] found a mean effect size of d = 0.56. Interventions including at least some guidance by a clinician were more effective (d = 0.78) than interventions without guidance (d = 0.36). Cuijpers (2010) showed that guided self-help interventions for depression (and anxiety disorders) could have comparable effects to traditional face-to-face therapy even when they are directly compared to each other [22].

However, a recent methodological analysis of 75 RCTs of computer- and Internet-based interventions for psychiatric disorders [23] criticised the mean methodological quality of published trials in the field as rather low. They criticised, for example, that many studies used weak control conditions (that is, wait-list control), that most studies relied solely on participants’ self-report, and that they failed to include an independent assessment such as blind ratings or biological indicators. Moreover, they found that low overall methodological quality scores were associated with higher reported effect sizes. In fact, the latest systematic review in the field on Internet-based treatments for depression [19] did not include any study that used an independent outcome assessment by assessors blind to the treatment condition. Moreover, half of the included studies used a wait-list control-only comparison. To the best of our knowledge no study on Internet-based treatment for major depression published so far, has applied an independent, blinded outcome evaluation.

Another important issue not adequately addressed to date is the potential negative effects of Internet-based treatments for depression. One particularly unfavourable outcome of psychotherapy is deterioration of symptoms during treatment. However, RCTs evaluating psychological treatments seldom report the number of patients who had deteriorated while being treated [24]. Critics of Internet-based treatments have often emphasised that treatment might be provided that is less intensive than required to treat severely affected or symptomatic individuals [23]. This inadequate treatment allocation may result in less treatment expectations for psychotherapy in general and discourage individuals from seeking more intensive treatment. Hence, not only is there a pressing need for research on potential adverse events of psychotherapy in general [24] but also specifically for Internet-based treatments.

Objective and research questions

The aim of this study is to evaluate the efficacy of a newly developed guided self-help Internet-based intervention (GET.ON Mood Enhancer) compared to an online psychoeducation on depression (OPD).

We expect that observer-rated depressive symptomatology assessed by raters blind to treatment conditions will be reduced to a greater extent in the intervention group than in the control condition. Moreover, we hypothesise that GET.ON Mood Enhancer is superior in terms of self-reported depressive symptoms, wellbeing, quality of life, symptoms of anxiety and problem-solving skills compared to the OPD-control condition. Potential negative effects with regard to (1) numbers of patients with symptom deteriorations, (2) attitudes towards seeking psychological help, and (3) other adverse events will be systematically evaluated. We hypothesise that both treatment conditions will not have a negative effect on attitudes towards seeking psychological help.

Methods/design

Design

A two-arm RCT will be conducted to compare GET.ON Mood Enhancer with an online psychoeducation on depression (OPD) condition. Both treatment arms will have full access to treatment as usual. Assessments will take place at baseline (T1), post-treatment (6 weeks, T2) and 12-week follow-up (T3; see Figure 1 for a detailed overview of assessments). All procedures involved in the study will be consistent with the generally accepted standards of ethical practice approved by the University of Marburg ethics committee (No. 2013–08 K). The trial is registered in the German clinical trials register under DRKS00005025.

Figure 1.

Figure 1

Open in a new tab

Study flow.

Trial status

By the time of submission, recruitment for the trial was still ongoing. Recruitment started in June 2013 and will approximately last until February 2014.

Participants and procedure

Inclusion and exclusion criteria

We will include adults (1) above the age of 18 years (2) with MDD according to DSM-5 criteria (3) who have Internet access, (4) have sufficient German skills in reading and writing (self-report), and (5) are willing to give informed consent. We will exclude subjects (1) with a history of manic/hypomanic episodes, (2) with a history of psychotic disorders, (3) currently receiving psychotherapy for any kind of mental health problems, (4) showing a notable suicidal risk as indicated by a score greater than 1 on BDI Item 9 (‘I feel I would be better off dead’), or (5) a change in antidepressive medication dosage/drug within 4 weeks before baseline.

Recruitment

Participants will be recruited via the GET.ON research website [25] that is announced in newspapers, on-air media and related websites. The research website provides information about the GET.ON Mood Enhancer training and details about another study evaluating GET.ON Mood Enhancer in a sample with subthreshold depression (Trial registration: DRKS00004709, [26]). Those participants who do fulfil criteria for a MDD are excluded from that study, and instead offered to participate in this trial.

Assessment of eligibility and randomisation

People who apply for study participation will receive an information letter with detailed information about the study procedures. Full written informed consent will be obtained from all participants. They will be informed that they can withdraw from the intervention and/or study at any time without any negative consequences. Nevertheless, we will ask those who withdraw from the trial treatment to attend all the remaining research appointments or at least provide minimal data (primary outcome measure). Applicants who continue to participate in the study will be asked to complete online screening questionnaires to assess the severity of their depressive symptoms (CES-D >16), whether they are currently receiving any kind of treatment for any mental health disease and whether they have a high suicidal risk (BDI Item 9 > 1). Subjects screened positive and are willing to give informed consent will be scheduled for a structured clinical interview (SCID) conducted via the telephone [27]. SCIDs will be conducted by trained clinicians. Participants meeting all of the inclusion and none of the exclusion criteria, who have completed the baseline assessment, and returned the informed consent form will enter the study and will be randomly allocated to study conditions. Randomisation will take place at an individual level. The allocation will be performed by an independent researcher not otherwise involved in the study using an automated computer-based random integer generator (randlist). The allocation will be concealed in advance from participants, researchers involved in recruitment, and therapists.

Assessments

Self-report and observer-rated assessments will take place at baseline, post-intervention and 12-week follow-up. See Figure 1 for a detailed overview. Self-report data will be collected using a secured online-based assessment system (AES, 256-bit encrypted), and observer-based assessments will be conducted via the telephone by trained interviewers. Observer-rated assessments will be recorded to examine inter-rater reliability. In case of disagreement, the two raters will discuss until a consensus is formed, and the agreed rating will be used for analysis. If no agreement is reached after discussion, the assessment will be rated by an experienced diagnostic-rater (gold standard), and this rating will be used for analysis.

Blinding

The research staff conducting the observer-based rating of depressive symptoms will be blinded to the condition the participants are assigned to. Considerable effort is undertaken to ensure blindness, including (1) an explanation to the participants why it is important not to inform the interviewer about the condition they were assigned to, (2) a written reminder in the interview manual for the interviewer to ask the participant not to mention anything about their randomisation status, (3) verbal reminders to the patient before the interview, and (4) a documentation after the assessment of whether or not the interviewer is still blind to the treatment condition. In the event of a blind violation, the interviewer will be changed to the following assessment. Participants know about the content of both conditions, but are blinded with regard to which intervention is the experimental and which the control condition.

Intervention

GET.ON Mood Enhancer

GET.ON Mood Enhancer is a brief intervention consisting of six lessons with modules concerning psychoeducation, behavioral activation (BA), problem solving (PS) and relapse prevention. Additionally, participants are offered four modules that can be chosen based on individual need and/or preference. Additional modules are directed at sleep problems, relaxation techniques and dealing with worrying thoughts (see Table 1 for a session overview). Lessons consist of text, exercises and testimonials and also include interactive elements such as audio and video clips. A strong focus of the intervention lies on transfer tasks (homework assignments) to integrate newly acquired strategies and techniques into daily life. In the beginning of each subsequent lesson, participants are invited to reflect on their experiences with the newly acquired skills. The training is adaptive as the content is tailored to the specific needs of the individual participant by continuously asking participants to respond by choosing among various response options. Subsequent content is then tailored to the participant’s response. Using responsive web design, participants can follow the program on the Internet, a tablet or mobile phone. An integrated read-aloud function allows participants to follow narrated lessons. Participants are advised to complete at least one but preferably two lessons per week, because a previous meta-regression analysis about the role of treatment intensity for treatment outcome in the treatment of depression indicated that more frequent therapy sessions might be associated with a better outcome compared to a lower frequency of sessions [28]. Consequently, the training lasts about 3 to 6 weeks. We decided that we will not include more modules, because (1) a recent meta-analysis [19] found that web-based interventions for depression including more than seven modules were less effective (d = 0.36) than interventions with seven or less modules (d = 0 .75), and (2) we want to lower the threshold for individuals as much as possible. In terms of a stepped-care model participants will be encouraged to seek more intensive help in routine mental health services, if their progress is not sufficient at the end of the intervention. During the last treatment session, they will be provided with information about evidence-based treatments in routine mental healthcare and how to obtain access to it.

Table 1.

Overview of GET.ON Mood Enhancer lessons

Session Topic Content
1
 Psychoeducation
 • Information about symptoms, causes and types of depression
• Role of motivation
• Introduction of the mood and activity diary
2
 Behavioural activation (BA) I
 • Relationship between activities and depression
• Introduction of daily activity scheduling
3
 Behavioural activation (BA) II
 • Reflection on experiences with newly acquired skills and on activities/behaviour that might have influenced the mood (from now on in every session at the beginning)
• Coping with difficulties with regard to daily activity scheduling
• Goal setting for the upcoming days
• Additional: sleeping problems
4
 Problem-solving techniques (PST) I
 • BA: reflection of goal attainment and goal setting for the upcoming days
• PST: distinction between solvable and unsolvable problems
• PST: introduction of 6-step PST procedure: (1) defining the problem, (2) defining the goal, (3) brainstorming about possible solutions and choosing the best one, (4) making a plan how to implement this solution, (5) putting the solution into practice, and (6) evaluating the outcome.
• PST: choosing one personal problem, filling out steps 1 to 4, step 5 as homework
5
 Problem-solving techniques (PST) II
 • BA: reflection of goal attainment and goal setting for the upcoming days
• PST: deepening the 6-step procedure: step 6 of the 6-step procedure: evaluating outcome; depending on the evaluation: coping with difficulties, revising steps 1 to 4 or choosing another problem
• Additional: stop-worrying techniques
6 Relapse prevention • BA: reflection goal attainment and goal setting for the future
• PST: step 6 of the 6-step procedure: evaluating outcome
• Evaluation: summarising gains and learned strategies
• Developing an individual relapse prevention plan
• Information about further healthcare services
Open in a new tab

The main modules used in GET.ON Mood Enhancer are based on evidence-based face-to-face manuals that have been found to be effective in the treatment of depression, such as behaviour therapy (BT) [29] and problem-solving therapy (PST) [30]. In BT, a strong focus rests on daily pleasurable activity scheduling that is integrated in each lesson. The PST elements implemented in GET.ON Mood Enhancer have been used in various web-based interventions, such as the Dutch web-based ‘Alles onder Controle’ course, which has been shown to be effective in reducing depressive symptomatology across several RCTs [31,32].

During the training, participants will be supported by an online therapist. Every participant will be assigned to one therapist for the duration of the study. The total time a therapist spends on a participant will be approximately 2 to 3 hours. Guidance is provided by psychotherapists in training supervised by an experienced clinician. Participants will communicate with their therapist through the internal messaging function of the system on which GET.ON Mood Enhancer is implemented. The guidance is mainly based on the supportive-accountability model of providing guidance in Internet-based interventions [33]. In the current study, the purpose of the guidance will be to support participants to adhere to the treatment modules, and therapists will not teach therapeutic techniques beyond techniques used in the treatment modules. All feedback from therapists is stored for supervision and adherence checks.

Psychoeducation on depression condition ( OPD )

The OPD intervention is also Internet-based and is implemented on the same platform as GET.ON Mood Enhancer. In the current study, the psychoeducational intervention is based on the patient version of the German S3-Guideline/National Disease Management Guideline Unipolar Depression [34]. It informs participants about the nature and evidence-based treatments of depression including information about symptoms, strategies to overcome depression and sources of help. In this study, the psychoeducational intervention requires no explicit homework assignments from the participants and provides no therapist support. Passive psychoeducational interventions have been shown to be effective in reducing depressive symptoms with a pooled standardised effect size of d = 0.26 [35].

Treatment as usual

There will be no restriction on the use of treatment as usual in routine mental health services during the study period, such as psychotherapeutic and psychiatric treatment or antidepressant medication. However, to control for potential confounding effects, treatment utilisation and changes in the dosage of antidepressant medication intake will be monitored during the study period.

Primary and secondary outcomes

The primary outcome will be observer-rated depression severity. In the secondary analyses, we will explore the effects of the treatments on self-reported depression severity, wellbeing, anxiety symptoms, quality of life and the number of patients who (1) responded and (2) are in remission. Moreover, we will also systematically evaluate potential negative treatment effects (that is, symptom deteriorations, negative effects on attitudes towards seeking psychological help; other adverse events).

Measures

Primary outcome

The primary outcome will be observer-based rating of depressive symptom severity measured by the Hamilton Rating Scale for Depression HRSD24[36-38]. The HRSD is likely to be the most widely used clinician-rated scale for measuring depression in research. The self-report measure assesses depressed mood, vegetative and cognitive symptoms of depression, and anxiety symptoms. Items are rated on either a 5-point or a 3-point scale, and the total score is derived by summing the individual item scores. Higher scores indicate greater symptom severity. The HRSD is sensitive to change, inter-rater reliability is 0.90 [36], and the scale corresponds well with overall clinical ratings of severity [39,40]. The cutoff points of 10, 19, 27 and 35 represent the threshold for mild, moderate, severe and very severe depression, respectively.

Secondary outcomes

Depressive symptoms

As the secondary outcome, observer-based rating of depression will also be measured using the Quick Inventory of Depressive Symptomatology - Clinician-Rating QIDS-CR16[41,42]. The 16-item QIDS-CR16 is a brief clinician-report rating scale developed from the 30-item Inventory of Depressive Symptomatology [41-43]. In contrast to the HRSD, it evaluates only the nine depression criterion symptom domains (that is, sad mood, concentration, self-criticism, suicidal ideation, interest/involvement, energy/fatigability, sleep disturbance, appetite/weight change and psychomotor agitation/retardation) from the Diagnostic and Statistical Manual of Mental Disorders [44] during the prior 7 days. Each item is scored on a scale from 0 to 3 points, with higher scores indicating higher symptom severity. This measure has shown good psychometric properties, such as strong internal consistency (α = 0.85), concurrent validity and sensitivity to symptom change in patients with MDD [42]. The cutoff points of 6, 11, 16 and 21 represent the threshold for mild, moderate, severe and very severe depression, respectively.

Self-reported depressive symptoms will be assessed with the Patient Health Questionnaire (PHQ-9). This measure has comparable sensitivity and specificity to many other depression measures, although it is only half the length. Its internal reliability reaches values between α = 0.86 and 0.89 [45,46]. Each item assesses the frequency of a symptom in the last 2 weeks on a scale ranging from 0 (‘not at all’) to 3 (‘nearly every day’). In contrast to the primary outcome measure HRSD24, the PHQ-9 only assesses frequency of symptoms, not their intensity. The total score ranges from 0 to 27, with a higher score indicating more frequent symptoms. The cutoff points of 5, 10, 15 and 20 represent the threshold for mild, moderate, moderately severe and severe depression, respectively [45,46].

Quality of life

Health-related quality of life will be assessed with the SF-12v1 Health Survey [47]. The SF-12v1 has 12 items covering eight health domains (physical functioning, physical and emotional role functioning, body pain, general health, vitality, social functioning and mental health). The SF-12 generates a physical and a mental health summary score.

Anxiety

Anxiety will be measured with the anxiety subscale of the Hospital Anxiety and Depression Scale HADS-A [48]. The anxiety subscale consists of seven questions and each is scored from 0 to 3 with total scores ranging from 0 to 21. A score between 0 and 7 indicates no anxiety, between 8 and 10 possible anxiety, and above 11 or 12 a clinical anxiety disorder. Psychometric properties are well established [49].

Problem-solving skills

Problem-solving ability (that is, generalised appraisal, beliefs, expectancies and emotional responses) will be measured with two subscales of the Social Problem-Solving Inventory-Revised (SPSI-R). The positive problem orientation (PPO) subscale represents a constructive dimension, and the negative problem orientation (NPO) subscale is viewed as a dysfunctional dimension. Cronbach’s alphas are α = 0.76 for the PPO dimension and 0.83 for the NPO dimension [50].

Behavioural activation

Participants’ activation towards goals or values and pleasant activities and avoidance behaviours will be measured with the BADS-Short Form BADS-SF [51]. The BADS-SF entails nine items comprising two subscales (activation and avoidance). The items are rated on a 7-point Likert-type scale. Higher scores indicate that the participant scores high on the area of interest. The internal consistency is α = 0.82 [51].

Wellbeing

Psychological wellbeing will be measured by the World Health Organization’s (WHO) 5 Wellbeing Index [52]. Participants indicate for each of the five statements, which one is closest to how they have been feeling over the prior 2 weeks. Each question is scored from 0 to 5 with the total score ranging from 0 to 25. Higher scores indicate better wellbeing.

Problematic alcohol use

The Alcohol Use Disorders Identification Test (AUDIT) is a 10-item questionnaire designed by the WHO to screen for hazardous alcohol intake. It assess three conceptual domains: alcohol intake (items 1 to 3), dependence (items 4 to 6) and adverse consequences (items 7 to 10). Sum-scores can range from 0 to 40, and the generally accepted cutoff point for identifying a potential alcohol problem is 8 [53].

Treatment credibility/patient expectancy

Training credibility and participants’ expectancy for improvement will be measured with the credibility and expectation questionnaire (CEQ). The CEQ consists of six items which are rated on a 9- or sometimes 10-point Likert scale. The psychometric properties of the instrument are well established [54].

Course evaluation

In absence of a standardised measure for evaluating course satisfaction in Internet-based treatments, user satisfaction will be measured with a self-designed questionnaire based on the Client Satisfaction Questionnaire CSQ-8, German Version [55,56]. This self-report measure consists of eight items measuring the global client satisfaction with the Internet-based training. Previous research indicated a high internal consistency [57].

Response and remission

Participants will be coded as responders when they demonstrate a reliable change according to the widely used reliable change index (RCI) [58]. Participants with an RCI of 1.96 or above on the primary outcome measure HRSD24 will be considered responders. Remission is defined a priori as a non-pathological score of ≤10 on the HRSD24.

Negative effects

Negative effects will be measured on (1) symptom deterioration during treatment (2), attitudes towards seeking psychological help, and (3) other adverse events.

(1) Symptom deteriorations

Patients will be classified as deteriorated when they display a reliable negative change (−1.96) in the primary outcome measure HRSD24 according to the reliable change index, proposed by Jacobsen & Truax (1991) [58].

(2) Negative effects on attitudes towards seeking professional psychological help

A potential negative influence on attitudes towards seeking mental healthcare service utilisation will be measured with the Attitudes Toward Seeking Professional Psychological Help Scale-SF ATSPPH-SF [59]. The ATSPPH-SF consists of 10 items that are rated on a 4-point Likert scale from 0 to 3, yielding a total score ranging from 0 to 30. High scores indicate more positive treatment attitudes. The instrument showed good psychometric properties [60].

Other adverse events

Other adverse events will be measured with the negative effects of psychotherapy inventory (INEP). The INEP is a relatively new measure and was developed for assessing systematic and potentially negative effects of psychotherapeutic interventions in different domains. The version used in this study consists of 15 items assessing any negative effects participants experienced within or after the completion of the Internet-based training on (1) negative intrapersonal changes (for example, ‘During treatment or since the end of my therapy, I suffered from suicidal thoughts or intentions for the first time ever’), (2) negative effects in an intimate relationship (for example, ‘My partner is or has been jealous of my therapist’), (3) family/friends (for example, ‘The relationships with my friends has worsened’), (4) perceived dependence on the psychotherapist/psychotherapeutic intervention (for example, ‘I feel dependent on my therapist’), and (5) stigmatisation (for example, ‘I am anxious that my colleagues or friends could find out about my psychotherapy’). The items are rated on a 4-point Likert scale (0 = no agreement at all; 3 = total agreement). Chronbach’s alpha is α = 0.85 (Table 2).

Table 2.

Overview of measurements

 
  
 Time of measurement
Instrument Aim T1 (Baseline) T2 (6 weeks) T3 (12 weeks)
SCID
 Diagnostic interview
 x
  
  
HRSD 24
Hamilton Rating Scale for Depression
 x
 x
 x
QIDS-C 16
Quick Inventory for Depressive Symptomatology
 x
 x
 x
PHQ-9
 Depressive symptomatology
 x
 x
 x
WHO-5
 Wellbeing
 x
 x
 x
SF-12
 Quality of life
 x
 x
 x
HADS-A
 Anxiety symptoms
 x
 x
 x
SPSI-R
 Problem-solving skills
 x
 x
 x
BADS-SF
 Behavioural activation
 x
 x
 x
CEQ
 Patient expectancy/treatment credibility
 x
  
  
ATSPPH-SF
 Attitudes toward seeking psychological help
 x
 x
  
BFI
 Big Five Inventory
 x
  
  
INEP
 Inventory of negative effects in psychotherapy
  
 x
 x
CSQ-8
 Clients’ satisfaction with the online training
  
 x
  
AUDIT
 Alcohol use disorder identification test
 x
  
  
Other questions Socio-demographics x    
Open in a new tab

Sample size calculation

We aim to include 128 participants. This sample will allow us to detect a between-group effect size (ES) of d = 0.50 with a power (1-ß) of 80% and an alpha of 0.05 (calculated using PASS 12). The most recent meta-analytic review found a mean ES of d = 0.78 for therapist-supported web-based interventions for depression [19]. However, most of the included studies used a waiting-list control comparison only, and these studies showed a considerably larger ES than those studies including a treatment-as-usual condition. Our comparison condition will include psychoeducation and full access to treatment as usual and participants are encouraged to seek further help. Thus, we expect a somewhat smaller ES of d = 0.50 at post-treatment.

Statistical analyses

The clinical trial will be conducted in compliance with the Declaration of Helsinki, the CONSORT guidelines, GCP and the protocol. Aiming at an intention-to-treat design [61] we will include all participants who will be randomly assigned to one of the two conditions. Additional per protocol analyses (PPA) will be conducted, including only participants’ satisfying protocol treatment (completed at least 4 of 6 modules). Mixed-model analyses of variance will be conducted to explore the effects of the treatments on all continuous outcomes. Missing data will be handled using multiple imputations (MI). MI is especially robust with respect to missing data [62]. Nevertheless, to assess systematic effects of non-ignorable missing data, pattern mixture analyses for multi-level longitudinal approaches [63] will be conducted. To determine if the treatment effect is dependent on missing data, the missing-data pattern of each participant will be first coded and then included in a three-way interaction (missing pattern × condition × change in depression severity) in the main outcome analyses. If no significant interactions between missing-data pattern and treatment outcome are found, we will conclude that no missing data bias occurred in the results. For all mixed-model analyses, Cohen’s d [64] will be calculated by standardising the differences between baseline and follow-up scores by the pooled standard deviation of the baseline scores. Response and remission rates will be compared across groups with the help of contingency tables and χ2 tests. We will also calculate the number needed-to-be-treated (NNT) with GET.ON Mood Enhancer to achieve one response and remission, respectively, compared to the control group. We will also calculate the number needed-to-harm, which indicates the number of participants treated in the experimental condition for one extra person to have a symptom deterioration. Because the differential risk of the intervention ideally needs to be set into relation with its benefits we will also calculate a benefit-risk ratio [65]. Benefit-risk ratio will be calculated by dividing the NNH to achieve one symptom deterioration through the NNT to achieve one response. If this benefit-risk ratio is greater than 1, the benefits outweigh the risks; if it is 1, the balance between benefit and risks are equal across the groups; if it is less than 1, the risks outweigh the benefits (LIT). Benefit-risk ratios will only be calculated when differences in risk for symptom deteriorations are statistically significant.

For other negative effects, we will use independent t-tests for continuous, χ 2 for categorical, and logistic regression for binary outcomes. In a pre-planned subgroup analysis, we will explore negative effects of GET.ON Mood Enhancer on attitudes towards seeking psychological help in patients who do not reach the response criteria up to post-assessment (6 weeks). For all statistical analyses, significance level will be set at P <0.05. All analyses will be conducted using SPSS 20.

Discussion

MDD is a highly prevalent disorder associated with a considerable loss in quality of life, increased mortality rates and substantial economic costs. Several reasons lead to the fact that patients remain untreated although they are in need of help. Internet-based guided self-help approaches could be an attractive, efficient and cost-effective approach to offer an evidence-based treatment alternative. Numerous studies have shown the acceptance and efficacy of guided self-help interventions in the treatment of depression. However, recent reviews have criticised that the methodological quality of studies are low and more high quality studies are needed, before such Internet-based therapies are widely disseminated. Moreover, as is also the case for face-to-face psychotherapy, hardly anything is empirically known about the potential negative effects of Internet-based guided self-help approaches. This study will evaluate a new Internet-based intervention for depression (GET.ON Mood Enhancer) compared to an online psychoeducation control in an investigator-blinded RCT. Special emphasis will be given to systematically evaluate potential negative effects, such as symptom deterioration, effects on attitudes towards seeking psychological help and other adverse events (that is, perceived intrapersonal negative change, relationships, friendships, family, therapeutic malpractice and stigmatisation).

This study will also have some limitations. First, as in most longitudinal studies we will need to deal with the problem of missing values. Although the planned adjustment for missing data (Multiple Imputation) is a highly recommended method to handle missing data [62], we will nevertheless additionally conduct pattern-mixture analyses [63] to minimise a possible risk for bias. Second, ideally the risk for negative effects when providing an Internet-based guided self-help intervention to patients with MDD needs to be compared with the risk for negative effects in face-to-face psychotherapy. However, a systematic evaluation of potentially negative effects of treatment has seldom been conducted for face-to-face psychotherapeutic interventions for Major Depression. Therefore, drawing conclusions on the differential risk of this guided self-help treatment compared to traditional psychotherapeutic treatment will not be possible. Third, the comparison condition does not control for unspecific effects through human support. Finally, the study sample will be too small to test for potential moderating effects. Thus, which patients are likely to profit from this type of treatment delivery or which patients are especially likely to experience negative effects will remain unclear.

There will also be several strengths of this study, including the strong methodology of a randomised controlled design with an active control condition, outcome assessment with validated assessments by independent raters blind to treatment condition, a sample defined by a standard diagnostic measure, an appropriate statistical analyses plan and handling of missing data with state of the art methods. Given these strengths, the results of the study should further enhance the evidence-base for Internet-based guided self-help interventions for Major Depression. Moreover, a systematic evaluation of potentially negative effects of an Internet-based treatment as it is conducted in the present study has to the best of our knowledge not been conducted before. This study will therefore provide valuable information to the field as the basis for a wide dissemination of such concepts.

Overall, to overcome the gap between the need for treatment and evidence-based treatment availability and treatment utilisation, (cost-) effective low-threshold interventions are needed that are accessible for as many people as possible. Internet-based guided self-help interventions might be a promising strategy not only as a first step in a stepped-care approach but by providing treatments that are more intensive when patients fail to respond. If at some point this strategy becomes classified as evidence-based treatment, such approaches could provide evidence-based services to patients in areas or countries where psychotherapeutic treatment is not readily available [66]. If the proposed trial shows that the investigated intervention is not only effective in reducing depressive symptoms but also shows an acceptable risk-benefit balance, it would further strengthen the arguments for a wide dissemination of Internet-based guided self-help approaches for Major Depression.

Competing interests

MB is a minority shareholder of Minddistrict GmbH, which will provide the technical platform for the web-based interventions.

Authors’ contribution

MB obtained funding for this study. DE, CB, JR, LB and DL contributed to the development of the GET.ON Mood Enhancer training. DE was responsible for the initial draft of design of the study, LB, PC, HR, HB, MB and DL further contributed to the study design. DE drafted the manuscript. LB, JR and CB are responsible for trial management and carrying out the diagnostic assessments. All authors contributed to the further writing of the manuscript. All authors read and approved the final manuscript.

Contributor Information

David Daniel Ebert, Email: ebert@inkubator.leuphana.de.

Dirk Lehr, Email: lehr@inkubator.leuphana.de.

Harald Baumeister, Email: harald.baumeister@psychologie.uni-freiburg.de.

Leif Boß, Email: boss@inkubator.leuphana.de.

Heleen Riper, Email: h.riper@vu.nl.

Pim Cuijpers, Email: p.cuijpers@vu.nl.

Jo Annika Reins, Email: reins@inkubator.leuphana.de.

Claudia Buntrock, Email: buntrock@inkubator.leuphana.de.

Matthias Berking, Email: berking@staff.uni-marburg.de.

Acknowledgements

This study is funded by the European Union (EFRE: ZW6-80119999, CCI 2007DE161PR001).

References

  1. Alonso J, Angermeyer MC, Bernert S, Bruffaerts R, Brugha TS, Bryson H, De Girolamo G, Graaf R, Demyttenaere K, Gasquet I, Haro JM, Katz SJ, Kessler RC, Kovess V, Lepine JP, Ormel J, Polidori G, Russo LJ, Vilagut G, Almansa J, Arbabzadeh-Bouchez S, Autonell J, Bernal M, Buist-Bouwman MA, Codony M, Domingo-Salvany A, Ferrer M, Joo SS, Martinez-Alonso M, Matschinger H. et al. Prevalence of mental disorders in Europe: results from the European Study of the Epidemiology of Mental Disorders (ESEMeD) project. Acta Psychiatr Scand Suppl. 2004;420:21–27. doi: 10.1111/j.1600-0047.2004.00327.x. [DOI] [PubMed] [Google Scholar]
  2. Kessler RC, McGonagle KA, Zhao S, Nelson CB, Hughes M, Eshleman S, Wittchen HU, Kendler KS. Lifetime and 12-month prevalence of DSM-III-R psychiatric disorders in the United States. Results from the National Comorbidity Survey. Arch Gen Psychiatry. 1994;51:819. doi: 10.1001/archpsyc.1994.03950010008002. [DOI] [PubMed] [Google Scholar]
  3. Waraich P, Goldner EM, Somers JM, Hsu L. Prevalence and incidence studies of mood disorders: a systematic review of the literature. Can J Psychiatry. 2004;49:124–138. doi: 10.1177/070674370404900208. [DOI] [PubMed] [Google Scholar]
  4. Ustun TB, Ayuso-Mateos JL, Chatterji S, Mathers C, Murray CJ. Global burden of depressive disorders in the year 2000. Br J Psychiatry. 2004;184:386–392. doi: 10.1192/bjp.184.5.386. [DOI] [PubMed] [Google Scholar]
  5. Saarni SI, Suvisaari J, Sintonen H, Pirkola S, Koskinen S, Aromaa A, Lonnqvist J. Impact of psychiatric disorders on health-related quality of life: general population survey. Br J Psychiatry. 2007;190:326–332. doi: 10.1192/bjp.bp.106.025106. [DOI] [PubMed] [Google Scholar]
  6. Cuijpers P, Smit F. Excess mortality in depression: a meta-analysis of community studies. J Affect Disord. 2002;72:227–236. doi: 10.1016/S0165-0327(01)00413-X. [DOI] [PubMed] [Google Scholar]
  7. Berto P, D'Ilario D, Ruffo P, Di Virgilio R, Rizzo F. Depression: cost-of-illness studies in the international literature, a review. J Ment Health Policy Econ. 2000;3:3–10. doi: 10.1002/1099-176X(200003)3:1&#x0003c;3::AID-MHP68&#x0003e;3.0.CO;2-H. [DOI] [PubMed] [Google Scholar]
  8. Greenberg PE, Birnbaum HG. The economic burden of depression in the US: societal and patient perspectives. Expert Opin Pharmacother. 2005;6:369–376. doi: 10.1517/14656566.6.3.369. [DOI] [PubMed] [Google Scholar]
  9. Smit F, Cuijpers P, Oostenbrink J, Batelaan N, De Graaf R, Beekman A. Costs of nine common mental disorders: implications for curative and preventive psychiatry. J Ment Health Policy Econ. 2006;9:193–200. [PubMed] [Google Scholar]
  10. Mathers CD, Loncar D. Projections of global mortality and burden of disease from 2002 to 2030. PLoS Med. 2006;3:e442. doi: 10.1371/journal.pmed.0030442. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Cuijpers P, Van Straten A, Andersson G, Van Oppen P. Psychotherapy for depression in adults: A meta-analysis of comparative outcome studies. J Consult Clin Psychol. 2008;76:909–922. doi: 10.1037/a0013075. [DOI] [PubMed] [Google Scholar]
  12. Cuijpers P, Van Straten A, Warmerdam L, Andersson G. Psychological treatment of depression: A meta-analytic database of randomized studies. BMC Psychiatry. 2008;8:36. doi: 10.1186/1471-244X-8-36. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Wittchen HU, Jacobi F, Rehm J, Gustavsson A, Svensson M, Jönsson B, Olesen J, Allgulander C, Alonso J, Faravelli C, Fratiglioni L, Jennum P, Lieb R, Maercker A, Van Os J, Preisig M, Salvador-Carulla L, Simon R, Steinhausen HC. The size and burden of mental disorders and other disorders of the brain in Europe 2010. Eur Neuropsychopharmacol. 2011;21:655–679. doi: 10.1016/j.euroneuro.2011.07.018. [DOI] [PubMed] [Google Scholar]
  14. Kessler RC, Berglund PA, Bruce ML, Koch RJ, Laska EM, Leaf PJ, Manderscheid RW, Rosenheck RA, Walters EE, Wang PS. The prevalence and correlates of untreated serious mental illness. Health Serv Res. 2001;36:987–1007. [PMC free article] [PubMed] [Google Scholar]
  15. Schulz H, Barghaan D, Harfst T, Koch U. Psychotherapeutische Versorgung. Berlin: Gesundheitsberichterstattung des Bundes; 2008. [Google Scholar]
  16. Klaw E, Humphreys K. In: Psychologists’ desk reference. 2. Koocher G, Norcross J, Hill S, editor. New York: Oxford University Press; 2005. Facilitating client involvement in self-help groups; pp. 502–505. [Google Scholar]
  17. Cuijpers P, Van Straten A, Warmerdam L, Van Rooy MJ. Recruiting participants for interventions to prevent the onset of depressive disorders: possible ways to increase participation rates. BMC Health Serv Res. 2010;10:181. doi: 10.1186/1472-6963-10-181. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Cavanagh K, Seccombe N, Lidbetter N. The implementation of computerized cognitive behavioural therapies in a service user-led, third sector self help clinic. Behav Cogn Psychother. 2011;39:427–442. doi: 10.1017/S1352465810000858. [DOI] [PubMed] [Google Scholar]
  19. Richards D, Richardson T. Computer-based psychological treatments for depression: A systematic review and meta-analysis. Clin Psychol Rev. 2012;32:329–342. doi: 10.1016/j.cpr.2012.02.004. [DOI] [PubMed] [Google Scholar]
  20. Spek V, Nyklicek I, Smits N, Cuijpers P, Riper H, Keyzer J, Pop V. Internet-based cognitive behavioural therapy for subthreshold depression in people over 50 years old: a randomized controlled clinical trial. Psychol Med. 2007;37:1797–1806. doi: 10.1017/S0033291707000542. [DOI] [PubMed] [Google Scholar]
  21. Ebert D, Tarnowski T, Gollwitzer M, Sieland B, Berking M. A transdiagnostic internet-based maintenance treatment enhances the stability of outcome after inpatient cognitive behavioral therapy: a randomized controlled trial. Psychother Psychosom. 2013;82:246–256. doi: 10.1159/000345967. [DOI] [PubMed] [Google Scholar]
  22. Cuijpers P, Donker T, Van Straten A, Li J, Andersson G. Is guided self-help as effective as face-to-face psychotherapy for depression and anxiety disorders? A systematic review and meta-analysis of comparative outcome studies. Psychol Med. 2010;40:1943–1957. doi: 10.1017/S0033291710000772. [DOI] [PubMed] [Google Scholar]
  23. Kiluk BD, Sugarman DE, Nich C, Gibbons CJ, Martino S, Rounsaville BJ, Carroll KM. A methodological analysis of randomized clinical trials of computer-assisted therapies for psychiatric disorders: toward improved standards for an emerging field. Am J Psychiatry. 2011;168:790–799. doi: 10.1176/appi.ajp.2011.10101443. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Lilienfeld S. Psychological treatments that cause harm. Perspect Psychol Sci. 2007;2:53–70. doi: 10.1111/j.1745-6916.2007.00029.x. [DOI] [PubMed] [Google Scholar]
  25. GesundheitsTraining.online. http://www.geton-training.de.
  26. Buntrock C, Ebert D, Berking M, Riper H, Smit P, Lehr D, Reins J, Boss L, Cuijpers P. Web-based prevention of major depression: study protocol of a randomised controlled trial. BMC Psychiatry. 2014;14:25. doi: 10.1186/1471-244X-14-25. doi: 10.1186/1471-244X-14-25. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Rohde P, Lewinsohn PM, Seeley JR. Comparability of telephone and face-to-face interviews in assessing axis I and II disorders. Am J Psychiatry. 1997;154:1593–1598. doi: 10.1176/ajp.154.11.1593. [DOI] [PubMed] [Google Scholar]
  28. Cuijpers P, Huibers M, Ebert DD, Sanders KL, Andersson G, Daniel Ebert D, Koole SL. How much psychotherapy is needed to treat depression? A metaregression analysis. J Affect Disord. 2013;149:1–13. doi: 10.1016/j.jad.2013.02.030. [DOI] [PubMed] [Google Scholar]
  29. Lewinsohn PM. In: The Psychology of Depression: Contemporary Theory and Research. Friedman RJ, Katz MM, editor. New York: Wiley; 1974. A behavioral approach to depression; pp. 157–185. [Google Scholar]
  30. D’Zurilla TJ, Nezu AM. In: Advances in cognitive-behavioral research and therapy. 1. Kendall PC, editor. New York: Academic Press; 1982. Social problem solving in adults; pp. 201–274. [Google Scholar]
  31. Warmerdam L, Van Straten A, Twisk J, Riper H, Cuijpers P. Internet-based treatment for adults with depressive symptoms: randomized controlled trial. J Med Internet Res. 2008;10:e44. doi: 10.2196/jmir.1094. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Van Straten A, Cuijpers P, Smits N. Effectiveness of a web-based self-help intervention for symptoms of depression, anxiety, and stress: randomized controlled trial. J Med Internet Res. 2008;10:e7. doi: 10.2196/jmir.954. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Mohr DC, Cuijpers P, Lehman K. Supportive accountability: a model for providing human support to enhance adherence to eHealth interventions. J Med Internet Res. 2011;13:e30. doi: 10.2196/jmir.1602. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. German S3-Guideline/National Disease Management Guideline Unipolar Depression. http://www.depression.versorgungsleitlinien.de/
  35. Donker T, Griffiths KM, Cuijpers P, Christensen H. Psychoeducation for depression, anxiety and psychological distress: a meta-analysis. BMC Med. 2009;16:79. doi: 10.1186/1741-7015-7-79. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Hamilton M. A rating scale for depression. J Neurol Neurosur Ps. 1960;23:56–62. doi: 10.1136/jnnp.23.1.56. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Hamilton M. Development of a rating scale for primary depressive illness. Brit J Clin Psychol. 1967;6:278–296. doi: 10.1111/j.2044-8260.1967.tb00530.x. [DOI] [PubMed] [Google Scholar]
  38. Miller IW, Bishop S, Norman WH, Maddever H. The modified Hamilton Rating Scale for Depression: Reliability and validity. Psychiatry Res. 1985;14:131–142. doi: 10.1016/0165-1781(85)90057-5. [DOI] [PubMed] [Google Scholar]
  39. Knesevich JW, Biggs JT, Clayton PJ, Ziegler VE. Validity of the Hamilton Rating Scale for depression. Br J Psychiatry. 1977;131:49–52. doi: 10.1192/bjp.131.1.49. [DOI] [PubMed] [Google Scholar]
  40. Miller IW, Bishop S, Norman WH, Maddever H. The Modified Hamilton Rating Scale for Depression: Reliability and Validity. Psychiat Res. 1984;14:131–142. doi: 10.1016/0165-1781(85)90057-5. [DOI] [PubMed] [Google Scholar]
  41. Rush AJ, Trivedi MH, Ibrahim HM, Carmody TJ, Arnow B, Klein DN, Markowitz JC, Ninan PT, Kornstein S, Manber R, Thase ME, Kocsis JH, Keller MB. The 16-Item Quick Inventory of Depressive Symptomatology (QIDS), Clinician Rating (QIDS-C), and Self-Report (QIDS-SR): A Psychometric Evaluation in Patients with chronic major depression. Biol Psychiatry. 2003;54:573–583. doi: 10.1016/S0006-3223(02)01866-8. [DOI] [PubMed] [Google Scholar]
  42. Trivedi M, Rush A, Ibrahim H, Carmody T, Biggs M, Suppes T, Crismon M, Shores-Wilson K, Toprac M, Dennehy E, Witte B, Kasher T. The Inventory of Depressive Symptomatology, Clinician Rating (IDS-C) and Self-Report (IDS-SR), and the Quick Inventory of Depressive Symptomatology, Clinician Rating (QIDS-C) and Self-Report (QIDS-SR) in public sector patients with mood disorders: a psychometric evaluation. Psychol Med. 2004;34:73–82. doi: 10.1017/S0033291703001107. [DOI] [PubMed] [Google Scholar]
  43. Rush AJ, Gullion CM, Basco MR, Jarrett RB, Trivedi MH. The Inventory of Depressive Symptomatology (IDS): psychometric properties. Psychol Med. 1996;26:477. doi: 10.1017/S0033291700035558. [DOI] [PubMed] [Google Scholar]
  44. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders IV-TR. Washington, DC: American Psychiatric Association; 2000. [Google Scholar]
  45. Kroenke K, Spitzer RL. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16:606–613. doi: 10.1046/j.1525-1497.2001.016009606.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Kroenke K, Spitzer RL. The PHQ-9: A New Depression Diagnostic and Severity Measure. Psychiatr Ann. 2002;32:1–7. [Google Scholar]
  47. Gandek B, Ware JE, Aaronson NK, Apolone G, Bjorner JB, Brazier JE, Bullinger M, Kaasa S, Leplege A, Prieto L, Sullivan M. Cross-validation of item selection and scoring for the SF-12 Health Survey in nine countries: results from the IQOLA Project. International Quality of Life Assessment. J Clin Epidemiol. 1998;51:1171–1178. doi: 10.1016/S0895-4356(98)00109-7. [DOI] [PubMed] [Google Scholar]
  48. Zigmond AS, Snaith RP. The hospital anxiety and depression scale. Acta Psychiatr Scand. 1983;67:361–370. doi: 10.1111/j.1600-0447.1983.tb09716.x. [DOI] [PubMed] [Google Scholar]
  49. Bjelland I, Dahl AA, Haug TT, Neckelmann D. The validity of the Hospital Anxiety and Depression Scale. An updated literature review. J Psychosom Res. 2002;52:69–77. doi: 10.1016/S0022-3999(01)00296-3. [DOI] [PubMed] [Google Scholar]
  50. D’Zurilla TJ, Nezu AM, Maydeu-Olivares A. Social Problem-Solving Inventory-Revised (SPSIR) New York: MHS; 2002. [Google Scholar]
  51. Manos RC, Kanter JW, Luo W. The behavioral activation for depression scale-short form: development and validation. Behav Ther. 2011;42:726–739. doi: 10.1016/j.beth.2011.04.004. [DOI] [PubMed] [Google Scholar]
  52. Primack BA. The WHO-5 wellbeing index performed the best in screening for depression in primary care. ACP J Club. 2003;139:50. [PubMed] [Google Scholar]
  53. Reinert DF, Allen JP. The Alcohol Use Disorders Identification Test (AUDIT): a review of recent research. Alcohol Clin Exp Res. 2002;26:272–279. doi: 10.1111/j.1530-0277.2002.tb02534.x. [DOI] [PubMed] [Google Scholar]
  54. Devilly GJ, Borkovec TD. Psychometric properties of the credibility/expectancy questionnaire. J Behav Ther Exp Psychiatry. 2000;31:73–86. doi: 10.1016/S0005-7916(00)00012-4. [DOI] [PubMed] [Google Scholar]
  55. Nguyen TD, Attkisson CC, Stegner BL. Assessment of patient satisfaction: development and refinement of a service evaluation questionnaire. Eval Program Plann. 1983;6:299–313. doi: 10.1016/0149-7189(83)90010-1. [DOI] [PubMed] [Google Scholar]
  56. Schmidt J, Lamprecht F, Wittmann WW. Satisfaction with inpatient management. Development of a questionnaire and initial validity studies. [Article in German] Psychother Psychosom Med Psychol. 1989;39:248–255. [PubMed] [Google Scholar]
  57. Attkisson CC, Zwick R. The client satisfaction questionnaire. Psychometric properties and correlations with service utilization and psychotherapy outcome. Eval Program Plann. 1982;5:233–237. doi: 10.1016/0149-7189(82)90074-X. [DOI] [PubMed] [Google Scholar]
  58. Jacobson NS, Truax P. Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol. 1991;59:12–19. doi: 10.1037//0022-006x.59.1.12. [DOI] [PubMed] [Google Scholar]
  59. Fischer EH, Farina A. Attitudes toward seeking professional psychological help: a shortened form and considerations for research. J Coll Stud Dev. 1995;36:368–373. [Google Scholar]
  60. Elhai JD, Schweinle W, Anderson SM. Reliability and validity of the Attitudes Toward Seeking Professional Psychological Help Scale-Short Form. Psychiatry Res. 2008;159:320–329. doi: 10.1016/j.psychres.2007.04.020. [DOI] [PubMed] [Google Scholar]
  61. Bryk R, Raudenbush SW, Bryk AS. Hierarchical Linear Models: Applications and Data Analysis Methods. 2. Newbury Park, CA: Sage Publications; 2002. [Google Scholar]
  62. Schafer JL, Graham JW. Missing data: our view of the state of the art. Psychol Methods. 2002;7:147–177. [PubMed] [Google Scholar]
  63. Hedeker D, Gibbons RD. Application of random-effects pattern-mixture models for missing data in longitudinal studies. Psychol Methods. 1997;2:64–78. [Google Scholar]
  64. Cohen J. Statistical power analysis for the behavioral sciences. Hillsdale, NY: Erlbaum; 1988. [Google Scholar]
  65. Willan AR, O’Brien BJ, Cook DJ. Benefit-risk ratios in the assessment of the clinical evidence of a new therapy. Control Clin Trials. 1997;18:121–130. doi: 10.1016/S0197-2456(96)00092-X. [DOI] [PubMed] [Google Scholar]
  66. Kessler D, Lewis G, Kaur S, Wiles N, King M, Weich S, Sharp DJ, Araya R, Hollinghurst S, Peters TJ. Therapist-delivered internet psychotherapy for depression in primary care: a randomised controlled trial. Lancet. 2009;22:628–634. doi: 10.1016/S0140-6736(09)61257-5. 374. [DOI] [PubMed] [Google Scholar]

Articles from Trials are provided here courtesy of BMC

RESOURCES