Skip to main content
PMC home page
Journal of Medical Internet Research logoLink to Journal of Medical Internet Research
. 2018 Nov 9;20(11):e10113. doi: 10.2196/10113

The Generalizability of Randomized Controlled Trials of Self-Guided Internet-Based Cognitive Behavioral Therapy for Depressive Symptoms: Systematic Review and Meta-Regression Analysis

Lorenzo Lorenzo-Luaces 1,✉, Emily Johns 1, John R Keefe 2
Editor: Gunther Eysenbach
Reviewed by: Nicolas Hoertel, Peter Bower, Nicholas Forand, Thomas Berger
PMCID: PMC6251981  PMID: 30413400

Abstract

Background

Self-guided internet-based cognitive behavioral therapies (iCBTs) for depressive symptoms may substantially increase accessibility to mental health treatment. Despite this, questions remain as to the generalizability of the research on self-guided iCBT.

Objective

We sought to describe the clinical entry criteria used in studies of self-guided iCBT, explore the criteria’s effects on study outcomes, and compare the frequency of use of these criteria with their use in studies of face-to-face psychotherapy and antidepressant medications. We hypothesized that self-guided iCBT studies would use more stringent criteria that would bias the sample toward those with a less complex clinical profile, thus inflating treatment outcomes.

Methods

We updated a recently published meta-analysis by conducting a systematic literature search in PubMed, MEDLINE, PsycINFO, and EMBASE. We conducted a meta-regression analysis to test the effect of the different commonly used psychiatric entry criteria on the treatment-control differences. We also compared the frequency with which exclusion criteria were used in the self-guided iCBT studies versus studies of face-to-face psychotherapy and antidepressants from a recently published review.

Results

Our search yielded 5 additional studies, which we added to the 16 studies identified by Karyotaki and colleagues in 2017. Few self-guided iCBT studies excluded patients with severe depressive symptoms (6/21, 29%), but self-guided iCBT studies were more likely than antidepressant (14/170, 8.2%) studies to use this criterion. However, self-guided iCBT studies did not use this criterion more frequently than face-to-face psychotherapy studies (6/16, 38%). Beyond this, we found no evidence that self-guided iCBTs used more stringent entry criteria. Strong evidence suggested that they were actually less likely to use most entry criteria, especially exclusions on the basis of substance use or personality pathology. None of the entry criteria used had an effect on outcomes.

Conclusions

A conservative interpretation of our findings is that the patient population sampled in the literature on self-guided iCBT is relatively comparable with that of studies of antidepressants or face-to-face psychotherapy. Alternatively, studies of unguided cognitive behavioral therapy may sample from a more heterogeneous and representative patient population. Until evidence emerges to suggest otherwise, the patient population sampled in self-guided iCBT studies cannot be considered as less complex than the patient population from face-to-face psychotherapy or antidepressant studies.

Keywords: depression, psychotherapy, CBT, internet-based therapy, pharmacotherapy, generalizability, exclusion criteria, cognitive therapy, telemedicine, drug therapy, patient selection

Introduction

Background

Depression is one of the leading causes, if not the leading cause, of disability worldwide [1,2]. A well-recognized barrier to reducing the level of disability associated with depression and other common mental disorders is the scarcity of available treatment providers [3]. Although there is a scarcity of providers, the research data support a wide variety of treatments for depression that vary in their modality (eg, self-management, psychotherapy, medications), as well as in how time and resource intensive they are for patients and providers. The combination of antidepressants and psychotherapy is widely seen as providing the best cost-benefit ratio for severe depression [4]. However, it is nearly impossible to advocate this or any other treatment approach as a first-line intervention because depression is highly heterogeneous in its prognosis [5-7]. For example, 50% to 75% of patients with first-onset major depressive episodes recover within a 3- to 6-month period, but 15% to 20% of cases are chronic [8-11]. Similarly, although 50% of patients recover and do not experience a relapse in 20- to 30-year follow-ups, approximately 35% of individuals experience a recurrent course [9,12,13]. Moreover, the presentation of depressive symptoms is highly heterogeneous. Approximately half of cases are classified as mild to moderate [14], but a proportion of cases involve psychotic symptoms [15] or suicidal risk (eg, 27%-53% [16]). As well, there is a substantial degree of comorbidity with other disorders [14].

A major advancement in the study of treatments for depression is the discovery that self-guided internet-based cognitive behavioral therapy (iCBT) can be more effective than control conditions [17]. In these interventions, patients complete interactive Web-based programs based on the principles of cognitive behavioral therapy with no therapeutic support, although sometimes technical support is available. The efficacy of these interventions can be enhanced by therapist support, although this effect may be smaller than previously thought (eg, standardized mean difference symptoms = 0.27 [18-20]). The high prevalence of depression and the ubiquity of internet access and mobile phone ownership make self-guided iCBTs hugely promising in reducing the burden of disability associated with depressive symptoms, even if they were somewhat less effective than guided iCBTs [21]. Kazdin and Blase [3] have called attention to the fact that the number of individuals with mental disorders such as depression far outnumbers the number of mental health providers available to deliver treatment. Thus, self-guided iCBT has the potential for a greater public health impact than guided iCBT or other forms of treatment that require contact with a trained professional [21]. However, a key concern in evaluating the self-guided iCBT treatment literature is the degree to which the patient population in randomized controlled trials (RCTs) is representative of the heterogeneous nature of the prognosis, severity, and comorbidity found in depressed patients. More than 20 years ago, Seligman [22] pointed out that the literature attesting to the efficacy of face-to-face psychotherapy was limited by virtue of the inclusion and exclusion criteria used, which excluded patients with subthreshold symptoms as well as comorbid conditions. Westen and Morrison [23] quantified the nature of this problem, reporting that most (68%) patients with depression were excluded from a typical psychotherapy study because of suicidality, comorbid disorders, especially substance use, and subclinical symptoms. Furthermore, these authors reported that the number of participants excluded from psychotherapy trials was related to the study-level effect size, such that studies that excluded more patients tended to find larger effect sizes for psychotherapy (see also van der Lem et al [24]). Zimmerman et al [25] made similar observations in the literature on antidepressants. More recently, Zimmerman and colleagues [26-29] reported on the inclusion and exclusion (henceforth entry) criteria for RCTs of antidepressant medications. They stated that most studies excluded patients on the basis of a minimum symptom threshold, suicidality, psychotic features, and substance use disorders.

Objective

We sought to review the entry criteria used in trials of self-guided iCBT. We did this by updating and coding the studies from the latest meta-analysis of self-guided iCBT, which was published by Karyotaki et al [17]. In addition to reporting the overall frequency with which specific criteria were used, we sought to explore the relationship between the specific entry criteria and the study-level outcome. Finally, to provide some context for evaluating the frequency with which self-guided iCBT studies used different exclusion criteria, we compared the frequency of use of entry criteria in self-guided iCBT trials versus the rate of inclusion criteria used in face-to-face psychotherapy trials and trials of antidepressant medications. The aim of this analysis was for face-to-face psychotherapy and antidepressant studies to serve as a kind of benchmark against which to compare the self-guided iCBT literature. A prior study suggested that face-to-face psychotherapy trials were less likely than antidepressant trials to use most of the specific entry criteria coded by the study authors [30]. Other studies similarly suggested that psychotherapy studies may be somewhat more generalizable than pharmacotherapy studies for adolescent depression [31], borderline personality disorder [32], social anxiety [33], and posttraumatic stress disorder [34], although not generalized anxiety disorder [35]. Based on prior data [23], we hypothesized that the use of more stringent entry criteria would be associated with better outcomes. However, because studies of self-guided iCBT are often fully remote and lack human support, we hypothesized that studies of self-guided iCBT would use more stringent psychiatric entry criteria than would face-to-face psychotherapy and pharmacotherapy trials.

Methods

Identification and Rating of Studies

We obtained RCTs exploring the efficacy of self-guided iCBT for depression in adults by referencing and updating the latest meta-analysis of self-guided iCBT for depression [17]. Those authors consulted a broad database of psychological treatments for depression that was constructed from a systematic literature search of free terms combining “psychotherapy” and “depression” in PubMed, EMBASE, PsycINFO, and the Cochrane Library published up to January 1, 2016. We updated the review by applying the same search criteria (see Multimedia Appendix 1), narrowing our search for internet-based interventions. We included studies if they included adults (>18 years of age) with a diagnosis of major depressive disorder on a psychiatric semistructured interview, with elevated symptoms of depression (ie, any specific cutoff score on a depression questionnaire), or who were seeking or undergoing treatment for depression. Two of the authors (LLL, EJ) rated all articles for the presence of common psychiatric inclusion and exclusion criteria used in treatment studies [26], resolving discrepancies by consensus. Descriptive analyses summarizing the specific entry criteria of the iCBT trials are presented. We rated risk of bias according to the Cochrane Collaboration risk-of-bias assessment tool [36]. We rated the primary outcome studies. When we could not determine a specific domain category from the main outcome study, we rated protocol articles if they were available or any publicly available study registry (eg, ClinicalTrials.gov).

Meta-Regression

We calculated Hedges g to quantify the difference between self-guided iCBT and control conditions on symptoms of depression. Hedges g is derived from the difference between the average posttreatment scores on self-reported measures of depressive symptoms in the 2 groups (ie, self-guided iCBT group vs the control group), divided by the pooled standard deviation while adjusting for small-sample bias. For entry criteria that had sufficient representation in the dataset (ie, ≥3 studies used them), we conducted meta-regressions, using the R statistical computing language version R-3.5.0. (R Foundation) package metafor [37], to examine the relationship between using specific entry criteria and outcomes. First, we conducted individual meta-regressions in which we regressed the study-level effect size on each of the exclusion criteria used. Then, we regressed the number of exclusion criteria used on the study-level outcomes. Finally, we conducted a simultaneous meta-regression in which we regressed outcomes on all entry criteria.

Meta-regressions were run in a random effects meta-analytic framework, using exclusion criteria either as dummy-coded categorical variables (0/1) or as an ordinal variable (ie, the number of exclusions). In line with current best practices for calculating confidence intervals with more accurate coverage and less-inflated type I error, we used the Sidik-Jonkman random effects estimator and the Hartung-Knapp adjustment [38,39].

Benchmarking Against Antidepressant and Face-to-Face Psychotherapy

To provide a rough index of how entry criteria for guided iCBT studies compare with those of other treatments, we drew on recent reviews of the entry criteria of trials of adult depression with antidepressants [26-29] and face-to-face psychotherapy [30]. These reviews employed searches in PubMed, EMBASE, and PsycINFO, as well as individual meta-analyses and specific journals, to identify acute treatment outcomes studies for depression. The search was limited to studies in which antidepressants or face-to-face psychotherapy was compared with a control condition, so studies of multiple treatments (eg, 2 psychotherapies) were only included if a control condition was used. There are systematic differences between antidepressant and face-to-face pharmacotherapy studies regarding the type of controls employed [40]. Virtually all pharmacotherapy studies use a pill placebo, whereas the efficacy of face-to-face psychotherapy, as well as self-guided iCBT, is tested with a more diverse mix of controls, including a waiting list, treatment as usual, pill placebos, and other conditions (eg, relaxation) that are intended as a control for nonspecific effects (eg, attention). Thus, we searched for face-to-face psychotherapy trials in which a waiting list, treatment as usual, or placebo control was used. We excluded trials if they focused exclusively on comorbidities (eg, only depression and alcohol use), whether psychiatric or general medical, as they are, by definition, less inclusive. As well, we did not include trials focused on subtypes of depression, inpatients, or patients with specific symptom profiles (eg, cognitive symptoms). The application of these criteria yielded 170 studies of antidepressants and 16 studies of face-to-face psychotherapy. To compare the differences between the self-guided iCBT studies and studies of antidepressants and face-to-face psychotherapies, we used a chi-square test, or Fisher exact test when we expected any cell to have a frequency lower than 5.

Results

Study Characteristics

We identified 5 new studies (see Multimedia Appendix 1) since the publication of the review by Karyotaki et al [17] that met our inclusion criteria (see Figure 1). Thus, we coded 21 RCTs comparing self-guided iCBTs versus a control in our updated review. These studies analyzed data for 4781 participants. Like Karyotaki et al, we found risk of bias to be low across most studies (see Multimedia Appendix 1).

Figure 1.

Figure 1

Open in a new tab

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart for self-guided internet-based cognitive behavioral therapy (iCBT) studies included in the systematic review and meta-regression analysis. CBT: cognitive behavioral therapy.

There were differences between the self-guided iCBT studies, the antidepressant studies, and the face-to-face psychotherapy studies in terms of sample size per study arm (F2,204=6.45, P=.002) and country of origin (χ21=48.3, P<.001), but no differences in the percentage of studies that excluded participants based on an upper age limit (χ21=1.5, P=.48). Antidepressant (mean 120.69, SD 74.10) and self-guided iCBT (mean 140.61, SD 109.16) studies tended to have more participants in each study arm than face-to-face psychotherapy studies did (mean 55.12, SD 39.48). Most self-guided iCBT studies originated from single sites within European countries (14/21, 67%). By way of contrast, only 5 of the psychotherapy studies were conducted in Europe (5/16, 31%) and, instead, the face-to-face psychotherapy studies were more likely to be published in the United States (8/16, 50%). Very few antidepressants studies were published from within single European countries (15/170, 8.8%), as most were published in the United States (101/171, 59.4%).

Table 1 lists the psychiatric inclusion and exclusion criteria used in the self-guided iCBT trials.

Table 1.

Psychiatric exclusion criteria of randomized controlled trials of self-guided internet-based cognitive behavioral therapy for depressive symptoms (N=21).

Exclusion criteria n (%)
Severity scale score below the cutoff (minimum score) 15 (71)
Severity scale score above the cutoff (maximum score) 6 (29)
Psychosis 12 (57)
Substance abuse or dependence 5 (24)
Significant suicidal ideation or risk 13 (62)
History of suicide attempt(s) 1 (5)
Episode length too long 1 (5)
Episode length too short 1 (5)
Any axis II disorder 0 (0)
Any axis I disorder 1 (5)
Borderline personality disorder 0 (0)
Antisocial personality disorder 0 (0)
Schizotypal personality disorder 0 (0)
Open in a new tab

At least half of the iCBT studies used the following 3 criteria: psychotic disorder or current psychotic symptoms (12/21, 57%), a minimum symptom severity on a depression scale (15/21, 71%), and significant suicidal ideation (13/21, 62%). The other criteria were used relatively infrequently. As has been noted of studies of antidepressants and face-to-face psychotherapy, there was considerable variability in the operationalization of these criteria. For example, of the 15 studies that excluded patients on the basis of a minimum score on a depression severity measure, 6 used the Patient Health Questionnaire-8 or -9 as an exclusion criterion, 4 used the Beck Depression Inventory (BDI), and 2 used the Kessler Psychological Distress Scale. The remaining 3 studies used other scales (eg, the Center for Epidemiologic Studies Depression Scale [CES-D]).

Effects of Entry Criteria on Outcomes

Only 5 of the exclusionary criteria were used with enough frequency (ie, >3 uses) that we could explore whether their use was associated with treatment outcomes: minimum symptom severity, maximum symptom severity, psychosis, substance misuse, and suicidality. Table 2 lists the results of the meta-regressions in which we regressed the study-level effect sizes in the 5 exclusionary criteria, individually as well as considered simultaneously. The updated meta-analysis found results similar to those by Karyotaki et al [17]. At the study level, self-guided iCBT was associated with more improvement in depression than were the control conditions (g=0.33, 95% CI 0.20-0.46, SE 0.06; P<.001).

Table 2.

Meta-regression coefficients for the relationship between individual exclusion criteria and study-level internet-based cognitive behavioral therapy controlled outcomes. No findings were significant at P<.05.

Exclusion criteria Single-predictor models, g (95% CI) Simultaneous-predictor model, g (95% CI)
Minimum symptom severity –0.12 (–0.41 to 0.17) –0.18 (–0.52 to 0.16)
Maximum symptom severity –0.04 (–0.33 to 0.25) –0.10 (–0.45 to 0.26)
Psychosis 0.07 (–0.20 to 0.34) 0.04 (–0.38 to 0.46)
Substance problems 0.02 (–0.29 to 0.33) 0.05 (–0.39 to 0.49)
Suicidality 0.09 (–0.18 to 0.36) 0.16 (–0.24 to 0.55)
Open in a new tab

We detected a high degree of between-study heterogeneity of effect sizes (I2=76%), indicating that exclusion criteria could potentially explain between-studies differences in effects. Despite this, the meta-regressions did not find any significant effects of specific exclusionary criteria (P values >.39) or the total number of criteria used on study-level treatment outcomes (B=–0.01, 95% CI –0.09 to 0.10, SE 0.05; P=.92). The study-level exclusion criterion that appeared to have the largest effect on study-level differences was the use of a minimum of symptom severity as an exclusion (B=–0.12, 95% CI –0.41 to 0.17, SE 0.14; P=.39). A simultaneous regression considering all of the exclusion criteria in tandem likewise did not find any significant effect of the use of specific entry criteria on outcomes (P values >.28).

Comparison of Entry Criteria in Internet-Based Cognitive Behavioral Therapies, Antidepressants, and Psychotherapy

Table 3 shows the comparisons of the frequency with which different entry criteria were used in studies of self-guided iCBT versus studies of face-to-face psychotherapy versus studies of antidepressant medications. Contrary to our hypothesis, virtually all the exclusion criteria coded were used less frequently in the iCBT trials than in face-to-face psychotherapy and antidepressant trials, though not all these differences were statistically significant.

Table 3.

Comparison of the frequency of use of different inclusion and exclusion criteria across self-guided internet-based cognitive behavioral therapy (iCBT), face-to-face psychotherapy (F2F), and antidepressant medication (AM) trials.

Exclusion criteria iCBT (n=21) F2F (n=16) AM (n=170) P value
iCBT vs F2F vs AM iCBT vs F2F iCBT vs AM
Minimum symptom severity 15 (71) 13 (81) 170 (100.0) <.001 .70 <.001
Maximum symptom severity 6 (29) 6 (38) 14 (8.2) <.001 .83 .01
Psychosis 12 (57) 14 (88) 143 (84.1) .02 .07 .01
Substance abuse or dependence 5 (24) 12 (75) 137 (80.6) <.001 <.001 <.001
Suicidal risk 13 (62) 9 (56) 128 (75.3) .14 .99 .29
Prior suicide attempt(s) 1 (5) 2 (12) 35 (20.6) .21 .57 .13
Episode length too long 1 (5) 2 (12) 34 (20.0) .21 .57 .13
Episode length too short 1 (5) 2 (12) 81 (47.6) <.001 .57 <.001
Any axis II disorder 0 (0) 3 (19) 60 (35.3) <.001 .07 <.001
Any axis I disorder 1 (5) 1 (6) 46 (27.1) .01 >.99 .03
Borderline personality disorder 0 (0) 5 (31) 70 (41.2) <.001 .01 <.001
Antisocial personality disorder 0 (0) 5 (31) 68 (40.0) <.001 .01 <.001
Schizotypal personality disorder 0 (0) 5 (31) 63 (37.1) <.001 .01 <.001
Open in a new tab

When specifically compared with studies of antidepressant medications, studies of self-guided iCBT were less likely to use almost all the exclusion criteria coded. For example, compared with studies of antidepressants (143/170, 84.1%), self-guided iCBT studies were less likely (12/21, 57%) to exclude patients on the basis of psychotic symptoms or a diagnosis (χ21=7.2, P=.01). Similarly, 41.2% (70/170) of studies on antidepressants excluded patients on the basis of borderline personality disorder, although no self-guided iCBT study used this criterion (Fisher P<.001). The only exception to this general pattern of self-guided iCBT studies being more, rather than less, inclusive was on the basis of a maximum symptom severity exclusion. Only 29% (6/21) of the self-guided iCBT studies excluded patients on the basis of high symptom severity. However, this exclusion occurred more frequently in the iCBT studies than in the antidepressant studies (14/170, 8.2%; χ21=6.2, P=.01). The self-guided iCBT studies were no more likely than face-to-face psychotherapy studies (6/16, 38%; χ21=.05, P=.83) to use this criterion.

Most of the differences between self-guided iCBT and face-to-face psychotherapy studies did not meet the statistical significance threshold of P<.05, although the arithmetic differences were often in favor of the self-guided iCBT studies. Face-to-face therapy studies were more likely to specifically exclude participants based on the diagnoses of borderline, schizotypal, or antisocial personality disorder (each 5/16, 31%) than were the iCBT studies, which did not use this exclusion (each 0/21, 0%; Fisher P<.001).

By far the largest observed difference between the iCBT studies and the face-to-face and antidepressants studies was that iCBT studies infrequently excluded patients on the basis of a substance use disorder (5/21, 24%), though this exclusion was typical in antidepressant (137/170, 80.6%; Fisher P<.001) and face-to-face psychotherapy studies (12/16, 75%; Fisher P=.003).

Discussion

Principal Findings

We summarized the exclusion criteria used in self-guided iCBT studies, explored the relationship of the use of these criteria and the treatment effect size reported in the trial, and compared the frequencies with which such criteria were used in antidepressant trials or trials of face-to-face psychotherapy. Overall, self-guided iCBT studies infrequently used exclusion criteria that are the norm in studies of depression. Contrary to our hypotheses, we did not find the type and number of exclusion criteria to be related to outcomes. Also contrary to our hypothesis, self-guided iCBT trials were either equally likely or less likely to use specific psychiatric entry criteria.

Limitations

A noteworthy limitation of this review is the small number of self-guided iCBT studies. It is possible that the entry criteria used had a small effect on outcomes, but a larger number of studies is needed to detect this effect. Our review was limited to studies of self-guided iCBTs. While these interventions may have a broader public health impact than guided iCBT [21], there is evidence that they have somewhat smaller effects and lower rates of treatment completion [18-20]. Future research should study the entry criteria in studies of guided iCBT.

The fact that studies did not use a specific entry criterion (eg, substance use disorder) does not imply that the trial actually had patients representative of that feature, which is another limitation of our study. Because studies do not uniformly report on all characteristics of their patient sample (eg, substance use, duration, suicide risk), it is impossible to test for the presence of these features of the sample across all the studies. It is also possible that studies used specific entry criteria but did not disclose them in the published report. Moreover, the psychiatric inclusion and exclusion criteria are only one aspect of the generalizability of the sample. Self-guided iCBT studies may have more selective patient samples by virtue of other criteria that they use explicitly (eg, access to the internet) or implicitly (eg, willingness to participate in an iCBT study).

While we updated the review by Karyotaki et al [17], we drew on published reviews by Lorenzo-Luaces et al [30], instead of updating this review as well. A difference between the 2 reviews is that the review by Lorenzo-Luaces and colleagues was limited to studies in which patients had a diagnosis of major depressive disorder. This difference specifically limited the number of face-to-face therapy studies eligible for coding. Our aim, however, was not to provide an exact estimate of the difference in frequency of use of exclusion criteria, but to provide a benchmarking context to evaluate the frequency with which self-guided iCBT studies excluded patients. The studies also differed by year of publications, as per the search strategy, country of origin, and size. These differences, along with the differences we identified as per our objectives, suggest that the studies are not directly or indirectly comparable (eg, as in a network meta-analysis). Our aim, however, was not to claim that the evidence base behind self-guided iCBTs is equivalent to the evidence base behind antidepressants or face-to-face psychotherapy but to provide a frame of reference against which to compare the entry criteria of self-guided iCBTs.

Implications

Only 29% of the self-guided iCBT studies excluded patients on the basis of severe depression, but it is still worth noting that self-guided iCBT studies appeared more likely than antidepressant studies to exclude participants on this basis. The perception that self-guided iCBT will not be effective for cases of more severely symptomatic depression aligns with common sense but is not supported by research data. For example, Bower et al [41] reported that the effects of self-guided internet-based therapies were more rather than less pronounced among patients high in symptom severity. In our analyses, we found no evidence that the effects of iCBT varied strongly according to exclusions by high or low symptom severity. It is possible that self-guided iCBTs are less effective for patients who have more complex presentations, with symptom severity being only one index of case complexity [5]. For example, the presence of anxiety and chronic depression duration have all been implicated in treatment outcomes in depression and may relate to lower response to iCBT [42]. Although we did not find the use of psychosis as an exclusion criterion to be related to outcomes, psychotic depression is relatively rare, and even studies allowing these patients into the trial may have had a low representation of psychotic depression. In contrast, approximately 50% of cases of depression are rated as severe or very severe. It is probable that patients self-select into treatment trials in a way that patients with severe depression avoid iCBT studies, but the data do not strongly support this conclusion. For example, Karyotaki et al [17] reported a mean score of 28 on the BDI for patients in their dataset along with a score of 26 on the CES-D, both of which are in the moderate to severe range for each measure.

Self-guided iCBT studies often excluded patients on the basis of suicidal risk. Our findings suggest that this exclusion criterion was not associated with outcomes, a finding that mirrors the results of van der Lem et al [24], as well as the fact that suicidal risk is not consistently reported as a predictor of treatment outcomes with face-to-face psychotherapy or antidepressants. A common exclusion criterion in antidepressant and face-to-face psychotherapy studies is substance use disorder, although this was missing from most self-guided iCBT studies. The findings of van der Lem et al [24] suggested that alcohol use disorder is not a strong predictor of outcomes in treatment for depression, and other studies have also not found an association between substance use disorder and outcomes in treatment for depression [43]. Given that there is no evidence to suggest negative outcomes for suicidal patients and those with substance use disorders, as well as the low rate of treatment utilization in these and other patient groups, more self-guided iCBT studies specifically recruiting from these patient groups are needed. Hoertel et al [44] have argued that using, or not using, specific criteria may influence not only the external validity and generalizability of a study but also its internal validity, by virtue of influencing outcomes. This runs counter to the prevailing logic that there is a trade-off between external and internal validity. Overall, our results suggest that the inclusion criteria commonly used in the field sacrifice external validity but provide no gains in internal validity.

Conclusions

To our knowledge, this is the first exploration of the effects of inclusion and exclusion criteria in self-guided iCBT studies, as well as the first comparison of the specific inclusion and exclusion criteria used in RCTs of iCBTs versus studies of face-to-face psychotherapy or antidepressants. Our findings can be taken to suggest that self-guided iCBT studies are more inclusive by design than studies of antidepressants or face-to-face psychotherapy. It is possible that, by using remote designs in which no individual face-to-face interviews can be conducted, self-guided iCBT trials limit their ability to exclude participants on specific features. For example, no self-guided iCBT trial excluded participants due to personality pathology broadly construed or borderline, schizotypal, or antisocial personality diagnosis, which are usually measured in psychiatric interviews, not self-reported questionnaires. By comparison, around a third of psychotherapy and around 40% of antidepressant trials used these exclusions. Given the differences in the entry criteria used, it stands to reason that, by design, studies of self-guided iCBT may be characterized by a more heterogeneous group of patients than studies of antidepressant or face-to-face psychotherapy. This level of heterogeneity in the underlying patient population increases the external validity of the research. Moreover, this variability can facilitate the discovery of process-outcome correlations [45], as well as effects of individual patient differences on outcomes [5]. This heterogeneity in the patient population may also contribute to heterogeneity in the overall effect size reported in treatment-control comparisons. Until evidence emerges to the contrary, however, it cannot be said that iCBT studies apply more stringent inclusion and exclusion criteria than studies of other treatments for depression or that the efficacy of the treatments is inflated by using a less complex or severe patient population.

Acknowledgments

We would like to thank Eirini Karyotaki, Pim Cuijpers, and the coauthors of the meta-analysis from which we drew the studies for this work, as well as the primary authors of the studies we analyzed.

Abbreviations

BDI

Beck Depression Inventory

CES-D

Center for Epidemiologic Studies Depression Scale

iCBT

internet-based cognitive behavioral therapy

RCT

randomized controlled trial

Multimedia Appendix 1

List of studies, risk-of-bias assessments, and example search terms.

Footnotes

Conflicts of Interest: None declared.

References

  • 1.World Health Organization . Depression and Other Common Mental Disorders: Global Health Estimates. Geneva, Switzerland: WHO; 2017. [2018-10-29]. http://apps.who.int/iris/bitstream/handle/10665/254610/WHO-MSD-MER-2017.2-eng.pdf;jsessionid=D11B5ED6C69A736C263041085DAEFD55?sequence=1 . [Google Scholar]
  • 2.Murray CJL, Vos T, Lozano R, Naghavi M, Flaxman AD, Michaud C, Ezzati M, Shibuya K, Salomon JA, Abdalla S, Aboyans V, Abraham J, Ackerman I, Aggarwal R, Ahn SY, Ali MK, Alvarado M, Anderson HR, Anderson LM, Andrews KG, Atkinson C, Baddour LM, Bahalim AN, Barker-Collo S, Barrero LH, Bartels DH, Basáñez M, Baxter A, Bell ML, Benjamin EJ, Bennett D, Bernabé E, Bhalla K, Bhandari B, Bikbov B, Bin AA, Birbeck G, Black JA, Blencowe H, Blore JD, Blyth F, Bolliger I, Bonaventure A, Boufous S, Bourne R, Boussinesq M, Braithwaite T, Brayne C, Bridgett L, Brooker S, Brooks P, Brugha TS, Bryan-Hancock C, Bucello C, Buchbinder R, Buckle G, Budke CM, Burch M, Burney P, Burstein R, Calabria B, Campbell B, Canter CE, Carabin H, Carapetis J, Carmona L, Cella C, Charlson F, Chen H, Cheng AT, Chou D, Chugh SS, Coffeng LE, Colan SD, Colquhoun S, Colson KE, Condon J, Connor MD, Cooper LT, Corriere M, Cortinovis M, de VKC, Couser W, Cowie BC, Criqui MH, Cross M, Dabhadkar KC, Dahiya M, Dahodwala N, Damsere-Derry J, Danaei G, Davis A, De LD, Degenhardt L, Dellavalle R, Delossantos A, Denenberg J, Derrett S, Des JDC, Dharmaratne SD, Dherani M, Diaz-Torne C, Dolk H, Dorsey ER, Driscoll T, Duber H, Ebel B, Edmond K, Elbaz A, Ali SE, Erskine H, Erwin PJ, Espindola P, Ewoigbokhan SE, Farzadfar F, Feigin V, Felson DT, Ferrari A, Ferri CP, Fèvre EM, Finucane MM, Flaxman S, Flood L, Foreman K, Forouzanfar MH, Fowkes FGR, Fransen M, Freeman MK, Gabbe BJ, Gabriel SE, Gakidou E, Ganatra HA, Garcia B, Gaspari F, Gillum RF, Gmel G, Gonzalez-Medina D, Gosselin R, Grainger R, Grant B, Groeger J, Guillemin F, Gunnell D, Gupta R, Haagsma J, Hagan H, Halasa YA, Hall W, Haring D, Haro JM, Harrison JE, Havmoeller R, Hay RJ, Higashi H, Hill C, Hoen B, Hoffman H, Hotez PJ, Hoy D, Huang JJ, Ibeanusi SE, Jacobsen KH, James SL, Jarvis D, Jasrasaria R, Jayaraman S, Johns N, Jonas JB, Karthikeyan G, Kassebaum N, Kawakami N, Keren A, Khoo J, King CH, Knowlton LM, Kobusingye O, Koranteng A, Krishnamurthi R, Laden F, Lalloo R, Laslett LL, Lathlean T, Leasher JL, Lee YY, Leigh J, Levinson D, Lim SS, Limb E, Lin JK, Lipnick M, Lipshultz SE, Liu W, Loane M, Ohno SL, Lyons R, Mabweijano J, MacIntyre MF, Malekzadeh R, Mallinger L, Manivannan S, Marcenes W, March L, Margolis DJ, Marks GB, Marks R, Matsumori A, Matzopoulos R, Mayosi BM, McAnulty JH, McDermott MM, McGill N, McGrath J, Medina-Mora ME, Meltzer M, Mensah GA, Merriman TR, Meyer A, Miglioli V, Miller M, Miller TR, Mitchell PB, Mock C, Mocumbi AO, Moffitt TE, Mokdad AA, Monasta L, Montico M, Moradi-Lakeh M, Moran A, Morawska L, Mori R, Murdoch ME, Mwaniki MK, Naidoo K, Nair MN, Naldi L, Narayan KMV, Nelson PK, Nelson RG, Nevitt MC, Newton CR, Nolte S, Norman P, Norman R, O'Donnell M, O'Hanlon S, Olives C, Omer SB, Ortblad K, Osborne R, Ozgediz D, Page A, Pahari B, Pandian JD, Rivero AP, Patten SB, Pearce N, Padilla RP, Perez-Ruiz F, Perico N, Pesudovs K, Phillips D, Phillips MR, Pierce K, Pion S, Polanczyk GV, Polinder S, Pope CA, Popova S, Porrini E, Pourmalek F, Prince M, Pullan RL, Ramaiah KD, Ranganathan D, Razavi H, Regan M, Rehm JT, Rein DB, Remuzzi G, Richardson K, Rivara FP, Roberts T, Robinson C, De LFR, Ronfani L, Room R, Rosenfeld LC, Rushton L, Sacco RL, Saha S, Sampson U, Sanchez-Riera L, Sanman E, Schwebel DC, Scott JG, Segui-Gomez M, Shahraz S, Shepard DS, Shin H, Shivakoti R, Singh D, Singh GM, Singh JA, Singleton J, Sleet DA, Sliwa K, Smith E, Smith JL, Stapelberg NJC, Steer A, Steiner T, Stolk WA, Stovner LJ, Sudfeld C, Syed S, Tamburlini G, Tavakkoli M, Taylor HR, Taylor JA, Taylor WJ, Thomas B, Thomson WM, Thurston GD, Tleyjeh IM, Tonelli M, Towbin JA, Truelsen T, Tsilimbaris MK, Ubeda C, Undurraga EA, van DWMJ, van OJ, Vavilala MS, Venketasubramanian N, Wang M, Wang W, Watt K, Weatherall DJ, Weinstock MA, Weintraub R, Weisskopf MG, Weissman MM, White RA, Whiteford H, Wiebe N, Wiersma ST, Wilkinson JD, Williams HC, Williams SRM, Witt E, Wolfe F, Woolf AD, Wulf S, Yeh P, Zaidi AKM, Zheng Z, Zonies D, Lopez AD, AlMazroa MA, Memish ZA. Disability-adjusted life years (DALYs) for 291 diseases and injuries in 21 regions, 1990-2010: a systematic analysis for the Global Burden of Disease Study 2010. Lancet. 2012 Dec 15;380(9859):2197–223. doi: 10.1016/S0140-6736(12)61689-4.S0140-6736(12)61689-4 [DOI] [PubMed] [Google Scholar]
  • 3.Kazdin AE, Blase SL. Rebooting psychotherapy research and practice to reduce the burden of mental illness. Perspect Psychol Sci. 2011 Jan;6(1):21–37. doi: 10.1177/1745691610393527.6/1/21 [DOI] [PubMed] [Google Scholar]
  • 4.Hollon SD, DeRubeis RJ, Fawcett J, Amsterdam JD, Shelton RC, Zajecka J, Young PR, Gallop R. Notice of Retraction and Replacement. Hollon et al. Effect of cognitive therapy with antidepressant medications vs antidepressants alone on the rate of recovery in major depressive disorder: a randomized clinical trial. JAMA Psychiatry. 2014;71(10):1157-1164. JAMA Psychiatry. 2016 Dec 01;73(6):639–40. doi: 10.1001/jamapsychiatry.2016.0756. http://europepmc.org/abstract/MED/27097060 .2515958 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Lorenzo-Luaces L, DeRubeis RJ, van Straten A, Tiemens B. A prognostic index (PI) as a moderator of outcomes in the treatment of depression: a proof of concept combining multiple variables to inform risk-stratified stepped care models. J Affect Disord. 2017 Apr 15;213:78–85. doi: 10.1016/j.jad.2017.02.010.S0165-0327(16)31214-9 [DOI] [PubMed] [Google Scholar]
  • 6.Parker G. Beyond major depression. Psychol Med. 2005 Apr;35(4):467–74. doi: 10.1017/s0033291704004210. [DOI] [PubMed] [Google Scholar]
  • 7.Lorenzo-Luaces L. Heterogeneity in the prognosis of major depression: from the common cold to a highly debilitating and recurrent illness. Epidemiol Psychiatr Sci. 2015 Dec;24(6):466–72. doi: 10.1017/S2045796015000542.S2045796015000542 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Eaton WW, Anthony JC, Gallo J, Cai G, Tien A, Romanoski A, Lyketsos C, Chen LS. Natural history of Diagnostic Interview Schedule/DSM-IV major depression. The Baltimore Epidemiologic Catchment Area follow-up. Arch Gen Psychiatry. 1997 Nov;54(11):993–9. doi: 10.1001/archpsyc.1997.01830230023003. [DOI] [PubMed] [Google Scholar]
  • 9.Eaton WW, Shao H, Nestadt G, Lee HB, Lee BH, Bienvenu OJ, Zandi P. Population-based study of first onset and chronicity in major depressive disorder. Arch Gen Psychiatry. 2008 May;65(5):513–20. doi: 10.1001/archpsyc.65.5.513. http://europepmc.org/abstract/MED/18458203 .65/5/513 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Patten SB, Wang JL, Williams JVA, Lavorato DH, Khaled SM, Bulloch AGM. Predictors of the longitudinal course of major depression in a Canadian population sample. Can J Psychiatry. 2010 Oct;55(10):669–76. doi: 10.1177/070674371005501006. [DOI] [PubMed] [Google Scholar]
  • 11.Spijker J, De Graaf R, Bijl RV, Beekman ATF, Ormel J, Nolen WA. Duration of major depressive episodes in the general population: Results from the Netherlands Mental Health Survey and Incidence Study (NEMESIS) Br J Psychiatry. 2018 Jan 2;181(03):208–213. doi: 10.1192/bjp.181.3.208. [DOI] [PubMed] [Google Scholar]
  • 12.Mattisson C, Bogren M, Horstmann V, Munk-Jörgensen P, Nettelbladt P. The long-term course of depressive disorders in the Lundby Study. Psychol Med. 2007 Jun;37(6):883–91. doi: 10.1017/S0033291707000074.S0033291707000074 [DOI] [PubMed] [Google Scholar]
  • 13.Steinert C, Hofmann M, Kruse J, Leichsenring F. The prospective long-term course of adult depression in general practice and the community. A systematic literature review. J Affect Disord. 2014 Jan;152-154:65–75. doi: 10.1016/j.jad.2013.10.017.S0165-0327(13)00746-5 [DOI] [PubMed] [Google Scholar]
  • 14.Kessler RC, Berglund P, Demler O, Jin R, Koretz D, Merikangas KR, Rush AJ, Walters EE, Wang PS, National CSR. The epidemiology of major depressive disorder: results from the National Comorbidity Survey Replication (NCS-R) JAMA. 2003 Jun 18;289(23):3095–105. doi: 10.1001/jama.289.23.3095.289/23/3095 [DOI] [PubMed] [Google Scholar]
  • 15.Ohayon MM, Schatzberg AF. Prevalence of depressive episodes with psychotic features in the general population. Am J Psychiatry. 2002 Nov;159(11):1855–61. doi: 10.1176/appi.ajp.159.11.1855. [DOI] [PubMed] [Google Scholar]
  • 16.Office of Applied Studies . Suicidal Thoughts, Suicide Attempts, Major Depressive Episode, and Substance Use among Adults. Rockville, MD: Substance Abuse and Mental Health Services Administration; 2006. p. 31. [Google Scholar]
  • 17.Karyotaki E, Riper H, Twisk J, Hoogendoorn A, Kleiboer A, Mira A, Mackinnon A, Meyer B, Botella C, Littlewood E, Andersson G, Christensen H, Klein JP, Schröder J, Bretón-López J, Scheider J, Griffiths K, Farrer L, Huibers MJH, Phillips R, Gilbody S, Moritz S, Berger T, Pop V, Spek V, Cuijpers P. Efficacy of self-guided internet-based cognitive behavioral therapy in the treatment of depressive symptoms: a meta-analysis of individual participant data. JAMA Psychiatry. 2017 Apr 01;74(4):351–359. doi: 10.1001/jamapsychiatry.2017.0044.2604310 [DOI] [PubMed] [Google Scholar]
  • 18.Baumeister H, Reichler L, Munzinger M, Lin J. The impact of guidance on Internet-based mental health interventions — a systematic review. Internet Interv. 2014 Oct;1(4):205–215. doi: 10.1016/j.invent.2014.08.003. [DOI] [Google Scholar]
  • 19.Cowpertwait L, Clarke D. Effectiveness of web-based psychological interventions for depression: a meta-analysis. Int J Ment Health Addict. 2013 Jan 18;11(2):247–268. doi: 10.1007/s11469-012-9416-z. [DOI] [Google Scholar]
  • 20.Königbauer J, Letsch J, Doebler P, Ebert D, Baumeister H. Internet- and mobile-based depression interventions for people with diagnosed depression: a systematic review and meta-analysis. J Affect Disord. 2017 Dec 01;223:28–40. doi: 10.1016/j.jad.2017.07.021.S0165-0327(17)30725-5 [DOI] [PubMed] [Google Scholar]
  • 21.Weisel KK, Zarski A, Berger T, Krieger T, Schaub MP, Moser CT, Berking M, Dey M, Botella C, Baños R, Herrero R, Etchemendy E, Riper H, Cuijpers P, Bolinski F, Kleiboer A, Görlich D, Beecham J, Jacobi C, Ebert DD. Efficacy and cost-effectiveness of guided and unguided internet- and mobile-based indicated transdiagnostic prevention of depression and anxiety (ICare Prevent): a three-armed randomized controlled trial in four European countries. Internet Interv. 2018 Apr; doi: 10.1016/j.invent.2018.04.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Seligman ME. The effectiveness of psychotherapy. The Consumer Reports study. Am Psychol. 1995 Dec;50(12):965–74. doi: 10.1037//0003-066x.50.12.965. [DOI] [PubMed] [Google Scholar]
  • 23.Westen D, Morrison K. A multidimensional meta-analysis of treatments for depression, panic, and generalized anxiety disorder: an empirical examination of the status of empirically supported therapies. J Consult Clin Psychol. 2001 Dec;69(6):875–99. [PubMed] [Google Scholar]
  • 24.van der Lem R, de Wever WWH, van der Wee NJA, van Veen T, Cuijpers P, Zitman FG. The generalizability of psychotherapy efficacy trials in major depressive disorder: an analysis of the influence of patient selection in efficacy trials on symptom outcome in daily practice. BMC Psychiatry. 2012 Nov 08;12:192. doi: 10.1186/1471-244X-12-192. https://bmcpsychiatry.biomedcentral.com/articles/10.1186/1471-244X-12-192 .1471-244X-12-192 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Zimmerman M, Mattia JI, Posternak MA. Are subjects in pharmacological treatment trials of depression representative of patients in routine clinical practice? Am J Psychiatry. 2002 Mar;159(3):469–73. doi: 10.1176/appi.ajp.159.3.469. [DOI] [PubMed] [Google Scholar]
  • 26.Zimmerman M, Clark HL, Multach MD, Walsh E, Rosenstein LK, Gazarian D. Have treatment studies of depression become even less generalizable? A review of the inclusion and exclusion criteria used in placebo-controlled antidepressant efficacy trials published during the past 20 years. Mayo Clin Proc. 2015 Sep;90(9):1180–6. doi: 10.1016/j.mayocp.2015.06.016.S0025-6196(15)00540-6 [DOI] [PubMed] [Google Scholar]
  • 27.Zimmerman M, Clark HL, Multach MD, Walsh E, Rosenstein LK, Gazarian D. Variability in the substance use disorder exclusion criterion in antidepressant efficacy trials. J Affect Disord. 2016 Dec 01;198:39–42. doi: 10.1016/j.jad.2016.03.024.S0165-0327(16)30126-4 [DOI] [PubMed] [Google Scholar]
  • 28.Zimmerman M, Clark HL, Multach MD, Walsh E, Rosenstein LK, Gazarian D. Symptom severity and the generalizability of antidepressant efficacy trials: changes during the past 20 years. J Clin Psychopharmacol. 2016 Apr;36(2):153–6. doi: 10.1097/JCP.0000000000000466. [DOI] [PubMed] [Google Scholar]
  • 29.Zimmerman M, Multach M, Walsh E, Rosenstein LK, Gazarian D, Clark HL. Problems in the descriptions of the psychiatric inclusion and exclusion criteria in publications of antidepressant efficacy trials: a qualitative review and recommendations for improved clarity. CNS Drugs. 2016 Mar;30(3):185–91. doi: 10.1007/s40263-016-0314-y.10.1007/s40263-016-0314-y [DOI] [PubMed] [Google Scholar]
  • 30.Lorenzo-Luaces L, Zimmerman M, Cuijpers P. Are studies of psychotherapies for depression more or less generalizable than studies of antidepressants? J Affect Disord. 2018 Dec;234:8–13. doi: 10.1016/j.jad.2018.02.066.S0165-0327(17)32356-X [DOI] [PubMed] [Google Scholar]
  • 31.Blanco C, Hoertel N, Franco S, Olfson M, He J, López S, González-Pinto A, Limosin F, Merikangas KR. Generalizability of clinical trial results for adolescent major depressive disorder. Pediatrics. 2017 Dec;140(6):e20161701. doi: 10.1542/peds.2016-1701.peds.2016-1701 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Hoertel N, López S, Wang S, González-Pinto A, Limosin F, Blanco C. Generalizability of pharmacological and psychotherapy clinical trial results for borderline personality disorder to community samples. Personal Disord. 2015 Jan;6(1):81–7. doi: 10.1037/per0000091. http://europepmc.org/abstract/MED/25580674 .2014-57733-003 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Hoertel N, de Maricourt P, Katz J, Doukhan R, Lavaud P, Peyre H, Limosin F. Are participants in pharmacological and psychotherapy treatment trials for social anxiety disorder representative of patients in real-life settings? J Clin Psychopharmacol. 2014 Dec;34(6):697–703. doi: 10.1097/JCP.0000000000000204. [DOI] [PubMed] [Google Scholar]
  • 34.Franco S, Hoertel N, McMahon K, Wang S, Rodríguez-Fernández JM, Peyre H, Limosin F, Blanco C. Generalizability of pharmacologic and psychotherapy clinical trial results for posttraumatic stress disorder to community samples. J Clin Psychiatry. 2016 Dec;77(8):e975–81. doi: 10.4088/JCP.15m10060. [DOI] [PubMed] [Google Scholar]
  • 35.Hoertel N, Le Strat Y, Blanco C, Lavaud P, Dubertret C. Generalizability of clinical trial results for generalized anxiety disorder to community samples. Depress Anxiety. 2012 Jul;29(7):614–20. doi: 10.1002/da.21937. [DOI] [PubMed] [Google Scholar]
  • 36.Higgins JP, Altman DG. Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series. Chichester, England: Wiley-Blackwell; 2008. Assessing risk of bias in included studies; pp. 187–241. [Google Scholar]
  • 37.Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010;36(3) doi: 10.18637/jss.v036.i03. [DOI] [Google Scholar]
  • 38.IntHout J, Ioannidis JPA, Borm GF. The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Med Res Methodol. 2014 Feb 18;14:25. doi: 10.1186/1471-2288-14-25. https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-14-25 .1471-2288-14-25 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Röver C, Knapp G, Friede T. Hartung-Knapp-Sidik-Jonkman approach and its modification for random-effects meta-analysis with few studies. BMC Med Res Methodol. 2015 Nov 14;15:99. doi: 10.1186/s12874-015-0091-1. https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-015-0091-1 .10.1186/s12874-015-0091-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Huhn M, Tardy M, Spineli LM, Kissling W, Förstl H, Pitschel-Walz G, Leucht C, Samara M, Dold M, Davis JM, Leucht S. Efficacy of pharmacotherapy and psychotherapy for adult psychiatric disorders: a systematic overview of meta-analyses. JAMA Psychiatry. 2014 Jun;71(6):706–15. doi: 10.1001/jamapsychiatry.2014.112.1865002 [DOI] [PubMed] [Google Scholar]
  • 41.Bower P, Kontopantelis E, Sutton A, Kendrick T, Richards DA, Gilbody S, Knowles S, Cuijpers P, Andersson G, Christensen H, Meyer B, Huibers M, Smit F, van SA, Warmerdam L, Barkham M, Bilich L, Lovell K, Liu ET. Influence of initial severity of depression on effectiveness of low intensity interventions: meta-analysis of individual patient data. BMJ. 2013;346:f540. doi: 10.1136/bmj.f540. http://www.bmj.com/cgi/pmidlookup?view=long&pmid=23444423 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Kessler RC, van Loo H, Wardenaar KJ, Bossarte RM, Brenner LA, Ebert DD, de JP, Nierenberg AA, Rosellini AJ, Sampson NA, Schoevers RA, Wilcox MA, Zaslavsky AM. Using patient self-reports to study heterogeneity of treatment effects in major depressive disorder. Epidemiol Psychiatr Sci. 2017 Feb;26(1):22–36. doi: 10.1017/S2045796016000020. http://europepmc.org/abstract/MED/26810628 .S2045796016000020 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Nunes EV, Levin FR. Treatment of depression in patients with alcohol or other drug dependence: a meta-analysis. JAMA. 2004 Apr 21;291(15):1887–96. doi: 10.1001/jama.291.15.1887.291/15/1887 [DOI] [PubMed] [Google Scholar]
  • 44.Hoertel N, Le Strat Y, Limosin F, Dubertret C, Gorwood P. Prevalence of subthreshold hypomania and impact on internal validity of RCTs for major depressive disorder: results from a national epidemiological sample. PLoS One. 2013;8(2):e55448. doi: 10.1371/journal.pone.0055448. http://dx.plos.org/10.1371/journal.pone.0055448 .PONE-D-12-24049 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.DeRubeis RJ, Gelfand LA, German RE, Fournier JC, Forand NR. Understanding processes of change: how some patients reveal more than others-and some groups of therapists less-about what matters in psychotherapy. Psychother Res. 2014;24(3):419–28. doi: 10.1080/10503307.2013.838654. http://europepmc.org/abstract/MED/24219275 . [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Multimedia Appendix 1

List of studies, risk-of-bias assessments, and example search terms.

jmir_v20i11e10113_app1.pdf (49.5KB, pdf)

Articles from Journal of Medical Internet Research are provided here courtesy of JMIR Publications Inc.

RESOURCES