An Evaluation of Cognitive Behavioral Therapy for Insomnia: A Systematic Review and Application of Tolin’s Criteria for Empirically Supported Treatments

Abstract

The American Psychological Association’s Society of Clinical Psychology recently adopted the “Tolin Criteria” to evaluate empirically supported treatments. These criteria better account for strength and quality of rapidly accumulating evidence bases for various treatments. Here we apply this framework to cognitive behavioral therapy for insomnia (CBT-I). Following procedures outlined by Tolin, McKay, et al. (2015), Step 1 included an examination of quantitative systematic reviews; nine met inclusion criteria. Step 2 evaluated review quality and effect size data. We found high-quality evidence that CBT-I produces clinically and statistically significant effects on insomnia and other sleep-related outcomes. Based on the Tolin Criteria, the literature merits a “strong” recommendation for CBT-I. This report is a working model for subsequent applications of the Tolin Criteria.

In 2015, the Society of Clinical Psychology (SCP; Division 12 of the American Psychological Association [APA]) adopted the “Tolin Criteria” as a way to classify the evidence for empirically supported treatments (ESTs) for mental disorders (Tolin, McKay, et al., 2015). These updated criteria replaced the previously established “Chambless Criteria” (Chambless & Hollon, 1998; Chambless & Ollendick, 2001). The Tolin Criteria were intended to address the concern that thresholds for “well established” and “probably efficacious” treatments, as defined in the Chambless Criteria, were too low. That is, the Chambless Criteria required a “well established” treatment to have (a) at least two independently conducted, well-designed primary studies or (b) a large series of well-designed and carefully controlled single-case design experiments. Critics felt that this was too lenient a bar for determining efficacy, that it did not adequately account for mixed or conflicting findings, and that it risks creating a “misrepresentative and highly selective impression of efficacy” (Tolin, McKay, et al., 2015, p. 320–321). In addition, the Tolin Criteria were meant to better account for the improved quality (and quantity) of clinical research and the adoption of more sophisticated research synthesis and evaluation methods (Tolin, McKay, et al., 2015). Indeed, in the years since the Chambless Criteria were implemented, more than 80 psychological treatments have been evaluated using the Chambless Criteria, and the corresponding list of ESTs are available to the public on the Division 12 website (www.div12.org/treatments/). To update the public’s understanding of the strength of the evidence for each treatment, Division 12 is leading the charge on applying the Tolin Criteria to each of the ESTs for which systematic reviews are available. The purpose of the present article is to describe the methods utilized in the application of the Tolin Criteria to cognitive behavioral therapy for insomnia (CBT-I), as well as to document the resulting recommendation regarding the strength of the evidence for this treatment.

Insomnia is the most prevalent of all sleep disorders and occurs in approximately one-third of adults in the general population, with approximately 6–10% of the population meeting criteria for insomnia (APA, 2013; Ohayon, 2002). Notably, 40–50% of individuals with insomnia also present with a comorbid mental disorder (APA, 2013). Chronic insomnia is associated with greater functional impairment, impaired daytime functioning (e.g., cognition, mood, or performance), increased loss of time from work (e.g., absenteeism or tardiness), and higher health care costs (e.g., greater medical service utilization; Morganthaler et al., 2006; Schutte-Rodin et al., 2008; Simon & VonKorff, 1997). As a result, chronic insomnia causes societal and personal burdens and is, therefore, an important treatment target.

In the past few decades, CBT-I, which is the most researched psychotherapeutic intervention for insomnia, has been named a first-line psychotherapy treatment for chronic insomnia by the National Institutes of Health and American Academy for Sleep Medicine (National Institutes of Health, 2005; Morganthaler et al., 2006; Schutte-Rodin et al., 2008). Consistent with a CBT framework (e.g., Spielman, 1987), CBT-I assumes that chronic sleep difficulties result from predisposing factors (e.g., anxious personality type), precipitating events (e.g., stressful life event), and perpetuating mechanisms (e.g., napping, automatic negative thoughts). CBT-I incorporates techniques to address these factors such as sleep restriction, stimulus control, relaxation, and cognitive therapy (Edinger, 2018). Several scientific reviews and meta-analyses have demonstrated the effectiveness of CBT-I for treating insomnia (e.g., Irwin et al., 2006; Morin et al., 2006; Taylor & Pruiksma, 2014), and the treatment has demonstrated efficacy across various age groups (e.g., Irwin et al., 2006), comorbid physical (e.g., cancer; Johnson et al., 2016) and mental health diagnoses (e.g., Ye et al., 2015), treatment modalities (e.g., group, online, and in-person formats; Koffel et al., 2018; Ye et al., 2015), and treatment settings (e.g., primary care; Davidson et al., 2019).

Given the abundance of research that has accumulated for CBT-I, it is an ideal treatment to apply the Tolin Criteria to, and in so doing, evaluate the strength of the treatment. Secondarily, the purpose of this work further serves to transparently document the procedures developed and outlined by Tolin, McKay, and colleagues (2015) as adapted to a new treatment evaluation. Originally evaluated as an EST using the 1998 Chambless Criteria, CBT-I was assigned a rating of “strong” based on the empirical support at that time. The current manuscript evaluates the quality of evidence for the use of CBT-I in individuals with insomnia using the recently ratified Tolin Criteria (see Table 1).

Table 1.

Summary of Tolin et al. 2015 Procedure v. Current Procedure for Evaluating Empirically Supported Treatments

Step	Tolin Process	Details	Current Process for CBT-I Evaluation	Details
Step 1	Systematic Review	Treatment is nominated	Systematic Review	Treatment was nominated by APA Division 12 Science and Practice Committee
		Existing systematic review is evaluated according to:		Existing systematic review is evaluated according to:
		PICOTS (population, intervention, comparison, outcomes, timeline, setting)  Treatment fidelity		PICOTS (population, intervention, comparison, outcomes, timeline, setting)-
		Risk of bias assessment (low, unclear, high)		-
Step 2	Committee-based evidence review	GRADE recommendation by committee: very strong, strong, weak	Committee-based evidence review	GRADE recommendation by committee: very strong, strong, weak

Note. CBT-I = Cognitive Behavioral Therapy for Insomnia. GRADE = Grading of Recommendations Assessment, Development, and Evaluation. Of note, we did not separately assess treatment fidelity or risk of bias given that these were evaluated using the AMSTAR 2 items.

Methods and Procedures (Step 1)

The methods and procedures utilized in the current treatment evaluation replicated what was originally delineated by Tolin, McKay, and colleagues (2015). When necessary, adaptations to the original procedure were made by the current authors, which are described in Table 1. The reviewers included seven volunteers (four full members and three student members), who served on the SCP Committee on Science and Practice. Step 1 of the treatment evaluation included an examination of quantitative systematic research reviews (i.e., meta-analyses) on CBT-I (see Table 1)¹. Consistent with Tolin, McKay, et al. (2015), the committee began by searching the literature and locating reviews eligible for inclusion. These reviews were subsequently evaluated for quality and relevant effect size data were extracted for Step 2 of the treatment evaluation. The final recommendation was determined by conducting a systematic review of published meta-analytic studies. Note that our methods described herein were conducted according to the checklist of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA; Moher et al., 2015) and the reporting standards of APA’s Publications and Communications Board Task Force Report (Levitt et al., 2018), as indicated by the Tolin Criteria.

Search Strategy and Data Sources

In the current evaluation, we limited our eligible reviews to quantitative meta-analyses. This is consistent with Tolin, McKay, et al. (2015), who proposed that candidate treatments be evaluated based on existing (published or unpublished) quantitative reviews. We, therefore, searched seven databases including Scopus, MEDLINE, ScienceDirect, PsycINFO, Social Sciences Citation Index, Science Citation Index, and Academic Search Ultimate. Search terms included the following: TI (“Cognitive Behavior* Therapy for Insomnia” or “CBT-I” or “CBT for Insomnia”) AND (TI [Review or “Systematic Review” or “Quantitative Review” or “meta analysis”]) OR (SU [Review or “Systematic Review” or “Quantitative Review” or “meta analysis”]) NOT TI (“qualitative review” or “narrative review”) where TI = title and SU = subject. There were no restrictions on language, type of publication, or year of publication. Additionally, we selected “reviews” from the “filter by format” option. Initial search results returned 25 records. Inclusion criteria included being a quantitative review focused on CBT-I (any delivery modality). Exclusion criteria included not reporting on insomnia- or sleep-related outcomes. The search strategy was conducted on August 1, 2018.

Study Selection

All reviewers coded each of the initial 25 meta-analyses as eligible, not eligible, or possibly eligible based on the title and abstract. For meta-analyses coded as eligible or possibly eligible, full texts were obtained and read by randomly assigned pairs of coders to determine eligibility. All discrepancies between pairs of coders were resolved via consensus among committee members. Of the 25 studies reviewed, 12 were excluded for not being quantitative, one was excluded for being a primary study, and three were excluded for being abstracts only (e.g., conference abstracts), resulting in nine eligible meta-analyses for inclusion in the current review (see Figure 1). For each of the nine eligible meta-analyses, two committee members were randomly assigned to double code the population, intervention, comparison, outcomes, timeline, setting (PICOTS) criteria (e.g., Schardt et al., 2007). PICOTS allows for a full consideration of review characteristics (see Table 2) and assists the reader in evaluating the appropriateness of the included reviews for answering the clinical question of interest. Discrepancies among coders were resolved by committee consensus.

Figure 1.

Flow Chart for CBT-I Search Process. This figure illustrates the search process for locating reviews eligible for inclusion in the treatment review of CBT-I. CBT-I = Cognitive Behavioral Therapy for Insomnia.

Table 2.

Meta-Analyses Included in the Review of CBT-I

Study	Intervention	Number of Studies	Population	Setting	Comparison Condition	Outcome	Time Points
Belleville et al., 2011	Individual In-person CBT-I; Group CBT-I; Internet/Web-based CBT-I	50	Insomnia/Self-reported Sleep Difficulties	Internet (Self-Help); NS*	Waitlist/ Inactive Control; Placebo/Active Control	Anxiety Symptoms	Post-Treatment
Ho et al., 2015	Internet/Web-based CBT-I	20	Insomnia/Self-reported Sleep Difficulties	Internet (Self-Help)	Waitlist/ Inactive Control; TAU; Placebo/Active Control	Insomnia Symptoms/ Sleep Outcomes; Depressive Symptoms; Anxiety Symptoms; Quality of Life; Adherence; Treatment Satisfaction/Acceptability; Dysfunctional Sleep Beliefs	Post-Treatment; Follow-up < 3-months, Follow-up ≥ 3-months
Koffel et al., 2015	Group CBT-I	8	Insomnia/Self-reported Sleep Difficulties	NS	Waitlist/ Inactive Control; TAU; Placebo/Active Control	Insomnia Symptoms/ Sleep Outcomes; Depressive Symptoms; Pain	Post-Treatment; Follow-up ≥ 3-months
Navarro-Bravo et al., 2015	Group CBT-I	9	Insomnia/Self-reported Sleep Difficulties	NS	Waitlist/ Inactive Control; TAU; Placebo/Active Control	Insomnia Symptoms/ Sleep Outcomes	Post-Treatment
Johnson et al., 2016	Individual In-person CBT-I; Group CBT-I; Internet/Web-based CBT-I	8	Insomnia/Self-reported Sleep Difficulties; Cancer	NS	Waitlist/ Inactive Control; TAU; Placebo/Active Control	Insomnia Symptoms/ Sleep Outcomes	Post-Treatment; Follow-up < 3-months
Taylor et al., 2014	Individual In-person CBT-I	16	Insomnia/Self-reported Sleep Difficulties; Non-SUD Psychiatric Diagnoses; Substance Use Disorder	NS	Waitlist/ Inactive Control	Insomnia Symptoms/ Sleep Outcomes; Depressive Symptoms; Anxiety Symptoms	Post-Treatment
Wu et al., 2015	Unspecified CBT-I	37	Insomnia/Self-reported Sleep Difficulties; Cancer; Chronic Pain; Non-SUD Psychiatric Diagnoses; Other Medical Conditions	NS	Waitlist/ Inactive Control; TAU; Placebo/Active Control	Insomnia Symptoms/ Sleep Outcomes; Medical/ Psychiatric Comorbid Conditions	Post-Treatment; Follow-up ≥ 3-months
Ye et al., 2015	Internet/Web-based CBT-I	10	Insomnia/Self-reported Sleep Difficulties	Internet (Self-Help)	Waitlist/ Inactive Control	Depression Symptoms; Anxiety Symptoms; Insomnia Symptoms, Sleep Outcomes	Post-Treatment
Zachariae et al., 2016	Internet, Web-Based, or Electronically Delivered	11	Insomnia/Self-Reported Sleep Difficulties	Internet (Self-Help)	Waitlist/Inactive Control	Insomnia Symptoms, Sleep Outcomes	Post-Treatment; Follow-up < 3-months, Follow-up ≥ 3-months

Note. CBT-I = Cognitive Behavioral Therapy for Insomnia; NS = not specified; NS* = sufficient detail on other settings not provided; SUD = substance use disorder; TAU = treatment as usual.

Data Extraction and Coding

Two coders reviewed each eligible review and compiled effect size values for every outcome reported. Any discrepancies in ratings were resolved by committee consensus. Sleep-rated outcomes included sleep efficiency, sleep onset latency, wake after sleep onset, total sleep time, sleep quality (assessed using the Pittsburgh Sleep Quality Index; Buysse et al., 1989), and a sleep questionnaire (Insomnia Severity Index [ISI]; Morin et al., 2011). Non-sleep-related outcomes included depressive symptoms, posttraumatic stress symptoms, anxiety symptoms, dysfunctional beliefs about insomnia, pain ratings, hypnotic dependence, and a broad grouping of measures labeled as “comorbid conditions,” which included medical and psychiatric comorbidities (from Wu et al., 2015).

Quality of Reviews

For each meta-analysis, committee members double coded the Assessing the Methodological Quality of Systematic Reviews 2 (AMSTAR 2; Shea et al., 2017) items. AMSTAR 2 is an instrument for critically appraising systematic reviews of randomized and non-randomized studies of healthcare interventions. Although the original Tolin, McKay, and colleagues (2015) publication used the AMSTAR, this committee used the subsequent revision, AMSTAR 2 (Shea et al., 2017). AMSTAR 2 includes 16 domains to consider when evaluating systematic reviews. For each domain, response options include “Yes” and “No,” and for some items, “Partial Yes.” A response of “Yes” denotes a positive result on the stated criterion. These domains are described in Table 3. Consistent with AMSTAR 2 guidelines, committee members identified the domains deemed critical for current review of CBT-I. These domains, bolded in Table 3, were: (a) including components of PICOTS in the research questions and inclusion criteria, (b) using a comprehensive search strategy, (c) describing the included studies in adequate detail, (d) using appropriate methods for statistical combination of results, (e) accounting for risk of bias in individual studies, and (f) providing explanation for a discussion for any heterogeneity observed in the results. Overall quality ratings were assigned based on the number of “No” responses (i.e., number of weaknesses) across domains for a given study as well as whether or not these domains were considered critical or non-critical (see bolded items in Table 3 as well as Table 4).

Table 3.

AMSTAR 2 Results for Eligible Studies

Item	Belleville et al., 2011	Ho et al., 2015	Koffel et al., 2015	Navarro-Bravo et al., 2015	Johnson et al., 2016	Taylor et al., 2014	Wu et al., 2015	Ye et al., 2015	Zachariae et al., 2016
1. Did the research questions and inclusion criteria for the review include the components of PICO?	Y	Y	Y	Y	Y	Y	Y	Y	Y
2. Did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol?	N	PY	N	N	PY	N	PY	N	PY
3. Did the review authors explain their selection of the study designs for inclusion in the review?	Y	N	Y	N	Y	N	Y	N	N
4. Did the review authors use a comprehensive literature search strategy?	PY	PY	PY	PY	Y	PY	PY	PY	PY
5. Did the review authors perform study selection in duplicate?	N	Y	N	Y	Y	N	Y	Y	Y
6. Did the review authors perform data extraction in duplicate?	Y	N	Y	Y	Y	N	Y	Y	N
7. Did the review authors provide a list of excluded studies and justify the exclusions?	N	N	N	N	Y	N	N	N	Y
8. Did the review authors describe the included studies in adequate detail?	PY	PY	PY	PY	PY	PY	PY	N	PY
9. Did the review authors use a satisfactory technique for assessing the risk of bias (RoB) in individual studies that were included in the review?
RCT	N	Y	Y	PY	Y	N	Y	N	PY
NRSI	N	NA	NA	NA	NA	NA	NA	NA	NA
10. Did the review authors report on the sources of funding for the studies included in the review?	N	N	N	N	NA	N	N	N	N
11. If meta-analysis was performed did the review authors use appropriate methods for statistical combination of results?
RCT	Y	N	Y	Y	Y	N	Y	Y	Y
NRSI	N	NA	NA	NA	NA	NA	NA	NA	NA
12. If meta-analysis was performed, did the review authors assess the potential impact of RoB in individual studies on the results of the meta-analysis or other evidence synthesis?	N	Y	Y	Y	Y	N	Y	Y	Y
13. Did the review authors account for RoB in individual studies when interpreting/ discussing the results of the review?	N	Y	Y	Y	Y	N	Y	Y	Y
14. Did the review authors provide a satisfactory explanation for, and discussion of, any heterogeneity observed in the results of the review?	Y	N	Y	Y	Y	N	Y	Y	Y
15. If they performed quantitative synthesis did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review?	Y	Y	Y	Y	Y	N	Y	Y	Y
16. Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review?	Y	Y	Y	N	Y	Y	Y	Y	Y
Overall Rating	Low	Critically Low	Moderate	Moderate	High	Critically Low	Moderate	Low	Moderate

Note. Y = yes, PY = partial yes, N = no. PICO: (P = population, I = intervention, C = comparator group, O = outcome). RCT = randomized controlled trial. NRSI = non-randomized studies of interventions. NA = not applicable. Items in bold are considered critical weaknesses if coded “no.” All studies were double-coded and discrepancies were resolved by consensus to arrive at the final ratings.

Table 4.

Overall AMSTAR 2 Confidence Ratings and Descriptions

Rating	Detailed Description
High (Zero or one non-critical weakness)	The systematic review provides an accurate and comprehensive summary of the results of the available studies that address the question of interest.
Moderate (More than one non-critical weakness*)	The systematic review has more than one weakness but no critical flaws. It may provide an accurate summary of the results of the available studies that were included in the review.
Low (One critical flaw with or without non-critical weaknesses)	The review has a critical flaw and may not provide an accurate and comprehensive summary of the available studies that address the question of interest.
Critically Low (More than one critical flaw with or without non-critical weaknesses)	The review has more than one critical flaw and should not be relied on to provide an accurate and comprehensive summary of the available studies.

Note.

^*Multiple non-critical weaknesses may diminish confidence in the review and it may be appropriate to move the overall rating down from moderate to low. Critical domains (indicating “critical weaknesses”) were determined by committee members and were described thoroughly in the study protocol prior to evaluation of quality. These are noted in bold in Table 3. Adapted from Shea et al., 2017.

Discrepancies among coders were resolved by committee consensus. Meta-analyses that were rated as low or critically low (n = 4; see Table 3) were excluded from the main effect size estimate summaries but were considered in supplemental sensitivity analyses (see Supplemental Materials). Of note, we did not separately assess treatment fidelity or risk of bias given that these were evaluated using the AMSTAR 2 items.

These nine meta-analyses, taken together in this systematic review, were considered adequate for drawing reasonable conclusions about the efficacy of CBT-I. The Johnson et al. (2016) meta-analysis is regarded as the strongest with an overall quality rating of “high” based on fully- or partially-met criteria for all AMSTAR 2 quality domains. Four meta-analyses (Koffel et al., 2015; Navarro-Bravo et al., 2015; Wu et al., 2015; Zachariae et al., 2016) were rated of “moderate” quality and two were rated as having “low” quality (Belleville et al., 2011; Ye et al., 2015). Two meta-analyses had critically-low quality ratings due to not accounting for risk of publication bias (Ho et al., 2015; Taylor & Pruiksma, 2014) and not providing explanation for heterogeneity observed in review of reported results (Taylor & Pruiksma, 2014). See Table 3 for detailed AMSTAR 2 ratings across all nine reviews.

Results (Step 2)

Before describing the results of the current systematic review, it is worth pointing out that although the authors of the current manuscript decided to estimate a combined effect size from the eligible meta-analyses included, the Tolin Criteria does not require this step. We chose to pursue this given the sole focus on meta-analyses for this treatment review which lends itself nicely to estimating a combined effect. It is our belief that the Tolin Criteria provides considerable flexibility in how effect size estimates from reviews are considered as part of the evaluation process and the current evaluation only reflects one approach.

Results from combined effect size estimates demonstrated important information regarding the effectiveness of CBT-I for insomnia and other outcomes. Descriptive results summarizing standardized effect size ratings (Hedge’s g) across studies are presented in two ways. First, we discuss results from five studies receiving AMSTAR 2 quality ratings of moderate or high. Second, as part of supplemental sensitivity analyses, we report on all nine studies, including the four studies receiving low or critically low AMSTAR 2 quality ratings (see Table 3). Studies that reported effect size metrics other than Hedge’s g were converted for ease of comparison and interpretation. Of note, Navarro-Bravo and colleagues (2015) provided three sets of ratings that included ratings comparing CBT-I to waitlist controls, ratings comparing CBT-I to alternative treatments, and ratings that summarized both sets of these comparisons. The current synthesis used the combined set of ratings as this was most consistent with the methodology used by the other meta-analyses.

Figures 2 and 3 depict the mean effect size estimates with 95% confidence intervals (CI) for the sleep-related outcomes and non-sleep related outcomes for the AMSTAR 2 highly rated studies, respectively. Using Cohen’s (1988) guidelines for classifying small effects as g = 0.2, medium effects as g = 0.5, and large effects as g = 0.8, the figures, along with Table 5, illustrate that CBT-I produces medium-to-large changes in most sleep outcomes (i.e., sleep efficiency, sleep onset latency, wake after sleep onset, sleep quality, and sleep questionnaire) across the meta-analyses reviewed. The high-quality meta-analyses did not provide enough information on other non-sleep-related outcomes (i.e., depressive symptoms, pain, and comorbid conditions) to make conclusive statements about the impact of CBT-I on these outcomes.

Figure 2.

Mean Effect Sizes (Hedge’s g) for Sleep-Rated Outcomes (N = 5). This figure depicts the mean effect size estimates across the five studies with moderate or higher quality ratings. CI = 95% confidence intervals. PSQI = Pittsburgh Sleep Quality Index. ISI = Insomnia Severity Index.

Figure 3.

Mean Effect Sizes (Hedge’s g) for Non-Sleep-Rated Outcomes (N = 5). This figure depicts the mean effect size estimates across the five studies with moderate or higher quality ratings. CI = 95% confidence intervals.

Table 5.

Effect Sizes for Relevant Outcomes in Moderate and High Quality Studies (N = 5)

Outcome	Number of Effect Sizes	Mean ES	SD ES	Min ES	Max ES	Skewness	SE	95% CI LL	95% CI UL
Sleep Efficiency	5	0.73	0.17	0.53	0.91	−0.31	0.074	0.583	0.873
Sleep Onset Latency	5	0.51	0.16	0.41	0.80	2.12	0.072	0.372	0.656
Wake After Sleep Onset	5	0.57	0.13	0.41	0.68	−0.62	0.059	0.457	0.690
Total Sleep Time	3	0.14	0.14	−0.04	0.29	−0.47	0.080	−0.020	0.295
Sleep Quality (PSQI)	3	0.61	0.22	0.40	0.84	0.42	0.127	0.360	0.859
Sleep Questionnaire(ISI)	3	0.85	0.21	0.70	1.09	1.51	0.121	0.616	1.089
Depressive Symptoms	1	0.53
Comorbid Conditions	1	0.39
Pain	1	0.35

Note. ES = Effect size; SD = Standard deviation; SE = Standard error; LL = Lower limit of 95% confidence interval; UL = Upper limit of 95% confidence interval; CI = Confidence interval.

Supplemental Table 1 and Supplemental Figures 1 and 2 summarize the effects of CBT-I on sleep and non-sleep-related outcomes for all comparisons, regardless of AMSTAR 2 quality. The most substantial sleep-related effects across all studies were seen on the ISI and sleep efficiency outcomes. The mean effect sizes were large, though with a fair amount of variability across studies (ISI: g = 0.96, SD = 0.23; sleep efficiency: g = 0.74, SD = 0.13). For non-sleep-related outcomes (Supplemental Figure 2), only one effect size was available for PTSD symptoms, dysfunctional beliefs about insomnia, pain, hypnotic dependence, and medical/psychiatric comorbid conditions, respectively, limiting the ability to draw concrete conclusions about the impact of CBT-I on these outcomes. There were, however, several effect sizes reported for depressive and anxiety symptoms, which demonstrated small to moderate effect sizes (g = 0.40 and g = 0.35, respectively).

Quality of the Evidence.

To assess the quality of relevant evidence and the degree to which benefits are established compared to potential harms, the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) system was used (e.g., Guyatt et al., 2008). This system considers factors such as methodological quality of the evidence, the importance of the outcome that the treatment improves, and the burden, costs, and potential risks associated with the therapy (see Table 6 for a full description). This information is combined to determine evidence quality as high, moderate, or low.

Table 6.

Judging the Quality of the Evidence for CBT-I

Quality	Criteria
High quality	All of the following: There is a wide range of studies included in the analyses with no major limitations. There is little variation between studies. The summary estimate has a narrow confidence interval.
Moderate quality	At least one of the following: There are only a few studies, and some have limitations but not major flaws. There is some variation between studies, or the confidence interval of the summary estimate is wide.
Low quality	Any of the following: The studies have major flaws. There is important variation between studies. The confidence interval of the summary estimate is very wide.

Note. CBT-I = cognitive behavioral therapy for insomnia. This table is reproduced from Tolin, Melnyk, & Marx, 2015.

Given that we prioritized the evaluation of high-quality studies from the outset (using AMSTAR 2), we are confident that, among these studies, there were no major limitations. Further, when examining sleep-related outcomes specifically, the effect sizes had little variation between studies (SD_range = 0.10–0.22) and generally had narrow CI (except those for the PSQI and ISI, which had fewer effect size estimates compared to the other sleep-related outcomes). Thus, we concluded there was a fairly robust effect of CBT-I on sleep-related factors among the high-quality studies included in our synthesis² Taken together, the quality of evidence for CBT-I was judged to be high, according to GRADE specifications.

Additional contextual factors.

Consistent with the recommendations of GRADE, Tolin, McKay, and colleagues (2015) note the importance of considering additional contextual factors when making an overall rating of the evidence (see Table 7 for contextual factors). Of note, the lack of assessment of contextual factors across the meta-analyses reviewed indicates that there is not enough data to make a firm conclusion in this category for CBT-I. We found that (a) CBT-I generates an effect that is efficient with regard to number of sessions (often taking place in 10 or fewer sessions across trials included in these meta-analyses; average number of sessions in included meta-analysis was 6 to 9), (b) CBT-I has been studied by several independent research groups without a strong allegiance to the treatment, and (c) CBT-I demonstrated efficacy across several patient populations (see Table 2). These three positive contextual factors supported our overall treatment recommendation.³

Table 7.

Additional Contextual Factors Considered in Increasing or Decreasing the GRADE recommendation for CBT-I

Positive	Negative
Treatment appears superior to other established and effective treatment(s)	There are other psychological treatments that have well-documented and much larger effects
The treatment generates an effect that is similar to other well-studied treatments, but requires a very small number of sessions or length of time to generate the same effect at a much lower cost	The treatment generates an effect that is similar to other well-studied treatments, but requires a very large number of sessions or length of time to generate the same effect at a much higher cost
Evidence supports the purported mechanism or active ingredient(s) of treatment	Evidence fails to support the purported mechanism or active ingredient(s) of treatment
Treatment has demonstrated good effects with minority groups	Treatment has demonstrated weak effects with minority groups
Treatment has been studied by a wide array of researchers without strong allegiance to the treatment	Treatment has been studied by a narrow array of researchers with strong allegiance to the treatment
Other: Demonstrated efficacy across several patient populations	Other:

Note. This table identifies additional positive contextual factors supported by the CBT-I literature and was adapted from Tolin et al., 2015.

Overall treatment recommendation.

There is high-quality evidence that CBT-I produces a clinically meaningful and statistically significant effect on symptoms of insomnia and other sleep-related outcomes, as evidenced by multiple meta-analytic reviews with acceptable quality. As such, based on the criteria outlined by Tolin, McKay, and colleagues (2015), the current status of the literature merits a strong recommendation for CBT-I in the treatment of insomnia per the GRADE criteria (see Table 8).

Table 8.

Overall Treatment Recommendation for CBT-I based on GRADE

Recommendation	Criteria
Very strong recommendation	All of the following: There is high-quality evidence that the treatment produces a clinically meaningful effect on symptoms of the disorder being treated There is high-quality evidence that the treatment produces a clinically meaningful effect on functional outcomes There is high-quality evidence that the treatment produces a clinically meaningful effect on symptoms and/or functional outcomes at least three months after treatment discontinuation At least one well-conducted study has demonstrated effectiveness in non-research settings
Strong recommendation	At least one of the following: There is moderate- to high-quality evidence that the treatment produces a clinically meaningful effect on symptoms of the disorder being treated There is moderate- to high-quality evidence that the treatment produces a clinically meaningful effect on functional outcomes
Weak recommendation	Any of the following: There is only low- or very low-quality evidence that the treatment produces a clinically meaningful effect on symptoms of the disorder being treated There is only low- or very low-quality evidence that the treatment produces a clinically meaningful effect on as well as on functional outcomes There is moderate- to high-quality evidence that the effect of the treatment, although statistically significant, may not be of a magnitude that is clinically meaningful

Note. This table was adapted from Tolin et al., 2015. GRADE = Grading of Recommendations Assessment, Development, and Evaluation (e.g., Guyatt et al., 2008).

There is additional evidence that CBT-I can be delivered in a variety of modalities (with initial support for online treatment) and that it can produce meaningful relief in patients with medical and psychiatric comorbidities, such as symptoms of depression, anxiety, and pain. Given the paucity of data related to non-sleep-related outcomes, we cannot conclude that the treatment produces a clinically meaningful effect on functional outcomes (beyond sleep), nor could we determine whether or not the treatment produces a clinically meaningful impact on symptoms or functional outcomes at least three months after treatment discontinuation. This type of data would be necessary for the treatment to earn a recommendation of “very strong.”

Our “strong” recommendation is further strengthened by several contextual factors, including the cost-effectiveness of the treatment, evaluation of the treatment by a range of researchers without known allegiance to the treatment, and evidence of efficacy in treating insomnia across various psychiatric and medical populations. There is some evidence of efficacy of CBT-I on secondary psychological symptoms (i.e., depressive symptoms), however this is based on an effect size estimate from one meta-analysis, suggesting further research is needed.

Discussion

Our evaluation is consistent with the recommendations from the National Institutes of Health (2005) and the American Academy of Sleep Medicine (Schutte-Rodin et al., 2008) for cognitive behavioral therapy as an effective intervention for the treatment of insomnia. This rating is strengthened by its consideration of meta-analytic data, contextual factors, and CBT-I’s impact on functional outcomes and symptom reduction.

This work is critical as it advances the field and public’s understanding of the strength of evidence for what has been deemed an EST. Indeed, given the abundance of individual trials, systematic reviews, and meta-analyses, it can be challenging to make sense of and integrate the literature. Thus, this review serves to strengthen our scientific understanding of the state and strength of the evidence for this treatment. Using transparent methods, we found that CBT-I merits a recommendation of “strong.” As the literature on CBT-I continues to develop (e.g., more work is conducted to evaluate the impact of CBT-I on non-sleep related outcomes), it is possible that the current recommendation will require an update to evaluate studies conducted after this systematic review. As such, recommendations under the Tolin Criteria can be considered evolving based on the most up to date literature.⁴ Taken together, although we demonstrate that CBT-I merits a “strong” recommendation, this can and should be updated to reflect the most recent empirical findings on CBT-I (as well as other treatments evaluated using the Tolin Criteria).

This work is also notable as it is the first attempt by Division 12 to have a treatment evaluated by members of our field who did not develop the Tolin Criteria. Indeed, the only previous treatment that has been evaluated to date is exposure and response prevention for obsessive compulsive disorder (Tolin, Melnyk, et al., 2015), which was conducted by Tolin and colleagues as the exemplar for the methods described in the seminal paper. Thus, a strength of this evaluation is that it was conducted by committee members (a) without allegiance to the Tolin Criteria and (b) without allegiance to the treatment being evaluated. Indeed, though we selected two committee members who could provide appropriate meta-analytic (CB) and statistical (MM) expertise, the remaining committee members were general psychology faculty and trainees interested in the integration of science and practice (per their role on the committee). Thus, we were able to test the portability and capacity of a standing, voluntary committee to evaluate a treatment using the Tolin Criteria. Though a full explanation on the process is beyond the scope of this paper, overall, we found that we were successfully able to provide a transparent and rigorous evaluation of CBT-I using the methods ratified by the Division.

That said, limitations of the current evaluation include the fact that we did not explicitly evaluate CBT-I treatment fidelity (e.g., therapist qualifications and training, use of a treatment manual) or risk of bias across eligible reviews, as suggested by Tolin, McKay, and colleagues (2015). This is partially due to time and resource constraints, but also because these factors were captured by the AMSTAR 2 criteria. For example, treatment fidelity is a primary-study level variable, and there are no standard instruments for assessing steps taken to ensure fidelity. As such, this should have been accounted for by the authors of the included meta-analyses. While it is possible that the lack of explicit consideration of these factors may limit our findings, it is doubtful that this would significantly impact our conclusions, given the robustness of the observed effects and the additional positive contextual factors evaluated. In addition, application of the Tolin Criteria relies on systematic reviews which, although efficient, presents several challenges. First, reliance on systematic reviews (which encompasses meta-analyses, as included here) can result in important nuances being overlooked. For example, because the eligible meta-analyses included in the current review report aggregates across several primary studies, the assessment of outcomes, such as sleep efficiency, varied both within and between the eligible meta-analyses. As such, depending on how sleep efficiency was assessed within and between the meta-analyses combined for evaluation using the Tolin Criteria, very different effect sizes could have resulted. In fact, in many cases, the meta-analyses included in the current review did not provide this level of detail regarding how certain outcomes were assessed. This is problematic as it could potentially mask important relationships between the treatment and outcome(s) of interest. Second, a limitation that is inherent to these criteria is the difficulty of evaluating the strength of the evidence base for treatments without published reviews. Although only one review is required to apply the Tolin Criteria, the absence of any systematic review on a given treatment makes applying these criteria difficult. As such, a lack of systematic reviews on a given treatment may be indicative that more research is needed to demonstrate the efficacy of such treatments. When existing reviews are lacking, it could be worthwhile for authors to consider the Tolin Criteria explicitly in their evaluation.

Despite these limitations, this effort by the SCP Committee on Science and Practice serves as the first independent application of the Tolin Criteria for evaluating and determining the strength of recommendation of one of the empirically supported treatment models featured on the SCP Psychological Treatments list: CBT-I. In addition, this work serves to further refine and standardize the application of the Tolin Criteria and, therefore, enhances the foundation for conducting subsequent evaluations to align the SCP empirically supported treatments list with the state-of-the-science. The SCP empirically supported treatments list is a frequently visited and highly cited web-based resource in the field of clinical psychology. Efforts to update and expand this information will undoubtedly prove valuable to clinical researchers and practitioners alike. These efforts serve to provide up-to-date information to the public that may facilitate clinicians, researchers, and patients utilizing and receiving the most empirically supported treatments in our field. As such, this work serves to improve dissemination of empirically supported treatments in order to help the largest number of individuals receive efficacious interventions.

Public Health Statement: Insomnia is the most prevalent of all sleep disorders and is associated with significant functional impairment along with personal and societal burdens. A systematic review of cognitive behavioral therapy for Insomnia found high quality evidence that it produces clinically meaningful and statistically significant improvements in insomnia and other sleep-related outcomes.