Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 Mar 2.
Published in final edited form as: Psychol Med. 2018 May 24;49(3):491–500. doi: 10.1017/S0033291718001113

Efficacy of digital CBT for insomnia to reduce depression across demographic groups: a randomized trial

Philip Cheng 1, Annemarie I Luik 2, Cynthia Fellman-Couture 1, Edward Peterson 1, Christine LM Joseph 1, Gabriel Tallent 1, Kieulinh Michelle Tran 1, Brian K Ahmedani 1, Timothy Roehrs 1, Thomas Roth 1, Christopher L Drake 1
PMCID: PMC7050476  NIHMSID: NIHMS1558410  PMID: 29792241

Abstract

Background.

Insomnia and depression are highly comorbid and mutually exacerbate clinical trajectories and outcomes. Cognitive behavioral therapy for insomnia (CBT-I) effectively reduces both insomnia and depression severity, and can be delivered digitally. This could sub-stantially increase the accessibility to CBT-I, which could reduce the health disparities related to insomnia; however, the efficacy of digital CBT-I (dCBT-I) across a range of demographic groups has not yet been adequately examined. This randomized placebo-controlled trial examined the efficacy of dCBT-I in reducing both insomnia and depression across a wide range of demographic groups.

Methods.

Of 1358 individuals with insomnia randomized, a final sample of 358 were retained in the dCBT-I condition and 300 in the online sleep education condition. Severity of insomnia and depression was examined as a dependent variable. Race, socioeconomic status (SES; household income and education), gender, and age were also tested as independent moderators of treatment effects.

Results.

The dCBT-I condition yielded greater reductions in both insomnia and depression severity than sleep education, with significantly higher rates of remission following treatment. Demographic variables (i.e. income, race, sex, age, education) were not significant moderators of the treatment effects, suggesting that dCBT-I is comparably efficacious across a wide range of demographic groups. Furthermore, while differences in attrition were found based on SES, attrition did not differ between white and black participants.

Conclusions.

Results provide evidence that the wide dissemination of dCBT-I may effectively target both insomnia and comorbid depression across a wide spectrum of the population.

Keywords: CBT-I, depression, insomnia, Internet

Introduction

Cognitive behavioral therapy for insomnia (CBT-I) has received significant attention as an effective non-pharmacological treatment for insomnia. In fact, the accumulation of substantial supporting evidence for the effectiveness of CBT-I (Brasure et al. 2016) has led to its recent recommendation as first-line treatment for chronic insomnia by the American College of Physicians (Qaseem et al. 2016). In addition to its effectiveness for insomnia, CBT-I also reduces concurrent depression without ostensibly targeting non-sleep depression symptoms (Manber et al. 2008, 2011; Taylor and Pruiksma, 2014). In fact, one study found that CBT-I alone resulted in a 37% decrease in depression severity (Manber et al. 2011). This has particular significance given the concordance between insomnia and depression, and their bi-directional relation (Lustberg and Reynolds III, 2000). Indeed, insomnia is a reliable precursor of depression (Mahowald, 2007; Li et al. 2010; McCall et al. 2010; Baglioni et al. 2011; Pigeon et al. 2012) and increases the risk for depression by nearly threefold compared to healthy sleepers (Zammit et al. 1999; National Institutes of Health, 2005; Baglioni et al. 2011; Hajak et al. 2011; Kessler et al. 2011). Given the pervasive health impacts of both insomnia and depression, the opportunity to address both via CBT-I is advantageous and cost-effective.

Despite its promise, CBT-I is not without limitations. These primarily include a scarcity of certified practitioners, geographic distance to providers, personal limitations on travel and costs, and the requirement of 6–8 weeks of direct patient contact. These factors can contribute to reduced usage and may remain as barriers even after treatment initiation (Vincent and Hameed, 2003; Espie et al. 2007). Above all, the limited availability of credentialed behavioral sleep medicine (BSM) clinicians is the most salient barrier: nearly 20% of US adults experience insomnia (Roth et al. 2006), yet there are well under 1000 board-certified BSM providers (Fields et al. 2013). The limited availability of certified clinicians is further compounded by disparities in access to health care (including CBT-I), which are particularly burdensome to those with lower socioeconomic status (SES) and racial minority groups. Furthermore, treatment dropout rates can be significantly higher in these vulnerable populations due to limited transport, insurance-coverage difficulties, child-care responsibilities, work schedule conflicts, and lack of knowledge about available treatments (Cooper and Conklin, 2015). Racial minorities also historically report significant distrust in the medical and health care systems, which may be associated with reduced health-seeking behaviors (Corbie-Smith et al. 2002; Armstrong et al. 2007; Kennedy et al. 2007).

To address some of these barriers of access to CBT-I, web and mobile technologies have been utilized to develop Internet-based or digital CBT-I (dCBT-I). dCBT-I confers the advantages of reduced cost, therapist time, and empowers end users with technology to manage their own care and health. Furthermore, there may be less stigma associated with a user-driven digital health intervention compared with traditional therapy, which may also increase the accessibility of insomnia treatment. Though dCBT-I is still nascent, support for its efficacy has been accumulating from randomized controlled trials comparing it with an attention control or face-to-face CBT-I (Ström et al. 2004; Ritterband et al. 2009, 2017; Espie et al. 2012; Zachariae et al. 2016). Other studies have also demonstrated the efficacy of dCBT-I programs in reducing depression, with effects sustained at both 6 and 18 months follow-up (Christensen et al. 2016; Batterham et al. 2017). This evidence suggests the potential for the wide dissemination of an accessible and low-cost intervention for both insomnia and depression.

Ultimately, the success of dCBT-I is predicated on its effectiveness for a wide range of individuals; however, the efficacy of dCBT-I on insomnia and depression in populations with health disparities (minority and low income) has not yet been adequately examined. As such, the objective of this study was to test the efficacy of dCBT-I in reducing insomnia and depression across diverse demographic groups, including race, SES (income and education), gender, and age. Additional analyses also examined demographic differences in attrition.

Methods

Recruitment for this study sampled from six hospitals, 38 medical centers, and subscribers of a major health insurance company in southeastern Michigan. Recruitment occurred between May and November of 2016, and utilized Internet-based methods, including health system-wide email newsletters, existing research databases (e.g. Qualtrics and prior research participants who have consented to future research recruitment), and clinic databases (e.g. health system chart review). Interested participants completed a screening survey via an online questionnaire platform (Qualtrics, Provo, UT) that assessed for study eligibility (see Fig. 1). Eligible participants had to meet the criteria for insomnia determined via Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) (American Psychiatric Association, 2013) diagnostic criteria, including the endorsement of sleep difficulty (i.e. difficulties falling asleep, staying asleep, or waking too early) for at least three nights a week lasting at least 3 months, accompanied by moderate distress and/or functional impairment consequential to the sleep difficulties. Exclusion criteria (assessed via the screening questionnaire) included lack of insomnia, reported diagnosis of sleep disorders other than insomnia (e.g. obstructive sleep apnea, restless legs, narcolepsy), and reported diagnosis of bipolar disorder or seizure disorder. Individuals who screened positive for signs of severe depression (i.e. self-reported daily or near daily depressed mood and anhedonia) were also excluded from study participation because a separate aim (not included in this manuscript) of this study examined the longer term prevention of depression incidence or relapse. As evidence for prevention or relapse requires longer term follow-up, additional analyses for this separate aim are planned after completion of follow-up visits.

Fig. 1.

Fig. 1.

Flow chart of study recruitment and enrollment.

Study design

This study utilized a placebo-controlled design with simple randomization into two parallel arms (dCBT-I and online sleep education). Randomization was computerized and conducted centrally through Qualtrics immediately after participants met eligibility criteria. A total of 1385 individuals with insomnia were enrolled in the study and randomized into either the dCBT-I or online sleep education conditions. The research staff was blinded to treatment allocation. Participants were randomized at a 2:1 ratio for the dCBT-I condition due to a higher anticipated attrition rate for the active compared with the sleep education condition, as has been previously demonstrated in Internet-based interventions (Christensen et al. 2009). The final sample in the analyses included 358 for the dCBT-I condition and 300 for the online sleep education condition (age range 18–92). Attrition was operationalized as those who did not engage with treatment (i.e. no-show) and those who discontinued after treatment onset (see Fig. 1 for enrollment flow chart). All procedures were approved by the Henry Ford Health System Institutional Review Board. Informed consent was also given by all participants immediately following study eligibility, before any study procedures were executed.

Outcome measures

Assessments of insomnia and depression were used as the primary outcomes and were obtained via the same questionnaire platform as the screening survey. Assessments were conducted at pre- and post-treatment, with the latter occurring approximately 1 week following the final dCBT-I session. Insomnia severity was measured using the Insomnia Severity Index (ISI) (Morin et al. 2011; Thorndike et al. 2011), with higher scores indicating increased insomnia severity (range 0–28). Remission at post-treatment operationalized as an ISI score of ⩽7, which corresponds to the threshold for non-clinically significant insomnia. Depression severity was measured using the Quick Inventory of Depressive Symptomatology (QIDS), with higher scores indicating increased depression severity (range 0–27). To examine the changes in non-sleep symptoms of depression, analyses only utilized non-sleep items from the QIDS (items 1–4 were dropped), unless otherwise specified. Both the ISI and QIDS have established validity and have been utilized in sleep- and depression-related clinical trials (Savard et al. 2005; Brown et al. 2008; Bernstein et al. 2010; Yeung et al. 2012; American Psychiatric Association, 2013; Collins et al. 2014) and clinical settings.

dCBT-I condition

Individuals randomized to the dCBT-I condition completed the Sleepio program via the Internet (www.sleepio.com, Big Health Ltd.). Sleepio is among several currently available dCBT-I programs, and was selected for this study because it is evidence-based, standardized, fully automated, and has been analyzed in multiple randomized controlled trials (Espie et al. 2012, 2016; Freeman et al. 2017). Participants received access for 12 weeks during which they could take the six core sessions of dCBT-I; each session was unlocked on a weekly basis and participants were advised to take a session once a week. The intervention covered behavioral components (e.g. sleep restriction, stimulus control) and cognitive components (e.g. cognitive restructuring, paradoxical intention), as well as relaxation strategies (e.g. progressive muscle relaxation and autogenic training) and sleep hygiene. Sessions are directed by an animated ‘virtual therapist’ who guides the sessions, conducts progress reviews with the participant, discusses diary data submitted during the week, and assesses progress achieved against previously set goals. In addition, the participant will have access to additional components such as a library with background information, a forum with other users of the program, their case file, and weekly live expert sessions.

Online sleep education

Individuals randomized to the online sleep education condition received six weekly e-mails containing information on the following topics: the basics of endogenous sleep regulation; the impact on sleep of health problems; the effects of sleep disruptive substances, such as caffeine, nicotine, and alcohol; and tips on creating a sleep-conducive bedroom environment.

Demographic moderators

Treatment response was also examined by demographics, which included annual household income, race, sex, age, and education. Annual household income was operationalized as an ordinal variable with four levels: poverty, low, middle, and high. Poverty was operationalized as an annual household income less than 15k, which is consistent with the poverty threshold for a two-person household in 2016 (US Census Bureau, 2016). The thresholds for low, middle, and high income were <35k, <75k, and ⩾75k, respectively. Race was categorized as white, black, or other (i.e. Asian, American Indian/Alaska Native, multiracial, and unknown). Education was also operationalized as an ordinal variable with four levels: high school or less, some college, college, and graduate school. These categories correspond to the International Standard Classification of Education (UNESCO Institute for Statistics, 2012) levels 3 or below, 4 and 5, 6, and 7 or higher.

Analytical approach

A per-protocol analysis was conducted to examine the efficacy of dCBT-I in reducing insomnia and depression. Two mixed-effects linear regression models were implemented with ISI and QIDS scores as the outcome variables. A random intercept was included in both models to account for individual variation in pre-treatment levels of insomnia and depression. Fixed effects included time (pre- and post-treatment), condition (dCBT-I, sleep education), and the interaction of time × condition.

To test for differences in the treatment based on demographic variables (SES, race, sex, age, education), each of the two mix-effects models was further tested with demographic variables entered into the model. Demographic variables were first tested as a moderator with a three-way demographic × time × condition interaction variable. In cases where the demographic variable was not a significant moderator, they were subsequently added as a covariate to the original model with time, condition, and time × condition as fixed effects. Power analyses indicated that the final sample size achieved 80% power to detect a small effect size (0.16) for a three-way interaction.

Results

The per-protocol analyses included a final sample of 358 individuals who completed the dCBT-I condition, and 300 who received online sleep education. See Table 1 for sample characteristics between conditions.

Table 1.

Experimental and control conditions stratified by demographic groups

Variables dCBT-I
(N = 358)
Sleep education
(N = 300)
Age 44.5 ± 15.8 s.d. 45.7 ± 15.1 s.d.
Sex 78.0% female 80.0% female
Race
 White 75.1% 67.0%
 Black 18.2% 25.0%
 Other  3.7%  8.0%
Education
 High school or less 14.5% 14.7%
 Some college 26.3% 33.7%
 College 38.8% 29.3%
 Graduate school 20.4% 22.3%
Household income
 Poverty (<15k) 14.3% 12.4%
 Low (<35k) 26.5% 32.0%
 Middle (<75k) 29.2% 28.3%
 Higher (75k+) 30.0% 27.3%
Insomnia (ISI) 17.9 ± 4.3 s.d. 17.7 ± 4.4 s.d.
Depression (QIDS) 10.8 ± 4.5 s.d. 10.8 ± 4.6 s.d.
 None (<6) 13.1% 12.3%
 Mild (<11) 38.5% 39.0%
 Moderate (<16) 32.4% 28.6%
 Severe (<21) 13.1% 18.7%
 Very severe (21+)  2.8%  1.3%
 QIDS sans sleep items  8.1 ± 4.5 s.d.  8.0 ± 4.4 s.d.

No statistical differences were detected at baseline between groups. ISI, Insomnia Severity Index; QIDS, Quick Inventory of Depressive Symptomatology.

Effectiveness of dCBT-I in reducing insomnia and depression

Insomnia

Results from the linear mixed model revealed a significant time × condition interaction, t(656) = −13.6, p < 0.001, indicating that change in ISI at post-treatment differed significantly between the dCBT-I and sleep education conditions. Specifically, the average decrease in ISI in the dCBT-I condition (−10.0 points ± 5.7 s.d.) was twofold greater than the decrease in the sleep education condition (−4.4 ± 4.6) (see Fig. 2A).

Fig. 2.

Fig. 2.

Change in ISI scores between the dCBT-I and sleep education conditions. Error bars indicate standard error of the mean. Experimental conditions in Panel A have been jittered for visual clarity and do not represent timing of treatments.

In addition to change in ISI, follow-up analyses also examined response and remission rates at post-treatment (response: reduction in ISI ⩾8; remission: post-treatment ISI ⩽7) using a χ2 test. Results indicated that more individuals in the dCBT-I condition exhibited a clinically significant treatment response (65.1%) compared with those in the sleep education condition (22.3%), χ2(1) = 16.9, p < 0.0001. Results also indicated that the remission rate at post-treatment was significantly higher in the dCBT-I condition, χ2(1) = 111.5, p < 0.0001. Specifically, the remission rate was almost four times higher in the dCBT-I condition [53.9%, 95% CI (48.7–59.1)] than in the sleep education condition [14.0%, 95% CI (10.4–17.6)] (see Fig. 2B).

Post-hoc analyses also examined if dCBT-I resulted in differential changes in difficulties with sleep onset, sleep maintenance, or early morning awakening. Results indicated comparable improvements between sleep onset difficulties (−1.25 pts), sleep maintenance difficulties (−1.31 pts), and early morning awakenings (−1.26 pts).

Depression

Models with depression symptoms as the outcome variable showed similar results compared with insomnia symptoms. A time × condition interaction, t(656) = −13.6, p < 0.001, indicated that change in QIDS at post-treatment differed significantly between the dCBT-I and sleep education conditions. Specifically, the average decrease in QIDS in the dCBT-I condition (−4.1 ± 4.7 s.d.) was 2.5 times greater than that of the sleep education condition (−1.6 points ± 3.7) (see Fig. 3 for change in QIDS). Overall, the effect size (Hedge’s g; thresholds for small, medium, and large effects are 0.2, 0.5, and 0.8, respectively) for improvements in depression (with sleep items) was 0.64 (medium effect size), which is higher than the average effect size documented for a range of antidepressants (Hedge’s g = 0.37) (Turner et al. 2008).

Fig. 3.

Fig. 3.

Change in QIDS scores between the dCBT-I and sleep education conditions.

Exploratory analyses also examined changes by item on the QIDS, which revealed greater reductions in the dCBT-I condition in all non-sleep items except for sadness, suicidality, and restlessness – these items included weight, energy, decision-making, general interest, self-perception, sluggishness, and appetite. The greatest group difference was found with reduced energy, with dCBT-I showing an additional half point decrease compared with sleep education. This was followed by improved self-perception and decision-making, with dCBT-I showing an additional 0.39 and 0.38 point improvement, respectively (see Fig. 4 for change in QIDS items).

Fig. 4.

Fig. 4.

Change in non-sleep items on the QIDS by experimental condition.

Post-hoc analyses also indicated that differences in treatment response by condition also increased linearly by baseline depression severity (sans sleep items), condition × initial severity (lower, middle, and upper tertiles), t(652) = −3.94, p < 0.0001. Those in the upper tertile of depression severity showed the greatest improvements in non-sleep symptoms of depression from dCBT-I (−8.0 ± 4.7 s.d.) relative to the sleep education condition (−3.7 ± 4.4 s.d.), compared with the middle (dCBT-I: −3.8 ± 3.7 s.d., sleep education: −1.65 ± 3.1 s.d.) and lower tertiles (dCBT-I: −1.0 ± 2.7 s.d., sleep education: 0.41 ± 2.4 s.d.). Finally, post-hoc analyses also indicated that the reduction in non-sleep depression symptoms was significantly associated with the reduction in insomnia symptoms across both groups, though the effect was stronger in the dCBT-I condition (r = 0.50, p < 0.001) compared with the sleep education condition (r = 0.36, p < 0.001).

Demographic moderators of treatment response

SES: household income

Household income did not significantly moderate the efficacy of dCBT-I in reducing either insomnia or depression, and the estimated effects of household income were small (ISI: B = 0.78 ± 0.94 s.e.; QIDS: B = 0.78 ± 0.77 s.e.). The time × condition interactions for both ISI and QIDS remained significant with household income as a covariate. Additionally, household income was a significant covariate for both insomnia and depression, indicating higher severity with lower income. Specifically, ISI scores increased by an average of 1.59 ± 0.35 points with each decrease in income bracket, whereas QIDS scores increased by an average of 2.29 ± 0.32 points with each decrease in income bracket.

SES: education

Education also did not significantly moderate the efficacy of dCBT-I in reducing either insomnia or depression (see Fig. 5). The estimated effects of education were moderate for ISI (B = 1.67 ± 0.94 s.e.) and small for QIDS (B = 0.83 ± 0.78 s.e.). The time × condition interactions for both ISI and QIDS remained significant with education as a covariate. Education was a significant covariate for both insomnia and depression, indicating higher severity with lower education. Specifically, ISI scores increased by an average of 0.90 ± 0.36 points with each decrease in education bracket, whereas QIDS scores increased 1.03 ± 0.34 points with each decrease in education bracket.

Fig. 5.

Fig. 5.

Change in insomnia and depression in the dCBT-I condition by household income brackets.

Race

Race also did not significantly moderate the efficacy of dCBT-I in reducing either insomnia or depression, and the estimated effects of race were small for both ISI (black v. white: B = 0.71 ± 1.00 s.e.; other v. white: B = 0.78 ± 1.57 s.e.) and QIDS (black v. white: B = −0.67 ± 0.83 s.e.); other v. white: B = 0.13 ± 1.30 s.e.). The time × condition interactions for both ISI and QIDS remained significant with education as a covariate. Race was a significant covariate for insomnia but not for depression. Specifically, black individuals reported an overall average of 1.15 ± 0.38 points higher on the ISI compared with white individuals.

Sex

Sex also did not significantly moderate the efficacy of dCBT-I in reducing either insomnia or depression, and the estimated effects of sex were small for both ISI (male v. females: B = 0.64 ± 0.99 s.e.) and QIDS (male v. females: B = −0.28 ± 0.82 s.e.). The time × condition interactions for both ISI and QIDS remained significant with sex as a covariate. Sex was not a significant covariate for insomnia or depression.

Age

Age also did not significantly moderate the efficacy of dCBT-I in reducing either insomnia or depression, and the estimated effects of age were small for both ISI (B = 0.40 ± 0.25 s.e.) and QIDS (B = 0.05 ± 0.21 s.e.). The time × condition interactions for both ISI and QIDS remained significant with age as a covariate. Age was not a significant covariate for insomnia but was significant for depression, though the effect was small. Each decade increase in age was associated with a 0.64 ± 0.09 point decrease in QIDS scores, t(655) = −7.08, p < 0.0001.

Demographic moderators of treatment uptake

Given the notable attrition rates of Internet-delivered interventions (Melville et al. 2010), additional analyses were also conducted to examine if treatment uptake of dCBT-I differed by demographic variables. Results indicated significant differences by household income, education, race, and age (see Table 2). Interestingly, logistic regression indicated that dropout rates did not differ between white and black participants; however, people of color who were not black (i.e. Asian, American Indian/Alaska Native, multiracial, and unknown) showed a 66% increase in odds of dropout compared with white participants, OR = 1.66, 95% CI (1.03–2.77). Individuals with less than a college education also demonstrated higher odds of dropout relative to college graduates; those with some college showed over two-thirds increase in odds of dropout, OR = 1.73, 95% CI (1.20–2.50), whereas those who graduated high school show more than a threefold increase in odds of dropout, OR = 3.26, 95% CI (2.13–5.04). Relative to the middle class, those at low or poverty levels of income also showed greater odds of dropout; those with low household income showed twice the odds of dropout, OR = 2.00, 95% CI (1.41–2.85), and those at the poverty level showed greater than twice the odds of dropout, OR = 2.20, 95% CI (1.46–3.33). Age was also a significant predictor of dropout, with each decade increase above the average age (44.47) associated with approximately a 20% decrease in odds of dropout, OR = 0.82, 95% CI (0.76–0.89).

Table 2.

Dropout rate for dCBT-I compared by demographic variables

Variables Attrition rate Odds ratio
Sex
Male 60.3% .
 Female 61.9% n.s.
Race
White 60.9% .
 Black 60.1% n.s.
 Other* 72.1% 1.66
Education
 High school*** 76.5% 3.26
 Some college*** 63.4% 1.73
College 50.0% .
 Graduate school 46.3% n.s.
Household income
 Poverty (<15k)*** 72.3% 2.20
 Low (<35k)*** 70.4% 2.00
Middle (<75k) 54.3% .
 Higher (75k+) 46.8% n.s.
Age
44.47 (mean age) .
 Decade increments*** 0.82

Differences in attrition were assessed via logistic regression stratified by demographic groups. Reference groups are italicized.

*

p < 0.05

**

p < 0.01

***

p < 0.001.

n.s., not significant.

Discussion

Findings from this study provide further evidence that dCBT-I reduces both insomnia and depression symptoms. As an Internet-based intervention, dCBT-I is less costly and more accessible than face-to-face interventions, particularly given the limited number of specialty providers trained in CBT-I. Given the potential benefits of dCBT-I, it is important that both efficacy and effectiveness are demonstrated across a range of demographic groups. This study was the first and largest to examine the efficacy of dCBT-I across different demographic groups, including those with significant health disparities. Results revealed no significant differences in the improvement of insomnia and depression between demographic groups despite having adequate statistical power to detect a small effect size. Furthermore, the estimates for the moderating effect of demographic groups were small (less than one point on the ISI and QIDS), suggesting limited clinical significance even if effects were to be detectable with a larger sample. Together, there is no strong evidence for reduced efficacy of dCBT-I for both insomnia and depression in the underserved populations studied here.

These results also support the potential for dCBT-I to serve as a first-line intervention for those with both insomnia and depression. The web-based delivery makes this approach highly scalable and sustainable, especially given the significant clinical impact. Because dCBT-I can be implemented with minimal to no involvement of a clinician, it can be easily integrated into a primary care setting where both insomnia and depression are typically first detected. Furthermore, integration at the primary care setting may also be ideal in reducing health disparities in access to mental health care. Future studies should focus on the prevention of incident depression in longitudinal follow-up designs to determine the potential impact on reducing the incidence and relapse of major depressive disorder.

dCBT-I can also be combined with CBT-I in a stepped-care framework by use of a fully automatized system with the potential for wide dissemination (Christensen et al. 2016). Stepped-care approaches also capitalize on the strengths of both face-to-face and Internet-delivered CBT-I treatment modalities while minimizing their disadvantages and inefficiencies. As opposed to a system where everyone with insomnia receives treatment by a specialist, a stepped-care approach begins with a least restrictive intervention, and only graduates limited- or non-responders to treatment with a specialist. This reserves specialist treatment for more complex cases, thus allowing specialists to practice at the top of their licenses.

We propose that this stepped-care model would be most impactful if integrated into the primary care system, as early identification and treatment of both insomnia and depression typically occur in primary care. The current integrated care processes can be leveraged to immediately implement treatment when sleep problems are initially identified in the primary care setting. This stepped-care approach to insomnia treatment has yet to be tested in large-scale and clinically-based effectiveness trials (i.e. real-world), but has the potential to significantly impact insomnia therapeutics and transform the limited approach currently in place. In particular, the large-scale clinical studies will need a particular focus on effectiveness (including treatment uptake and adherence) in underserved minority and low-income populations to establish real-world generalizability and wide dissemination. Such trials are also needed to determine if a reduction of insomnia and depression symptoms leads to long-term mitigation of depression incidence (i.e. secondary prevention).

Demographic differences in attrition for dCBT-I

One critical consideration for the implementation of dCBT-I is the utilization and uptake by individuals. This is a critical gap in the literature as very few studies have examined demographic differences in attrition for insomnia interventions, and this is the first study that has done so for dCBT-I. As such, we were unable to assess how attrition rates by demographics for dCBT-I compare to face-to-face CBT-I or other digital insomnia interventions. However, the overall attrition rate for the dCBT-I group found in this study was within the expected range in comparison with other randomized controlled trials of Internet-delivered psychotherapy interventions (50–83%) (Yeung et al. 2015; Batterham et al. 2017; Watson et al. 2017). Notably, the rate of dropout was much lower for those who engaged in CBT-I (i.e. completed at least one session) compared with those who did not engage in treatment (i.e. no-shows).

While we found no differences in the treatment effects of dCBT-I between demographic groups, results did show demographic differences in the dropout rates consistent with other intervention studies (Wierzbicki and Pekarik, 1993; Melville et al. 2010; Watson et al. 2017). In particular, low SES (i.e. education and income) was associated with greater risk for dropout from dCBT-I. This is consistent with prior evidence from a meta-analysis indicating that lower SES was the strongest predictor of psychotherapy dropout among demographic variables (Wierzbicki and Pekarik, 1993). Importantly, results from the current study suggest that increased accessibility alone does not sufficiently mitigate treatment barriers for many individuals with limited education and financial resources. Further evidence must be collected to elucidate additional barriers to engagement and persistence for dCBT-I associated with low SES.

The finding of high attrition among those with low SES also represents an opportunity to examine if and how dCBT-I may be refined to enhance the appropriateness and feasibility across a diverse range of people. This evaluation is among the critical steps required prior to the wide dissemination of dCBT-I. The flexibility afforded via the digital platform means that both content and implementation of dCBT-I can be readily tailored to address the needs of a low SES population. Additionally, gains in retention could be further enhanced with a stepped-care model. For example, even a modest improvement of 5–10% in attrition could extend accessibility of dCBT-I to one-third of those in the lowest SES bracket. The remaining individuals can be triaged to enhanced interventions that are socially integrative and multipronged. For instance, a recent study successfully achieved a fourfold reduction in attrition in low-income racial minority participants via a strategic framework that targeted cultural and linguistic competency, relationship building, leveraging existing social networks, contingency management, and other relevant domains (Flores et al. 2017). Because these approaches are generally more resource intensive, they are well matched for a stepped-care model that enables redistribution of existing resources. As such, future research should test the feasibility of stepped-care models for insomnia, particularly in those with limited education and financial resources.

Importantly, attrition rates from this study did not differ between black and white participants despite historically established differences in treatment engagement and adherence in traditional forms of psychotherapy (Yamamoto et al. 1967; Sue et al. 1974). In fact, an early study found that attrition within 6 weeks of treatment (a match to the length of dCBT-I in this study) was twice as high for black compared with white patients (Rosenthal and Frank, 1958). More recent studies have continued to show similarly higher attrition rates in black compared with white individuals (Murphy et al. 2013; Johnson et al. 2014). As such, it is notable that attrition rates for dCBT-I were comparable between white and black participants in this study. Interestingly, results did indicate that those in the ‘other’ category appeared to be at higher risk for dropout. The interpretation of this result is less clear because this category is significantly smaller in sample size (<25 in each condition) and comprises multiple racial identities (including multiracial identities). Future research with greater representation of non-black people of color is necessary to further understand this relationship.

While gender was not associated with differences in dropout, our final sample was predominantly female. Exploratory analyses revealed comparable gender distribution throughout the enrollment process, suggesting that the gender differences may be associated with initial interest in participating in intervention research for insomnia. There is evidence that females are more likely to engage in health-seeking behaviors in both a traditional clinical setting (Möller-Leimkühler, 2002) and on the Internet (Ybarra and Suman, 2006). The gender differences in help-seeking behaviors are also further compounded by the higher prevalence of insomnia in women (Roth and Roehrs, 2003). Finally, the gender distribution found in this study was also comparable to that found in another recent study of dCBT-I (Christensen et al. 2016).

Results from this study also found that older age was associated with reduced attrition from dCBT-I, which is consistent with the previous studies that have examined intervention dropout (DeMaris, 1989; Lange et al. 2005). The replication of this finding in the context of an Internet-based intervention is significant, particularly given general concerns that technological literacy in older adults would be a barrier to dCBT-I. While results do not speak specifically to the role of technological literacy, they do support the feasibility of dCBT-I in older populations, and also indicate comparable efficacy across the age ranges. This should be further examined in a larger effectiveness study in a community sample.

While the depression severity in this sample ranged from none to very severe (max QIDS = 23), the majority of depression scores fell in the ‘mild’ and ‘moderate’ range (QIDS score 7–15). As such, one limitation of this study is the generalizability to those with more severe forms of depression. Another limitation is the use of a per-protocol analysis, which can be vulnerable to bias and also precludes generalizability of the described effects to all who would be prescribed dCBT-I. However, given that our proposed stepped-care model would integrate dCBT-I as a first-line intervention, those who were unable to complete dCBT-I would be additionally triaged for other forms of intervention. Additionally, as is the case with most efficacy trials, these results describe the effect of the treatment under ideal circumstances (e.g. minimal comorbidities, ability to complete the treatment) and should be interpreted as such. Finally, there were no statistical differences in demographics between the two experimental groups, suggesting that any bias associated with the per-protocol analysis was minimal. These data also point to a clear need for further implementation research to examine how uptake of Internet-delivered interventions may be enhanced in the community, particularly in those with socioeconomic disadvantages.

Overall, results from this study lend support that dCBT-I could be an effective first-line intervention for insomnia and depression. The evidence suggests that there are little to no differences in efficacy for underserved minorities who are able to complete the intervention. Furthermore, the lack of differences in attrition between black and white participants suggests that dCBT-I may also have potential in improving the racial disparities of access to care for insomnia and depression. Future research should examine the effectiveness of dCBT-I in a community sample because this evidence was collected in the context of a research intervention and thus may have limited generalizability to a naturalistic help-seeking population. Furthermore, treatment barriers in lower SES populations need to be further characterized in order to inform interventions strategies that are feasible and appropriate.

Conclusion

Findings from this study provide further evidence for the efficacy of dCBT-I in treating insomnia and concurrent depression in a wide range of individuals from various demographic groups, including race, SES (income and education), sex, and age. Furthermore, dCBT-I appears to have comparable uptake in both black and white individuals, suggesting that it could have potential in reducing racial disparities in access to care for insomnia and depression. Together, these evidence support the further examination of the utility of dCBT-I in large-scale reduction of insomnia and depression.

Acknowledgments.

Support for this study was provided from the Robert Wood Johnson Foundation, and from the National Institute of Mental Health R56MH115150 awarded to CLD. Funding for PC was provided from the National Heart Lung and Blood Institute (K23HL138166). We would also like to thank David Adler for his continued support of our research program.

Footnotes

Clinical Trial Registry: clinicaltrials.gov; Clinical Trial Identifier:

References

  1. American Psychiatric Association (2013) Diagnostic and Statistical Manual of Mental Disorders (DSM-5®). Arlington, VA: American Psychiatric Pub. [Google Scholar]
  2. Armstrong K, Ravenell KL, McMurphy S and Putt M (2007) Racial/ethnic differences in physician distrust in the United States. American Journal of Public Health 97, 1283–1289. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Baglioni C, Battagliese G, Feige B, Spiegelhalder K, Nissen C, Voderholzer U, Lombardo C and Riemann D (2011) Insomnia as a predictor of depression: a meta-analytic evaluation of longitudinal epidemiological studies. Journal of Affective Disorders 135, 10–19. [DOI] [PubMed] [Google Scholar]
  4. Batterham PJ, Christensen H, Mackinnon AJ, Gosling JA, Thorndike FP, Ritterband LM, Glozier N and Griffiths KM (2017) Trajectories of change and long-term outcomes in a randomised controlled trial of Internet-based insomnia treatment to prevent depression. British Journal of Psychiatry Open 3, 228–235. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Bernstein IH, Rush AJ, Trivedi MH, Hughes CW, Macleod L, Witte BP, Jain S, Mayes TL and Emslie GJ (2010) Psychometric properties of the Quick Inventory of Depressive Symptomatology in adolescents. International Journal of Methods in Psychiatric Research 19, 185–194. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Brasure M, Fuchs E, MacDonald R, Nelson VA, Koffel E, Olson CM, Khawaja IS, Diem S, Carlyle M, Wilt TJ, Ouellette J, Butler M and Kane RL (2016) Psychological and behavioral interventions for managing insomnia disorder: an evidence report for a clinical practice guideline by the American College of Physicians. Annals of Internal Medicine 165, 113. [DOI] [PubMed] [Google Scholar]
  7. Brown ES, Murray M, Carmody TJ, Kennard BD, Hughes CW, Khan DA and Rush AJ (2008) The Quick Inventory of Depressive Symptomatology-Self-report: a psychometric evaluation in patients with asthma and major depressive disorder. Annals of Allergy, Asthma & Immunology 100, 433–438. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Christensen H, Griffiths KM and Farrer L (2009) Adherence in Internet interventions for anxiety and depression: systematic review. Journal of Medical Internet Research 11, e13. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Christensen H, Batterham PJ, Gosling JA, Ritterband LM, Griffiths KM, Thorndike FP, Glozier N, O’Dea B, Hickie IB and Mackinnon AJ (2016) Effectiveness of an online insomnia program (SHUTi) for prevention of depressive episodes (the GoodNight Study): a randomised controlled trial. The Lancet Psychiatry 3, 333–341. [DOI] [PubMed] [Google Scholar]
  10. Collins LM, Nahum-Shani I and Almirall D (2014) Optimization of behavioral dynamic treatment regimens based on the sequential, multiple assignment, randomized trial (SMART). Clinical Trials 11, 426–434. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Cooper AA and Conklin LR (2015) Dropout from individual psychotherapy for major depression: a meta-analysis of randomized clinical trials. Clinical Psychology Review 40, 57–65. [DOI] [PubMed] [Google Scholar]
  12. Corbie-Smith G, Thomas SB and George DMMS (2002) Distrust, race, and research. Archives of Internal Medicine 162, 2458–2463. [DOI] [PubMed] [Google Scholar]
  13. DeMaris A (1989) Attrition in batterers’ counseling: the role of social and demographic factors. Social Service Review 63, 142–154. [Google Scholar]
  14. Espie CA, Macmahon KM, Kelly H-L, Broomfield NM, Douglas NJ, Engleman HM, McKinstry B, Morin CM, Walker A and Wilson P (2007) Randomized clinical effectiveness trial of nurse-administered small-group cognitive behavior therapy for persistent insomnia in general practice. Sleep 30, 574–584. [DOI] [PubMed] [Google Scholar]
  15. Espie CA, Kyle SD, Williams C, Ong JC, Douglas NJ, Hames P and Brown JS (2012) A randomized, placebo-controlled trial of online cognitive behavioral therapy for chronic insomnia disorder delivered via an automated media-rich web application. Sleep 35, 769–781. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Espie CA, Luik AI, Cape J, Drake CL, Siriwardena AN, Ong JC, Gordon C, Bostock S, Hames P and Nisbet M (2016) Digital cognitive behavioural therapy for insomnia versus sleep hygiene education: the impact of improved sleep on functional health, quality of life and psychological well-being. Study protocol for a randomised controlled trial. Trials 17, 257. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Fields BG, Schutte-Rodin S, Perlis ML and Myers M (2013) Master’s-level practitioners as cognitive behavioral therapy for insomnia providers: an underutilized resource. Journal of Clinical Sleep Medicine: JCSM 9, 1093. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Flores G, Portillo A, Lin H, Walker C, Fierro M, Henry M and Massey K (2017) A successful approach to minimizing attrition in racial/ethnic minority, low-income populations. Contemporary Clinical Trials Communications 5, 168–174. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Freeman D, Sheaves B, Goodwin GM, Yu L-M, Nickless A, Harrison PJ, Emsley R, Luik AI, Foster RG, Wadekar V, Hinds C, Gumley A, Jones R, Lightman S, Jones S, Bentall R, Kinderman P, Rowse G, Brugha T, Blagrove M, Gregory AM, Fleming L, Walklet E, Glazebrook C, Davies EB, Hollis C, Haddock G, John B, Coulson M, Fowler D, Pugh K, Cape J, Moseley P, Brown G, Hughes C, Obonsawin M, Coker S, Watkins E, Schwannauer M, MacMahon K, Siriwardena AN and Espie CA (2017) The effects of improving sleep on mental health (OASIS): a randomised controlled trial with mediation analysis. The Lancet Psychiatry 4, 749–758. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Hajak G, Petukhova M, Lakoma MD, Coulouvrat C, Roth T, Sampson NA, Shahly V, Shillington AC, Stephenson JJ, Walsh JK and others (2011) Days-out-of-role associated with insomnia and comorbid conditions in the America Insomnia Survey. Biological Psychiatry 70, 1063–1073. [DOI] [PubMed] [Google Scholar]
  21. Johnson S, Price M, Mehta N and Anderson PL (2014) Stereotype confirmation concerns predict dropout from cognitive behavioral therapy for social anxiety disorder. BMC Psychiatry 14, 233. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Kennedy BR, Mathis CC and Woods AK (2007) African Americans and their distrust of the health care system: healthcare for diverse populations. Journal of Cultural Diversity; Lisle 14, 56–60. [PubMed] [Google Scholar]
  23. Kessler RC, Berglund PA, Coulouvrat C, Hajak G, Roth T, Shahly V, Shillington AC, Stephenson JJ and Walsh JK (2011) Insomnia and the performance of US workers: results from the America insomnia survey. Sleep 34, 1161–1171. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Lange A, Rietdijk D, Hudcovicova M and Others (2005) Interapy: a controlled randomized trial of the standardized treatment of posttraumatic stress through the Internet. Year Book of Psychiatry & Applied Mental Health 2005, 174. [DOI] [PubMed] [Google Scholar]
  25. Li SX, Lam SP, Yu MW, Zhang J and Wing YK (2010) Nocturnal sleep disturbances as a predictor of suicide attempts among psychiatric outpatients: a clinical, epidemiologic, prospective study. The Journal of Clinical Psychiatry 71, 1440–1446. [DOI] [PubMed] [Google Scholar]
  26. Lustberg L and Reynolds III CF (2000) Depression and insomnia: questions of cause and effect. Sleep Medicine Reviews 4, 253–262. [DOI] [PubMed] [Google Scholar]
  27. Mahowald M (2007) Book review sleep disorders and sleep deprivation: an unmet public health problem by the committee on sleep medicine and research. New England Journal of Medicine 356, 199–200. [Google Scholar]
  28. Manber R, Edinger JD, Gress JL, San Pedro-Salcedo MG, Kuo TF and Kalista T (2008) Cognitive behavioral therapy for insomnia enhances depression outcome in patients with comorbid major depressive disorder and insomnia. Sleep 31, 489–495. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Manber R, Bernert RA, Suh S, Nowakowski S, Siebern AT and Ong JC (2011) CBT for insomnia in patients with high and low depressive symptom severity: adherence and clinical outcomes. Journal of Clinical Sleep Medicine 7, 645–652. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. McCall WV, Blocker JN, D’Agostino R, Kimball J, Boggs N, Lasater B and Rosenquist PB (2010) Insomnia severity is an indicator of suicidal ideation during a depression clinical trial. Sleep Medicine 11, 822–827. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Melville KM, Casey LM and Kavanagh DJ (2010) Dropout from Internet-based treatment for psychological disorders. British Journal of Clinical Psychology 49, 455–471. [DOI] [PubMed] [Google Scholar]
  32. Möller-Leimkühler AM (2002) Barriers to help-seeking by men: a review of sociocultural and clinical literature with particular reference to depression. Journal of Affective Disorders 71, 1–9. [DOI] [PubMed] [Google Scholar]
  33. Morin CM, Belleville G, Bélanger L and Ivers H (2011) The Insomnia Severity Index: psychometric indicators to detect insomnia cases and evaluate treatment response. Sleep 34, 601–608. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Murphy EJ, Kassem L, Chemerinski A, Rush AJ, Laje G and McMahon FJ (2013) Retention and attrition among African Americans in the STAR*D study: what causes research volunteers to stay or stray? Depression and Anxiety 30, 1137–1144. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. National Institutes of Health (2005) National Institutes of Health State-of-the-Science Conference statement: management of menopause-related symptoms. Annals of Internal Medicine 142, 1003. [PubMed] [Google Scholar]
  36. Pigeon WR, Pinquart M and Conner K (2012) Meta-analysis of sleep disturbance and suicidal thoughts and behaviors. The Journal of Clinical Psychiatry 73, e1160–7. [DOI] [PubMed] [Google Scholar]
  37. Qaseem A, Kansagara D, Forciea MA, Cooke M and Denberg TD (2016) Management of chronic insomnia disorder in adults: a clinical practice guideline from the American College of Physicians Management of Chronic Insomnia Disorder in Adults. Annals of Internal Medicine 165, 125–133. [DOI] [PubMed] [Google Scholar]
  38. Ritterband LM, Thorndike FP, Gonder-Frederick LA, Magee JC, Bailey ET, Saylor DK and Morin CM (2009) Efficacy of an Internet-based behavioral intervention for adults with insomnia. Archives of General Psychiatry 66, 692–698. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Ritterband LM, Thorndike FP, Ingersoll KS, Lord HR, Gonder-Frederick L, Frederick C, Quigg MS, Cohn WF and Morin CM (2017) Effect of a web-based cognitive behavior therapy for insomnia intervention with 1-year follow-up: a randomized clinical trial. JAMA Psychiatry 74, 68–75. [DOI] [PubMed] [Google Scholar]
  40. Rosenthal D and Frank JD (1958) The fate of psychiatric clinic outpatients assigned to psychotherapy. Journal of Nervous 127, 330–343. [DOI] [PubMed] [Google Scholar]
  41. Roth T and Roehrs T (2003) Insomnia: epidemiology, characteristics, and consequences. Clinical Cornerstone 5, 5–15. [DOI] [PubMed] [Google Scholar]
  42. Roth T, Soubrane C, Titeux L and Walsh JK (2006) Efficacy and safety of zolpidem-MR: a double-blind, placebo-controlled study in adults with primary insomnia. Sleep Medicine 7, 397–406. [DOI] [PubMed] [Google Scholar]
  43. Savard M-H, Savard J, Simard S and Ivers H (2005) Empirical validation of the Insomnia Severity Index in cancer patients. Psycho-Oncology 14, 429–441. [DOI] [PubMed] [Google Scholar]
  44. Ström L, Pettersson R and Andersson G (2004) Internet-based treatment for insomnia: a controlled evaluation. Journal of Consulting and Clinical Psychology 72, 113. [DOI] [PubMed] [Google Scholar]
  45. Sue S, McKinney H, Allen D and Hall J (1974) Delivery of community mental health services to black and white clients. Journal of Consulting and Clinical Psychology 42, 794. [DOI] [PubMed] [Google Scholar]
  46. Taylor D and Pruiksma K (2014) Cognitive and behavioural therapy for insomnia (CBT-I) in psychiatric populations: a systematic review. vol 26. [DOI] [PubMed]
  47. Thorndike FP, Ritterband LM, Saylor DK, Magee JC, Gonder-Frederick LA and Morin CM (2011) Validation of the insomnia severity index as a web-based measure. Behavioral Sleep Medicine 9, 216–223. [DOI] [PubMed] [Google Scholar]
  48. Turner EH, Matthews AM, Linardatos E, Tell RA and Rosenthal R (2008) Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine 358, 252–260. [DOI] [PubMed] [Google Scholar]
  49. UNESCO Institute for Statistics (2012) International Standard Classification of Education: ISCED 2011. Montreal, Quebec: UIS. [Google Scholar]
  50. US Census Bureau (2016) Preliminary Estimate of Weighted Average Poverty Thresholds for 2016.
  51. Vincent NK and Hameed H (2003) Relation between adherence and outcome in the group treatment of insomnia. Behavioral Sleep Medicine 1, 125–139. [DOI] [PubMed] [Google Scholar]
  52. Watson HJ, Levine MD, Zerwas SC, Hamer RM, Crosby RD, Sprecher CS, O’Brien A, Zimmer B, Hofmeier SM, Kordy H, Moessner M, Peat CM, Runfola CD, Marcus MD and Bulik CM (2017) Predictors of dropout in face-to-face and Internet-based cognitive-behavioral therapy for bulimia nervosa in a randomized controlled trial. International Journal of Eating Disorders 50, 569–577. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Wierzbicki M and Pekarik G (1993) A meta-analysis of psychotherapy dropout. Professional Psychology: Research and Practice 24, 190–195. [Google Scholar]
  54. Yamamoto J, James QC, Bloombaum M and Hattem J (1967) Racial factors in patient selection. The American Journal of Psychiatry 124, 630–636. [DOI] [PubMed] [Google Scholar]
  55. Ybarra ML and Suman M (2006) Help seeking behavior and the Internet: a national survey. International Journal of Medical Informatics 75, 29–41. [DOI] [PubMed] [Google Scholar]
  56. Yeung A, Feldman G, Pedrelli P, Hails K, Fava M, Reyes T and Mundt JC (2012) The Quick Inventory of Depressive Symptomatology, clinician rated and self-report: a psychometric assessment in Chinese Americans with major depressive disorder. The Journal of Nervous and Mental Disease 200, 712. [DOI] [PMC free article] [PubMed] [Google Scholar]
  57. Yeung WF, Chung K-F, Ho F and Ho L-M (2015) Predictors of dropout from Internet-based self-help cognitive behavioral therapy for insomnia. Behaviour Research and Therapy 73, 19–24. [DOI] [PubMed] [Google Scholar]
  58. Zachariae R, Lyby MS, Ritterband LM and O’Toole MS (2016) Efficacy of Internet-delivered cognitive-behavioral therapy for insomnia – a systematic review and meta-analysis of randomized controlled trials. Sleep Medicine Reviews 30, 1–10. [DOI] [PubMed] [Google Scholar]
  59. Zammit GK, Weiner J, Damato N, Sillup GP and McMillan CA (1999) Quality of life in people with insomnia. Sleep: Journal of Sleep Research & Sleep Medicine 22(Suppl 2), S379–385. [PubMed] [Google Scholar]

RESOURCES