The Implementation and Effectiveness of Digital Cognitive Behavioral Therapy for Insomnia in Primary Care: A Pilot Study

Abstract

Background

Digital interventions delivering Cognitive Behavioral Therapy for insomnia (Digital CBTi) may increase utilization of effective care for a common and serious condition. A low-intensity implementation strategy may facilitate digital CBTi use in healthcare settings. This pilot study assessed the feasibility of implementing a digital CBTi in Veterans Health Administration (VA) primary care through iterative modifications to a low-intensity implementation strategy, while evaluating clinical outcomes of a specific digital CBTi program.

Methods

A self-directed digital CBTi was implemented in the primary care clinics of a single VA facility using a cohort trial design that iteratively modified an implementation strategy over three 8-month phases. The phase 1 implementation strategy included (1) provider education; (2) point-of-care information via pamphlets; and (3) provider referral to digital CBTi through phone calls or messages. Phases 2 and 3 maintained these activities, while (1) adding a clinic-based coach who performed initial patient education and follow-up support contacts, (2) providing additional recruitment pathways, and (3) integrating the referral mechanism into provider workflow. Implementation outcomes included provider adoption, patient adoption, and acceptability. Clinical outcomes (insomnia severity, depression severity, and sedative hypnotic use) were compared among enrollees at baseline and 10 weeks.

Results

Across all phases 66 providers (48.9%) made 153 referrals, representing 0.38% of unique clinic patients. Of referrals, 77 (50.3%) enrolled in the study, 45 (29.4%) engaged in the program, and 24 (15.7%) completed it. Provider and patient adoption did not differ meaningfully across phases. Among enrollees, digital CBTi was acceptable and the Insomnia Severity Index decreased by 4.3 points (t = 6.41, p < 0.001) and 13 (18.6%) reached remission. The mean number of weakly sedative-hypnotic doses decreased by 2.2 (35.5%) (t = 2.39, p < 0.02).

Conclusions

Digital CBTi implementation in VA primary care is feasible using low-intensity implementation strategy, resulting in improved clinical outcomes for users. However, iterative implementation strategy modifications did not improve adoption.

The trial was registered at clinicaltrials.gov (NCT03151083).

Plain Language Summary

Cognitive Behavioral Therapy is a gold-standard treatment for insomnia, a common and serious health condition. Internet-based or mobile applications that deliver Cognitive Behavioral Therapy for insomnia (Digital CBTi) may increase the use of this treatment. However, strategies for getting healthcare providers and patients in health care systems to use digital CBTi are needed. This study developed and tested such a strategy in five Veterans Health Administration (VA) primary care clinics over three consecutive phases. In the first phase, implementation activities included (1) provider education, (2) information pamphlets for providers and patients, and (3) a referral system to connect patients to the program. In phases 2 and 3, these activities were modified and expanded, primarily through a coach who performed initial patient education and follow-up contacts to increase use of the program. Over the entire study, approximately one-half of providers made a total of 153 patient referrals to the program. Of these referrals, approximately one-half enrolled in the study, 30% engaged in the program, and 15% completed it. Modifications to this strategy did not increase provider or patient use of the program. Use of the program by enrollees resulted in reduced insomnia and less use of medications for sleep. In conclusion, a relatively simple strategy helped VA providers and patients use a digital CBTi program, which improved patient health.

Introduction

Insomnia is a highly prevalent health condition associated with mortality, impaired functioning, and adverse health outcomes, which include mental health disorders and suicidality (Pigeon et al., 2012; Pigeon et al., 2017). Among Veterans, up to one-half entering Veterans Health Administration (VA) care report clinically significant insomnia, and diagnosis rates have increased 7-fold in the last 20 years. (Colvonen et al., 2020; Hom et al., 2016; Jenkins et al., 2015).

The most common insomnia treatments, sedative-hypnotic medications, have well-known risks and limited long-term effectiveness (Glass et al., 2005). In contrast, behavioral treatments such as Cognitive Behavioral Therapy for insomnia (CBTi) are highly effective and recommended as first-line in treatment guidelines (Qaseem et al., 2016). Medium to large effect sizes for CBTi have been observed in multiple systematic reviews covering a wide range of patent populations (Trauer et al., 2015). However, access to and utilization of CBTi is limited by a host of barriers such as a lack of trained providers, travel, scheduling, and stigma associated with mental health treatment (Mojtabai et al., 2011). Of the 1.8 million VA service users who are estimated to meet criteria for insomnia disorder, as few as 0.4% received CBTi in 2020 (C. Ulmer, personal communication, October 1, 2020). Unfortunately, underutilization of effective behavioral treatments is not unique to insomnia or CBTi, and routinely observed for other common mental health conditions and evidence-based practices (Finley et al., 2015).

The deployment of digital interventions may address this gap in care. Digital interventions are internet-based or mobile programs, which can deliver evidence-based psychosocial treatments in a virtual format, enabling varying levels self-management for common mental health conditions (Kumar et al., 2017). The potential value of such programs has been highlighted during the recent COVID-19 pandemic. There are multiple programs delivering digital versions of CBTi, referred to here as digital CBTi, which are efficacious as demonstrated in several research trials and systematic reviews (Seyffert et al., 2016; Zachariae et al., 2016). While trials directly comparing digital self-care with in-person provider-led formats of CBTi have not been undertaken, effect sizes for the two forms of therapy are comparable.

Although digital interventions hold promise for increasing the utilization of evidence-based practices, their use is uncommon in healthcare delivery settings where there are multiple barriers to implementation. For providers, staff, and healthcare organizations, key barriers include inadequate awareness and understanding of such programs as well as the absence of processes to facilitate program access for appropriate patients (Hermes et al., 2019a; Lattie et al., 2020). For patients treated in healthcare systems such as VA, their practical ability to use digital interventions may be limited by inadequate awareness of programs, technology literacy, technology access, as well as competing demands (Hermes et al., 2018). Implementation strategies, addressing these barriers at the patient, provider, and healthcare organization levels are needed to support digital intervention use. At the provider, staff, and organizational level, effective implementation strategies are likely to include making digital interventions available through funding, integrating their delivery in current clinical workflows, and educating providers and staff on their content and availability (Powell et al., 2015). Among users of digital interventions within healthcare systems, implementation strategy components should include educational, motivational, and technical support provided through digital intervention coaching, an activity supported by a broad evidence base (Baumeister et al., 2014; Schueller et al., 2017).

These implementation strategy components are consistent with both the small but growing literature on digital intervention use within healthcare systems (Graham et al., 2020b; Greenhalgh et al., 2017) and prescriptive implementation frameworks such as Replicating Effective Programs (REP), which has been used to facilitate mental health program implementation (Kilbourne et al., 2007). REP is a low-intensity implementation strategy developed by the U.S. Centers for Disease Control, which promotes activities such as provider training, technical assistance, and a program champion (Neumann & Sogolow, 2000). REP has been modified and supplemented for the implementation of behavioral interventions in previous studies as demonstrated by Kilbourne et al. (2007).

The primary objective of this pilot study was to assess the feasibility of implementing a digital CBTi in VA primary care clinics by iteratively modifying of a low-intensity implementation strategy for digital interventions derived from REP. As a second objective, the study examined clinical outcomes and acceptability among those VA service users who received digital CBTi after referral from their provider.

Methods

Design

Between June 2017 and November 2019, a pilot implementation trial took place at a single primary care site, utilizing three phases of active implementation lasting 8-months each. Implementation was paused in the two intervening periods between active implementation to assess progress and iteratively modify strategy activities to improve implementation outcomes. The study was approved by the (human subjects committee of the VA Connecticut Healthcare System) and registered at clinicaltrials.gov (NCT03151083).

Implementation Context

Implementation occurred in five primary care clinics collocated within one geographic site within (the VA Connecticut Healthcare System). The site included three general primary care clinics, one exclusively serving women Veterans, and an integrated primary care and mental health clinic. During the study, these clinics provided patient care to over 15,000 Veterans annually.

Participants

Trial participants were Veterans who were referred by their primary care providers and met enrollment criteria, which included the diagnosis of insomnia and referral by a primary care provider, as well as willingness to participate in the study. Patients taking any medication, as well as those with comorbid mental health disorders, substance use disorders, or obstructive sleep apnea, were not excluded from participation. Patients were excluded only if they were psychiatrically unstable or lacked adequate decision-making capacity or English literacy. Providers were all physicians, physician assistants, advanced practice registered nurses, psychologists, and trainees providing care in the clinics. Providers were not enrolled, but identified and referred patients to digital CBTi after receiving education by the research team as part of the implementation strategy.

Digital Intervention

The “Sleep Healthy Using the Internet” (SHUTi)® digital CBTi program was provided through a contract with Pear Therapeutics Inc. SHUTi delivers the CBTi components of sleep restriction, stimulus control, cognitive therapy and sleep hygiene education through six internet-based modules and sleep diaries, which use text, graphics, animations, vignettes, quizzes, and brief games. The program is designed to be completed over 6 weeks without clinical assistance from a provider. Automated e-mails remind users to complete modules, enter data in sleep diaries, and apply sleep strategies. In research trials, SHUTi participants have experienced reduced insomnia severity, improved sleep measures, and reduced depression symptoms (Christensen et al., 2016; Ritterband et al., 2009). The program had not been used previously in VA.

Implementation Strategy

Specific implementation strategy activities were based on components suggested by REP and findings from the digital intervention implementation literature (Graham et al., 2020a, 2020b; Schueller & Torous, 2020). Pre-implementation planning occurred during meetings between the research team and primary care leaders. Implementation strategy activities were iteratively modified over three phases (Table 1). Modifications to the implementation strategy occurred during periods between phases of active implementation and were based on review of implementation outcomes as well as informal feedback from providers, staff, leadership, and coaches.

Table 1.

Implementation strategy activities across study phases.

Strategy Activity		Definition	Target	Dose	Phase			Implementation Outcome
					1	2	3
Education		Organization-Facing Education About the Program: Group & Individual Sessions, E-mail Contacts	Providers, Staff	Goal of Each Provider Receiving at Least One Education Contact	X	X	X	Provider Adoption
Point-of-Care Information		Pamphlets in Treatment Rooms as a Reminder and Information Resource	Providers, Staff, Patients	Ensure Continual Presence in Treatment Locations	X	X	X	Provider & Patient Adoption
Basic Referral Pathway		Phone & Messaging Contacts Between Providers and Research Team	Providers, Organization	Planned and Organized Prior to Active Implementation	X	X		Provider & Patient Adoption
Coach Contacts	Initial Education	Initial Session Describing the Content and Process of Digital CBTi	Patients	Multiple Outreach Attempts, One Session Per Patient		X	X	Patient Adoption
	Follow-up Support	Motivational Support to Start and Complete Digital CBTi	Patients	Multiple Outreach Attempts, One Session Per Patient		X	X	Patient Adoption
	Ad Hoc Contacts	As Needed, Patient-Initiated Coach Contact	Patients	Available for Multiple Sessions Per Patient		X	X	Patient Adoption
Integrated Referral Path		Digital CBTi Referrals Initiated as Consults in the EMR	Providers, Organization	Planned and Organized Prior to Active Implementation			X	Provider & Patient Adoption
Direct-to-patient Advertisement		Targeted Information and Invitation Letter Sent to At-Risk Patients	Patients	355 Total Letters in 3 Batches			X	Provider & Patient Adoption
Warm-Handoffs		Simultaneous Referral and Digital CBTi Education by Mental Health Providers Integrated in Primary Care	Providers, Staff, Patients	Planned and Organized Prior to Active Implementation			X	Provider & Patient Adoption

Note. Phase 2 coaching was provided by a primary care nurse, and Phase 3 coaching was provided by a peer support specialist. EMR  =  Electronic Medical Record; CBTi  =  Cognitive Behavioral Therapy for Insomnia.

In Phase 1, low-intensity implementation activities were used, consisting of (1) provider education, (2) point-of-care information, and (3) a basic process to navigate referred patients to the program. All providers in the five clinics received education, which focused on insomnia characteristics and health impact, CBTi efficacy, discussing digital CBTi with patients, and the digital CBTi referral/enrollment process. Training occurred via staff meetings, individual contacts, and emails. Point-of-care information consisted of pamphlets in all treatment rooms, which served as a prompt for providers to consider referral to SHUTi and as a patient information resource. Referral involved a provider contacting research staff by phone or messaging. Once referred, a research assistant met with referred patients to facilitate informed consent, which included brief education about the digital CBTi program, and complete baseline measures.

Phase 2 maintained the activities of Phase 1 and added a clinic-based coach (a primary care nurse as recommended by clinic leadership) to support digital CBTi use after low patient adoption was noted in phase 1. Coaching to support digital intervention engagement is a common practice in digital CBTi research trials (Schueller et al., 2017; Seyffert et al., 2016). The coach provided (1) an initial patient education session covering program access and content, which also addressed common technology literacy issues; (2) two follow-up contacts supporting motivation to engage, technical aspects of engagement, and questions about program content; (3) ad hoc technical support as requested by the patient. The coach was trained by the principal investigator over three 1-h sessions, which consisted of review of SHUTi content, common patient questions and answers, and role-playing coaching scenarios. Coaches already had training and experience in health education and motivational interviewing. Subsequently, each coach contact was reviewed during weekly meetings.

Phase 3 maintained the activities of Phase 2 but, upon recommendation by clinic leadership, changed the coach role to a peer specialist because of high workload demands on the primary care nurse. The coach was trained and supervised using the above process. In addition, primary care providers were invited to make direct, time-sensitive referrals, termed “warm-handoffs,” to providers working in the integrated mental health clinic who administered the initial digital CBTi education sessions. For warm-handoffs, the coach provided follow-up and ad hoc support contacts. In addition, direct-to-patient advertising via opt-in letters were sent to all (VA Connecticut Healthcare System) primary care patients identified on VA-developed clinical decision support dashboards for insomnia and sedative-hypnotic treatment. Finally, the referral process was integrated into the electronic medical record as a digital CBTi consult, replicating provider workflow for referrals to other services.

Outcomes

Implementation feasibility was determined by clinic leadership, in consultation with the research team and evaluation of outcomes, who decided if and how the program should be sustained following the trial. During the trial, implementation outcomes representing provider and patient adoption of digital CBTi and patient acceptability were measured, as well as patient clinical outcomes (Hermes et al., 2019b). Provider adoption was operationalized as the proportion of providers who made a referral among all providers in the five primary care clinics during the implementation phase. Patient adoption was operationalized as five stages representing progression from referral to program completion (Figure 1): (1) Referred - the proportion of patients referred to SHUTi among all unique patients receiving care in the five primary care clinics during each phase; (2) Educated - the proportion of those referred who had an initial contact with the coach or research staff for orientation and education about digital CBTi; (3) Enrolled - the proportion among those educated who agreed to enroll in the study; (4) Engaged - the proportion of those enrolled who completed at least one SHUTi module; and (5) Completed - the proportion of those who started the sixth and final SHUTi module. Additional implementation measures included referral characteristics and fidelity to coaching contact protocols (i.e., three phone contact attempts before discontinuation for both initial education and follow-up support sessions). Patient technology acceptability with respect to digital CBTi was measured at baseline and follow-up from twelve elements of the Unified Theory of Acceptance and Use of Technology (UTAUT) (Venkatesh et al., 2003). Common reasons for attrition from digital CBTi use were measured at follow-up (Hermes et al., 2016).

Figure 1.

Patient adoption of digital CBTi and trial consort diagram. Note. “Referred” is the number of referrals out of the total number of patients seeking primary care services in each phase. The denominator for the proportion given in the categories of “Educated” through “Completed” represent in the number in the preceding category.

Clinical outcomes were measured by self-report at enrollment and 10-weeks after enrollment. The Insomnia Severity Index (ISI), a widely used seven-question self-report measure of insomnia symptoms (score range 0–28), was the primary clinical outcome (Bastien et al., 2001; Morin et al., 2011). A score reduction of 6 or more was considered clinically meaningful and a follow-up score of less than 8 was considered (Yang et al., 2009). Secondary clinical outcomes included self-report of the number of sedative-hypnotic medication doses taken in the prior week (Stirratt et al., 2015) and the Beck Depression Inventory II (Beck et al., 1996). Patient socio-demographics, clinical characteristics, and self-reported internet use were measured at baseline. Subjects received a small monetary compensation for completing measures.

Analysis

Evaluation of implementation and clinical outcomes relied primarily on descriptive statistics. Inferential statistics were used to explore potential key associations. Given the pilot trial's location at a single site and small size, p-values should be interpreted as suggesting potential trends to explore in future research rather than identifying definitive statistical significance. Implementation outcomes were analyzed as proportions and compared across trial phases using chi square. The association of patient baseline characteristics with trial phases, levels of program engagement, and clinical outcomes were explored using chi square and Analysis of Variance. Analyses of clinical outcomes utilized the sample of all enrollees and are reported as mean summary scores. Change from baseline to follow-up in clinical outcomes were explored using paired T-Tests. Total ISI score at follow-up was explored in linear regression models across levels of digital CBTi adoption after enrollment, adjusting for baseline clinical scores and race/ethnicity. Analyses of sedative-hypnotic use and BDI score were completed in a similar manner. Additional exploratory analyses focused on the characteristics of referral, fidelity to the coaching plan, and reasons for declining enrollment and not completing the digital CBTi. Analyses were completed using SAS 9.4.

Results

Implementation Outcomes

Provider Adoption

A total of 66 providers made referrals, representing 48.9% of the 135 providers (60 primary care providers, 11 mental health in primary care providers, and 64 trainees) working at the five clinics over the three trial phases. Differences in the proportion of providers making referrals across the three phases did not vary meaningfully (Chi Square = 1.1, DF = 3, p = 0.583). A total of 32 (53.3%) of primary care, 5 (45.5%) mental health in primary care, and 29 (45.3%) trainee providers made referrals.

Patient Adoption

A total of 153 patients were referred to digital CBTi across the three phases, representing an average of 0.38% of the unique patients cared for in the primary care clinics over those periods (Phase 1 n = 12,658; Phase 2 n = 13,592; Phase 3 n = 13,572; Figure 1 and Table 2). Of all those referred, 132 (86.3%) participated in initial education about digital CBTi. Of those educated, 77 (58.3%) enrolled in the study. Of those enrolled, 45 (58.4%) engaged in the program, and among those engaged, 24 (53.3%) completed it. The mean number of modules completed among all subjects who enrolled was 2.4 (SD = 2.6) out of six. There were no differences in any level of patient adoption across the three trial phases.

Table 2.

Socio-demographic, clinical, health service Use, and internet Use characteristics of enrollees by phase.

	Intent to Treat n = 77		Phase 1 n = 27 (35.1%)		Phase 2 n = 20 (26.0%)		Phase 3 n = 30 (39.0%)
	n	(%)	n	(%)	n	(%)	n	(%)
Socio-demographic Characteristics
Age
< = 55	22	(28.6)	8	(36.4)	7	(31.8)	7	(31.8)
>55, < = 65	22	(28.6)	9	(40.9)	4	(40.9)	9	(40.9)
>65	33	(42.9)	10	(30.3)	9	(42.4)	14	(42.2)
Gender (Male)	66	(85.7)	23	(34.9)	19	(28.8)	24	(36.4)
Race/Ethnicity
White/Non-Hispanic	58	(75.3)	21	(36.2)	13	(22.4)	24	(41.4)
AA/Non-Hispanic	9	(11.7)	2	(22.2)	4	(44.4)	3	(33.3)
Hispanic	7	( 9.1)	3	(42.9)	2	(28.6)	2	(28.6)
Other	3	( 3.9)	1	(33.3)	1	(33.3)	1	(33.3)
Marital Status
Married	42	(54.6)	17	(40.5)	11	(26.2)	14	(33.3)
Not Married	34	(45.5)	10	(28.6)	9	(25.7)	16	(45.7)
Highest Education Level
Up to Some College	46	(59.7)	18	(39.1)	13	(28.3)	15	(32.6)
Bachelor's	19	(24.7)	4	(21.1)	6	(31.6)	9	(47.4)
Master's or above	12	(15.6)	5	(41.7)	1	( 8.3)	6	(50.0)
Employment
Full Time	27	(35.1)	11	(40.7)	5	(18.5)	11	(40.7)
Retired or Disability	37	(48.1)	7	(18.9)	13	(35.1)	17	(46.0)
Unemployed or Part time	13	(16.9)	9	(69.2)	2	(15.4)	2	(15.4)
Travel Time to Primary Care
< = 15min	22	(28.6)	4	(19.1)	5	(23.8)	12	(57.1)
>15, < = 30min	33	(42.9)	13	(39.4)	11	(33.3)	9	(27.3)
>30min	22	(28.6)	10	(45.5)	4	(18.2)	8	(36.4)
Clinical History
Total Number of Diagnoses
<5	17	(22.1)	4	(23.5)	6	(35.3)	7	(41.2)
> = 5, <10	21	(27.3)	10	(47.6)	4	(19.1)	7	(33.3)
> = 10	39	(50.7)	13	(33.3)	10	(25.6)	16	(41.0)
Any Mental Health Diagnosis	35	(45.5)	12	(34.3)	8	(22.9)	15	(42.9)
Depression Diagnosis	15	(19.5)	6	(40.0)	2	(13.3)	7	(46.7)
Posttraumatic Stress Diagnosis	10	(13.0)	3	(30.0)	3	(30.0)	4	(40.0)
Substance Use Diagnosis	9	(11.7)	1	(11.1)	3	(33.3)	5	(55.6)
Insomnia Diagnosis	21	(27.3)	8	(38.1)	6	(28.6)	7	(33.3)
Respiratory Diagnosis	16	(20.8)	5	(31.3)	2	(12.5)	9	(56.3)
Circulatory Diagnosis	36	(46.8)	11	(30.6)	8	(22.2)	17	(47.2)
Endocrine Diagnosis	15	(19.5)	5	(33.3)	5	(33.3)	5	(33.3)
Musculoskeletal Diagnosis	49	(63.6)	21	(42.9)	12	(24.5)	16	(32.7)
Gastrointestinal Diagnosis	31	(40.3)	15	(48.4)	7	(22.6)	9	(29.0)
Neurological Diagnosis	24	(31.2)	10	(41.7)	6	(25.0)	8	(33.3)
Sleep Apnea Diagnosis	7	( 9.1)	4	(57.1)	1	(14.3)	2	(28.6)
Other Medical Diagnosis	56	(72.7)	22	(39.3)	15	(26.8)	19	(33.9)
Number of Prescriptions
<5	15	(19.5)	5	(33.3)	4	(26.7)	6	(40.0)
> = 5, <10	27	(35.1)	12	(44.4)	6	(22.2)	9	(33.3)
> = 10	35	(45.5)	10	(28.6)	10	(28.6)	15	(42.9)
Any Sedative-Hypnotic Prescription	22	(28.6)	6	(27.3)	4	(18.2)	12	(54.6)
Any Past Psychotherapy	24	(31.6)	12	(50.0)	3	(12.5)	9	(37.5)
Any Past CBT for insomnia	8	(10.4)	2	(25.0)	2	(25.0)	4	(50.0)
Internet Use
How often do you use the Internet?
At least once per day	72	(93.5)	25	(34.7)	17	(23.6)	30	(41.7)
At least once/week, not once a day	4	( 5.2)	1	(25.0)	3	(75.0)	0	( 0.0)
Les often	1	( 1.3)	1	(100)	0	(0.0)	0	( 0.0)
Where do access the Internet most?
Home	67	(87.0)	22	(32.8)	17	(25.4)	28	(36.4)
Public Places	1	( 1.3)	0	( 0.0)	1	(100)	0	( 0.0)
Work	9	(11.7)	5	(55.6)	2	(22.2)	2	(22.2)

Note. No proportion differed by phase in chi square tests at a significance level of p < 0.05. CBT  =  Cognitive Behavioral Therapy, AA = african american.

In examination of factors associated with referral to digital CBTi, the following results are noted. In 21 referrals (13.7%), an information pamphlet was used in the referral process. While pamphlet use did not differ by trial phase, a higher proportion of all pamphlet-associated referrals went on to engage in the program (Chi Square = 11.1, DF = 4, p = 0.025). Only three referrals stemming from direct-to-patient advertising in Phase 3 occurred (0.8% of 355 letters mailed) and were not associated with subsequent levels of patient adoption. When the period between provider referral and initial patient education was short (i.e., when coach education sessions happened at the time of referral [n = 11] and warm handoffs from primary care to mental health providers happened in phase 3 [n = 4]), a higher proportion of enrollments occurred (Chi Square  =  4.7, DF = 1, p = 0.03 and Chi Square = 4.3, DF = 1, p = 0.038), but not subsequent digital CBTi engagement or completion.

Patient Acceptability of Digital CBTi

All self-report items pertaining to the acceptance and use of technology were either affirmed or marked as neutral by 90% or more of enrollees at baseline and follow-up. These proportions did not differ meaningfully by baseline or follow-up assessment, trial phase, or level of digital CBTi adoption after enrollment. Patients who declined enrollment indicated that competing demands, absence of insomnia, deficits in technology access/literacy, and preference to focus on obstructive sleep apnea treatment prohibited involvement (Figure 1). The most frequently reason cited for not completing digital CBTi after enrollment was competing demands for time, endorsed by 14 enrollees (26.9%, Table 3).

Table 3.

Self-Reported reasons for Not completing SHUTi after enrollment.

	Strongly or Somewhat Disagree		Neutral		Strongly or Somewhat Agree
	n	(%)	n	(%)	n	(%)
I did not feel I would receive any more benefit	30	(57.7)	11	(21.2)	11	(21.2)
Using the website was too difficult	35	(67.3)	6	(11.5)	11	(21.2)
I had a difficult time understanding the content of the program	36	(69.2)	7	(13.5)	9	(17.3)
The homework was too difficult	38	(73.1)	12	(23.1)	2	( 3.9)
I did not have enough time to complete the program or homework	28	(53.9)	10	(19.2)	14	(26.9)
Finding a computer or other device with which to access the internet was too difficult	42	(80.8)	4	( 7.7)	6	(11.5)

Note. Among n = 52, 98.1% of enrollees who did not complete SHUTi after enrollment.

Coaching Fidelity

In phases 2 and 3, the digital coaches made 783 outreach efforts and 259 successful contacts or a mean of 2.4 contacts (SD = 2.1) per referred patient, which did not differ by phase. Fidelity to the coach's initial education contact protocol was lower for the nurse coach in phase 2 compared to the peer support specialist coach in phase 3 (65.2% vs. 98.3%; Chi Square  =  21.2, DF = 1, P < 0.001). Differences in fidelity to the coach's protocol for support contacts after initial education did not differ meaningfully between phase 2 and 3 (70.0% vs. 90%; Chi Square = 3.3, DF = 1, P < 0.071).

Clinical Outcomes

Patient Characteristics

The 77 enrollees (Table 2) were older (mean of 67 years; SD = 16.9; range = 24 to 91), majority male (n = 66, 85.7%), White/non-Hispanic (n = 58, 75.3%), and married (n = 42, 54.3%). Many had complicated medical histories: 39 (50.7%) carried 10 or more diagnoses and 35 (45.5%) were prescribed 10 or more medications, while 35 (45.5%) carried a mental health diagnosis. At baseline, enrollees reported insomnia levels of moderate severity (mean ISI = 16.4, SD = 5.7), depression levels between minimal and mild (mean BDI = 13.1, SD = 10.6), and almost daily sedative-hypnotic use with an average of 6.2 (SD = 6.7) doses in the prior week (Table 4). As for the enrollee's technology use (Table 2), 72 (93.5%) stated they access the internet at least once per day, primarily from home (n = 67, 87.0%). Baseline demographic, clinical, or technology use characteristics did not differ by phase, nor were these variables associated with a subsequent level of patient adoption using a Bonferroni correction accounting for 15 independent baseline variables (p < 0.003; Table 5). No adverse events were reported among enrollees.

Table 4.

Clinical outcomes for total sample and by engagement level after enrollment.

Clinical Outcomes Measures by Level of Engagement	Baseline		10-Weeks		Paired T-Test			Change
	mean	(SD)	mean	(SD)	T	DF	P	mean	(SD)
Insomnia Severity Index	16.4	( 5.7)	12.2	( 6.9)	6.4	69	<0.001	−4.3	(5.6)
Enrolled but did not Engage	16.2	( 6.1)	13.4	( 7.8)	2.2	25	0.041	−2.9	(6.7)
Engaged but did not Complete	18.2	( 6.3)	15.5	( 6.4)	3.1	19	0.006	−3.0	(4.3)
Completed	15.0	( 4.0)	8.1	( 3.9)	8.1	23	<0.001	−6.9	(4.2)
Beck Depression Inventory	13.7	(10.6)	10.2	( 9.8)	4.7	69	<0.001	−3.1	(5.5)
Enrolled but did not Engage	15.6	(11.5)	12.4	(10.9)	2.6	25	0.016	−2.7	(5.3)
Engaged but did not Complete	14.5	(11.5)	12.9	(10.9)	1.4	19	0.173	−1.6	(5.1)
Completed	10.4	( 8.0)	5.8	( 5.5)	3.9	23	<0.001	−4.7	(5.9)
Past Week Sedative-Hypnotic Use	6.2	( 6.7)	4.3	( 5.2)	2.4	69	0.020	−2.2	(6.6)
Enrolled but did not Engage	6.3	( 6.7)	5.4	( 5.3)	0.5	25	0.595	−2.7	(5.3)
Engaged but did not Complete	5.6	( 6.4)	4.7	( 5.9)	0.9	19	0.374	−1.6	(5.1)
Completed	6.5	( 7.1)	2.8	( 4.0)	2.5	23	0.019	−4.7	(5.9)

Table 5.

Enrollee characteristics by level of engagement in digital CBTi.

	Completed n = 24 (31.2%)		Engaged, but did not Complete n = 21 (27.3%)		Enrolled, but Did not Engage n = 32 (41.6%)
	n	(%)	n	(%)	n	(%)
Socio-demographic Characteristics
Age
< = 55	4	(16.7)	4	(19.1)	14	(43.8)
>55, < = 65	8	(33.3)	8	(38.1)	6	(18.8)
>65	12	(50.0)	9	(42.9)	12	(37.5)
Gender (Male)	20	(83.3)	18	(85.7)	28	(87.5)
Race/Ethnicity
White/Non-Hispanic	23	(95.8)	15	(71.4)	20	(62.5)
AA/Non-Hispanic	1	( 4.2)	2	( 9.5)	6	(18.8)
Hispanic	0	( 0.0)	3	(14.3)	4	(12.5)
Other	0	( 0.0)	1	( 4.8)	2	( 6.3)
Marital Status
Married	18	(56.3)	11	(52.4)	13	(54.2)
Not Married	14	(43.8)	10	(47.6)	11	(45.8)
Highest Education Level
Up to Some College	20	(62.5)	15	(71.4)	20	(62.5)
Bachelor's	8	(25.0)	3	(14.3)	8	(25.0)
Master's or above	4	(12.5)	3	(14.3)	4	(12.5)
Employment
Full Time	8	(33.3)	8	(38.1)	11	(34.4)
Retired or Disability	12	(50.0)	10	(47.6)	15	(46.9)
Unemployed or Part time	4	(16.7)	3	(14.3)	6	(18.8)
Travel Time to Primary Care
< = 15min	6	(25.0)	8	(38.1)	8	(25.0)
>15, < = 30min	13	(40.6)	7	(33.3)	13	(40.6)
>30min	5	(34.4)	6	(28.6)	11	(34.4)
Clinical History
Total Number of Diagnoses
<5	8	(33.3)	3	(14.3)	6	(18.8)
> = 5, <10	5	(20.8)	6	(28.6)	10	(31.3)
> = 10	11	(45.8)	12	(57.1)	16	(50.0)
Any Mental Health Diagnosis	7	(29.2)	9	(42.9)	19	(59.4)
Depression Diagnosis	5	(20.8)	2	( 9.5)	8	(25.0)
Posttraumatic Stress Diagnosis	1	( 4.2)	2	( 9.5)	7	(21.9)
Substance Use Diagnosis	0	( 0.0)	3	(14.3)	6	(18.8)
Insomnia Diagnosis	5	(20.8)	8	(38.1)	8	(25.0)
Respiratory Diagnosis	5	(20.8)	6	(28.6)	5	(15.6)
Circulatory Diagnosis	10	(41.7)	12	(57.1)	14	(43.8)
Endocrine Diagnosis	5	(20.8)	7	(33.3)	3	( 9.4)
Musculoskeletal Diagnosis	13	(54.2)	16	(76.2)	20	(62.5)
Gastrointestinal Diagnosis	10	(41.7)	6	(28.6)	15	(46.9)
Neurological Diagnosis	7	(29.2)	7	(33.3)	10	(31.3)
Sleep Apnea Diagnosis	2	( 8.3)	3	(14.3)	2	( 6.3)
Other Medical Diagnosis	15	(62.5)	17	(81.0)	24	(75.0)
Number of Prescriptions
<5	6	(25.0)	2	( 9.5)	7	(21.9)
> = 5, <10	10	(41.7)	3	(14.3)	14	(43.8)
> = 10	8	(33.3)	16	(76.2)	11	(34.4)
Any Sedative-Hypnotic Prescription	6	(28.1)	7	(33.3)	9	(25.0)
Any Past Psychotherapy	9	(37.5)	8	(40.0)	7	(21.9)
Any Past CBT for insomnia	2	(8.3)	3	(14.3)	3	(9.4)
Internet Use
How often do you use the Internet?
At least once per day	24	(100)	20	(95.2)	28	(87.5)
At least once/week, not once a day	0	( 0.0)	1	( 4.8)	3	( 9.4)
Les often	0	( 0.0)	0	( 0.0)	1	( 3.1)
Where do access the Internet most?	18	(75.0)	18	(85.7)	31	(96.9)
Home	0	( 0.0)	0	( 0.0)	1	( 3.1)
Public Places	6	(25.0)	3	(14.3)	0	( 0.0)
Work	24	(100)	20	(95.2)	28	(87.5)

Note. No query was associated with the subsequent level of reach category using a Bonferroni correction accounting for 18 independent baseline variables (p < 0.003). CBT  =  Cognitive Behavioral Therapy.

Clinical Outcomes

Follow-up clinical outcomes were available for 70 (90.9%) of the 77 enrollees (Table 4). All clinical outcomes displayed a decrease from baseline to follow-up: the ISI by a mean of 4.3 (SD = 5.6, t = 6.41, p < 0.001); the BDI by 3.1 (SD = 5.5, t = 4.65, p < 0.001), and the mean number of sedative-hypnotic doses by 2.2 per week (SD = 6.6, t = 2.39, p = 0.020). There were no differences in follow-up scores or change in scores by trial phase for any clinical outcome. For all enrollees, 14 (20.0%) demonstrated clinically meaningful change (treatment responders) and 13 (18.6%) reached clinical remission on total ISI score. In exploratory regression models evaluating clinical outcomes by level of patient engagement, those who completed SHUTi showed more clinical improvement at follow-up compared to those who did not engage or engaged but did not complete SHUTi: 5.1 fewer points on the ISI (F = 17.8, DF = 6, p < 0.0001), 3.1 fewer points on the BDI (F = 69.7, DF = 3, p < 0.0001), and 2.8 fewer weekly sedative hypnotic doses (F = 7.1, DF = 3, p = 0.0003). This improvement in the primary clinical outcome, insomnia severity, suggested a potentially large effect size (Cohen's D = 0.9).

Discussion

This study demonstrated that the implementation of digital CBTi is feasible in a VA primary care clinic. A total of 50% of primary care providers made 153 referrals, resulting in 45 patients receiving at least one session and 24 completing the program. Moreover, those enrolled in digital CBTi found it to be acceptable and attained clinically significant improvements in insomnia levels and reduced sedative-hypnotic use, especially among those who completed the entire program. This data was used by leadership to determine that digital CBTi should be sustained as an intervention in the clinic following the trial. However, iterative modifications to the low-intensity implementation strategy, which added coaching support as well as enhancements to the referral process and targeted advertising over the trial's course, did not improve implementation outcomes. Key trial findings and their implications for future digital CBTi implementation efforts are presented in Table 6.

Table 6.

Impact on future digital intervention implementation efforts: lessons learned from pilot trial characteristics and findings.

Topic	Pilot Trial Characteristics and Findings	Implications for Future Implementation Efforts
Feasibility, Acceptability, and Clinical Outcomes	Digital CBTi Implementation is Feasible in VA Primary Care	Digital CBTi Implementation is Ready for Scale-Up to Multiple Sites within VA. Pilot Trial Findings can be Confirmed and Expanded in Randomized, Controlled, and Adequately Powered Hybrid Implementation and Effectiveness Trials
	Digital CBTi was Acceptable to VA Service Users who Enrolled
	Digital CBTi Enrollment Associated with Improved Clinical Outcomes
Patient Adoption	29% of Referred Patients Engaged in and 16% Completed Digital CBTi	Digital CBTi Implementation Research Should Rigorously Explore Factors associated with and Methods to Increase Patient Adoption, while also Comparing Adoption Levels to Traditional Provider-Delivered forms of CBTi
	No Participant Characteristics Measured were Associated with Digital CBTi Adoption
Sample	Enrollees were on Average 63 Years of Age with Complex Medical Histories	Implementing Digital CBTi in Healthcare Systems, such as VA, May Recruit Users who are Different from Those Participating in Digital Intervention Research Trials
Provider Workflow	A Digital CBTi Referral Process that Utilized Current Provider Workflows May have Induced More Referrals	Fully Integrating Digital CBTi in Current Provider Workflows May Improve Provider Adoption
Study Design	Subjects May Have Received Adequate Digital CBTi Support from Research Personnel in Phase 1	Research Processes such as Consent, Enrollment, Compensation, and Data Acquisition May Affect Subject Adoption, a Key Outcome in Implementation Research
	Implementation Strategy Activities were Modified Iteratively Over Time	Future Research Should Definitively Assess the Effect of Individual Implementation Strategy Activities Using Randomization and Controls
	Single Site Pilot Trial with Small Sample Size	Future Research Trials May Require Large Sample Sizes and Multiple Sites to Assess the Effect of Individual Implementation Strategy Activities
Digital Intervention Coaching	Digital CBTi Coaching was Integrated into the Duties of Local Primary Care Personnel, who Experienced Competing Demands for Time	More Research is Needed on (1) which Patients Benefit Most from Coach Contact, (2) who within Different Treatment Contexts is Best Positioned to Provide Coaching, as well as (3) what Schedule, Content, and Model for Digital Coaching Contacts is Most Effective
	The Technology Literacy of Subjects Played an Unclear Role in Patient Digital CBTi Adoption	Future Research Should Explore the Balance of Technology Literacy Support within Digital Coaching when Alternative, Provider-led forms of Care are also Available

Several possible explanations exist for the lack of improvement in provider and patient adoption with iterative changes to the implementation strategy. First, as a relatively small pilot study at a single site, the objectives of which were to evaluate feasibility and explore implementation, this trial was underpowered (Table 6). In a large and appropriately powered multi-site trial, the higher provider referral rate observed in Phase 3 (44%), likely induced by better integration of digital CBTi into provider referral workflow (Graham et al., 2020a), may have demonstrated significant differences with referral rates in Phase 1 (37%) and Phase 2 (34%).

Second, the intent of the Phase 1 was to start with a low-intensity implementation strategy to promote essential provider and patient awareness of digital CBTi and provide a basic process for patient referral. Formal coaching was not included in Phase 1 as this increases the resources needed to implement digital CBTi. However, as part of the consent and enrollment process, the research assistant provided basic information about the digital CBTi and education about how to use the program in Phase 1. The research assistant also served as a source of technical assistance for patients who were having difficulty engaging in the program during this phase. In retrospect, these activities were similar to the initial education and follow-up coaching sessions added in Phases 2 and 3 to boost patient engagement and completion, but likely obfuscated finding an effect of patient adoption. More research is needed to clearly define the effects of coaching, which patients benefit most from coach contact, and who, within different VA treatment contexts, is best positioned to serve in a coach role (Table 6).

Third, the study design may have contributed to the null implementation findings. While a cohort design at a single site allows for iterative strategy changes over time based on stakeholder feedback and observed outcomes, it does not allow for random assignment of different implementation activities or to control for time effects. The first phase may have involved both providers and patients who were early adopters and more likely to promote and adopt digital CBTi (Valente & Rogers, 1995). In subsequent phases, the pool of patients who were most in need and most appropriate for digital CBTi may have been reduced, leading to relatively lower adoption at all levels (Table 6). The extent to which coaching, referral enhancements, and targeted advertising may have been evaluated under more challenging implementation circumstances in phases 2 and 3 is not known.

Overall, the proportion of patients who engaged in digital CBTi (29.4%) and completed the program (15.7%) was low. These proportions are slightly lower than those observed during the implementation of digital CBTi in a VA outpatient substance use disorder treatment clinic where engagement and completion were 35% and 22%, respectively (Hermes and Rosenheck, 2015). They are also slightly lower than those observed in a recent digital intervention implementation effort in the Kaiser Health System, where 58% of patients referred to a digital intervention enrolled and 27% used it more than three times (Mordecai et al., 2021).

While technology literacy and access are often hypothesized to be associated with digital intervention adoption, the association of these factors with patient adoption of digital CBTi in this study is unclear. Although technology access and literacy issues were voiced as reasons for not enrolling in and not finishing digital CBTi (11% in Figure 1 and 21% in Table 2, respectively), self-reported internet use at baseline was not associated with subsequent patient adoption after enrollment. In addition, while the frequency of Internet use differed dramatically between this VA digital CBTi implementation trial (94% reported daily internet use) and the previous trial (39% reported daily internet use), there was more digital CBTi adoption in the prior trial (Hermes and Rosenheck, 2015). While differences between the two trials may be related to a slightly lower insomnia severity in the current primary care sample, it also suggests that as yet undetermined factors may best predict digital CBTi engagement and completion among Veterans. More research is needed into how to best mitigate the factors associated with low digital intervention adoption (Greenhalgh et al., 2017), including the level of technology literacy and access support including digital coaching interventions, especially when traditional provider led CBTi is also a treatment option within the health care system (Table 6).

In addition, the patient adoption levels observed here are considerably lower than those observed in research trials (Seyffert et al., 2016). In prior trials, digital CBTi programs have been implemented outside of traditional healthcare delivery contexts and recruitment is accomplished through community and internet advertising. Moreover, this study did not compare patient adoption levels for digital CBTi to those for traditional provider led CBTi at the same facility (Karlin et al., 2013). The levels of patient engagement observed in this trial may be par for the course outside of highly controlled research trials. More research in this area is needed (Table 6).

This pilot trial did not demonstrate a clear effect for the addition of coaching on patient adoption of digital CBTi for the reasons discussed above. However, coaching for digital interventions is widely supported in the literature (Lattie et al., 2020; Schueller & Torous, 2020), to the extent that the most important current question may not be “Should digital coaching be used?” but “What coaching content and models of coaching delivery are most feasible and effective?” (Lattie et al., 2021). For instance, two different deployment models for coaching may be considered: a “centralized” model where coaches spend most of their effort making virtual coaching contacts from a centralized location and are provided more extensive training; and a “diffused” model where coaching activities are only one part of an individual's duties and multiple coaches are trained, similar to the model used in this trial. Research into the relative effectiveness of these two models would more clearly guide the field in this area.

Additionally, this study found that no demographic, clinical, or technology use characteristics were associated with subsequent levels of patient digital CBTi adoption Table 5). The most common identifiable reason given by patients for either not enrolling, or not completing after enrollment was competing demands for time, similar to findings in our prior implementation of digital CBTi in VA care (Table 6; Hermes et al., 2016). Again, more research exploring the characteristics and attitudes of patients who do not adopt digital interventions is needed to develop education and recruitment strategies that better target these interventions and support their full adoption.

Notwithstanding these limitations, this study is one of the few to show improved clinical outcomes when digital CBTi is implemented in a traditional healthcare delivery context (Seyffert et al., 2016; Zachariae et al., 2016). Notably, the sample of participants was fundamentally different from those of previous digital CBTi trials, consisting of a high proportion of older men with mental health and medical comorbidities. In prior trials, such co-morbidities were relatively rare and participants were younger and potentially more comfortable with technology (Christensen et al., 2016). These findings show that digital CBTi is a promising intervention for VA users and suggests the need for continued research to maximize adoption of these effective interventions in regular practice (Table 6). Additionally, a qualitative understanding of implementation needs was identified in this trial through ad hoc feedback from primary care providers, administrative staff, and clinic leadership. This informal process was used because clinic leadership discouraged burdening busy primary care providers with more formal formative evaluation procedures. A systemic formative evaluation process in primary care, such as those suggested by Rubenstein et al. (2014), although requiring more resources, may have more clearly identified key barriers and facilitators to implementation and improved implementation strategy development as well as execution.

In conclusion, this study demonstrated that it is feasible to implement an evidence-based digital intervention delivering CBT for insomnia in VA primary care. The study showed that patients who engage in and, more importantly complete the digital CBTi, will likely have improved insomnia outcomes. While iterative changes to the implementation strategy did not improve implementation outcomes, a low-intensity strategy that included provider education, point-of-care information resources, and a referral mechanism integrated into healthcare workflow did facilitate digital CBTi use. The added benefit and specific aspects of coaching for some digital interventions remains unclear and requires further investigation. The clear specification of implementation strategy components and the implications from this pilot trial's findings (Table 6) should influence future digital intervention implementation research.