Utilization, Patient Characteristics, and Longitudinal Improvements among Patients from a Provincially Funded Transdiagnostic Internet-delivered Cognitive Behavioural Therapy Program: Observational Study of Trends over 6 Years

Abstract

Objective:

There is strong evidence supporting internet-delivered cognitive behaviour therapy (iCBT) and consequently growing demand for iCBT in Canada. Transdiagnostic iCBT that addresses both depression and anxiety is particularly promising as it represents an efficient method of delivering iCBT in routine care. The Online Therapy Unit, funded by the Saskatchewan government, has been offering transdiagnostic iCBT for depression and anxiety since 2013. In this article, to broadly inform implementation efforts, we examined trends in utilization, patient characteristics, and longitudinal improvements for patients receiving transdiagnostic iCBT over 6 years.

Methods:

Patients who completed telephone screening between November 2013 and December 2019 were included in this observational study. Patients provided demographics and mental health history at screening and completed measures at pre-treatment, post-treatment and at 3- to 4-month follow-up. Treatment engagement and satisfaction were assessed.

Results:

A total of 5,321 telephone screenings were completed and 4,283 of patients were accepted for treatment over the 6-year period (80.5% acceptance). The most common reason for referral to another service was high suicide risk/severe symptoms (47.1%). Examination of trends showed growing use of transdiagnostic iCBT over time (37% increase per year). There was remarkable stability in patient characteristics across years. Most patients were concurrently using medication (57.3%) with 11.9% reporting using iCBT while on a waiting list for face-to-face treatment highlighting the importance of integrating iCBT with other services. Consistent across years, large improvements in depression and anxiety symptoms were found and maintained at 3- to 4-month follow-up. There was strong patient engagement with iCBT and positive ratings of treatment experiences.

Conclusions:

As there is growing interest in iCBT in Canada, this large observational study provides valuable information for those implementing iCBT in terms of likely user characteristics, patterns of use, and improvements. This information has potential to assist with resource allocation and planning in Canada and elsewhere.

Introduction

Depression and anxiety contribute significantly to the global disease burden, ¹ but many Canadians do not receive treatment for their concerns. ² Individuals face both structural (e.g., location) and attitudinal (e.g., stigma) barriers to treatment. ³ Public health crises, like COVID-19, highlight the need for digital mental health treatments. ⁴ Consequently, the Canadian government announced “an investment of $240.5 million to develop, expand, and launch virtual care and mental health tools to support Canadians.” ⁵

Within the digital mental health field, there is enormous evidence supporting both diagnostic-specific and transdiagnostic (concurrent treatment of depression and anxiety) therapist-assisted internet-delivered cognitive behaviour therapy (iCBT). ^{6 –11} Although heterogeneity among iCBT programs exists, in routine practice, it is common for programs to begin with screening and for treatment materials to be presented over several months using online weekly lessons. In routine practice, it is also common for trained providers (e.g., therapist, psychologist) to offer support via weekly asynchronous emails and or telephone calls ¹⁰ and for programs to include persuasive design features such as automated emails. ¹²

While iCBT research emerged over 20 years ago, ¹³ within Canada, there has been limited access to publicly funded therapist-assisted iCBT programs coordinated with healthcare services. ¹⁴ One question that emerges is what can be expected longer-term when therapist-assisted iCBT is routinely and freely available.

One example of a well-researched publicly funded iCBT program in Canada is the Online Therapy Unit in Saskatchewan. The unit screens patients for, and delivers, an 8-week transdiagnostic iCBT program called the Wellbeing Course that was originally, developed, validated, and then implemented by MindSpot, an Australian digital mental health service. ¹¹ The transdiagnostic program creates efficiency in implementing iCBT (e.g., reduces time to train therapists in and to allocate patients to diagnostic-specific programs) and has been integrated into Saskatchewan mental health care by encouraging physicians and clinicians to direct patients to this service when this meets patients’ needs or when patients are on waiting lists. Furthermore, community mental health clinics have designated clinicians who deliver iCBT via the unit platform.

The Online Therapy Unit also has a research arm. Past research on the Wellbeing Course has shown outcomes are strong whether delivered by therapists who specialise in iCBT or by therapists who primarily deliver face-to-face treatment. ^15,16 Furthermore, outcomes are comparable with varying support (i.e., once-weekly support compares to optional support, ^17,18 twice-weekly support, ¹⁹ and one-business-day support ²⁰ ). Still, other trials have studied therapist behaviours in iCBT ^{21 –23} and outcomes of iCBT for alcohol misuse, ²⁴ for various chronic health conditions, ^{25 –30} and for specific populations like postpartum depression/anxiety ³¹ or university students. ³²

With the growing imperative to offer iCBT in Canada, the purpose of this article is to examine patient characteristics as well as longitudinal improvements among patients screened for and offered transdiagnostic iCBT via the Online Therapy Unit over ∼6 years; there is no other such report of iCBT in routine care in Canada and to our knowledge only one other similar report from Australia, ³³ which deserves replication. In this 7-year Australian study, utilization increased in the first 3 years and subsequently stabilized, with a higher demand for assessment services than iCBT. iCBT effects were stable across years. Over time, there was an increase in young adult users, Aboriginal or Torres Strait Islander, and individuals with anxiety.

As has been noted, “evaluation of trends over time in how mental health services are used can help guide decisions about resource allocation” ³⁴ and program planning. ³³ Specifically, monitoring trends helps identify user changes and gaps in care and informs new programming or care models (e.g., growth in younger users could suggest offering texting rather than telephone support). Examining trends also adds to evidence regarding iCBT in routine care as there are few reports of iCBT outcomes sustained over longer periods after initial implementation efforts conclude. ³⁵ Research questions explored in this study were as follows: What growth will be observed in iCBT over time? Will users of iCBT change? Will there be stability in longitudinal symptom improvements over 6 years? Drawing on findings from Australia, ³³ it was hypothesized there would be growth in iCBT over time and effects would be consistently large over the 6-year period. Hypotheses regarding changes in users of iCBT were exploratory.

Methods

Design and Ethics

Patients who requested iCBT between November 2013 and December 2019 are included in this uncontrolled observational cohort analysis. Ethics approval was obtained from the University of Regina Research Ethics Board. The article combines data across and between registered trials that examined the Wellbeing Course outcomes across settings (ISRCTN42729166), with motivational interviewing as a pre-treatment (NCT03684434) or with varying levels of support (ISRCTN14230906; NCT03304392; NCT03957330).

Participants

Participants learn about iCBT through various sources (which have remained consistent) including medical professionals (55.0%; n = 2,928), mental health professionals (16.0%; n = 851), word of mouth (13.0%; n = 689), online sources (6.9%; n = 367), printed materials (2.2%; n = 115), media (2.5%, n = 132), insurance providers (0.3%, n = 14), or non-specified means (4.0%; n = 214). Consistent across years, to be eligible for iCBT, patients have to identify as (1) ≥18 years of age, (2) Saskatchewan residents, (3) having >5 Patient Health Questionnaire 9-item (PHQ-9) ³⁶ and/or >5 Generalized Anxiety Disorder Questionnaire 7-item (GAD-7) ³⁷ , (4) having Internet access and computer comfort, and (5) a healthcare provider emergency contact. Individuals meeting any of the following are excluded: (1) high risk of suicide, past year suicide attempt, or mental health hospitalization; (2) unmanaged psychosis, mania, and or drug/alcohol problems; (3) > twice-monthly face-to-face therapy; (4) not in Saskatchewan for 8-week treatment; (5) seeking help for different problem; or (6) not interested in iCBT. As individuals with subthreshold symptoms benefit from iCBT, diagnosis is not required for iCBT. ³⁸

Online and Telephone Screening

Prospective participants complete an online screening questionnaire capturing demographics and questionnaires listed below. Thereafter, participants complete a telephone interview further assessing inclusion/exclusion criteria. Individuals ineligible for iCBT are referred to alternative resources. Eligible participants provide verbal treatment consent over the telephone followed by digital consent on the platform.

Intervention

All patients received the transdiagnostic iCBT Wellbeing Course (see Titov et al. for details ¹¹ ), consisting of 5 lessons targeting anxiety and depression symptoms, released gradually over 8 weeks. Lessons consist of educational content, patient stories, and homework. Core strategies include monitoring physical, behavioural, and cognitive symptoms; thought challenging; controlled breathing; behavioural activation; graduated exposure; and relapse prevention. Additional resources (i.e., assertiveness, beliefs, communication, managing emergencies, mental skills, panic, post-traumatic stress, sleep, and structured problem solving/worry time) are also openly accessible throughout treatment. ¹¹

Therapists

The Online Therapy Unit trained all therapists (n = 132) who have a primary affiliation with the unit (n = 46) or a government-funded Saskatchewan mental health clinic (n = 86 therapists from 11 clinics). Therapists have backgrounds in social work (n = 80), psychology (n = 36), addictions (n = 9), or nursing (n = 7) and are either registered professionals (n = 96) or supervised graduate students (n = 36). On average, therapists included in this dataset treated 31 patients (SD = 89.23; range = 1 to 626). In March 2018, to improve coordination of iCBT, the Unit increased the numbers of therapists working half-to full-time and reduced the numbers of therapists with small caseloads. ²²

Therapist Support

Therapists contact clients on a designated day once-weekly for 8 weeks, although in some trials the amount of contact has varied (e.g., optional, twice-weekly). ^{17 –20} Most contact includes asynchronous secure emails, with therapists spending 15 to 20 minutes composing messages after reviewing patient iCBT progress, symptom measures, and messages. Therapists also phone patients when there is a large symptom increase, messages suggest suicide risk, patients have not logged in for a week, and or a phone call would more easily address patient concerns. Across trials, therapists follow a treatment manual; regular audit and feedback is used to ensure compliance with procedures. ²⁴

Measures

All measures are administered pre-treatment, at 8 weeks, and 3 to 4 months post-treatment. There has been some minor variability in measures administered resulting in variable sample size for some measures.

Primary Follow-up Measures

PHQ-9

The PHQ-9 is a 9-item questionnaire assessing depression over the past 2 weeks (range 0 to 27). ³⁶ Scoring ≥10 suggests a likely depression diagnosis. ³⁸

GAD-7

The GAD-7 is a 7-item questionnaire measuring anxiety over the past 2 weeks (range 0 to 21). ³⁷ Scoring ≥8 suggests a likely generalized anxiety disorder diagnosis. ³³

Secondary Follow-up Measures

Panic Disorder Severity Scale Self-report (PDSS-SR)

The PDSS-SR is 7-item panic disorder questionnaire (ranges 0 to 28). ³⁹ Scoring ≥8 suggests a likely panic disorder diagnosis. ⁴⁰

Social Interaction Anxiety Scale and Social Phobia Scale (SIAS-6/SPS-6)

The SIAS-6/SPS-6 are 6-item social anxiety measures (ranges 0 to 48). ⁴¹ SIAS-6 ≥7 and SPS-6 ≥ 2 suggest a likely social anxiety disorder diagnosis. ⁴¹

Kessler Distress Scale (K10)

The K10 includes10 items of psychological distress (ranges 10 to 50). ⁴² Score > 21 suggests significant distress. ³³

Sheehan Disability Scale (SDS)

The SDS includes 3 items measuring disruption to work/school, social life, and family life ⁴³ creating a 0 (unimpaired) to 30 (highly impaired) score.

Working Alliance Inventory–Short Revised (WAI-SR)

The 12-item WAI-SR, administered at 8 weeks, resulted in a total working alliance score (range 15 to 60), and task, goal, and bond sub-scores (range 5 to 20). ⁴⁴

Healthcare use

From October 2017 to September 2018, at pre-treatment and 4-month follow-up, patients completed 3-month healthcare use questions (i.e., physician, psychiatrist, mental health provider, ER/hospital admission and other medical/health provider).

Treatment Satisfaction

At 8 weeks post-treatment, patients answered yes/no items about whether they would recommend the treatment to a friend, the course was worth their time, and they experienced negative treatment effects. Patients also rated satisfaction (1 to 5) with the course, materials, frequency and quality of therapist contact, confidence to manage symptoms, and motivation to seek future treatment.

Engagement

iCBT engagement was assessed by examining number of patient messages, therapist messages, phone calls, and lessons accessed.

Statistical Analyses

Data were analysed using SPSS version 26.0 with chi-square analyses of linear-by-linear associations (categorical variables) and analysis of variance (continuous variables) to assess yearly trends in patient characteristics, satisfaction, and engagement. As this study used data collected during and between trials, secondary analyses examined whether these variables differed by trials (see Supplementary Material).

Generalized estimating equation (GEE) models were run in R and examined symptoms from assessment to post-treatment to follow-up after controlling for enrolment year and therapist location. ⁴⁵ Secondary analyses also controlled for trials. Consistent with intention-to-treat analyses, missing data were imputed before analysis.

Clinically significant change was examined by year (and secondarily by trials) through percentage symptom changes from baseline, ⁴⁶ and within-group Hedges g effect sizes using GEE estimated marginal means. Separate analyses were conducted for the overall sample and for patients in the clinical range at pre-treatment (i.e., PHQ-9 > 10, GAD-7 > 8, PDSS > 8, SIAS >7 and SPS > 2, K10 > 21, ³³ and SDS > 11)^. The SDS value was based on 1st quartile cut-off as no cut-off was identified. Clinical deterioration was defined as PHQ-9 increasing ≥ 6 and GAD-7 increasing ≥ 5 ³³ ; clinical recovery was defined as PHQ-9 decreasing ≥ 6 and GAD-7 decreasing ≥ 5, resulting in a patient moving from the clinical to the non-clinical range. ³³ A Bonferroni correction was used for each set of analyses as a control for number of tests performed.

Results

Patient Flow

Figure 1 shows the number and proportion of patients who were screened (n = 5,321), started (n = 4,283) and completed all treatment materials (n = 2,831).

Figure 1.

Patient flow from screening to follow-up. Trial 1 (November 1, 2013, to July 20, 2015): Cohort comparison of specialized and nonspecialized therapists. Trial 2 (February 1, 2016, to July 25, 2016): Randomized controlled trial comparing optional support versus once-weekly support. Trial 3 (December 2, 2016, to June 8, 2017): Preference trial comparing patient preference for optional versus once-weekly support. Trial 4 (October 1, 2017, to September 14, 2018): Randomized trial comparing one-business-day support versus once-weekly support. Trial 5 (September 2018, to March 1, 2019): Randomized trial comparing once-weekly support with or without pre-treatment lesson on motivational interviewing. Trial 6 (May 2, 2019, to November 5, 2019): Randomized factorial trial comparing once-weekly versus twice-weekly support and homework reflection record versus no reflection record.

Background

Table 1 provides demographics for patients who started iCBT by year. There were no differences in patient characteristics over time, except for the proportion who indicated having children (range: 51.0% to 63.8%). Patient characteristics also did not differ by trial period (See Supplementary Table 1).

Table 1.

Demographic Characteristics by Year and Overall for Patients Who Started the Wellbeing Course.

	2013	2014	2015	2016	2017	2018	2019	Total	Significance
Patients started course (n)	28	275	407	571	715	812	1,195	4,003
Age
Mean age (SD)	38.14 (11.99)	38.97 (12.42)	38.94 (13.22)	37.36 (12.50)	37.25 (12.90)	37.56 (13.12)	36.58 (12.76)	37.42 (12.89)	F ( 6,4000 ) = 2.51, P = 0.02
[Age range]	[22 to 73]	[19 to 73]	[18 to 81]	[18 to 81]	[18 to 86]	[18 to 84]	[18 to 88]	[18 to 88]
Gender
Men	10.7%	24.9%	29.70%	25.40%	23.60%	24.80%	25.60%	25.3%	χ2 (1, N = 4,001) = 0.17, P = 0.68
	(3/28)	(68/273)	(121/407)	(145/571)	(169/715)	(201/812)	(306/1,195)	(1,013/4,001)
Women	89.3%	74.7%	69.8%	74.30%	76.10%	75.20%	73.20%	74.20%
	(25/28)	(204/273)	(284/407)	(424/571)	(544/715)	(611/812)	(875/1,195)	(2,967/4,001)
Non-binary, two-spirit, not listed, or prefer not to disclose	0	0.40%	0.50%	0.40%	0.30%	0	1.20%	0.40%
		(1/273)	(2/407)	(2/571)	(2/715)		(21/1,195)	(21/4,001)
Location
Large city (>200,000)	50.00%	54.2% (148/273)	46.40%	42.60%	40.80%	41.70%	45.10%	44.10%	χ2 (1, N = 3,999) = 3.25, P = 0.07
	(14/28)		(189/407)	(243/571)	(292/715)	(339/812)	(539/1,194)	(1,764/4,000)
Small-medium city (10,000 to 200,000)	10.70%	19.00%	23.80%	29.10%	31.20%	25.20%	25.80%	26.40%
	(3/28)	(52/273)	(97/407)	(166/571)	(223/715)	(205/812)	(308/1,194)	(1,054/4,000)
Town or village	35.70%	19.40%	20.90%	20.80%	20.70%	23.30%	20.00%	21.10%
	(10/28)	(53/273)	(85/407)	(119/571)	(148/715)	(189/812)	(239/1,194)	(843/4,000)
Farm	3.60%	7.30%	8.60%	7.40%	6.70%	9.10%	8.30%	8.00%
	(1/28)	(20/273)	(35/407)	(42/571)	(48/715)	(74/812)	(99/1,194)	(319/4,000)
Reserve	0	0	0.20%	0.20%	0.60%	0.60%	0.80%	0.50%
			(1/407)	(1/571)	(4/715)	(5/812)	(9/1,194)	(20/4,000)
Ethnicity
White/Caucasian	92.90%	93.40%	92.00%	89.50%	88.10%	91.30%	89.80%	90.3%	χ2 (1, N = 3,992) = 5.14, P = 0.02
	(26/28)	(253/271)	(368/400)	(511/571)	(630/715)	(743/812)	(1,073/1,195)	(3,604/3,992)
Asian	3.60%	0.40%	1.50%	2.10%	1.30%	0.60%	1.70%	1.40%
	(1/28)	(1/271)	(6/400)	(12/571)	(9/715)	(5/812)	(20/1,195)	(54/3,992)
First Nations/Métis	3.60%	2.20%	3.50%	2.00%	5.10%	5.40%	5.70%	4.50%
	(1/28)	(6/271)	(14/400)	(11/571)	(36/715)	(44/812)	(68/1,195)	(180/3,992)
Other	0	4.10%	3.00%	6.50%	5.60%	2.50%	2.80%	3.90%
		(11/271)	(12/400)	(37/571)	(40/715)	(21/812)	(34/1,195)	(155/3,992)
Employment (most applicable)
Paid employment	67.90%	70.00%	65.40%	64.80%	63.50%	61.50%	68.10%	65.30%	χ2 (1, N = 4,000) = 4.62, P = 0.03
	(19/28)	(191/273)	(266/407)	(370/571)	(454/715)	(499/812)	(814/1,195)	(2,613/4,001)
Student	3.60%	1.80%	4.20%	7.40%	7.60%	6.90%	6.40%	6.30%
	(1/28)	(5/273)	(17/407)	(42/571)	(54/715)	(56/812)	(76/1,195)	(251/4,001)
Unemployed	10.70%	9.20%	11.30%	6.80%	7.40%	7.80%	6.30%	7.60%
	(3/28)	(25/273)	(46/407)	(39/571)	(53/715)	(63/812)	(75/1,195)	(304/4,001)
Other (retired, home duties, disability)	17.90%	19.00%	19.20%	21.00%	21.50%	26.80%	19.25%	20.89%
	(5/28)	(54/273)	(78/407)	(120/571)	(154/715)	(194/812)	(230/1,195)	(835/4,001)
Education
University degree	57.20%	36.20%	35.30%	37.30%	33.40%	33.90%	33.60%	33.7%	χ2 (1, N = 4,001) = 1.62, P = 0.20
	(16/28)	(99/273)	(104/407)	(213/571)	(239/715)	(276/812)	(401/1,195)	(1,348/4,001)
Relationship status
Married	50.0% (14/28)	48.0% (131/273)	46.2% (188/407)	46.8% (263/562)	45.5% (325/715)	45.2% (367/812)	44.4% (531/1,195)	45.6% (1,819/3,992)	χ2 (1, N = 3,992) = 0.93, P = 0.44
Children
Have children	53.60%	63.8% (173/271)	57.6% (234/406)	57.50%	59.00%	58.40%	51.0%	56.4%	χ2 (1, N = 3,988) = 13.50, P < 0.001a
	(15/28)			(323/562)	(422/715)	(474/812)	(609/1,195)	(2,250/3,989)

Note. Bonferroni correction P < 0.006.

^a Over time, there was a lower number of patients reporting they had children. See Supplementary Materials Table 1 showing patient characteristics did not differ by trial.

Clinical Profile at Screening

Table 2 shows baseline symptom severity for patients who started iCBT each year remained stable (Supplementary Material Table 2 also shows the clinical profile was stable across trials). Across years, 83.7% (n = 3,352) of patients had depression and or anxiety in the clinical range and 60.7% (n = 2,290) reported psychotropic medication use in the last 3 months.

Table 2.

Baseline Symptom Severity and Current and Lifetime Mental Health Service Use by Year and Overall for Patients Who Started the Wellbeing Course.

	2013	2014	2015	2016	2017	2018	2019	Total	Significance
Baseline symptom scores
Mean PHQ-9 (SD)	11.86 (5.05)	12.59 (5.67)	12.05 (5.66)	11.79 (5.95)	12.34 (5.92)	12.02 (5.79)	11.91 (5.62)	12.05 (5.76)	F (6,3995) = 1.03, P = 0.40
Mean GAD-7 (SD)	12.07 (5.66)	12.07 (5.10)	11.13 (5.27)	11.81 (5.20)	12.31 (5.00)	12.05 (5.14)	11.62 (4.95)	11.85 (5.09)	F (6,3995) = 3.02, P = 0.006
Mean K10 (SD)	26.36 (6.84)	27.82 (7.45)	27.14 (7.30)	27.52 (7.34)	28.46 (7.80)	28.20 (7.60)	27.64 (7.58)	27.84 (7.55)	F (6, 3995) = 2.14, P = 0.04
Mean SDS (SD)	18.11 (8.10)	17.37 (8.00)	17.58 (7.72)	17.11 (7.67)	17.01 (7.65)	17.02 (7.87)	16.89 (7.92)	17.09 (7.81)	F (6, 3995) = 0.56, P = 0.76
Mean PDSS-SR (SD)	9.39 (8.16)	7.22 (6.22)	7.41 (6.21)	7.36 (6.12)	7.57 (5.96)	7.53 (5.92)	7.14 (5.93)	7.38 (6.03)	F (6,3995) = 1.07, P = 0.38
Mean SPS/SIAS-6 (SD)	12.18 (9.78)	13.19 (10.39)	14.23 (10.24)	14.70 (10.37)	15.01 (10.84)	15.06 (11.29)	14.91 (10.69)	14.71 (10.73)	F (6,3995) = 1.61, P = 0.14
Symptoms in the clinical range
Depression or anxiety	85.7% (24/28)	84.0% (231/275)	83,3% (339/407)	81.6% (466/571)	84.6% (605/715)	84.4% (685/812)	83.8% (1,002/1,195)	83.7% (3,352/4,003)	χ2 (1, N = 4,003) = 0.26, P = 0.61
Mental health service use (last 3 months)
Receiving other mental health treatment	32.1% (9/28)	49.6% (134/270)	47.2% (191/405)	40.8% (233/571)	43.0% (307/714)	48.9% (397/812)	49.2% (587/1,194)	46.5% (1,858/3,994)	χ2 (1, N = 3,994) = 3.90, P = 0.05
On wait-list for other services	—	—	8.6% (16/187)	9.6% (55/571)	14.4% (103/714)	14.2% (115/812)	10.5% (125/1,194)	11.9% (414/3,478)	χ2 (1, N = 3,532) = 0.16, P = 0.69
Psychotropic medication	42.9% (12/28)	62.4% (171/274)	60.4% (246/407)	57.3% (327/571)	56.2% (401/713)	55.6% (451/811)	56.9% (682/1,194)	57.3% (2,292/3,999)	χ2 (1, N = 3,999) = 1.72, P = 0.19
Mental health service use (lifetime)
Family doctor/walk-in clinic	—	—	—	—	88.6% (109/123)	84.7% (688/812)	86.9% (1,037/1,194)	86.1% (1,834/2,129)	χ2 (1, N = 2,129) = 0.31, P = 0.58
Psychiatrist	—	—	—	—	30.9% (38/123)	30.4% (247/812)	32.5% (388/1,194)	31.61% (673/2,129)	χ2 (1, N = 2,129) = 0.78, P = 0.38

Note. PHQ-9 = Patient Health Questionnaire 9-item; GAD-7 = Generalized Anxiety Disorder Questionnaire 7-item; K10 = Kessler Distress Scale; SDS = Sheehan Disability Scale; PDSS-SR = Panic Disorder Severity Scale Self-Report; SIAS-6/SPS-6 = Social Interaction Anxiety Scale and Social Phobia Scale (SIAS-6/SPS-6). Bonferroni correction P < 0.004.

Primary and Secondary Follow-up Measures

Table 3 shows means, tests of symptom reduction, effect sizes, and percentage reductions for the overall sample and those in the clinical range. Analysis revealed significant symptom reductions on all measures at post-treatment, which were maintained at follow-up. For those in the clinical range, effect sizes were large for all post-treatment and follow-up measures, except for social anxiety, which had a moderate post-treatment effect. A significant portion of patients demonstrated post-treatment recovery and very few experienced deterioration across years (see Table 4).

Table 3.

Treatment Outcomes in Overall Sample.

	Estimated Means, Standard Deviations, and Test of No Change from Pre-treatment			Hedges g Effect Sizes from Pre-treatment		Percentage Change in Estimated Mean from Pre-treatment		Reliable Treatment Outcomes
	Pre-treatment	Post-treatment	Follow-up	to Post-treatment	to Follow-up	to Post-treatment	to Follow-up	Deterioration	Recovery
Overall sample
PHQ-9	12.05 (5.76)	6.43 (5.39) χ2 = 1,115.7, P < 0.001	5.86 (5.04) χ2 = 1,563.5, P < 0.001	1.01 [0.97, 1.05]	1.14 [1.10, 1.18]	47% [45% to 49%]	51% [50% to 53%]	1.7%	36.3%
GAD-7	11.85 (5.09)	5.91 (4.89) χ2 = 1,339.6, P < 0.001	5.21 (4.51) χ2 = 2,101.5, P < .001	1.19 [1.15, 1.23]	1.38 [1.34, 1.42]	50% [48% to 52%]	56% [54% to 58%]	2.0%	45.0%
K10	27.84 (7.55)	20.72 (7.73) χ2 = 1,074.4, P < 0.001	19.32 (7.63) χ2 = 2,169.7, P < 0.001	0.93 [0.89, 0.97]	1.12 [1.08, 1.16]	26% [24% to 27%]	31% [29% to 32%]	—	—
SDS	17.12 (7.81)	10.52 (8.37) χ2 = 741.9, P < 0.001	9.15 (8.67) χ2 = 972.2, P < 0.001	0.81 [0.78, 0.85]	0.97 [0.93, 1.00]	39% [36% to 41%]	47% [44% to 49%]	—	—
PDSS-SR	7.38 (6.03)	4.51 (4.84) χ2 = 444.4, P < 0.001	3.46 (4.23) χ2 = 774.6, P < 0.001	0.53 [0.49, 0.56]	0.75 [0.72, 0.79]	39% [36% to 42%]	53% [50% to 56%]	—	—
SIAS-6/SPS-6	14.71 (10.73)	11.15 (9.31) χ2 = 474.2, P < 0.001	9.79 (9.34) χ2 = 421.1, P < 0.001	0.35 [0.32, 0.39]	0.49 [0.46, 0.52]	24% [22% to 27%]	33% [30% to 36%]	—	—
Clinical samplea
PHQ-9 (≥ 10, N = 2,607)	15.39 (3.98)	8.09 (5.58) χ2 = 1,123.2, P < 0.001	7.12 (5.29) χ2 = 1,567.1, P < 0.001	1.51 [1.45, 1.56]	1.77 [1.70, 1.83]	47% [45% to 49%]	54% [52% to 56%]	1.3%	55.7%
GAD-7 (≥ 8, N = 3,068)	13.94 (3.76)	6.86 (4.98) χ2 = 1,463.8, P < 0.001	5.96 (4.64) χ2 = 2,008.8, P < 0.001	1.60 [1.55, 1.66]	1.89 [1.83, 1.95]	51% [49% to 53%]	57% [56% to 59%]	1.7%	58.7%
K10 (≥ 21, N = 3,279)	30.21 (6.10)	22.05 (7.73) χ2 = 1,061.7, P < 0.001	20.45 (7.75) χ2 = 2,045.8, P < 0.001	1.17 [1.13, 1.22]	1.40 [1.35, 1.45]	27% [26% to 28%]	32% [31% to 33%]	—	—
SDS excluding lower quartile (≥ 11, N = 3,105)	20.40 (5.27)	12.25 (8.31) χ2 = 781.5, P < 0.001	10.46 (8.86) χ2 = 1,014.0, P < 0.001	1.17 [1.12, 1.22]	1.36 [1.31, 1.41]	40% [38% to 42%]	49% [47% to 51%]	—	—
PDSS-SR (≥ 8, N = 1,832)	12.91 (3.99)	7.24 (5.18) χ2 = 606.4, P < 0.001	5.41 (4.72) χ2 = 1,120.2, P < 0.001	1.23 [1.17, 1.29]	1.72 [1.64, 1.79]	44% [41% to 47%]	58% [56% to 60%]	—	—
SIAS-6 ≥7 and SPS-6 ≥ 2 (N = 2,073)	22.59 (8.58)	16.48 (9.23) χ2 = 590.2, P < 0.001	14.66 (9.74) χ2 = 507.5, P < 0.001	0.69 [0.64, 0.73]	0.86 [0.81, 0.91]	27% [25% to 29%]	35% [32% to 38%]	—	—

Note. PHQ-9 = Patient Health Questionnaire 9-item; GAD-7 = Generalized Anxiety Disorder Questionnaire 7-item; K10 = Kessler Distress Scale; SDS = Sheehan Disability Scale; PDSS-SR = Panic Disorder Severity Scale Self-Report; SIAS-6/SPS-6 = Social Interaction Anxiety Scale and Social Phobia Scale (SIAS-6/SPS-6). Bonferroni correction P < 0.002.

^a For each measure, estimated means, effects sizes, and percentage change are calculated based on those who scored in the clinical range on the measure at pre-treatment.

Table 4.

Reliable Change, Program Engagement, Working Alliance, and Treatment Ratings by Year and Overall.

	2013	2014	2015	2016	2017	2018	2019	Total	Significance
Reliable change
PHQ-9 post-treatment recovery	42.0%	39.4%	36.8%	35.9%	37.1%	35.2%	35.7%	36.3%	F (6, 2609) = 0.33, P = 0.92
PHQ-9 post-treatment deterioration	4.7%	1.3%	1.8%	1.4%	1.9%	2.0%	1.4%	1.7%	F (6,1434) = 0.36, P = 0.90
GAD-7 post-treatment recovery	50.7%	43.7%	39.5%	44.3%	47.5%	45.5%	45.5%	45.0%	F (6, 2252) = 1.00, P = 0.43
GAD-7 post-treatment deterioration	0.8%	1.3%	3.7%	2.1%	2.0%	1.6%	1.7%	2.0%	F (6,1117) = 0.76, P = 0.60
Program engagement
Accessed > 4 lessons	85.7% (24/28)	85.8% (236/275)	79.4% (323/407)	79.3% (453/571)	78.0% (558/715)	81.8% (664/812)	80.3% (959/1,195)	80.4% (3,217/4,003)	χ2 (1, N = 4,003) = .48, P = 0.49
Accessed 5 lessons	75.0% (21/28)	78.9% (217/275)	70.3% (286/407)	70.6% (403/571)	66.4% (475/715)	71.1% (577/812)	71.3% (852/1,195)	70.7% (2,831/4,003)	χ2 (1, N = 4,003) = 1.21, P = 0.27
Mean number of therapist emails to patient (SD)	9.68 (2.98)	9.44 (2.29)	8.33 (1.98)	7.07 (2.70)	8.15 (2.94)	10.01 (3.14)	11.29 (3.97)	9.43 (3.52)	F (6, 3996) = 152.29, P < 0.001a
Mean number of emails from patient to therapist (SD)	6.93 (8.02)	4.78 (3.30)	4.13 (3.70)	3.34 (3.44)	3.53 (3.20)	4.09 (3.76)	4.14 (3.57)	3.97 (3.60)	F (6, 3996) = 11.04, P < 0.001a
Mean number of phone calls (SD)	1.21 (1.69)	0.77 (1.23)	0.88 (1.29)	0.54 (0.89)	0.68 (1.04)	0.96 (1.32)	0.98 (1.21)	0.84 (1.19)	F (6, 3996) = 18.00, P < 0.001a
Working alliance
WAI-SR total score	—	—	—	43.67 (11.92)	45.36 (10.89)	46.30 (10.56)	47.22 (10.01)	46.19 (10.62)	F (3, 1973) = 7.81, P < 0.001b
WAI-SR bond score	—	—	—	15.80 (4.76)	16.58 (3.91)	16.88 (3.79)	17.14 (3.47)	16.80 (3.84)	F (3, 1966) = 8.14, P < 0.001b
WAI-SR task score	—	—	—	14.78 (3.73)	14.89 (3.66)	15.08 (3.58)	15.34 (3.40)	15.11 (3.55)	F (3, 1966) = 2.25, P = 0.08
WAI-SR goal score	—	—	—	13.09 (5.03)	13.84 (4.99)	14.30 (4.57)	14.73 (4.45)	14.25 (4.68)	F (63, 1966) = 8.69, P < 0.001b
Treatment ratings
Satisfied/very satisfied overall	90.9% (20/22)	84.7% (171/202)	81.3% (222/273)	80.4% (300/373)	86.3% (402/466)	84.4% (487,576)	88.6% (675/762)	85.2% (2,277/2,674)	χ2 (1, N = 2,674) = 3.23, P = 0.07
Satisfied/very satisfied with materials	95.5% (21/22)	93.1% (188/202)	89.0% (243/273)	86.9% (324/373)	90.8% (423/466)	88.7% (511/576)	90.4% (689/762)	89.7% (2,399/2,674)	χ2 (1, N = 2,674) = 0, P = 0.99
Satisfied/very satisfied with telephone calls	71.4% (5/7)	77.3% (85/110)	71.9% (110/153)	67.9% (133/196)	71.2% (183/257)	75.0% (252/336)	74.8% (357/477)	73.2% (1,125/1,536)	χ2 (1, N = 1,536) = 4.93, P = 0.03
Satisfied/very satisfied with emails	90.9% (10/11)	83.7% (169/202)	78.0% (213/273)	74.0% (276/373)	77.0% (359/466)	83.8% (482/575)	84.1% (641/762)	80.7% (2,150/2,662)	χ2 (1, N = 2,622) = 8.344, P = 0.004
Increased/greatly increased confidence	95.5% (21/22)	91.0% (184/202)	86.4% (236/273)	87.9% (328/373)	93.6% (436/466)	90.1% (519/576)	93.0% (709/762)	91.0% (2,433/2,674)	χ2 (1, N = 2,674) = 2.78, P = 0.10
Increased/greatly increased motivation for other treatment	77.3% (17/22)	79.2% (160/202)	75.8% (207/273)	77.2% (288/373)	81.5% (380/466)	83.5% (481/576)	86.9% (636/732)	81.1% (2,169/2,674)	χ2 (1, N = 2,674) = 4.90, P = 0.03
Course was worth the time (%)	100% (22/22)	98.0% (198/202)	94.5% (258/273)	94.6% (353/373)	96.8% (451/466)	95.8% (552/576)	97.2% (741/762)	96.3% (2,575/2,674)	χ2 (1, N = 2,674) = 0.77, P = 0.38
Would recommend course to a friend (%)	100% (22/22)	97.5% (197/202)	93.8% (256/273)	94.9% (354/373)	97.2% (453/466)	97.0% (559/576)	97.2% (741/762)	96.6% (2,582/2,674)	χ 2 (1, N = 2,674) = 2.78, P = 0.10

Note. PHQ-9 = Patient Health Questionnaire-9-item; GAD-7 = Generalized Anxiety Disorder-7-item; WAI-SR = Working Alliance Inventory-Short Report. Bonferroni correction p < .002.

^aPost-hoc comparisons revealed some year-to-year differences in emails sent, emails received, and phone contact. These differences are attributed to trial designs investigating different levels of therapist support. For detailed comparisons by trial, see Supplementary Materials.

^bPost-hoc comparisons revealed the WAI-SR Total, Bond subscale, and Goal subscale were lower in 2016 (year patients offered optional support) than in 2018 or 2019 (years offered once-weekly or greater support). For detailed comparisons by trial, see Supplementary Materials.

Program Engagement

Year-to-year analysis of showed high engagement across measures and years (e.g., 80.4%; n = 3,217 of patients accessed at least 4 lessons; see Table 4). Some significant differences were observed in number of emails and phone calls between therapists and patients over the years and across trials reflecting research trials that varied support levels (see Table 4; Supplementary Material Table 3).

Working Alliance and Patient Satisfaction

Working alliance and satisfaction variables show strong patient satisfaction across all indicators over time (see Table 4). There was a significant increase in patients’ WAI-SR total, bond, and goal scores over the years and related to research trials where there was increased therapist contact (see Table 4, Supplementary Material Table 3).

Secondary Analysis of Healthcare Use

For the sub-sample who completed healthcare questions at screening (n = 655) and follow-up (n = 453), there was no change in the proportion of patients who reported visits to a psychiatrist (16.6% vs. 16.6%), mental health professional (27% vs. 21.6%), medical or health professional (7.2% vs. 4.9%), or having an emergency room/hospital admission (4.7% vs. 2.6%). There was a reduction in visits to family physicians (61.2% vs. 32.5%; χ2 (1, N = 655) = 88.16, P < 0.001) and patients not utilizing any service for mental health concerns (50.3% vs. 24.0%; χ2 (1, N = 655) = 81.82, P < 0.001).

Discussion

This observational cohort study adds to evidence related to patient features, engagement, satisfaction and longitudinal improvements in therapist-assisted transdiagnostic iCBT in routine care in Canada. Consistent with studies on iCBT in routine care in Australia, ^{10, 33} large effect sizes were found across years on primary measures, distress, and disability that were maintained at follow-up. A considerable number of patients met criteria for recovery (55.7% for depression; 58.7% for anxiety), and deterioration rates were very low (1.3% for depression; 1.7% for anxiety). These findings demonstrate the stability in iCBT symptom improvements over time in routine care and after initial implementation efforts, which has rarely been reported in the literature with such large sample sizes and has not been examined in Canada to date. ³⁵

The Online Therapy Unit showed significant growth in the numbers of patients completing telephone screens (average annual increase: 37.7%) and starting iCBT (average annual increase: 35.3%), indicating increasing demand for digital mental health services. This finding supports the Canadian Government announcement to fund virtual mental health tools, especially iCBT. It would be valuable to explore whether increased use of digital mental services is part of an observed trend for increased use of mental health services ⁴⁷ or whether increased use of iCBT is associated with generally decreased use of other services. Examining trends suggests that characteristics of those seeking services have remained stable. Patients seeking iCBT show considerable diversity in terms of age (18 to 88) and location of residence (with ∼56% of patients residing outside large cities). Regarding ethnic diversity, there was a trend toward increased iCBT use by patients who identified as First Nations or Métis across the years. Comparison to 2016 provincial statistics ⁴⁸ suggests, however, First Nations and Métis people are still under-represented in iCBT (4.5% of the sample compared with 16.3% of Saskatchewan’s First Nations or Métis population). Also noteworthy, across years only about one-quarter of patients identified as male. The under-representation of First Nations or Métis individuals ⁴⁹ and males ⁵⁰ reflects a broader concern about lower mental health utilization in these populations and is not necessarily specific to iCBT.

Of interest, there is considerable diversity in symptom severity among those seeking iCBT and iCBT programs need to take this into account (∼16% did not have scores in the clinical range; 57.3% of patients were taking medication). Over 50% of patients were referred by physicians, suggesting iCBT is a valuable resource for physicians in managing patient needs. In the subsample of patients who completed healthcare use questions pre and post-ICBT, there was a decreased report of physician visits after completing iCBT (61.2% vs. 32.5%), highlighting the importance of coordinating iCBT with medical care and draws attention to the need for research on how iCBT impacts healthcare use.

Across years, patient engagement and satisfaction were excellent (e.g., 80% review most materials; 96.3% report iCBT was worth their time). There were some differences in the patient/therapist emails, phone calls, and working alliance observed across years, with supplementary analyses suggesting differences reflected research trials that offered varying degrees of therapist support. Importantly, this past research shows increasing therapist support beyond once weekly does not improve effects and in fact is associated with higher costs and operational challenges. ^20,21

Comparison with Wellbeing Course in Australia

The sample characteristics are similar to the Australian sample, ³³ except a greater proportion of our patients reported concurrent medication use and treatment. Notably, patients in our sample were more likely to complete most treatment materials (80.4% vs. 67.1%), post-treatment measures (70.8% vs. 60.0%), and follow-up measures (60.3% vs. 39.3%). Lower completion rates in the Australian sample may be explained by higher pre-treatment symptom severity or differences in practices. For example, a large proportion of the Australian patients were seeking symptom consultation rather than iCBT, and users were permitted to access iCBT without first speaking to a therapist. A greater focus on consultation and system navigation is a potential avenue for the Online Therapy Unit in the future, especially since this represents an area of growth in Australia. Of note, the Australian group identified an increase in proportion of young adult users, Aboriginal or Torres Strait Islander, and those with anxiety, which we did not identify; these may be trends to watch that could impact programming. Notwithstanding the differences, the similarity in patient characteristics and longitudinal effects across the two countries is striking. The results provide promising evidence that with minor adaptations, iCBT may be used effectively across countries.

Limitations

Post-treatment and follow-up data were missing for approximately 30% and 40% of patients, respectively, although rates of follow-up are higher than what is typical of routine care studies, ^{9,51 -53} and a conservative approach was used to replace missing data. This study did not include a control group, although previous RCTs on the Wellbeing Course, ⁵⁴ and many randomized controlled trials of iCBT have been undertaken. ⁶ Self-report measures were used rather than diagnostic interviews and key demographic characteristics (e.g., gender, sexual orientation) were not tracked, so certain groups may be under-represented. The results need to be considered in light of the fact that many patients use other services while using iCBT, and we used a brief measure of healthcare use for a subsample only; as such, it is impossible to fully ascertain whether participating in iCBT is associated with decreased use of other services, including face-to-face CBT, at follow-up. Finally, it is important to note we did not track additional resource use in the current study, and future research should explore how these resources contribute to outcomes.

Strengths and Conclusions

In light of research showing many patients do not receive adequate quality or quantity of treatment ⁵⁵ intensified by the COVID-19 pandemic, ⁵⁶ the spotlight is on digital mental health services to improve treatment access. This study highlights growth can be expected in iCBT over time, and it is possible to sustain large effects, patient engagement, and satisfaction over time after initial iCBT implementation efforts have been employed. A strength of this study is its large sample size spanning 6 years, and the stability of symptom improvements from post-treatment to 3- to 4-month follow-up, engagement, and satisfaction over time. Publication of large studies like this one assists groups who are planning changes to mental health systems ³³ and is important as there is no large health service databases to draw on related to iCBT. The findings highlight iCBT is not a replacement for other services, as ∼20% of patients seeking iCBT are referred to other services, and a high percentage of patients concurrently take medication; as such, iCBT needs to be woven into existing healthcare systems. The study identifies some gaps in service users, suggesting directions for improving reach of iCBT that represent important considerations for others implementing iCBT in Canada. The longitudinal effects observed in this large study may serve as a benchmark for the growing number of iCBT programs that are emerging in Canada. ¹⁴ Benchmarking is a common technique in routine practice that will allow groups implementing iCBT programs to compare effects across different programs ⁵⁷ and can guide quality improvement efforts (e.g., encouraging programs with lower effects to improve care) as well as funding decisions.