Insights into internet-delivered cognitive behavioural therapy for public safety personnel: Exploration of client experiences during and after treatment

Abstract

Canadian public safety personnel (PSP) experience high rates of mental health problems and barriers to receiving care. Internet-delivered cognitive behavioural therapy (ICBT) may help reduce barriers to care; however, there is no literature involving qualitative analyses of client feedback to describe PSP experiences with ICBT. Identifying these experiences is important because it can inform future use of ICBT with this group that has unique needs. The current study was designed to explore how clients (N = 82) experienced ICBT that had been tailored to meet their needs; specifically, the study assessed their perceptions of program impacts, what clients found helpful, and client suggestions for improvements. The ICBT course included five core lessons, client stories, and nine initial additional resources, as well as flexible frequencies (optional, once weekly, or twice weekly) and durations (8 to 16 weeks) of therapist support. A qualitative reliability thematic analysis was used to analyze client communications and feedback. Responses to a Treatment Satisfaction Questionnaire administered at eight weeks post-enrollment were available for 57 clients. Client emails with therapists were also examined among all clients, including an additional 25 clients who did not complete the Treatment Satisfaction Questionnaire. Themes identified in the qualitative analyses were related to: reported impacts and hindering events, helpful and challenging course skills and content, helpful aspects of the course, and areas for improvement. Clients who completed the Treatment Satisfaction Questionnaire and those who did not reported beneficial impacts from the program, with the most commonly endorsed themes being skill development and normalizing mental health issues. Hindering events were experienced by both groups and included timeline challenges, technology challenges, and negative effects. Comments from both groups suggested that clients had more success than challenges when practicing the skills. Thought challenging was the skill most frequently identified as helpful. Clients described many aspects of the program as helpful with the most frequently endorsed themes being the course format and content, the flexible nature of the course, access to additional materials and case stories, and therapist assistance. Clients also provided suggestions for improving the course (e.g., case stories, additional resources, timelines audio and videos). Overall, client communications suggest that ICBT is accepted and perceived as beneficial among PSP. These results informed rapid improvements to the ICBT program tailored for PSP and may inform others seeking to provide digital mental health services to PSP.

Highlights

• •Public safety personnel (PSP) report benefits of ICBT, such as improved wellbeing.
• •PSP value therapist assistance, content, and activities of tailored ICBT.
• •The timeline to complete ICBT is a common challenge for PSP.
• •The most common recommendation to modify ICBT is further tailoring ICBT to PSP.
• •PSP who do not complete post-measures experience ICBT challenges and benefits.

PSP include border services officers, public safety communicators, correctional workers, firefighters (career and volunteer), Indigenous emergency managers, operational intelligence personnel, paramedics, police (municipal, provincial, federal), and search and rescue personnel (Canadian Institute for Public Safety Research and Treatment, 2019). PSP are frequently exposed to diverse potentially psychologically traumatic events (PPTEs; Carleton et al., 2019). PSP also have much higher rates of mental health disorders, including generalized anxiety disorder, major depressive disorder, and posttraumatic stress disorder (PTSD)—which are often collectively referred to as posttraumatic stress injuries (PTSIs; CIPSRT, 2019)—than the general population (Carleton et al., 2018). PSP have also reported logistical (e.g., geographic, time) and attitudinal barriers (e.g., stigma, desire to manage mental health problems independently) to accessing mental healthcare (Jones et al., 2020; McCall et al., 2020a).

Internet-delivered cognitive behaviour therapy (ICBT) is a form of cognitive behavioural therapy that can help to overcome barriers to mental healthcare because it is private, accessible at any location, and cost-effective (Andersson, 2016; Donker et al., 2015). ICBT retains the traditional concepts and skills included in face-to-face cognitive behaviour therapy; however, ICBT is delivered online, typically through weekly lessons, and often with therapist support through phone or email (Andersson, 2016). ICBT is convenient for users and shows treatment outcomes similar to face-to-face cognitive behavioural therapy, especially when combined with provider support (Andersson, 2016; Andersson et al., 2019; Karyotaki et al., 2021; Sijbrandij et al., 2016).

In 2019, the Government of Canada responded to the high rates of PTSIs among PSP by initiating a National Action Plan (Public Safety Canada, 2019) in which ICBT was identified as a potential solution to overcome barriers to care and provide mental healthcare to PSP. The Canadian Government resourced a research unit called PSPNET at the University of Regina to develop and pilot test ICBT tailored for PSP as part of the National Action Plan.

The current study is one of a series of studies designed to evaluate PSPNET's tailored ICBT programs. We have conducted two previous studies to garner insight into how best to tailor ICBT for PSP. The first study examined PSP's openness to using ICBT. The results indicated that diverse PSP rated ICBT as highly acceptable and their second most preferred mental health service option after face-to-face psychological services (McCall et al., 2020b). The second study indicated that PSP perceive ICBT to have advantages (e.g., privacy, access) and disadvantages (e.g., less accountability, demanding, security concerns) but overwhelmingly perceived ICBT as necessary (McCall et al., 2020a). PSP from this second study also provided feedback on how to tailor ICBT to PSP, such as by providing flexible timelines and frequencies of therapist support and by addressing PSP-specific issues, such as frequent PPTE exposures, public scrutiny, low workplace support, high standards for toughness and control, pessimism, and discomfort with vulnerability (McCall et al., 2020a). The PSPNET team used the results of this study to help tailor a transdiagnostic ICBT course (the PSP Wellbeing Course) specifically to meet the needs of PSP. Preliminary quantitative outcomes (e.g., symptom change scores) from the PSP Wellbeing Course are promising and are described in a separate paper (Hadjistavropoulos et al., 2021). However, there is still no published qualitative research on PSP's experiences using ICBT. The current study contributes to the larger PSPNET research program by analyzing qualitative data from PSP concerning their experiences with the PSP Wellbeing Course.

Past research has used a qualitative approach to evaluate ICBT by gathering feedback through post-treatment interviews (e.g., Lillevoll et al., 2013; Richards et al., 2016) or post-treatment questionnaires (e.g., Hadjistavropoulos et al., 2018). Researchers have consistently found qualitative evidence that ICBT has positive impacts on the wellbeing of clients (e.g., Earley et al., 2017; Lillevoll et al., 2013; Richards et al., 2016). For example, ICBT users with subclinical symptoms of depression have reported improved wellbeing, an enhanced sense of empowerment, feelings of validation, positive behavioural changes, and increased awareness and insight (Earley et al., 2017). In another study, most ICBT users described participation as beneficial (Richards et al., 2016). Qualitative research evidence also indicates that most ICBT clients like the fact that ICBT is flexible, is available online, includes therapist support, and provides CBT techniques (e.g., Earley et al., 2017; Richards et al., 2016); however, ICBT clients have also reported difficulties navigating websites and frustrations with therapist feedback (Earley et al., 2017; Richards et al., 2016). Complementing these findings, a recent meta-analysis of qualitative studies on the acceptability and usability of digital mental health interventions concluded that users generally prefer interventions that offer support, are personalized to meet their preference and needs, and are accessible and interactive with simple interfaces and succinct content (Patel et al., 2020).

The PSP Wellbeing Course was based on an existing course for which clients have reported liking the flexibility, convenience, ability to print materials, engagement with therapists, course content, case stories, and format, but suggested increasing timeline flexibility, matching therapist availability to client needs, and improving case stories (Hadjistavropoulos et al., 2018). Results from previous qualitative studies of ICBT have provided valuable information for evaluating and improving programming; however, the extant results have not provided recommendations or examples of how client feedback can be used to improve services. In the current study, we show how evaluating the perceived impacts and likes and dislikes of the PSP Wellbeing Course contributed to the ongoing tailoring of PSPNET's ICBT services for PSP.

Use of client feedback to continuously improve a program is consistent with the learning health systems (LHS) model (Smith and Institute of Medicine (U.S.), 2013). The LHS approach emphasizes using a person-centered and collaborative research approach and highlights the importance of client input for improving treatment outcomes and experiences (Government of Canada, 2016; Menear et al., 2019). This approach also emphasizes using research results to drive rapid, evidenced-based, point-of-care innovations and improvements. The LHS approach uses an iterative process whereby “evidence informs practice, and practice informs evidence” (Greene et al., 2012: 207), which is particularly well-suited for rapidly improving digital mental health technologies (e.g., Graham et al., 2020; Mohr et al., 2015, Mohr et al., 2017, Mohr et al., 2018). Making changes during trials is encouraged and necessary, so long as the core components of the technology under investigation remain intact (Mohr et al., 2015). The current study uses an LHS approach to evaluate clients' feedback on or experiences using the program to make improvements to the program with the ultimate goal of improving client outcomes and experiences.

1. Purpose and research questions

The current study was designed as a qualitatively-driven evaluation of an ICBT program tailored for, and now being offered to, Canadian PSP. The study explores clients' experiences using the program to address four main objectives: 1) to identify client perspectives of the initial impacts of the program 2) to identify the perceived strengths and challenges of the program, and suggestions for improvement 3) to identify differences in experiences between those who completed the Treatment Satisfaction Questionnaire and those who did not (i.e., those who offered less feedback and appeared to be less engaged in the program), and 4) to use the results to drive improvements to the program.

To meet these objectives, the current study addresses seven exploratory questions: 1) What are the reported benefits of the PSP Wellbeing Course? 2) What are reported negative effects of the course? 3) What aspects of the course content did clients find helpful? 4) What aspects of the course content did clients find challenging? 5) What aspects of the course did clients like most? 6) What aspects of the course did clients dislike or want improved? and 7) What are the demographic and experiential differences between those who completed the Treatment Satisfaction Questionnaire and those who did not?

The analyses were congruent with an LHS framework that allowed the PSPNET team to identify areas for further tailoring to meet the needs of PSP. Consistent with an LHS approach, we examined available data (e.g., weekly open-ended reflection questions about ICBT, emails sent to therapists, and the Treatment Satisfaction Questionnaire) from all Saskatchewan clients enrolled in the PSP Wellbeing Course from December 5, 2019 to June 5, 2020 in an effort to rapidly learn from patients and adapt the program to better meet their needs. As such, we report examples of how we used the results of our analyses to improve PSPNET's ICBT services. Given the exploratory nature of the study, we did not formulate any hypotheses.

2. Methods

2.1. Context

The PSP Wellbeing Course is a transdiagnostic ICBT course tailored for PSP. The course is offered by PSPNET at the University of Regina. PSPNET's ICBT is currently available to PSP in the Canadian provinces of Saskatchewan, Quebec, New Brunswick, and Nova Scotia. The PSP Wellbeing Course was tailored by the PSPNET team based on results from extensive interviews with PSP stakeholders (McCall et al., 2020a). The original course was developed, validated, and implemented broadly in Australia (Dear et al., 2015; Fogliati, 2016; Titov et al., 2015a, Titov et al., 2015b, Titov et al., 2020), before being implemented and validated in Saskatchewan over the past 7 years (Hadjistavropoulos et al., 2016, Hadjistavropoulos et al., 2017, Hadjistavropoulos et al., 2019, Hadjistavropoulos et al., 2020). The current study draws on data collected as part of a registered observational trial of the PSP Wellbeing Course (Clinicaltrials.gov NCT04127032) and was approved by the University of Regina's Research Ethics Board (REB#: 2019-157).

2.2. Course description and eligibility criteria

Clients access the course by creating a PSPNET account, completing an online screening questionnaire, and participating in a follow-up telephone call. Eligible clients are (a) 18 years of age or older; (b) comfortable using and able to access the Internet; (c) willing to provide a local medical contact in case of emergencies; and (d) residents of either Saskatchewan (enrollment began December 2019), Quebec (enrollment began August 2020), New Brunswick and Nova Scotia (enrollment began November 2021). Clients who report severe problems with alcohol or drugs, mania, or psychosis, or clients at high risk of suicide, are referred to face-to-face services. Eligible clients are given the option to work through the course with once-weekly, twice-weekly, or optional, as-needed therapist support via secure emails or telephone calls. Therapist support is offered for eight weeks, with an option to extend support for up to 16 weeks.

Clients work through five psychoeducational lessons addressing (a) symptom identification and the cognitive behavioural model; (b) thought monitoring and challenging; (c) de-arousal strategies and pleasant activity scheduling; (d) graded exposure; and (e) relapse prevention. Each lesson consists of slides with text, images, and diagrams, additional downloadable and printable readings and activities (i.e., do-it-yourself guides), frequently asked questions, and case stories, which feature stories of PSP and their thoughts, behaviours, successes, and challenges working through the materials. Clients can also work through additional materials tailored with examples for PSP. The nine additional resources included when the course was initially launched covered various topics and strategies related to problem solving, panic, assertiveness, communication, beliefs, worry, sleep, PTSD, and mental skills (e.g., calculating risks). Clients are asked to complete weekly questionnaires about their symptoms and experiences with the course. After eight weeks of enrollment in the course, clients are asked to complete a Treatment Satisfaction Questionnaire.

2.3. Participants

The current study included 82 clients who enrolled in the PSP Wellbeing Course in Saskatchewan between December 5, 2019 and June 5, 2020 (clients from other provinces were not included as we had not begun offering ICBT in other provinces at the time this study was carried out). Most participating clients completed the eight-week Treatment Satisfaction Questionnaire (n = 57). The clients who did not complete the Treatment Satisfaction Questionnaire consisted of 15 clients who officially withdrew from the course and 10 who did not. The reasons provided for withdrawal included: the client decided to seek alternative services (n = 7) or the client had personal issues arise (e.g., a busy work schedule) that prevented full participation (n = 7). Several clients expressed interest in taking the course again at a later date (n = 8). One client did not access any materials, could not be reached, and was consequently withdrawn from the course. All open-ended data provided by clients within the first eight weeks of treatment was included in the current study and analyzed. The sample size is appropriate for exploratory qualitative research that does not seek to generalize results to larger populations (Boddy, 2016).

2.4. Measures and data

Before accessing the PSP Wellbeing Course, clients completed the Patient Health Questionnaire-9 (PHQ-9; Kroenke et al., 2001), Generalized Anxiety Disorder-7 (GAD-7; Spitzer et al., 2006), and PTSD Checklist for DSM-5 (PCL-5; Blevins et al., 2015) as part of the online screening (in addition to other measures, the results of which are reported elsewhere; Hadjistavropoulos et al., 2021). The current qualitative data were collected from three sources and de-identified prior to analyses. First, we analyzed open-ended responses to the Treatment Satisfaction Questionnaire, in which clients were asked why the course was (or was not) worth their time and what they liked and disliked about the course (see Appendix A). Second, we analyzed the content of emails clients sent to therapists. Third, we analyzed the two optional weekly homework reflection questions wherein clients were asked to share any difficulties they had during the previous week and any examples of areas on which they focused.

2.5. Analyses

2.5.1. Quantitative analyses

We used a series of Chi-Square tests and independent samples t-tests to compare the clinical and demographic characteristics of clients who completed the Treatment Satisfaction Questionnaire at eight weeks post-enrollment and clients who did not. These two groups were compared on gender, community size, relationship status, race/ethnicity, years of experience as a PSP, level of education, age, PSP occupation, prior treatments, and symptoms of depression, anxiety, and PTSD. We conducted these analyses using SPSS Version 23 (IBM Corp, 2015).

2.5.2. Qualitative analyses

Qualitative data from all three sources were combined for each client and entered into the qualitative analysis software NVIVO 12.0 (QSR International, 2018). Analysis of client responses to the Treatment Satisfaction Questionnaire highlighted salient comments at post-enrollment. Examination of the emails and reflection questions allowed for assessing comments during the course, which was particularly important for clients who did not complete the Treatment Satisfaction Questionnaire. Data from participating clients who completed and those who did not complete the Treatment Satisfaction Questionnaire were divided into separate groups to capture any differences in the experiences of these two groups. There were too few clients who officially withdrew (n = 15) to make comparisons with clients who remained in the program but did not complete Treatment Satisfaction Questionnaire (n = 10). Therefore, those who officially withdrew and those who did not complete the Treatment Satisfaction Questionnaire were combined to create a Treatment Satisfaction Questionnaire non-completers group as both groups offered minimal data and appeared to be less engaged in the program. This allowed us to account for the experiences of those who appeared less engaged with the course (i.e., non-completers) without reducing the frequencies of responses among those who offered more feedback and appeared more engaged with the course (i.e., completers).

Throughout the qualitative analyses, we used a coding reliability thematic analysis approach (Boyatzis, 1998; Braun et al., 2018) suitable for answering questions about clients' perceptions of treatments and their likes and dislikes. Coding reliability thematic analyses focuses on developing overarching themes that largely reflect client responses and use relatively little interpretation. Themes are often grouped under larger domains to create summary lists of grouped themes. Frequencies of responses were included in the analyses to represent general trends within the data in order to provide insights into the frequency with which specific themes were endorsed. The codebook was developed over time using deductive and inductive approaches. The initial framework of summary domains was derived from the questions asked of clients (e.g., benefits, likes, dislikes, challenges, negative effects). Themes were then created by examining patterns in the open-ended responses through the following process. The data were coded by meaning units defined as a program impact, strength, or challenge. The initial codebook was created by the primary coder, author J.D.B., and was revised based on reviews and discussions with author H.D.H. The discussions led to codes being grouped together into larger themes and organized under larger domains. The new codebook was applied to a new set of clients and was further refined through discussion between author J.D.B. and author H.D.H.

Given the descriptive (rather than interpretative) nature of the qualitative analyses, efforts were made to ensure descriptive themes reflected clients' own words. The primary coder, author J.D.B., was not involved in the therapeutic activities of ICBT and sought to create domains and themes based on client comments within the data by creating themes that were as close to the textual data as possible. The primary investigator, author H.D.H., oversees the therapeutic activities of PSPNET and performed the initial check of the coding and codebook to ensure the coding accurately reflected client perceptions.

Inter-coder reliability was calculated using a percent agreement approach established by Campbell et al. (2013) to support trustworthiness in the coding process. In the first phase of the reliability coding, a reliability coder coded 20% of the data after the development of the initial codebook. An initial percent agreement was calculated based on total agreements divided by the total number of coded pieces of data (agreements and disagreements; Campbell et al., 2013). The initial percent agreement was 72.6%. In the second phase of the reliability coding, a negotiated percent agreement was calculated after discussions between the primary coder and the reliability coder took place to settle disagreements from the first phase. A negotiated agreement percent was calculated at 98.9%. The reliability coder deferred to the primary coder 81.7% of the time and the primary coder deferred to the reliability coder 17.2% of the time. The problem of unitization can occur as part of conducting inter-coder reliability (Kurasaki, 2000). Unitization occurs when units of analysis are not given naturally and coders may code different portions of the text. In the current study, unitization was handled with the program NVIVO by capturing agreements anytime there was overlap between the coders at some point in a passage of text. After resolving discrepant coding, final revisions were made to the codebook, which was then used to code the remaining data and to recode previously coded data.

3. Results

3.1. Client characteristics and completion rates: quantitative analyses

Client characteristics and the results of comparisons between the Treatment Satisfaction Questionnaire completer and non-completer client groups are summarized in Table 1. Non-completers were significantly more likely to identify as married and as ethnic minorities. Non-completers also reported significantly more severe symptoms of major depressive disorder (PHQ-9) and generalized anxiety disorder (GAD-7). Additional results, including the number of lessons accessed within eight weeks, are displayed in Table 1. Only one client in the completer and one client in the non-completer group accessed lessons after eight weeks.

Table 1.

Participant characteristics.

Characteristics	Completers (n = 57)	Non-completers (n = 25)	Total sample (N = 82)	Test of group differencesa
Gender, n (%)				χ2(1, N = 81) = 1.55, p = .21
Male	23 (40)	14 (56)	37 (45)
Female	33 (58)	11 (44)	44 (54)
Nonbinary	1 (2)	0	1 (1)
Community size, n (%)				χ2(1, N = 82) = 0.15, p = .70
<100,000	27 (47)	13 (52)	40 (49)
>100,000	30 (53)	12 (48)	42 (51)
Relationship status, n (%)				χ2(1, N = 82) = 4.26, p = .04
Not married or common law	25 (44)	5 (20)	30 (37)
Married or common law	32 (56)	20 (80)	52 (63)
Race/ethnicity, n (%)				χ2(1, N = 82) = 5.14, p = .02
White	52 (91)	18 (72)	70 (85)
First Nations, Inuit, or Metis	3 (5)	5 (20)	8 (10)
Other ethnic minority	2 (4)	2 (8)	4 (5)
Years of PSP experience				t(79) = 1.01, p = .32
Mean (SD)	13.27 (9.29)	11.12 (7.84)	12.60 (8.88)
0–9 years, n (%)	21 (37)	12 (48)	33 (40)
10+ years, n (%)	35 (61)	13 (52)	48 (59)
No response, n (%)	1 (2)	0	1 (1)
Highest level of education, n (%)				χ2(1, N = 81) = 0.66, p = .42
No degree	27 (47)	10 (40)	37 (45)
College diploma	13 (23)	10 (40)	23 (28)
University degree	16 (28)	5 (20)	21 (26)
No response	1 (2)	0	1 (1)
Age				t(80) = 0.72, p = .47
Mean (SD)	39.99 (11.26)	38.14 (9.38)	39.43 (10.70)
19–29, n (%)	12 (21)	5 (20)	17 (21)
30–39, n (%)	16 (28)	10 (40)	26 (32)
40–49, n (%)	13 (23)	6 (24)	19 (23)
50–59, n (%)	14 (25)	4 (16)	18 (22)
60+, n (%)	2 (4)	0	2 (2)
PSP occupation, n (%)
Border Services	0	1 (4)	1 (1)	χ2(1, N = 82) = 2.31, p = .13
Corrections	5 (9)	4 (16)	9 (11)	χ2(1, N = 82) = 0.93, p = .34
Dispatch/Communications	4 (7)	2 (8)	6 (7)	χ2(1, N = 82) = 0.03, p = .88
Fire	4 (7)	2 (8)	6 (7)	χ2(1, N = 82) = 0.03, p = .88
Paramedicine	23 (40)	6 (24)	29 (35)	χ2(1, N = 82) = 2.03, p = .15
Police	16 (28)	6 (24)	22 (27)	χ2(1, N = 82) = 15, p = .70
Other	5 (9)	4 (16)	10 (12)	χ2(1, N = 82) = 0.93, p = .34
Treatments within past 3 months, n (%)
Taken mental health medication	15 (26)	6 (24)	21 (26)	χ2(1, N = 82) = 0.20, p = .66
Seen mental healthcare provider	33 (58)	14 (56)	47 (57)	χ2(1, N = 82) = 0.03, p = .87
Pretreatment PHQ-9				t(80) = − 3.00, p = .004
Mean (SD)	10.39 (5.90)	14.52 (5.36)	11.65 (6.02)
Not clinically significant (0–9), n (%)	28 (49)	5 (20)	33 (40)
Clinically significant (10–27), n (%)	29 (51)	20 (80)	49 (60)
Pretreatment GAD-7				t(79) = − 2.02, p < .05
Mean (SD)	9.60 (5.54)	12.29 (5.36)	10.40 (5.59)
Not clinically significant (0–9), n (%)	28 (49)	7 (28)	35 (43)
Clinically significant (10−21), n (%)	29 (51)	17 (68)	46 (56)
No response, n (%)	0	1 (4)	1 (1)
Pretreatment PCL-5				t(80) = −1.89, p = .06
Mean (SD)	28.26 (20.63)	37.04 (15.87)	30.94 (19.63)
Not clinically significant (0−32), n (%)	34 (60)	12 (48)	36 (44)
Clinically significant (33–80), n (%)	23 (40)	13 (52)	40 (49)
Lessons accessed at 8 weeks, n (%)
Lesson 1	57 (100)	22 (88)	79 (96)
Lesson 2	57 (100)	17 (68)	74 (90)
Lesson 3	56 (98)	11 (44)	67 (82)
Lesson 4	55 (96)	6 (24)	61 (74)
Lesson 5	53 (93)	5 (20)	58 (71)

^aFor categorical variables, we conducted Chi-Square tests using the categories displayed in this table, with three exceptions: (a) we omitted the client who identified as nonbinary from the comparison for gender; (b) we dichotomized race/ethnicity into white and non-white; (c) we dichotomized level of education into university degree and no degree.

3.2. Summary of qualitative domains

The findings from the qualitative analyses were grouped under the following domains: reported impacts, hindering events, and negative effects; helpful and challenging course skills and content; and helpful aspects of the course and areas for improvement. The following results describe themes identified within each domain. Reported frequencies and percentages are presented to reflect general patterns and trends which PSPNET used to guide insights and quality enhancement (rather than to make generalizations about clients within the program).

3.3. Reported impacts, hindering events, and negative effects

Throughout communications with therapists, clients conveyed that the course had a variety of impacts. Reported positive impacts of the program were grouped into the following themes: increased skills and/or improved wellbeing, normalization of mental health issues, improved communication or relationships, first steps towards improved wellbeing, a beneficial program for PSP, and a good reminder of previously learned skills. Clients' comments about how the course had negative or limited impacts were amalgamated into two themes: the program did not meet the specific needs of the user, and the program made the client feel neglected by the government. Hindering events described by clients fell into the following three themes: timeline challenges, technical difficulties, and negative effects. Table 2 provides an impact summary including frequency of comments for completers and non-completers, as well as client quotes.

Table 2.

Reported impacts and hindering events.

Domain/theme	Example quote	Completers (n = 57)	Non-completers (n = 25)	Total sample (N = 82)
Perceived benefits, n (%)		44 (77)	6 (24)	50 (61)
Increased skills or improved wellbeing	“I'm so glad I signed up for this course. It really has made a big difference in the quality of my day to day life and I feel very optimistic for the future. While many of my symptoms developed in my teens I feel that the work I do contributes to the worsening of those symptoms and lately I had been considering whether it was feasible to do this job until I retired. Now, having done this course, I haven't had those thoughts and feel like I can do this job until I retire without it taking such a toll on my mental health.” #48	37 (65)	2 (8)	39 (48)
Normalizes mental health issues	“Once I began the course, I found myself feeling hopeful and even a little more understood in how I am struggling and the different ways people can struggle with mental health as a PSP.” #211	13 (23)	5 (20)	18 (22)
Improved communication or relationships	“I have found that since beginning this course, the skills I have learned have helped me not only decrease my symptoms but also help with my relationships within my family and friends.” #75	10 (18)	0 (0)	10 (12)
First steps towards improved wellbeing	“I have a better understanding of myself and my symptoms and what I can do to manage them. Even though I am not feeling 100%, I am confident with time and practice I can learn to manage my symptoms long term.” #121	7 (12)	0 (0)	7 (9)
Beneficial for PSP	“I can see how practicing this course would be a benefit to anyone in the first responder world.” #38	7 (12)	0 (0)	7 (9)
Good reminder of skills	“So nothing I've read is new but it's all a good reminder and a good reminder of perspective.” #136	5 (9)	0 (0)	5 (6)
Limited or negative impacts, n (%)		5 (9)	2 (8)	7 (9)
Didn't meet specific needs	“Felt like my problems were outside of this course. Not that the course is bad just need to work in different stuff.” #136	4 (7)	2 (8)	7 (9)
Feel neglected by government	“I am sure that the government is very pleased that they are able to place a check mark next to the box that reads ‘sparing no expense to make sure our veterans’ mental health is being looked after.’” #139	2 (4)	0 (0)	2 (2)
Hindering events		50 (88)	11 (44)	61 (74)
Timeline challenges	“My biggest problem is making the time, the content itself has been useful when I get to it.” #60	45 (79)	9 (36)	54 (66)
Technical issues	“I haven't been able to access the course as I am locked out of the lessons. I still have a lot of time on my hands and would like to complete the course.” #71	17 (30)	5 (20)	22 (27)
Negative effects	“The course just made be deal with the issues head on. There was a bit of a tough section, but that is necessary to be able to heal.” #222	14 (25)	1 (4)	25 (18)

Note. Non-questionnaire completers include 10 clients who chose not to complete the Treatment Satisfaction Questionnaire and 15 clients who withdrew.

The majority of clients (n = 50/82; 61%) made at least one comment describing benefits from the course. Almost all clients who completed the Treatment Satisfaction Questionnaire indicated (in response to a multiple choice, yes/no question) that the course was worth their time (55/57; 97%) and that they would feel confident recommending the course to others (56/57; 98%). Most comments on beneficial impacts came from the completers group (n = 44/57; 77%), yet 24% (n = 6/25) of clients in the non-completers groups made at least one unsolicited comment about the benefits of the course. Notably, clients in the non-completers group most frequently commented that their experience in the course helped them to normalize their symptoms.

Less than 10% of clients (n = 7/82; 9%) reported that the course had little or no impact on their wellbeing. In the completer group, four clients (7%) reported thinking the course was worth their time but believed the course content was not specific to their needs and wanted more personalized content. Two clients (4%) in the completers group expressed mistrust or skepticism towards the program. The two clients described online programs as too general for the complex and varied mental health issues PSP face. They reported that being provided with an online program and being part of research made them feel neglected by the government or their work organization. Two clients (8%) in the non-completers group also stated that the course was not helping them manage their symptoms. Several (n = 7, 28%) clients in this group withdrew from the course to receive a different form of therapy and did not provide comments on the course impacts.

Hindering events affecting clients' progress in the course were reported by 88% (n = 50/57) of completers and by 44% (n = 11/25) of non-completers. These clients reported having difficulties keeping up with an eight-week timeline to complete the course because of their schedules or because of concentration problems. A few clients in the completers group (n = 3/57, 5%) indicated that they found it helpful to know that timelines for accessing the course were flexible but disliked the automatic notification system and the release of materials according to the eight-week timeline. The notifications and automation were described as putting undue pressure on clients to complete the course within eight weeks. Clients in both groups reported having technical difficulties, including issues with logging into the PSPNET web application, at similar rates.

Some clients in the completers group (n = 14/57, 25%) and one client in the non-completers group reported experiencing negative effects from working on the course, such as increased or novel symptoms. The clients reported that working on the course and directly focusing on their symptoms and experiences increased their symptoms. Yet, all of the clients in the completers group who experienced negative effects suggested the course skills helped them to cope with their increased symptoms, or that the negative effects decreased as they progressed throughout the course. Two clients reported that their increased symptoms were expected or necessary for them to recover. No clients reported that negative effects impacted their overall experience in the course; however, one client from the non-completer group withdrew from the course, reporting a need for face-to-face therapy. Client comments about transient symptom increases that were not directly related to working on the course were excluded from analyses.

3.4. Helpful and challenging course skills and content

Table 3 provides a summary of helpful and challenging content. Clients made comments about finding diverse skills helpful. Overall, thought challenging was the skill most frequently cited as helpful. However, recognizing triggers and cycles of symptoms was commented on at a higher rate than thought challenging within the non-completers group. Few (n = 4/25; 16%) clients in the non-completers group made comments about skills that were introduced beyond Lesson 2. Clients in the non-completers group also made few comments (n = 4/25; 16%) on finding specific skills from the additional resources helpful, whereas about 30% (n = 17/57) of clients in the completers group commented on finding at least one specific additional resource helpful. Client comments on challenging content reflected questions about or struggles with implementing the skills rather than not finding them helpful; however, two clients in the completers group reported the thought challenging skill was not helpful. Clients in the non-completers group made no comments about finding skills challenging beyond Lesson 2. Among both groups, there were more comments about finding skills and resources helpful than there were about finding them challenging. Two clients in the completers group but no clients in the non-completers group made comments about finding the skills within the additional resources challenging.

Table 3.

Helpful and challenging skills.

Domain/theme	Example quote	Completers (n = 57)	Non-completers (n = 25)	Total sample (N = 82)
Helpful lesson skills, n (%)		53 (93)	10 (40)	63 (77)
Thought challenging	“The thought challenging has been the best as I have learned to be more aware of the thoughts, how they affected me and the need for a change in the outcome.” #77	43 (75)	4 (16)	47 (57)
Recognize cycle of symptoms and triggers	“By filling out the cycle of symptoms worksheet I was able to see that my thoughts, symptoms, and behaviours were all connected in a way and that there are ways to cope with them that I will learn throughout this course.” #75	35 (61)	6 (24)	44 (54)
Controlled breathing	“I used breathing techniques after a scary call that helped otherwise I might have gone home.” #173	25 (44)	1 (4)	26 (32)
Activity scheduling	“I worked on daily walks or other activities which I can already tell has helped my motivation.” #208	25 (39)	1 (4)	26 (32)
Graduated exposure	“Graded exposure therapy helps out a ton for me and learning all the new skills to go along with it.” #65	14 (25)	2 (8)	16 (20)
Relapse plan	“I have read through lesson 5 and am looking forward to making a relapse prevention plan and goal setting. I was glad to see it was a part of the course as I wasn't sure how to approach wellness maintenance. #48	5 (9)	0 (0)	5 (6)
Helpful resources, n (%)		17 (30)	3 (12)	20 (24)
Anger	“I have found the section on anger to be very helpful.” #186	6 (11)	0 (0)	6 (7)
Beliefs	“I found managing beliefs helpful.” #52	6 (11)	0 (0)	6 (7)
Communication and assertiveness	“I am still working on Assertiveness in communication, but the DIY Guide has been effective for my communication.” #106	4 (7)	0 (0)	4 (5)
PTSD	“I still use the grounding exercise from the additional resource to be the most helpful skill in the course so far.” #75	4 (7)	1 (4)	4 (5)
Worry	“With controlled breathing and setting aside worry time not before bed my thoughts prior to bedtime haven't been racing which has allowed me to get to sleep quicker.” #99	4 (7)	0 (0)	4 (5)
Relationships	“I read the additional resources on relationships with significant others and it was spot on! It explained so many things I have been questioning about why I do and why I react the way I react and it helped so much!” #249	3 (5)	1 (4)	4 (5)
Sleep	“I've also found the extra resource on sleep interesting. I've heard a lot of the info before but am encouraged to put it in to practice.” #99	2 (4)	0 (0)	2 (2)
Grief	“I look forward to trying to apply this to my own life and just actively thinking about the idea more I feel has helped me start doing this somewhat naturally when I am struggling with grief.” #211	1 (2)	0 (0)	1 (1)
Problem solving	“Lesson 2 I have been working on a bit over the last week along with belief challenging and Problem Solving. These are all tools I will find helpful going forward.” #99	1 (2)	1 (4)	1 (1)
Challenging skills, n (%)		33 (58)	5 (20)	38 (46)
Thought challenging	“Remembering to use Thought Challenging. I will keep practicing.” #38	27 (42)	3 (4)	30 (37)
Graduated exposure	“Exposure can be hard to do and set up. Just thinking about it can bring on a bit of anxiety.” #226	16 (28)	0 (0)	16 (20)
Recognizing cycle of symptoms and triggers	“Just slight difficulty in reflecting on symptoms and how my thoughts lead to the behaviours and physical symptoms.” #208	11 (19)	3 (4)	14 (17)
Controlled breathing	“I have been trying the controlled breathing. Right now I'm struggling with the “3 min” portion as it always feels much longer than it actually is, which I would assume is a good thing.” #38	7 (12)	0 (0)	7 (9)
Activity scheduling	“I am working towards goal setting and daily routines, which given my work schedule has been very difficult. I find it very frustrating the days I attempt a routine and get called in.” #204	7 (12)	0 (0)	7 (9)
Relapse plan	“It's hard to work on a relapse plan when I just can't figure out my triggers but I will figure it out!” #208	3 (5)	0 (0)	3 (4)
Challenging resources, n (%)		2 (4)	0 (0)	2 (2)
Beliefs	“My difficulty was not so much with the lesson but with the beliefs additional resource. It applied to me very much, and like the resource says, it is difficult to challenge beliefs.” #208	2 (4)	0 (0)	2 (2)
Anger	“The Anger resource is one I have read, but again hard to focus on. I rarely let my anger out, I keep it bottled up. I am thinking if I can overcome the feelings of guilt that the resentment and anger will be corrected. Not sure if that is the way to approach this?” #106	1 (2)	0 (0)	1 (1)

Note. Non-questionnaire completers includes 10 clients who chose not to complete the Treatment Satisfaction Questionnaire and 15 clients who withdrew.

3.5. Helpful aspects of course and areas for improvement

Findings from the qualitative analyses indicated that clients found the following aspects of the course helpful: therapist check-ins and communications, the course design and format, DIY guides, stories, additional resources, and the convenience and flexibility. Some clients (n = 22/82, 27%) even reported that the course did not need any improvements. Feedback offered for enhancing the course included suggestions that fell within the following themes: general comments on improving course design and materials, improving case stories, improving resources or including more topics in resources, adjustment of timelines, including audio or video, addressing technical issues, and increasing contact or accountability. Table 4 provides a summary of helpful aspects of the course and suggestions for improvement.

Table 4.

Likes and suggestions for improvement.

Domain/theme	Example quote	Completers (n = 57)	Non-completers (n = 25)	Total sample (N = 82)
Likes, n (%)		55 (96)	5 (20)	60 (73)
Therapist check-ins and communications	“I enjoyed the one-on-one counselling. My assigned counsellor was very encouraging and always had a suggestion if I needed one.” #38	31 (54)	5 (20)	36 (44)
Course design and format	“I liked how the course was laid out with the lesson first and then I did the DIY Guide right after.” #75	33 (58)	0 (0)	33 (40)
DIY guides	“I really like the DIY guide as I can put some things into practice and it's a good resource to look at throughout the week.” #30	28 (49)	1 (4)	29 (35)
Stories	“The stories were good as they made me feel more normal.” #209	19 (33)	2 (8)	21 (26)
Additional resources (general comments)	“All of the Resources were helpful and I will refer to them in the future.” #251	19 (33)	1 (4)	20 (24)
Convenience and flexibility of access	“It was good to be able to do it on my time and in my own home. Helps with the comfort of reaching out for help while still being in my own safe zone.” #83	12 (21)	1 (4)	13 (16)
Suggestionsa, n (%)		25 (44)	0 (0)	25 (30)
No suggestions	“No suggestion here. I see it as a very good program as it is.” #155	22 (39)	0 (0)	22 (27)
Course design and materials (general)	“Sometimes I thought the lessons were over repetitive. Scrolling through many pages before there was much added content. Sometimes this helped reinforce a specific point, but could be condensed by a few pages.” #106	19 (33)	0 (0)	19 (23)
Improve case stories	“I did not find the stories to be that useful to me as I found it hard to read about other people's symptoms and stories as I found that I was just comparing myself to them in an unhelpful way.” #75	11 (19)	0 (0)	11 (13)
Improve resources or include more topics	“More resources.” #186	6 (11)	0 (0)	6 (7)
Timelines	“I found the time frame to be challenging even though a couple of the lessons were two weeks at a time.” #52	5 (9)	0 (0)	5 (6)
Include audio and video	“It would have been nice if the lesson had, “Read to” button. I found myself “skipping” over things. IF the lessons read to me I may have listened a bit more closely. I struggle with reading too.” #51	4 (7)	0 (0)	4 (5)
Technical issues	“Also on a technical stand point there were often times the site wouldn't save my progress or my email and I'd end up starting all over.” #173	4 (7)	0 (0)	4 (5)
Increased contact or accountability	“I think having some expectation to submit each of the exercises would give the student and the assigned therapist more springboards for discussion.” #87	2 (4)	0 (0)	2 (2)

Note. Non-completers include 10 clients who chose not to complete the Treatment Satisfaction Questionnaire and 15 clients who withdrew.

^aTotals reported in this row do not include clients who commented that they had no suggestions.

Feedback regarding what clients found helpful about the program came primarily from clients who completed the Treatment Satisfaction Questionnaire, who were directly asked for input on the course. Among this group, 96% (n = 55/57) made a comment about liking at least one aspect of the course. Yet, in the non-completers group, 20% (n = 5/25) described liking at least one aspect of the course without being asked directly. Therapist check-ins were one of the most frequently cited helpful aspects of the course among both groups.

All of the clients who offered suggestions for improvement came from the completer group, primarily through responses to questions in the Treatment Satisfaction Questionnaire. A few suggestions (e.g., on the need for audio or improving the sleep resource) from the completer group were made in the weekly reflections and emails. While 33% (n = 19/57) of clients in the completer group shared that they liked the stories and additional resources, 19% (n = 11/57) requested further tailoring of case stories, and 11% (n = 6/57) described a need for improving additional resources and topics (e.g., adding information to the original sleep resource) and adding new ones. No clients reported disliking the therapists or feeling a lack of support from or connection with therapists.

3.6. Using data for quality enhancement

Client suggestions for improving the course were diverse and often inconsistent; for example, some clients reported liking the stories, while other clients described the stories as unhelpful. Nonetheless, we followed the LHS approach and used the results from analyzing clients' feedback to further tailor the PSP Wellbeing Course (see Table 5 for a summary of changes suggested and made).

Table 5.

Improvements made.

Suggestions	n	Description of change	Level of change	Change status
Course design and materials (general), n (%)	19
		Change colour, size, and font style on some slides and additional resources.	Revise content	Complete
		Add graphs showing symptom changes to further help clients track their symptoms.	New content	Complete
		Add in activity on self-awareness to lesson one.	New content	Complete
		Requests to download lesson slides.	Revise Delivery	Under Discussion
		Requests to change repetitiveness/lengthiness of materials.	Revise Content	Under Discussion
Improve case stories	11
		Add client quotes about their experiences with the lesson to the landing page of each lesson and to automated emails.	New content	Complete
		Add new case stories and or examples.	Revise content	Partially complete
Improve resources or include more topics	6
		Revise sleep resource.	Revise content	Complete
		Add examples of how to use course skills to manage wellbeing during COVID-19.	New content	Complete
		Add new additional resources including: anger, pain, grief, alcohol change, and a resource for family or friends.	New content	Complete
		Add mindfulness resources including information and meditation recordings to the course.	New content	Complete
Timelines	5
		Add reminders about flexibility of the timelines to emails.	Revise content	Complete
		Monitor the appropriateness of a 16-week timeline and if process of delivering content needs to be revised.	Revise delivery	In progress
Include audio or video	4
		Add audio version of each lesson.	New content	Complete
		Add video summaries of each lesson.	New content	Complete
Technical issues	4
		Add an email notification to the client's external email saying there is a message from PSPNET for them in the client portal.	Technical	Complete
		Add an email notification for clinicians to notify them if a staff member or client has sent them an in-system message.	Technical	Complete
		Added in document aid outlining the account creation process.	New content	Complete
		Add a feature to save clients progress in the lesson slides.	Technical	Complete
Increased contact or accountability	2
		Clarify process for increasing to twice per week contact with clients. Offer additional support at week three depending on client level.	Revise delivery	Complete

4. Discussion

There is increasing recognition worldwide that PSP's mental health needs are going unmet (e.g., Duff et al., 2020; Public Safety Canada, 2019). ICBT has been proposed as one solution to better support PSP mental health needs in Canada (Public Safety Canada, 2019). There is ample research on ICBT for the general population but little research on how PSP respond to ICBT. PSP have unique work experiences (e.g., repeated exposures to PPTE) (Carleton et al., 2019, Carleton et al., 2020), which PSPNET has attempted to address by tailoring ICBT with PSP specific examples. Qualitative research may help iteratively tailor ICBT for PSP. The current study was designed to examine in-program communication and responses to the Treatment Satisfaction Questionnaire administered at eight weeks post-enrollment. The examination was used to evaluate client perceptions of how the PSP Wellbeing Course impacted their lives, what clients found helpful or wanted improved about the course, and what specific skills they found helpful or challenging.

4.1. Implications of client feedback

Clients' comments suggested that most clients perceived the PSP Wellbeing Course as beneficial and that they had more successes than challenges when practicing various skills throughout the course. The findings suggest that ICBT is suitable for skill development for PSP and normalizing experiences with mental health. This is important because prior research indicates that PSP want to learn new coping skills (McCall et al., 2020a) and experience substantial workplace stigma related to mental health (Krakauer et al., 2020; Ricciardelli et al., 2020).

Consistent with previous research (e.g., Richards et al., 2016), some clients appreciated the course but felt they needed a different form of treatment, suggesting that ICBT is not suitable for all clients. Facilitating clearer expectations about the program among prospective clients could help redirect clients who are not suitable for ICBT to other services; however, clients may also need to explore a program before knowing if the program will meet their needs. A few clients expressed skepticism towards the course based on the government sponsorship and the research orientation, which highlights the importance of developing trust with PSP populations when developing digital mental health interventions. Like other programs (e.g., Rozental et al., 2015), some clients reported negative effects, highlighting the importance of therapists reminding clients that increased or novel symptoms are normal and part of an ultimately beneficial process.

Even though clients consistently reported having difficulties completing the course within the initial eight-week timeframe, most clients who accessed all five lessons did so within eight weeks, and only a few clients suggested the timeline needed adjusting (see Table 1, Table 4). Only one client completed the lessons after eight weeks, suggesting that an extended timeline may not solve timeline challenges. Therefore, the PSPNET team is continuing to monitor whether having the option of a 16-week timeframe as suggested by our stakeholders (McCall et al., 2020a) is necessary because previous research indicates that deadline flexibility is inversely associated with ICBT outcomes (Paxling et al., 2013). The PSPNET team is also continuing to monitor whether there is a better timeline for releasing materials to clients (e.g., waiting until clients complete previous lessons before releasing new lesson materials) to address comments about undue pressure from the notification system.

Clients had more positive feedback than suggestions for improvement for the PSP Wellbeing Course and their responses to the program were similar to responses from clients who took the original version of the Wellbeing Course (Hadjistavropoulos et al., 2018). The positive feedback clients provided regarding their engagement with PSPNET therapists was encouraging because prior research indicates that PSP distrust mental healthcare providers or are concerned that providers lack sufficient knowledge and understanding of PSP (McCall et al., 2020a). The overall positive feedback suggests that the PSP Wellbeing Course was appropriately tailored and well-received by PSP.

4.2. Principal findings from the treatment satisfaction questionnaire non-completer group

Completers and non-completers were compared in the analyses to explore whether there were any differences in experiences between the two groups for the purpose of guiding insights for program enhancement. Most non-completers accessed only the first few lessons (see Table 1), so results from non-completers generally reflect only their initial impressions of the course. The relative absence of suggestions for improvement may have been due to the fact that we only solicited suggestions from clients during the Treatment Satisfaction Questionnaire. This suggests a possible need for soliciting feedback about user experiences earlier on in programs. Nonetheless, by including data from non-completers we were able to gather some insights into their experiences that would not have been captured had we only analyzed data from clients who completed the Treatment Satisfaction Questionnaire. Given the limited data provided by non-completers, separating completers from non-completers in the analyses allowed us to capture initial perceptions and impacts related to the course among both groups without diluting overall response rates of completers.

Given that most clients in the non-completers group stopped engaging after the first two lessons, clients may benefit from additional engagement early on in the course to encourage completion. Nonetheless, the results were consistent with prior evidence that many non-completers of ICBT benefit from their partial engagement (Hilvert-Bruce et al., 2012). Among the non-completers, the most frequently cited benefit was that the course improved their awareness about mental health issues and helped to normalize how common mental health concerns are among PSP. This suggests that even partial completion of the course can help to reduce stigma surrounding mental health concerns, which is a frequently cited concern impeding PSP from acknowledging their mental health concerns (Krakauer et al., 2020). Additionally, a number of non-completers expressed satisfaction with the therapist support they received. Thus, not having adequate therapist support may not explain why clients did not complete the course or the Treatment Satisfaction Questionnaire.

4.3. Applying suggestions for improvement

Decisions concerning which suggested changes to implement were largely related to the PSPNET team's capacity to make each change, rather than the frequency with which each change was suggested. Suggestions that came from even one client were sometimes implemented if the changes were not onerous (e.g., changes to font sizes and colour based on feedback from one client). Our approach illustrates how singular comments can be used to make program adjustments in the pursuit of excellence (Patton, 2014). Some suggested changes are currently under discussion (e.g., requests to download slides as the course was not originally designed to have downloadable slides). Adapting digital technologies drawing on user feedback throughout a clinical trial is an accepted and encouraged practice (e.g., Graham et al., 2020; Mohr et al., 2017). The current study provides an example of how to systematically use qualitative research to make iterative reflexive improvements to a program (see Table 5).

Many changes made to the PSP Wellbeing Course were consistent with or guided by Oinas-Kukkonen and Harjumaa's (Oinas-Kukkonen and Harjumaa, 2009) persuasive systems design (PSD) framework, which consists of 28 design principles to facilitate greater user engagement for a program or intervention. For example, providing clients with graphs displaying their symptom change over time is consistent with the PSD framework principle of self-monitoring. Likewise, providing clients with email notifications when they receive messages in the client portal is consistent with the PSD framework principle of reminders, and sharing clients' quotes is consistent with several PSD framework principles, including social learning. The alignment of our changes with PSD framework principles is noteworthy given the growing research on persuasive design in eHealth interventions and emerging evidence that persuasive design predicts efficacy among ICBT interventions (McCall et al., 2021).

4.4. Limitations and future research

Conducting interviews may have been a better approach for collecting data to determine course perceptions from the non-completers. Future research could also systematically collect information from therapists in order to gather additional data on clients who do not complete the Treatment Satisfaction Questionnaire. The current study is also limited by the sample size and relative homogeneity of the sample. The current study clients were from similar occupations (primarily policing and paramedicine) and predominantly identified as white. More research is required to determine how clients from other occupational or demographic backgrounds perceive the impacts, strengths, and weaknesses of the PSP Wellbeing Course. Lastly, it is possible that client comments in the Treatment Satisfaction Questionnaire, administered at eight weeks post-enrollment, reflect a response bias (e.g., courtesy bias) and do not accurately reflect their experiences using the program. However, a strength of this study is that client communications throughout the program were examined, thus, capturing client's experiences as they occurred rather than solely through reflection.

Additional further research is needed to inform practices for tailoring ICBT to specific populations. There have been previous recommendations to use feedback to iteratively and reflexively tailor programs; however, the literature provides little guidance on how to approach tailoring. The current study provides a preliminary example of how feedback can be used to systematically improve an intervention for a specific population. Future mixed-methods research is needed to examine the impact of program improvements on patient engagement and outcomes over time (e.g., exploring if there are fewer drop-outs, fewer dislikes, or better outcomes after improvements are made; Mohr et al., 2017).

Other approaches to analyzing client communications with therapists may also help improve programs. For example, examining client communications for the types of questions clients ask therapists could provide insight into client challenges with the course and reveal further areas for improvement (e.g., Soucy et al., 2019). Examining client communications to identify how clients use therapist support (e.g., rapport building or client venting; Soucy et al., 2018) can show ways to improve support. Future research may also examine how and if clients are using the program with other forms of mental health care. Finally, it may be beneficial to systematically study whether ICBT outcomes vary with programs of different lengths.

5. Conclusion

Canadian PSP experience higher rates of mental health problems than the general population and report multiple barriers to receiving mental healthcare. The current study examined feedback from clients of the PSP Wellbeing Course, a transdiagnostic ICBT course tailored to PSP. The study identified how the PSP Wellbeing Course impacted clients, what clients found helpful, and what improvements clients recommended. The results suggest that most clients found ICBT to be beneficial, especially in terms of developing skills to improve wellbeing. The skills most commonly cited as helpful were thought challenging and increasing awareness of the relationships among thoughts, behaviours, and physical sensations; however, at times the same skills were described as challenging. Negative effects were rare and did not appear to impact clients' overall experiences of the course. The greatest challenge encountered by clients with respect to the PSP Wellbeing Course appears to be meeting the course timeline, yet clients typically still completed the course within the timeframe. Clients valued therapist assistance, content, and activities, with their likes outweighing their dislikes. The course materials and stories of PSP were valued but were also areas that clients commonly identified as needing improvement. In general, clients who did not complete Treatment Satisfaction Questionnaire provided little information during the program regarding their perceptions of ICBT. The available information suggests some of the clients who did not complete the course still experienced benefits, and limited time spent using the course appeared to be a factor hindering their progress. Clients who did not complete the program also presented with greater symptom severity; as such, PSPNET therapists might consider following up with clients who seem less engaged early in the course to help encourage them to continue using the course. We used an LHS approach to iteratively and reflexively incorporate client feedback into the PSP Wellbeing Course with most attention given to making adjustments to the additional resources. The current study and results may help inform other groups who are developing and providing mental health services to PSP.

Declaration of competing interest

Titov and Dear are authors of the original Wellbeing Course but received no financial benefit from its use in this study. The other authors have no conflicts of interest to declare.

Appendix A. Supplementary data

Treatment Satisfaction Questionnaire