Digital mental health intervention for schizophrenia spectrum and psychotic disorders: Protocol for a pragmatic feasibility study of Horyzons-Canada

Abstract

Background

Schizophrenia spectrum and other psychotic disorders (SSPD) are among the most debilitating of all mental disorders. While the evidence for psychosocial interventions such as cognitive behavioral therapy and peer support has significantly improved, access to these services remains limited. This paper describes a protocol for a pragmatic feasibility study of a digital mental health intervention (HoryzonsCa) that provides access to evidence-based psychosocial interventions, social networking, and clinical and peer support services through a secured, web-based platform for adults diagnosed with SSPD.

Objective

The objectives are: (1) Adapt and translate HoryzonsCa for implementation in English and French; (2) Develop an implementation and training strategy; (3) Assess the acceptability, safety, and demand of HoryzonsCa; (4) Assess clinical outcomes and perceived impacts; (5) Examine the experiences and process of adapting and implementing HoryzonsCa; (6) Explore the role of sociocultural and demographic factors on HoryzonsCa outcomes and implementation.

Methods

This feasibility study will use a single-group, pre-post, mixed-methods (QUAN–QUAL convergent) research design, with assessments at baseline and 12 weeks. The study aims to recruit 100 individuals (ages 18–50) diagnosed with SSPD from two healthcare settings in Canada. Data collection includes interview-based psychometric measures, self-reports, focus groups, and interviews with participants. The study will also collect qualitative data from moderators and the research team, and will be conducted entirely remotely.

Conclusions

This study has been prospectively registered and is underway. It will provide timely information on the feasibility and potential impacts of using digital mental health services for individuals with chronic mental health conditions.

Trial Registration

ISRCTN12561259; https://doi.org/10.1186/ISRCTN12561259 (250/max 250 words)

Introduction

Schizophrenia spectrum and other psychotic disorders, ¹ referred to hereafter as SSPD, are among the most debilitating of all chronic and severe mental health conditions.^2–4 Individuals affected by these conditions are particularly vulnerable to the adverse impacts of COVID-19.^2,5–9 SSPD have a lifetime prevalence of 3% ¹⁰ and include symptoms such as hallucinations, delusions, disorganized thoughts and behaviors, poverty of thought and affect, apathy, as well as deficits in verbal memory and executive functioning.^1,11,12 The onset of SSPD is typically between the ages of 15 and 25, often leading to significant impairments in social and occupational functioning and derailing critical transitions towards life goals. Research also shows that both sex (i.e., the biological attributes that distinguish male from female) and gender (i.e., socially constructed roles, identities, and behaviors) influence the onset, persistence, and recurrence of SSPD, as well as attitudes and responses towards treatment.^13–15

The most serious implications associated with SSPD include reduced life expectancy, high rates of concurrent mental health problems (e.g., substance abuse, severe social anxiety, depression, suicide, suicide ideation and behavior), increased physical health problems (e.g., cardiovascular and type II diabetes), and a deteriorating trajectory in functioning (e.g., unemployment, legal issues).^16–23 Thus, SSPD are consistently ranked among the top leading causes of disability in the world ⁴ despite pharmacological advances. Moreover, these disorders continue to impose an enormous strain on individuals, families, and communities, costing countries billions of dollars each year.^24–26

In recent decades, evidence has accumulated on the effectiveness of psychosocial interventions (e.g., cognitive behavioral therapy, psychoeducation, and peer support) for treating SSPD. However, universal access to these interventions remains challenging.^27–30 Several factors contribute to why psychosocial interventions are at the treatment periphery for this population, including (but not limited to): focus on acute inpatient services, limited availability of highly trained professionals, and related human resource costs^27,31 mainly because they are predominantly offered using traditional models of service delivery (live, in-person, individual-based format), which have limitations for scalability. Furthermore, the COVID-19 restrictions (e.g., limits on non-essential visits to health care settings) exposed the limitations of the mental health care system and disproportionately impacted this population ⁸ and young people overall. Thus, innovation is needed to address the mental health needs and critical treatment gap faced by this population.

Leveraging technology to improve access to psychosocial interventions

There is emerging evidence of the potential of digital interventions in supporting clinical and social outcomes for individuals living with SSPD. ³¹ For example, one of the first systematic reviews ³¹ investigating the evidence regarding the acceptability, feasibility, safety and benefits of online and mobile-based interventions for individuals diagnosed with SSPD showed that the majority (i.e., 70% or more) used the web-based interventions efficiently, perceived them as positive and useful, and were engaged with the interventions over the follow-up. Findings indicated that online and mobile-based interventions show promise in improving positive psychotic symptoms, hospital admissions, socialization, social connectedness, depression and medication adherence. Scholars concur with these findings and indicate the continued need for more extensive research on specific digital interventions for individuals coping with SSPD.^32,33

In terms of access to and use of technology in this population, one of the first studies with young individuals receiving services for first-episode psychosis (FEP) found that the majority had access to cellphones (88%, most of which were smartphones) and laptop computers (70%), and more than half had searched the Internet for mental health information. ³⁴ Another study with 71 participants (age range: 17 to 39) with FEP found that the majority reported frequent use (i.e., daily to three times per week) of computers (96%), cell phones and smartphones (70%); and many (68%) reported regular use of social media (i.e., daily or weekly). ³⁵ Research has also shown that the majority (85%) of young adults receiving treatment for psychosis expressed interest in using the Internet, social media, and mobile technologies for receiving mental health services. ³⁶ A third study with 51 adults (ages 18 to 38) with FEP recruited from a Canadian hospital center had similar results in technology access and use. ³⁷ In terms of older individuals and those diagnosed with more chronic forms of SSPD, a study with 457 adults (median age: 41.3) reporting a diagnosis of schizophrenia showed that the majority had access to at least one technological device and that they used web-based technologies to receive support from family and friends, to identify coping strategies to manage symptoms, and to assist with self-care (e.g., medication reminders). ³⁸

HoryzonsCa: A digital mental health intervention to increase access to psychosocial interventions and supports in the treatment of SSPD

There is limited implementation research on specific digital mental health interventions for individuals coping with SSPD. This type of research is needed to identify effective, sustainable, and scalable interventions in real-world settings. ³⁹ Moreover, previous research has shown that interventions adapted to the contextual needs of another population (e.g., considering sociocultural and demographic factors such as culture, language, age, and gender) can yield better outcomes than providing interventions without adaptation.^40–43 As such, scholars have highlighted the importance of considering implementation factors at the beginning stages of researching a digital mental health intervention rather than leaving it as a post hoc consideration. ³⁹

Horyzons is a digital mental health intervention, originally developed in Australia through participatory design methodologies in close collaboration with young people receiving specialized services for FEP, to support the process of recovery. ⁴⁴ Horyzons is a web-based application comprising interactive, evidence-based-strengths-focused psychosocial interventions, online social networking, and clinical and peer support moderation. It is powered by an adaptive platform that can host multiple sites and therapy content can be tailored to provide treatment to a variety of cohort groups in relation to clinical condition, severity of illness, age, and various other factors such as culture, geographical location, and health care setting.^44–49 Horyzons was first piloted in Australia with 20 young adults with FEP over one month.^44,50 The findings showed significant reductions in depressive symptoms. ⁴⁴

Subsequently, a randomized controlled trial (18-month, parallel-group, single-blind) examined Horyzons’ impact on recovery following discharge from specialized services over 18 months with 170 participants (mean age 21, SD 2.9). Participants were allocated to the Horyzons + Treatment as Usual (TAU) group (which typically consists of general outpatient psychiatry follow-up) or (TAU) group (including information on free e-mental health resources). Over 70% of the participants completed follow-up. Participants in the TAU group were twice as likely to visit emergency services or experience hospital admissions during their follow-up. Participants in the Horyzons group had a 5.5 times greater increase in their odds of finding employment or enrolling in education from baseline to 18 months compared with those in the TAU (OR = 5.5, 1.09–28.23), and almost half sustained engagement in the intervention over several months. ⁵¹ A recent economic evaluation demonstrated that Horyzons was both cost-effective and cost-saving with evidence of a dose–response effect (the more young people used the platform the higher the cost-savings). ⁵² While these results are promising, it is important to note that the RCT did not find evidence for the impact of Horyzons on several outcome variables that the platform targets, including social functioning, depression, social support, and loneliness. ⁵¹ The varying outcomes across Australian studies on Horyzons in patients with FEP indicate the importance of ongoing research on the intervention for this clinical population.

Horyzons has also been adapted and piloted in Canada. The first study explored what adaptations would be needed to prepare the digital health intervention for implementation and testing in Canada.^53,54 In the context of this preliminary research, the authors defined digital health adaptation as “a systematic, purposeful, and collaborative process of making changes to a digital health innovation to increase its relevance and acceptability for a local community” and developed an adaptation framework to guide the process of digital health adaptation, informed by the literature on adapting behavioral and psychosocial interventions (in-person-delivery).^55,56 Based on the perspectives of patients and service providers recruited from two Canadian early intervention programs for psychosis, the authors identified a range of adaptations needed to prepare Horyzons for testing in Canada, including, for example, adjustment of therapy content on employment and local information on health services; modification of safety protocols and moderation features to fit the local health care context; and adjustment of language where needed. ⁵⁴ The adapted version of the intervention is called Horyzons-Canada (short form: HoryzonsCa).

Subsequently, HoryzonsCa was piloted with young adults with SSPD^57,58 to evaluate the feasibility of implementing and evaluating it in a Canadian context and its potential benefits using an uncontrolled single-group, pre-post (8 weeks), mixed-methods (QUAL–QUAN convergent) design. The recruited sample included 23 young adults with FEP (52% female, ages 18 to 36). Participants logged onto the site an average of 7.10 times over the 8-week follow-up period (SD = 7.30; Mdn = 5.00; range = 0–30), with 65% logging into HoryzonsCa four times or more. The majority reported a positive experience with HoryzonsCa (85%) and perceived it as easy to use (95%). Preliminary pre-post outcome analyses estimated by linear mixed models showed a nonsignificant increase in social functioning and no deterioration on the Clinical Global Impression Scale from baseline to 8 weeks follow-up. There were no adverse incidents, and 90% perceived the platform as safe and confidential.

Rationale for a larger pilot study, conceptual framework, objectives, and hypotheses

While the international body of research on the impact and implementation of Horyzons with a population receiving treatment for psychotic disorders is promising, it remains specific to individuals in the early course of treatment and with access to specialized early intervention services. Additional research, in different contexts, and with larger and diverse samples is required to understand the feasibility of scaling its implementation and evaluation of this intervention in real-world settings, as well as its effects and impact. For example, is it feasible to implement the intervention with those diagnosed with more chronic forms of SSPD alongside those that are earlier in the course of illness, what factors need to be considered when the intervention is implemented with diverse populations (e.g., individuals representing ethnic minority groups or limited educational backgrounds), what are the barriers and facilitators when implementing and evaluating this intervention in a bilingual healthcare context, and if provided with longer duration of access, how to characterize engagement on the platform (e.g., what is considered minimal exposure) and treatment effect sizes.

This project therefore aims to scale up our feasibility study on HoryzonsCa to a larger group of patients (100 individuals) with SSPD receiving services in two Canadian healthcare settings. The rationale for scaling up the pilot study is to more closely reflect implementation in a real-world setting. For example, rather than restricting recruitment to individuals receiving specialized services for FEP and to only a younger age group (as in previous Horyzons research), the study will provide insights on the feasibility of providing this service to individuals across the continuum of treatment for psychotic disorders. We also aim to gain insights on the feasibility of delivering the implementation over a longer duration, in a bilingual context (i.e., in English and French), and the implications this has on user engagement and human resources and capacities. The larger pilot will also provide knowledge on our data collection procedures (e.g., efficiency, participant issues), which will, for the first time, be conducted entirely remotely. The longer duration will allow us to better understand engagement patterns over time, and more time for exposure to the intervention, which includes contact with peers and the moderators. This is particularly relevant as our previous research showed limited use of the social networking features, which may have been due to the small number of participants recruited to the platform at the same time. ⁵⁸ A larger sample size will also allow for an increase in sample diversity, potential to identify outliers of engagement and use, and statistical power (thereby reliability of results).

The conceptual frameworks that informed our research objectives, design, and methods are: (1) the Readiness Assessment for Pragmatic Trials Model: ⁵⁹ (2) the Technology Acceptance Model,^60,61 a well-established model that has been used in digital health research that suggests users may or may not accept a technology, depending on a variety of factors including perceptions of the usefulness and ease of use of the innovation. Personal and organizational factors have also been considered in adaptations to this model ⁶² ; and (3) the Bowen Feasibility Framework, ⁶³ a well-cited and practical framework that proposes eight key aspects to address in feasibility studies: adaptation, practicality, integration, acceptability, demand, potential efficacy, implementation, and expansion. This protocol is informed by the SPIRIT 2013 guidance for protocols of clinical trials, ⁶⁴ and key elements of the methodology are described in the following sections.

Our six objectives and related hypotheses (where applicable), informed by Bowen's focus on feasibility studies ⁶³ and Sex and Gender Equity in Research guidelines, ⁶⁵ are:

1. Adapt and translate HoryzonsCa for implementation in a bilingual health services context for adults.

2. Develop an implementation and training strategy for clinicians, managers, and peer moderators to support the integration of HoryzonsCa in real-world settings.

3. Assess the acceptability, safety, and demand of HoryzonsCa.

   Primary hypotheses: We hypothesize that HoryzonsCa will be acceptable to patients, defined by: (a) at least 70% will provide positive self-reports on the general experience of the platform, perceived usefulness, ease of use, and safety; (b) there will be no adverse reactions from the use of HoryzonsCa from baseline to 12 weeks follow-up; and, (c) at least 70% agree or strongly agree with perceived safety and confidentiality of HoryzonsCa.

   We will also explore acceptability, safety, and demand through website usage analytics and semi-structured interviews to generate a better understanding of patients’ and moderators’ use, perceptions and experiences of HoryzonsCa (e.g., perceived usefulness, ease of use), and barriers and facilitators of the intervention for patients and moderators.

4. Assess benefits of HoryzonsCa on psychosocial recovery based on clinical outcome measures and participant perceptions of perceived impact.

   Secondary hypotheses: We hypothesize that patients using HoryzonsCa will show improvements on secondary outcome measures of symptoms of psychosis, social functioning, social support, loneliness, depression, anxiety, self-esteem, and psychological well-being from baseline to 12 weeks follow-up. These outcomes are based on areas targeted by the platform, including those that have shown improvements in previous Horyzons research.

5. Examine the experiences and process of adapting, implementing, and evaluating HoryzonsCa in two healthcare settings.

6. Explore the role of sociocultural and demographic factors on the implementation of HoryzonsCa and related outcomes.

Methods

Study design

This feasibility study will use a single-group, pre-post, mixed-methods (QUAN–QUAL convergent) research design, ⁶⁶ with assessments at baseline and 12 weeks. The trial was registered prospectively: ISRCTN12561259, https://doi.org/10.1186/ISRCTN12561259.

Participants

The study will recruit English and French-speaking patients (ages 18 to 50) receiving follow-up care in the outpatient clinics of CIUSSS de l’Ouest-de-l’Île-de-Montréal and CHUM. Representation of participants regarding gender and other sociocultural and demographic factors based on SSPD literature and the 2016 Canadian Census ⁶⁷ will be ensured. Other inclusion criteria are: diagnosis of SSPD (affective or non-affective psychosis); symptomatically stable and capable of interacting online as judged by their primary clinician; low or at most moderate suicidal risk measured by the Ask Suicide-Screening Questions (ASQ) Toolkit ⁶⁸ and based on responses to item 4 (suicidality) on the Brief Psychiatric Rating Scale (BPRS), version 4; ⁶⁹ and agreement to nominate an emergency contact. Exclusion criteria are individuals diagnosed with antisocial and/or borderline personality disorder, intellectual disability, hospitalization at the time of recruitment, and inability to speak or read English or French. The rationale for excluding individuals with antisocial and/or borderline personality disorder is based on the intention to minimize the risk of potential disruptions to group dynamics and to participant safety. Moreover, given that the moderators will be learning how to moderate this intervention for the first time, we cautiously exclude people with these primary diagnoses given their interpersonal issues, to reduce additional challenges in ensuring effective group dynamics and participant safety. Future phases of the research on the intervention could consider fewer exclusion criteria. This would require added expertise to the moderation team in delivering psychological interventions to individuals with antisocial and personality disorders, in a heterogeneous clinical population context, and in virtual setting.

Clinicians and peer support moderators will be recruited to deliver the intervention and participate in interviews, focus groups, and journal entries regarding their experiences and perspectives on HoryzonsCa implementation. Clinician moderators will have relevant experience working in SSPD treatment as a psychologist or mental health occupational therapist and may include graduate students (e.g., doctoral students in clinical psychology, graduate students in occupational therapy). Peer support moderators will have received training in peer support, relevant lived experience, and professional experience in the role. Recruitment will occur at: (1) CIUSSS de l’Ouest-de-l’Île-de-Montréal, which includes the Douglas, one of Canada's largest mental health institutions (semi-urban Montreal) and has a Psychotic Disorders program providing services to over 2000 individuals across multiple outpatient clinics including a FEP clinic, four Assertive Community Treatment teams, and the Centre for Psychological Intervention for Psychosis. This CIUSSS also includes other hospitals in its jurisdiction providing outpatient psychiatry services, such as the Lakeshore; (2) the CHUM, an urban super-hospital (downtown Montreal) which consists of the Clinique JAP, a well-established clinical and research program specializing in the treatment of FEP for over 350 diverse clients per year.

Sample size calculation is based on the study's main purpose: examining the feasibility of implementing and evaluating HoryzonsCa in real-world healthcare settings. Viechtbauer et al.'s ⁷⁰ formula was used to calculate the sample size needed to obtain a sufficient number of individuals per cohort (young adults; adults) that would lead to the identification of unanticipated problems regarding our intervention and evaluation methods (e.g., misinterpretation of questionnaire items, issues with the delivery of the intervention). The sample size calculation suggests that 60 participants must be recruited to ensure that a problem with 5% probability of occurring in a participant (i.e., one of 20 participants) could be identified (with 95% confidence). Considering the results of this sample size calculation, in addition to the median sample size of 76 found in a review of related pilot studies, ⁷¹ the diversity of the sample (e.g., in terms of age groups, language, recruitment setting), and possible attrition, a target sample size of 100 is considered here to be sufficient.

The timeline for the project is approximately three years and will unfold iteratively in three phases:

• Phase 1, Preparation and Training: Prepare intervention content and training strategy. The training strategy will help create capacity for scaling the intervention and support our ability to quickly on-board moderators in case of staff turnover.

• Phase 2, Recruitment and Data Collection: Progressive recruitment of 100 participants aged 18 to 50. Rate of recruitment on average: six participants/month. Data will also be collected from moderators (e.g., focus group discussions, journal entries).

• Phase 3, Analysis and Reporting: This phase includes analysis, stakeholder meetings to discuss preliminary findings, reporting, dissemination, and next steps.

The entire study and delivery of the intervention will be conducted remotely using web-based technologies. Eligible patients will be identified by treating clinicians and through individuals on waitlists for psychological therapy. A trained research team member (not involved in participant clinical care) will conduct informed consent, which will be signed electronically using an electronic data management platform. The research assistant will administer baseline measures and send participants links to the questionnaires through the electronic data management platform. Participants will be oriented to HoryzonsCa with login information and can access the website at their convenience, with a follow-up assessment at 12 weeks. They will be encouraged to use the website at least 10 minutes/week. Translation of the therapeutic modules has been completed through professional services and validated by team members for whom French is the primary language. Shorter content (e.g., moderator posts) is conducted through the assistance of computer-assisted artificial intelligence software and human review for equivalency in conceptual meaning following principles and guidelines from the World Health Organization. ⁷² Intervention staff have been trained, and data collection is in process.

Data collection

Data collection will include quantitative and qualitative data using interviewer-administered and self-report measures. For the quantitative data, outcomes of interest and outcome measures are described below. Additional details are provided in Supplementary Material 1. Evaluation Protocol for Key Measures.

Socio-demographic Information: To explore the influence of participant factors on acceptability and clinical outcomes, socio-demographic information from participants (patients and moderators) will be collected at baseline, including age, sexual orientation, gender, duration of receiving services at the site (or providing services—in the case of moderators), education level, origins, primary language, citizenship, employment and education status, living situation, and annual income (see Supplementary Material 2. Socio-Demographic Questionnaire for details). Data collection will also include information on access, use, and perceived competency with Internet and mobile technology at baseline, using TAUC-Q (Technology Access, Use, and Competency-Questionnaire), which has been used in our previous work,^{36,37,53,54,57} and which was further developed based on Statistics Canada's Canadian Internet Use Survey conducted in 2018. ⁷³

Primary Outcome Measures: Acceptability (objective 3) will be assessed by: (1) HoryzonsCa Acceptability, Usability, Safety, and Impact Questionnaire (HC-AUSI-Q), at 12-week follow-up; an interviewer-administered questionnaire including closed and open-ended questions on perceived ease of use, usefulness, enjoyment, and safety. The HC-AUSI-Q was adapted from our previous research on Horyzons;^44,53,54,57 (2) website usage analytics (Secondary Outcome Measure) (i.e., frequency of logins; views of therapeutic content), at monthly intervals; and (3) adverse events, incidents, reactions will be carefully monitored and quantified, from baseline to 12-week follow-up. Participants who drop out of the study will be contacted to assess the reason for dropping out (e.g., hospitalization). Data will also be reported on participant withdrawals and dropouts, disaggregated by gender and other sociocultural and demographic factors. ⁶⁵ Withdrawal from the trial will occur if: (1) participation in the study interferes with clinical management of risk of harm to self or others, (2) adverse events that could be associated with the intervention, 3) failure to comply with the Terms of Use of the intervention.

Secondary Outcome Measures: To assess benefits (objective 4), several measures will be implemented at baseline and 12 weeks. Most of the measures are self-report, except where specified. Our primary outcome of interest is Social Functioning, defined as overall performance and engagement in social relationships and productive activities, and by individuals’ perceptions of their ability to meet roles in these domains. ⁷⁴ Next to symptom remission, social functioning is also considered as an outcome of high relevance. ⁷⁵ The interviewer-administered Personal and Social Performance Scale (PSP) ⁷⁶ will be used to assess social functioning; it has sound psychometric properties and assessment procedures. ⁷⁷ To complement this data, an interviewer-administered question on Employment and Education Status (Average number of hours/week employed or pursuing school or vocational-related activities (e.g., internship) in the past 12 weeks) will also be included; Other secondary outcome measures include: positive and negative symptoms of psychosis, measured by the interviewer-administered six-item Positive and Negative Syndrome Scale (PANSS-6), ⁷⁸ depression using the nine-item Patient Health Questionnaire (PHQ-9), ⁷⁹ anxiety, measured by the five-item Overall Anxiety Severity and Impairment Scale (OASIS), ⁸⁰ social support, measured by the 12-item Multidimensional Scale of Perceived Social Support (MSPSS), ⁸¹ loneliness using the 20-item UCLA Loneliness Scale (Version 3), ⁸² self-esteem, measured by 20-item Self-Esteem Rating Scale Short Form, ⁸³ the eight-item Strengths Knowledge Scale (SKS) ⁸⁴ and the 14-item Strengths Use Scale (SUS),^84,85 and psychological well-being, measured by the 14-item Warwick-Edinburgh Mental Well-Being Scale (WEMWBS). ⁸⁶ Suicidality will be assessed at follow-up using the same tools at screening (e.g., item 4 in BPRS). The summary of all outcome measures used in this study is provided in Supplementary Material 3.

Analysis of these outcomes will help determine pre–post effect sizes on variables targeted by HoryzonsCa, which can be used to inform statistical power calculation and estimation of the sample size for a future larger trial.

Other Outcome Measures and Potential Covariates: (1) Other website usage analytics (aside from frequency of logins), such as duration and patterns of use over time, use of the live chat feature, and completed therapeutic content, will be automatically collected through the web application; (2) Hospitalizations and ER visits will be monitored and quantified; (3) During the evaluation meetings (i.e., the Baseline Interview and HoryzonsCa Orientation Meeting, and the HoryzonsCa Final Interview), the research assistant will take field notes on any additional comments from participants related to their experiences and perspectives of the website.

Stakeholder Experiences, Feedback, and Implementation Process: To examine stakeholder (patients and moderators) experiences of HoryzonsCa and the process of implementation (objectives 4, 5, and 6), a descriptive qualitative approach will be applied using the following methods: (1) HoryzonsCa Meet-Ups (focus group meetings with patients), (2) open-ended questions on the interviewer-administered HC-AUSI-Q with patients, (3) reflective narratives of the moderators and meeting notes from the research team, (4) moderator interviews (individual or group format), and (5) project documentation. The latter will contain recruitment outcomes (e.g., number of potential participants approached, number consented, number of dropouts and reasons). This combination of methods (individual and group-based interviews; reflective narratives) will help to elicit variation and depth of perspectives held by participants. HoryzonsCa Meet-Ups will include focus group discussions with patients (90 minutes, 15-minutes break) that will serve three purposes: (1) to better understand participant experience and perspectives of HoryzonsCa, including barriers and facilitators to using it, (2) to answer participant questions about the intervention and to provide tips on “how to make the best use of HoryzonsCa,” and (3) to obtain suggestions for moderators on how to improve participant experiences of HoryzonsCa (e.g., topics that participants would like to have a group chat on; recovery issues participants are seeking support on, etc.). These meetings will be conducted online using video-conferencing solutions (our team already has experience delivering group-based interventions online), will be facilitated by two team members, and will have approximately four to eight participants per meeting. Approximately ten meetings are planned throughout the project and will prioritize new participants recruited to the study. This is a general guide for the Meet-Ups; however, the content and structure may be subject to change (e.g., based on participant feedback from initial focus groups and implementation experiences and observations). Qualitative data will also be collected from participant responses to the open-ended questions on the interviewer-administered HC-AUSI-Q that address questions on acceptability, safety, and impact of HoryzonsCa and recommendations for refining the intervention.

All individuals involved in implementing HoryzonsCa (i.e., moderators, moderator trainers, and coaches) will be invited to participate in an interview (individual or group format). These interviews will address moderators’ experiences, actions, and perspectives on implementing HoryzonsCa (including barriers, facilitators, strategies to overcome the obstacles, and lessons learned), impact, sustainability, scale-up potential, and future study recommendations. Moderators will also be invited to reflect on their experiences and perspectives of implementing HoryzonsCa for English and French-speaking participants, for different age groups, and for participants representing diversity in terms of various characteristics (e.g., gender, ethnicity, education level, access and use of technology). These discussions will provide insights on the feasibility of conducting a larger trial across multiple sites as well as optimal research design (i.e., best type of trial, such as adaptive RCT or patient-preference RCT, or other). The interviews will be recorded and transcribed verbatim. Each member of the moderator team will also be invited to write reflective narratives (maximum two pages) on their experiences (e.g., baseline, middle, and end of implementing the intervention). Reflective narratives are helpful for elucidating lessons learned from an implementation project. ⁸⁷ Minutes of project meetings will also be considered for insights regarding the implementation of the intervention and the study protocol.

Data management and analysis

Qualitative data will be transcribed verbatim using automated speech-to-text software and human review for accuracy of transcriptions and de-identification. Once the transcriptions are finalized, they will be stored in an institutional server, and the recordings will be destroyed. The qualitative data (including project documentation) will be managed using qualitative data management software (e.g., the latest version of Atlas.ti). Quantitative data from the questionnaires and outcome measures will be directly entered into an electronic data capture system and analyzed using SPSS (or R). Website usage data (e.g., number of logins per week, number of posts) will be exported to an Excel file that will be stored on an institutional server.

Several measures are in place to enhance data quality and reduce the likelihood of missing data occurrence (e.g., all questionnaires and measures at baseline and 12 weeks follow-up will either be interviewer-administered or self-reported, and those that the participant completes will be checked by a member of the research team for completeness as sometimes questions are not answered simply because of rushing through, filling in errors, or oversight. Any missing data will be discussed using a team approach to understand its nature (e.g., participant non-response, error entering information while administering the questionnaires, human error in any other type of data entry, participant dropout, etc.). In consultation with a statistician, the extent and type of the missing data (e.g., missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR) will be considered to determine the best approach (e.g., multiple imputation, regression imputation) to handle the missing data.

The quantitative data from the questionnaires and website use will be assessed using descriptive statistics (e.g., counts, frequencies, medians, ranges, means, standard deviation) and outcomes using inferential statistics. Between-group factors will be assessed using linear mixed models. Linear mixed models will allow us to evaluate potential associations with factors and changes in clinical measures from baseline to 12 weeks follow-up. To account for the correlation present for measurements within the same individual, the linear mixed model will include a random factor by individual. Linear mixed models have an advantage over paired methods in that all measurements can be included in the model, including individuals who do not have complete measurements over time. Reliable Change Index (RCI) on clinical measures (proportion improved, deteriorated, and no change) will also be reported. In addition, exploratory analysis of website usage, gender-based analysis,^88,89 and potential mediators and moderators of treatment effects will be conducted. For example, a moderating effect of gender and other sociocultural and demographic factors on the outcomes (e.g., clinical benefits, website use, and satisfaction) of Horyzons will be assessed. Checks of assumptions will be completed before conducting an analysis, including the normality of the data, and data will be transformed if needed.

Following the convergent mixed-methods model, the quantitative and qualitative data will be analyzed separately and then considered for an integrated analysis of the findings. ⁹⁰ The qualitative data will be analyzed using content analysis ⁹¹ or thematic analysis where applicable for patterns related to acceptability, perceived benefits, safety, barriers, and facilitators of using HoryzonsCa. We will also consider how gender and other sociocultural and demographic factors relate to the themes during the qualitative analysis. At least two team members will review the qualitative data, develop a coding framework, and analyze the data in collaboration with the lead investigator. The coding framework will be developed based on an inductive and deductive coding approach (the latter being informed by the research objectives, the conceptual framework for the feasibility study, and related interview guides). The quantitative and qualitative results will be considered for an integrated analysis to address the research objectives. Participants (patients and moderators) will also be invited to discuss preliminary findings from the project (as a form of member feedback). ⁹²

Internet access, use, and digital literacy

All participants are asked to complete an internet access and use questionnaire prior to being given access to the system. Any concerns about technology access and use will be reported to the moderation team that will investigate the issue further and identify strategies to address concerns. Participants will be provided with videos that help with orienting them to the website and will meet with the moderators to receive additional support in navigating and using the website. Initial group meet-ups and occasional website posts from the moderators will also focus on how to get the most out of the platform, ensuring that participants are aware of its various aspects. The role of socio-demographic characteristics, such as education, will be considered in the engagement outcomes of the intervention, and qualitative data from the moderators will also be considered to better understand the impact of these elements on intervention use.

Clinical safety

A comprehensive safety protocol (including criteria and the process for withdrawal from the study) previously used in our pilot research and detailed in international research on Horyzons will be implemented.^44,51,58 All team members interacting with participants will be trained in this protocol, including assessment procedure for risk of suicide attempt.

To address online and clinical safety, daily monitoring of new website posts will be conducted by clinician moderators. Clinical safety risks will be managed through both manual and automated procedures, including for example, screening for evidence of psychotic, depression, or suicidal symptoms in participant posts, participant concerns or complaints about other users’ posts, automatic detection and blocking of self-harm related words, and any posts or activity that places participants at risk. The system also has an automatic word screening function which detects information consistent with increased risk of relapse or suicide or potentially harmful communication. It also allows users to report any concerns about material posted by a user.

Moderators will monitor and follow-up any participant safety issues derived from online activity or communication. Any detected risk will activate safety procedures, including risk assessment with the participant, informing the research team, and liaising with emergency services or treatment team, if necessary. They will have the administrative authority to address any issues with online posts (e.g., remove offending material from the system) and, if needed, can pause the participant's access. All users will be asked to nominate an emergency contact person that can be contacted in the event of a major concern. Contact information of the primary treating team will also be accessible to members interacting with participants. The website also has emergency resources and contact information easily accessible through every section of the website and an electronic reporting system for participants to report issues.

Initially, moderators will be supervised by an experienced clinical psychologist or senior clinician-researcher. As the intervention delivery team gains experience, moderation may be conducted by senior health professional students or mental health workers with relevant experience. Moderators will meet weekly to discuss new participants, participant engagement and progress, and to plan group activities (meet-ups, focus groups). They will have access to senior clinical psychologists in situations where the need arises for additional support or de-briefing and they will report to the lead investigator (also a clinician-researcher) on a weekly basis regarding the status of the intervention delivery and any clinical safety issues encountered.

Privacy, security, and technical considerations

HoryzonsCa has been developed, and continues to be refined, maintained, and monitored actively by an information technology (IT) team and group of researchers based at Orygen Digital ¹ (digital division of the youth mental health organization Orygen—https://www.orygen.org.au/) and the University of Melbourne. Liason with the Orygen Digital development team is primarily via a project manager for international projects and partnerships. Technological issues with the platform will be reported to the IT team through an online service desk. If the system is inaccessible due to technical issues, participants will be updated through their preferred method of contact. A range of measures are in place to ensure the security of the HoryzonsCa website, and the data generated by users. The application will be hosted on an Amazon AWS server based in Canada and monitored actively. Access to the HoryzonsCa website database is restricted to the Orygen Digital IT team by both password and/or key-based authentication. Measures conforming to industry best practice as defined by the Open Web Application Security Project (https://www.owasp.org) are within the application to secure the application and database against unauthorized access. Moreover, participants will be encouraged to use pseudonyms on the website and discouraged to post identifiable information (e.g., age, address). All posts and profile information will also be monitored by moderators daily to ensure the risk of privacy issues is minimized. In addition, when participants are oriented to the intervention, they will be asked to review the Terms of Use, which outline several elements regarding privacy and security, including, for example: what to do in case of emergency, where data is stored and how it will be used, behaviors that will help to keep the intervention safe (e.g., not forwarding content to third parties, not posting content that is hateful or threatening), strategies to maintain account security, and general information on online safety. The system allows any participant to temporarily “hide” their profile and all of their online communications at any time, for example, if participants develop heightened concerns about their communications on the website. The safety protocol (including criteria and the process for withdrawal from the study) has been successfully used in previous research on Horyzons.^44,51

Discussion

In this paper, we have described a protocol for evaluating feasibility and impact of providing a digital mental health intervention in a real-world clinical setting to individuals diagnosed with SSPD in Canada. The protocol benefits from the following: (1) the intervention is already developed and piloted (e.g., preliminary feasibility study in Canada and RCT in Australia); (2) there is a well-established Canadian–Australian collaboration, including leading researchers in digital mental health research and innovation and working with individuals diagnosed with SSPD; (3) presence of active clinician leads at each recruitment site; (4) ethics protocol already approved, with minor amendments submitted; (5) partner cash contributions already awarded; and (6) the team includes people with diverse clinical, research, lived experiences, and knowledge users, representing multiple disciplines, career stages, and specializing in mental health, digital health, non-pharmacological interventions, and mental health services evaluation. For example, the lead investigator has extensive front-line and leadership experience (trained in occupational therapy) in providing psychosocial interventions (in individual and group formats) to individuals diagnosed with psychotic disorders and in digital mental health research, with a focus in mixed-methods approaches. The research team includes several clinician-researchers (trained in psychology or psychiatry) with extensive experience in successfully completing psychological and psychiatric intervention trials. The Australian co-developers of the intervention (who lead a multidisciplinary team of content writers, graphic artists, software developers and related IT experts, and researchers), medical directors at the recruitment sites, and a senior clinician-researcher also have a leadership role in delivering psychological services for individuals diagnosed with SSPD at the main recruitment site. We also have a lead software engineer for the web application that operates HoryzonsCa who will be responsible for the technical aspects of the project. Postdoctoral and/or graduate trainees with an interest in the role of socio-demographic characteristics in the acceptability and outcomes of digital mental health intervention will also be involved in the study's implementation. Moderators and research team members will be recruited, hired, and supervised during the study. In general, the intervention is designed as such that one full-time moderator can manage up to 100 users (including a mix of high and low users). In our pilot, one moderator working one day/week successfully managed 20 participants. ⁵⁸ For the current study, we anticipate hiring one part-time peer moderator at 14 hours/week and approximately two part-time moderators working 7 hours/week each. The moderators will meet on a weekly basis to discuss new cases and engagement strategies and to plan for therapeutic activities; a research coordinator will also be present to facilitate these meetings and to present new participants to the clinical team. Moderators and research staff will be supervised by senior clinician-researchers through weekly meetings and communications.

While this study presents unique opportunities, it is not devoid of challenges. For example, in terms of recruitment, participants may be deterred by the level of time commitment the study involves (e.g., comprehensive baseline and follow-up assessments, focus group discussions, and an intervention that has multiple activities) over several months. As with many mental health services (including digital interventions), there may be disengagement or low engagement of participants, as well as symptom relapse that may affect the ability to participate. In terms of intervention delivery, providing digital mental health services will be a new skill set for most, if not all, the moderators, and may pose challenges in terms of training and supervision. The integration of peer support workers is also relatively new in this healthcare context, and this may present new challenges in optimizing collaboration and clarifying roles between moderators. There may be new socio-relational scenarios that moderators will need to navigate considering the peer-to-peer component of the intervention (as well as the peer-to-peer moderator virtual context). The technical aspects include participants’ digital literacy and their access to the needed Internet and technology; providing effective orientation to the platform; and, navigating any technical issues with the platform itself, including communication with the technical support team. Given the intervention will be delivered in both English and French, there will be challenges in terms of recruiting a bilingual moderation team and managing the content in both languages. These implementation challenges will be monitored and examined through the various data collection methods, such as the moderator reflective journals.

The study also has limitations. In terms of understanding effectiveness, the results of pre–post outcomes using a single-group design will not be conclusive without randomization procedures and a lack of control group. Through our focus group discussions with moderators and project documentation, we will gather feedback and insights on the feasibility of conducting a larger trial across multiple sites as well as optimal research design and service delivery model. Additionally, limited recruitment of diverse participants may pose challenges in addressing one of the secondary objectives regarding the role of socio-demographic characteristics on acceptability and outcomes. There is also a possibility of recruitment bias towards higher functioning FEP. Moreover, the decision to exclude individuals with antisocial and personality disorders may limit understanding of the intervention's broader applicability. Finally, while the study has a longer implementation duration of 12 weeks compared to eight weeks in the pilot, it may fall short of what is needed to encourage engagement and integration with routine care within healthcare settings (for example,18 months in Horyzons RCT study). ⁵¹

In terms of knowledge transfer and dissemination, a Knowledge User Stakeholders Committee (KUSC) will be established to contribute to the project dissemination. We will use several dissemination strategies, including diffusion in open-access scientific journals, meetings with key knowledge user groups, tailored executive summaries of the project, multi-media video clips, infographics, and social media, virtual conferences to discuss the results; and conference presentations. The KUSC will determine other activities as needed. Principles of knowledge translation will be considered throughout these activities to ensure that the findings are accessible to a range of stakeholder groups. We will also leverage evidence-based and theory-informed frameworks focused on the evaluation of digital health innovations, such as the NASSS framework, to facilitate the discussion, dissemination, and implementation of results. ⁹³

Conclusion

This study will generate new knowledge on the feasibility of implementing HoryzonsCa on a broader scale for individuals with SSPD in English and French (i.e., across the continuum of care such that the intervention will be delivered to both young adults and adults in the same online therapeutic setting). In addition, this study will help to inform factors to consider in the design of a future trial. These factors include: a replicable and feasible implementation and evaluation protocol (including training and supervision strategy, moderation activities); acceptability of the intervention (how likely are service users and providers to adopt the intervention); safety and risks (e.g., when delivering it to a larger cohort of participants, in a study recruiting across multiple settings); service delivery model when integrating into routine care; and impact of the results (the likelihood that the results of the trial will be helpful to inform clinical care). Project results will provide clinicians, hospital administrators, policymakers, patients, and families with important and timely information on using evidence-based digital mental health interventions to improve access to psychosocial therapies for individuals living with chronic mental health conditions. Additionally, measuring and analyzing sex, gender, and other sociocultural and demographic differences will help with discovering how acceptable are digital mental health interventions across diverse populations, thereby improving both the practice and science of digital mental health research and reducing gender and other diversity-related inequalities.