Skip to main content
NCBI home page
Internet Interventions logoLink to Internet Interventions
. 2019 Aug 15;18:100266. doi: 10.1016/j.invent.2019.100266

Predicting engagement with an online psychosocial intervention for psychosis: Exploring individual- and intervention-level predictors

Chelsea Arnold a,⁎, Kristi-Ann Villagonzalo a, Denny Meyer a, John Farhall b,c, Fiona Foley a, Michael Kyrios d, Neil Thomas a,e
PMCID: PMC6926321  PMID: 31890619

Abstract

Background

Individuals with psychosis demonstrate positive attitudes towards utilising digital technology in mental health treatment. Although preliminary research suggests digital interventions are feasible and acceptable in this population, little is known about how to best promote engagement with these resources. Candidate predictors include therapist support, sources of motivation and recovery style. Understanding what factors predict engagement will aid more effective design and implementation of digital interventions to improve clinical benefits.

Objective

This study aimed to investigate demographic, psychological, and treatment variables that predict overall and type of engagement with a psychosocial, online intervention for individuals with psychosis.

Methods

Ninety-eight participants with a history of psychosis were given access to a web program containing modules on self-management and recovery, which they were asked to use flexibly at their own pace. Activity was automatically logged by the system. Baseline measures of demographics, recovery style and motivation were administered, and participants were randomised to receive either website access alone, or website access plus weekly, asynchronous emails from an online coach over 12 weeks. Log and baseline assessment data were used in negative binomial regressions to examine predictors of depth and breadth of use over the intervention period. A logistic regression was used to examine the impact of predictor variables on usage profiles (active or passive).

Results

Depth and breadth of engagement were positively predicted by receiving email support, low levels of externally controlled motivations for website use, older age, and having a tertiary education. There was a significant interaction between level of controlled motivation and condition (+/−email) on breadth and depth of engagement: receiving asynchronous emails was associated with increased engagement for individuals with low, but not high, levels of externally controlled motivations. Receiving email support and more autonomous motivations for treatment predicted more active use of the website.

Conclusions

Asynchronous email support can promote engagement with online interventions for individuals with psychosis, potentially enabling self-management of illness and improving clinical outcomes. However, those using online interventions due to external motivating factors, may have low levels of engagement with the intervention, irrespective of coaching provided. These findings may guide design and implementation of future online interventions in this population.

Abbreviations: SMART, Self-Management and Recovery Technology; SCID, Structured Clinical Interview for DSM-IV-TR Axis I Disorders; RSQ, Recovery Style Questionnaire; ACMTQ, Autonomous and Controlled Motivations for Treatment Questionnaire; IRR, Incidence rate ratio

Keywords: Digital technology, Digital mental health, Engagement, Intervention, Psychosis

Highlights

  • •

    Asynchronous email can promote engagement with online interventions for psychosis.

  • •

    Individuals motivated by external factors (e.g., wanting to please others) are unlikely to have high levels of engagement.

  • •

    Older age and a tertiary education were associated with higher levels of engagement amongst individuals with psychosis.

1. Introduction

1.1. Online interventions for psychosis

In the past decade there has been increasing research interest into the utilisation of online interventions for individuals with more serious mental illness, such as psychosis (Bonet et al., 2017; O'Hanlon et al., 2016). Preliminary research suggests that online interventions for psychosis are feasible and acceptable (Alvarez-Jimenez et al., 2014; Berry et al., 2016; Firth et al., 2016; Gay et al., 2016; Thomas et al., 2017), particularly for symptom monitoring and clinical management (Bell et al., 2017). The Internet may also enable provision of adjunctive psychosocial treatments, progressing beyond clinical management to broader service delivery goals such as promoting personal recovery (i.e., living a satisfying life despite potential limitations posed by mental illness; Anthony, 1993). Whilst highly individualised, recovery-related areas of focus may include greater participation in community activities and relationships, as well as increasing autonomy and illness self-management (Drake and Whitley, 2014).

1.2. Engagement with online interventions

Engagement broadly refers to a user's involvement and interaction with an online intervention (Baltierra et al., 2016). Issues with limited engagement have arisen consistently across online psychological interventions (Christensen et al., 2009; Eysenbach, 2005). Whilst there may not be a simple dose-response relationship between engagement and outcome, some level of engagement with an intervention is required to obtain any clinical benefit (Donkin et al., 2013). As such, understanding what factors promote engagement and can inform more effective design and implementation of future online interventions to improve associated outcomes (Scherer et al., 2017).

Online interventions provide a unique opportunity to objectively examine users' involvement with, and exposure to an intervention via log data (Couper et al., 2010). Previous research has largely focused on non-usage attrition or the proportion of intervention completed by users (Killikelly et al., 2017; Simco et al., 2014). However, recent studies have begun to utilise log data to examine more in-depth measures of engagement, such as breadth or depth of use, to inform understanding of engagement (Couper et al., 2010; Short et al., 2018). Type of engagement, such as ‘active’ (requiring more direct involvement from the user, e.g., posting on a forum, self-monitoring exercises) or ‘passive’ (requiring less direct involvement, e.g., viewing a forum comment or watching a video) intervention use has also demonstrated different relationships with outcomes (Short et al., 2018).

1.3. Predicting engagement

Engagement with online interventions may be influenced by environmental, intervention and individual factors (Short et al., 2015). As a group, individuals with psychotic disorders may have low computer literacy and limited Internet access secondary to low socio-economic status (Robotham et al., 2016). Further, individuals with psychosis have unique challenges, such as cognitive impairment and positive and negative symptoms, potentially impacting engagement (Ben-Zeev et al., 2014; Granholm et al., 2012; Rotondi et al., 2010). Given these unique challenges, it is important to develop an understanding of factors associated with engagement in this population specifically. To date, there has been limited investigation of factors associated with engagement with online interventions amongst individuals with psychosis (Killikelly et al., 2017).

1.3.1. Email support

In both psychosis and broader mental illness populations, interventions with some form of guidance or adjunct support have higher levels of engagement than those without support (Baumeister et al., 2014; Killikelly et al., 2017). In other clinical populations, communication medium (i.e., telephone vs email) and qualification of support provider have not been associated with engagement to online interventions (Berger et al., 2011a; Lindner et al., 2014; Titov et al., 2010). As such, asynchronous emails from persons without clinical expertise may provide a cost-effective opportunity to increase engagement with online interventions for psychosis.

According to the theory of supportive accountability, the provision of additional support to online interventions increases engagement through a sense of accountability to a support person (Mohr et al., 2011). This framework highlights the importance of providing a supportive relationship, encouraging use of the intervention and providing practical assistance with applications (Mohr et al., 2011; Mohr et al., 2013). This model has been utilised in supporting individuals with first-episode psychosis (Alvarez-Jimenez et al., 2013; Lederman et al., 2014) and has successfully increased engagement with online interventions for individuals with depression, stress, and weight management (Dennison et al., 2014; Mohr et al., 2013; Zarski et al., 2016). Adjunct email support may also encourage additional exploration and provide assistance with issues associated with use of self-guided online interventions. This may improve engagement by facilitating further use and overcoming potential barriers.

1.3.2. Motivations for treatment

A person's motivational orientation for treatment may also relate to engagement with online interventions. Self-determination theory is a broad framework that describes the relationship between motivations and behaviour (Deci and Ryan, 1985). Individuals may have autonomous and/or controlled motivations for engaging with treatment (Ryan and Deci, 2000, Ryan and Deci, 2008). Autonomous motivations are driven by personal interest and choice, with behaviour identified as of personal value and importance (Ryan and Deci, 2008). In relation to treatment, autonomous motivations may reflect key recovery processes including self-agency, improvement and learning (Farkas, 2007). In contrast, controlled motivations relate to external motivating factors such as praise from others, or avoidance of punishment or some other negative reinforcer (Ryan et al., 2008). For example, individuals with psychosis may use an online intervention upon the recommendation of their mental health care worker or family member, to avoid perceived negative consequences or guilt.

According to self-determination theory, maintenance of behaviour over time requires internalisation of values and skills required for change, i.e., self-determination (Deci and Ryan, 1985). Therefore, autonomous motivations are associated with enhanced and maintained engagement, whereas controlled motivations do not support long term engagement (Ryan and Deci, 2008). In their theory of supportive accountability, Mohr et al. (2011) proposed that motivational orientation may moderate the impact of support on engagement. When participants are more autonomously motivated, they require less support with online interventions. Thus, there may be an interplay between individuals' motivation and support received on engagement with online interventions. However, to the best of the authors' knowledge this has not been empirically examined in the context of online interventions for mental health.

1.3.3. Recovery style

Individuals' coping style and adaptation to psychosis may be conceptualised in terms of their ‘recovery style’ (McGlashan et al., 1975). An individual's recovery style may be more integrated – i.e., they demonstrate interest and curiosity towards their mental health – or sealing-over – where individuals may seek to minimise the significance of, or demonstrate a lack of interest in their mental illness (McGlashan et al., 1975). Individuals with a more integrated recovery style may have more positive clinical and functional outcomes (McGlashan, 1987; Staring et al., 2011). Conversely, in congruence with an avoidant style of coping, a sealing over recovery style has been associated lower levels of engagement (including both attendance and collaboration) in face-to-face mental health treatments for psychosis (Startup et al., 2006; Tait et al., 2003). Although not yet examined, congruent with face-to-face treatments, recovery style may be associated with engagement with digital interventions in individuals with psychosis. A more integrated recovery style may promote engagement with digital interventions, in accordance with an interest in mental health and treatment.

1.3.4. Demographics

Demographic variables such as age, gender, and education may also be associated with engagement with online interventions for psychosis. Congruent with other clinical populations (Beatty and Binnion, 2016), preliminary research suggests that females may be more highly engaged than males (Ben-Zeev et al., 2016; Moitra et al., 2017). There have been mixed findings on the association between age and engagement, with the relationship currently unclear (Ben-Zeev et al., 2016; Granholm et al., 2012; Maijala et al., 2015). Higher levels of education have been associated with Internet use amongst individuals with psychosis (Baup and Verdoux, 2017; Välimäki et al., 2017), but investigations to date have found no significant association with engagement (Granholm et al., 2012; Maijala et al., 2015).

1.4. Objectives

The objective of this study was to investigate demographic, psychological, and treatment variables that may predict engagement with a self-guided, recovery-focused online intervention for people with psychosis. We hypothesised that asynchronous email support, high autonomous and low controlled motivations for treatment, and an integrated recovery style would be positively associated with engagement. Consistent with the theory of supportive accountability, we also hypothesised that controlled motivation would moderate the impact of email support on engagement.

2. Methods

2.1. Design

This study was conducted as part of the Self-Management and Recovery Technology (SMART) research program in Australia. As part of the broader research into digital mental health treatment for psychosis, the SMART website was developed as a resource on self-management and personal recovery (Thomas et al., 2016a). Separate studies were conducted with parallel examinations of: efficacy (Thomas et al., 2016b); use as a tool in routine practice by mental health workers; and predictors of engagement in self-guided use (this study).

2.2. Participants

Participants were recruited via publicity within community-based adult mental health services in Victoria, Australia or via social media (Facebook) advertising. Inclusion criteria were: (a) age between 18 and 65 years inclusive; (b) currently residing in Australia; (c) diagnosis of a nonorganic psychotic disorder (schizophrenia-related disorder or bipolar disorder or major depressive disorder with psychotic features present within the past 2 years), confirmed using the Structured Clinical Interview for DSM-IV-TR Axis I Disorders (SCID: First et al., 2002); (d) proficiency in the English language; and (e); willingness to access and use the Internet. Exclusion criteria were: (a) psychosis only during substance intoxication, (b) no Internet access.

2.3. Procedure

Interested participants were directed to the SMART website, where they completed an online, informed consent process and created an account using an email address. Eligibility was confirmed via telephone administration of the SCID. Eligible participants then completed baseline questionnaires via website login, prior to receiving access to website content. Participants received $30AUD for completing baseline questionnaires. Questionnaires to assess recovery style and motivational orientation were introduced following the beginning of recruitment, after the first four participants had completed their baseline assessments.

The website automatically randomised participants on a 1:1 basis to receive either a) access to the SMART website only (website only condition) or b) access to the SMART website supplemented with weekly, asynchronous emails from an online coach over a 12 week period (+ email condition). Follow-up questionnaires were also administered at 1-, 3-, and 6-months post-baseline, but will be reported separately.

2.4. Intervention

2.4.1. The SMART website

The SMART website aims to promote personal recovery and self-management of mental health in individuals with a history of psychosis (Thomas et al., 2016a). The website was developed in consultation with mental health consumers and is based on the Connectedness-Hope-Identity-Meaning-Empowerment (CHIME) framework of personal recovery (Leamy et al., 2011). Website content is organized into modules on six topics: Recovery (promoting hope), Managing Stress (common stressors, approaches to coping), Health (self-management, medication, and sleep), Me (identity, stigma, personal strengths), Relationships (interpersonal relationships) and Life (values, goals). The modules contain videos of peers with lived experience of psychosis discussing their experiences and opinions in relation to the relevant content (see Fig. 1). Additional website features included exercises pertaining to module content, self-management tools (including charts for sleep, stress, mood) and social networking features (a peer moderated forum, and opportunities to comment publicly and interact with other site users).

Fig. 1.

Fig. 1

Open in a new tab

SMART website video and features.

In accordance with guidelines on website design for individuals with serious mental illness (Rotondi et al., 2007): navigation was simplified; content was developed to be simple, clear and organized logically; pages were designed to have minimal distractions with clearly labelled links. During original consultation, consumers highlighted flexibility and interactivity as important design aspects, which were accordingly incorporated throughout. Finally, as consultation highlighted a positive emotional experience as important in website engagement, we aimed to incorporate this throughout. A full description of the intervention is provided elsewhere (Thomas et al., 2016a).

As recovery is a highly individual process and consumers may have differing needs and interests, the SMART website was developed as a selection of recovery-based resources that users can explore. Participants were not required to work through materials in any order or utilise any specific aspect of the resource. Rather, participants were able to utilise all aspects of the website at any time in accordance with personal preference. For the current study, participants accessed the SMART website independently using personal or public devices accessible to them. Access was possible via smartphone, tablet computer or computer with an Internet connection.

2.4.2. Asynchronous email

In addition to website access, participants in the +email condition received weekly, asynchronous emails from an online coach over 12-weeks. The online coaches were helpline staff and volunteers at a community mental health organisation who were provided training and supervision for the project. In line with previous research (Berger et al., 2011b), online coaches sent one asynchronous email at a scheduled time each week, and participants were invited to contact their online coach as much as they wanted. Messages between participants and coaches were sent through the SMART system, with notifications sent to participants' email addresses. In alignment with end-user consultation during the development phase and consistent with promoting consumer autonomy within a recovery framework (Drake and Whitley, 2014), online coaches did not receive information about participants website use and provided feedback based on participants' self-reported activity.

Consistent with the Supportive Accountability framework (Mohr et al., 2011), emails were designed to encourage participants to work through the website content rather than be therapeutic. Emails were written using a protocol and templates adapted from an online mental health service for high prevalence disorders (Klein et al., 2011). Content included introductions, goal setting, and orientation to various topics and features of the website. Emails mirrored consumers' tone, provided positive feedback and aimed to reduce cognitive burden via succinct and clear communication. Emails incorporated individualised feedback, strategies for goal achievement and practical support as required. Emails were tailored as required to meet the individual participants' interests.

2.5. Measures

2.5.1. Engagement

The main dependent variable was engagement with the SMART website over the 12-week intervention period. The SMART system automatically logged participants' use of the website. Specifically, for each participant, the system recorded dates and times of logins and activities. Activities included: watching videos; listening to audio content; downloading content (e.g., audio content); accessing external links for further reading; commenting on content; viewing or posting in the peer forum; adding contacts; completing exercises; using the charting or list features, adding content to ‘favourites’; and viewing topic pages within modules. This log data was extracted for each participant and synthesised into summary indices of engagement.

Engagement is a multifaceted concept and different operationalisations may lead to different results (Alfonsson et al., 2016). In accordance with previous examinations of depth and breadth of use (Couper et al., 2010), we were primarily interested in how much participants used the website and how varied their website use was. Depth of use was assessed on the basis of participants' total amount of website use over the intervention period. This was calculated as the total number of activities completed during that time. Breadth of use was assessed on the basis of the variation in participants' use of the website. This was calculated as the total number of unique activities participants completed during the 12 weeks (i.e., the number of different website components utilised).

To examine types of engagement, activities were also categorised as either active or passive. Active clicks included for example, posting in the forum, commenting on videos or completing exercises in the website. In contrast, passive clicks included for example, reading a forum post without making a contribution, viewing a section, or watching a video. Individuals were characterised as ‘active’ or ‘passive’ users based on the predominant activity type (i.e., >50% of completed activities).

2.5.2. Demographics and internet use

Basic demographic information (age, gender, educational attainment, employment status) was collected. Additionally, Internet use and access was measured using a 6-item questionnaire adapted from previous research by the research team (Thomas et al., 2017). Items measure confidence and frequency of Internet and email use, membership of mental-health related forums or social networks, and frequency of online social networking.

2.5.3. Recovery style

The Recovery Style Questionnaire (RSQ: Drayton et al., 1998) was used to measure recovery style. The RSQ is a 39-item measure on which participants responses are coded as either sealing over or integrated. Based on responses, participants receive a scaled score from 1 to 6. A score of 1 represents an integrated recovery style and a score of 6 represents a sealing over recovery style. The RSQ has good construct validity, test-retest reliability and internal reliability correlations (Drayton et al., 1998). In this study Cronbach's α = 0.67.

2.5.4. Motivational orientation

A modified version of the Autonomous and Controlled Motivations for Treatment Questionnaire (ACMTQ: Zuroff et al., 2007) was used to assess motivations for using SMART website. The ACMTQ includes two 6-item subscales assessing autonomous and controlled motivations for CBT for depression. The ACMTQ has good internal consistency (Zuroff et al., 2007). Wording of the ACMTQ was modified to refer to the website rather than psychological treatment. For example “Using CBT is an important choice that I really want to make to become well” was modified to “Using the website is an important choice that I really want to make to recovery.” Scaled scores on both the autonomous scale (Cronbach's α = 0.78) and controlled scale (Cronbach's α = 0.75) were utilised in all analyses, with higher scores representing more autonomous or more controlled motivations on the relevant scales.

2.6. Data analysis

Prior to analysis, data were screened for normality via visual inspection of histograms and examination of skewness and kurtosis values. Data were screened for univariate and multivariate outliers through standardised Z-scores and Mahalanobis distances. Correlations between predictor variables were screened for issues with multicollinearity. As some baseline measures were introduced following the beginning of recruitment, four participants had missing data for the RSQ and ACMTQ. This data was missing at random (Little's MCAR test: χ2(1) = 0.864, P = .353) and represented a small proportion of total data (<5%). Therefore, expectation-maximisation strategies were utilised to impute missing data and avoid a loss of power.

To check for baseline differences between randomised groups, t-tests were used for scales and chi-square (χ2) tests for categorical data. Descriptive statistics were used to describe patterns of use with the SMART website. As usage data were positively skewed, we used Mann-Whitney U tests to examine differences in usage metrics between groups.

Predictor variables were identified based on theoretical grounds and included: age (years), gender, condition (+/− email), education (non-tertiary/tertiary), autonomous motivation (ACMTQ, autonomous scale average score), controlled motivation (ACMTQ controlled scale, average score), recovery style (RSQ scaled score). Spearman's rho correlations were calculated to examine associations between candidate predictors and depth and breadth of use. Negative binomial regressions were conducted to predict the total number of activities and number of unique activities completed by participants over the 12-week intervention period. In addition to the above mentioned predictor variables, an interaction term between controlled motivation and condition was also included in these regressions on theoretical grounds. As an exploratory investigation, a logistic regression was performed to examine whether any of the measured variables predicted a more ‘active’ or ‘passive’ user profile.

All analyses were conducted utilising IBM SPSS statistics V25. An a-priori alpha level of <0.05 was adopted. As a sensitivity analysis, we conducted the above analyses with and without EM-imputed data.

3. Results

3.1. Participant flow and characteristics

Participant flow is outlined in Fig. 2. Of the 154 participants who enrolled online, 133 were assessed for eligibility and 113 were eligible to participate. Seven eligible participants did not complete the baseline questionnaires. A further 6 participants completed the questionnaires but did not access any website content. These participants were characterised as ‘non-starters’ and excluded from analyses. Two participants received the incorrect condition in error, and were also excluded from analyses. This yielded a final sample of 98 participants for all subsequent analyses.

Fig. 2.

Fig. 2

Open in a new tab

Participant flow diagram.

Table 1 provides demographic characteristics of both groups and the total sample. The mean age of all participants was 42 years (SD = 11.1). Approximately half of all participants were unemployed (49%) and the majority were receiving a disability support pension (72%). The majority of participants used the Internet more than once a day (85%) and were confident doing so independently (82%). t-tests and chi-square test revealed no significant differences between groups at baseline (P > .05).

Table 1.

Demographic characteristics of participants.

Variable SMART only
(n = 51)		 SMART + email
(n = 47)		 Total
(n = 98)
Age (years), mean (SD) 41.3 (11.4) 42.7 (10.8) 42.0 (11.1)
Gender, n (% female) 29 (57) 25 (53) 54 (55)
Level of education, n (%)
 Primary school 0 (0) 2 (4) 2 (2)
 Secondary school 22 (43) 14 (30) 36 (37)
 Apprenticeship/diploma/certificate 11 (22) 10 (21) 9 (9)
 Bachelor's degree 9 (18) 18 (38) 27 (28)
 Post graduate degree 9 (18) 3 (6) 12 (12)
Recent employment status, n (%)
 Paid or self-employment 16 (31) 12 (26) 28 (29)
 Voluntary employment 3 (6) 5 (11) 8 (8)
 Unemployed 24 (47) 24 (51) 48 (49)
 Student 8 (16) 5 (11) 13 (13)
 Home duties 0 (0) 1 (2) 1 (1)
SCID diagnosis, n (%)
 Bipolar disorder w psychotic feat. 13(25) 16 (34) 29 (30)
 Major depression w psychotic feat. 5 (10) 4 (9) 9 (9)
 Schizoaffective disorder 17 (33) 11 (23) 27 (28)
 Schizophrenia 16 (31) 13 (28) 30 (31)
 Delusional disorder 0 (0) 3 (6) 3 (3)
Disability support recipient, n (%) 34 (67) 37 (79) 71 (72)
Confidence using Internet, n (%)
 Confident without assistance 40 (78) 40 (85) 80 (82)
 Occasionally need assistance 10 (20) 7 (15) 17 (17)
 Assistance required 1 (2) 0 (0) 1 (1)
Frequency of internet use, n (%)
 More than once a day 41 (80) 42 (89) 83 (85)
 Once a day 6 (12) 2 (4) 8 (8)
 A few times a week 1 (2) 2 (4) 3 (3)
 Once a week 1 (2) 1 (2) 2 (2)
 Every few weeks 2 (4) 0 (0) 2 (2)
Open in a new tab

3.2. Engagement

Over the 12-week intervention period, all participants logged in to the website a total of 985 times and completed 4698 activities. ‘Recovery’ was the most viewed module, with 543 activities completed. ‘Life’ was the least viewed module, with 101 activities completed. The most common type of activity was “view website content or sections” (N = 1527). The chart feature was also used frequently with 526 total entries. Forum posts received many views (N = 427), but few comments (N = 88): Forty-six participants viewed the forum, but only 18 participants contributed.

3.3. Group differences

Table 2 presents descriptive statistics of engagement metrics for both groups. Individuals in the +email group had significantly more logins, total activities, unique activities and active activities than those in the website only group. Individuals in the +email group also used the website for, on average, almost 2 weeks longer than those in the website only group but this result did not reach statistical significance (P = .058).

Table 2.

Comparison of engagement metrics by group.

Engagement metric SMART only (n = 51) SMART+email (n = 47) Total group (n = 98) P
Total logins, mean (SD) 7.0 (5.3) 13.3 (14.8) 10.1 (11.4) .005
Total activities, mean (SD) 36.3 (39.1) 60.6 (64) 47.9 (53.6) .033
Total unique activities, mean (SD) 18.9 (20.0) 30.2 (28.9) 24.3 (25.2) .032
Total active activities, mean (SD) 8.43 (14.8) 16.7 (21.9) 12.4 (18.9) .006
Weeks until final activity, mean (SD) 7.3 (4.7) 9.0 (4.2) 8.1 (4.5) .058
Open in a new tab

Note. Groups were compared by Mann-Whitney U tests.

Examining engagement over time, both groups had a large number of activities in the first week of use and a large drop in number of activities from the second week onwards. Participants in the +email group completed more activities on average each week, than individuals in the website only group. Fig. 3 provides a graphical representation of activities completed per group over each of the 12 weeks.

Fig. 3.

Fig. 3

Open in a new tab

Comparison of activities per week by group.

3.4. Predictors of engagement

Table 3 provides correlations between each of the relevant predictor variables and depth and breadth of use over the intervention period. Mann-Whitney U tests demonstrated that depth of use was significantly greater amongst individuals with a tertiary education than those without (U = 853.0, P = .036) but breadth of use was not associated with education (U = 896, P = .076). Gender was not associated with depth (U = 974.5, P = .127) or breadth of use (U = 998.5, P = .176).

Table 3.

Spearmans rho correlations between continuous predictors and depth and breadth of use.

Predictors Depth of use
 Breadth of use
r P r P
Age 0.19 .059 0.12 .239
RSQb 0.01 .895 0.00 .994
ACMTQ-AUTONc −0.07 .526 −0.08 .415
ACMTQ-CNTRLd −0.24 .019 −0.24 .019
Open in a new tab

a Spearmans rho.

b Recovery style questionnaire.

c Autonomous and controlled motivations for treatment questionnaire, autonomous scale.

d Autonomous and controlled motivations for treatment questionnaire, control scale.

3.4.1. Depth of use

Table 4 presents the model effects of the negative binomial regression predicting depth of use over 12 weeks. The overall model was significant (Likelihood ratio χ2(8) = 35.65, P < .001). When controlling for other predictor variables, significant effects were found for additional email support (Incidence rate ratio (IRR) = 0.24), having a tertiary education (IRR = 0.61), older age (IRR = 1.02) and less controlled motivations for treatment (IRR = 0.66). The activity rate was on average 4 times higher for participants receiving emails and 39% lower for participants without a tertiary education. On average the activity rate increased by 2% for each year of age and decreased by 34% for each unit of controlled motivation. The interaction between controlled motivations and condition was also statistically significant (IRR = 1.36). As highlighted in Fig. 4, engagement levels were higher for those receiving emails when controlled motivations for treatment were low, and engagement was lower for emails when controlled motivations were high. Recovery style and autonomous motivations were not significant predictors of depth of use.

Table 4.

Negative Binomial Regression for predictors of depth and breadth of use.

Predictors Depth of use
 Breadth of use
B (SE) IRRa (95% CI) P B (SE) IRRa (95% CI) P
Condition (+/− email) −1.41 (0.59) 0.24 (0.08–0.77) .016 −1.43 (0.56) 0.24 (0.08–0.73) .012
Gender −0.34 (0.22) 0.72 (0.47–1.09) .121 −0.28 (0.20) 0.75 (0.50–1.13) .168
Tertiary education −0.50 (0.22) 0.61 (0.40–0.93) .020 −0.39 (0.20) 0.68 (0.46–1.01) .057
Age 0.02 (0.01) 1.02 (1.01–1.04) .013 0.02 (0.01) 1.02 (1.00–1.04) .036
RSQb 0.08 (0.13) 1.08 (0.85–1.38) .535 0.05 (0.12) 1.05 (0.83–1.34) .668
ACMTQ-AUTONc 0.17 (0.12) 1.18 (0.93–1.50) .166 0.13 (0.12) 1.14 (0.90–1.44) .286
ACMTQ-CNTRLd −0.42 (0.10) 0.66 (0.54–0.80) <.001 −0.34 (0.10) 0.71 (0.59–0.86) .001
Condition*ACMTQ-CNTRL 0.31 (0.16) 1.36 (1.00–1.85) .049 0.31 (0.15) 1.36 (1.01–1.83) .042
Open in a new tab
a

Incidence rate ratio = exp.(B).

b

Recovery style questionnaire.

c

Autonomous and controlled motivations for treatment questionnaire, autonomous scale.

d

Autonomous and controlled motivations for treatment questionnaire, control scale.

Fig. 4.

Fig. 4

Open in a new tab

Interaction between controlled motivation and condition on total activities.

3.4.2. Breadth of use

When controlling for other predictor variables, significant effects were found for additional email support (IRR = 0.24), older age (IRR = 1.02) and less controlled motivations for treatment (IRR = 0.71). The interaction term was also significant (IRR = 1.36). Autonomous motivations and recovery style did not predict breadth of use (see Table 4).

3.4.3. Active/passive use

The logistic regression demonstrated that condition, and autonomous motivations were significant predictors (see Table 5). Individuals receiving additional email support were four times more likely on average to be ‘active’ users. With every additional unit increase in autonomous motivations for treatment, individuals were 50% less likely on average to be ‘passive’ users.

Table 5.

Logistic regression for active use profile.

Predictors B (SE) ExpB
(95% CI)		 P
Condition (+/− email) 1.41 (0.59) 4.08 (1.28–12.94) .017
Gender 0.096 (0.59) 2.61 (0.82–8.31) .105
Tertiary education 0.11 (0.56) 1.12 (0.37–3.36) .842
Age 0.001 (0.02) 1.00 (0.96–1.05) .958
RSQa 0.41 (0.36) 1.50 (0.74–3.03) .260
ACMTQ-AUTONb −0.69 (0.34) 0.50 (0.25–0.98) .044
ACMTQ-CNTRLc 0.17 (0.24) 1.18 (0.74–1.90) .490
Open in a new tab
a

Recovery style questionnaire.

b

Autonomous and controlled motivations for treatment questionnaire, autonomous scale.

c

Autonomous and controlled motivations for treatment questionnaire, control scale.

3.5. Sensitivity analyses

When analyses were rerun excluding four participants with missing baseline data, findings with regards to predictor variables and active and passive use remained robust. However, retaining all other variables, female gender became a significant predictor of a greater total number of activities (B(SE) = −0.49(0.22), P = .024) and number of unique activities (B(SE) = −0.46(0.21), P = .024). Nonetheless, due to the possibility of type I error, the original planned analyses using imputed variables were considered the more conservative and reliable.

4. Discussion

4.1. Principal findings

This study demonstrated that both individual- and intervention- related factors were predictive of engagement with a self-guided online intervention for psychosis. Individuals who received weekly, asynchronous emails from an online coach were more engaged, and used the website more actively than those who did not receive email support. Individuals with more controlled, i.e., external, motivations for using the website had lower levels of engagement, which moderated the impact of email support. Conversely, participants who were autonomously motivated to use the site, were more actively engaged with the resource. Demographic variables including age and education were also predictive of engagement. The findings of this study highlight important factors that should be considered when designing and implementing digital interventions for individuals with psychosis.

4.2. Predictors of engagement

4.2.1. Asynchronous email

This is the first study to our knowledge to systematically investigate the impact of asynchronous email support on engagement with a self-guided online intervention for individuals with psychosis. The results indicate that the presence of additional emails had a substantial impact on engagement: individuals receiving emails used the website on average approximately 40% more than those accessing the website independently. When controlling for other measured variables, participants receiving emails completed approximately 75% more total activities, than those using the website independently. This is consistent with findings with reviews in both other clinical populations and psychosis, where online mental health interventions with some form of guidance have significantly higher levels of adherence compared to interventions without guidance (Baumeister et al., 2014; Killikelly et al., 2017).

Asynchronous email may be an especially appealing modality of support due to the limited resource requirements and potentially low costs. Previous investigation has highlighted that qualification of the support provider does not impact on adherence (Baumeister et al., 2014; Titov et al., 2010). In the current study, emails were sent by volunteers and staff of a community mental health service helpline, who were not registered health practitioners. Whilst we did not specifically investigate qualification of coaches on engagement in this study, our results are congruent with previous findings and suggest that clinical expertise is not required to successfully promote engagement. Emails were also drafted from pre-developed templates, reducing the amount of time required to provide support. Previous economic evaluation has highlighted guided online interventions as a cost-effective method of mental health service delivery (Donker et al., 2015). Although we did not conduct a formal cost analysis in this study, similar support could be provided to future interventions in a cost-effective manner.

In the current study, participants were sent emails on a weekly basis during the intervention period. However, the optimal ‘dose’ of support required to promote engagement with online interventions is not yet clear. Increasing the frequency of supportive contact from one to five days a week resulted in increased engagement with a mobile application for schizophrenia during initial evaluation (Schlosser et al., 2016). However, further research is required to specifically investigate the impact of varying support on engagement. It may be that some initial contact, coupled with intermittent support is also adequate to promote engagement with self-guided digital interventions. Different individuals may require more or less contact depending on their motivations and preferences (Mohr et al., 2011). Determination of this will further aid to increase effectiveness of email support in promoting engagement.

4.2.2. Motivations for treatment

Congruent with hypotheses, high levels of controlled motivations, (reflecting external motivations to use the site) predicted lower rates of engagement. Our findings are consistent with previous qualitative investigation into online mental health seeking behaviour by individuals with psychosis (Aref-Adib et al., 2016). When online resources were recommended by family members or mental health clinicians, consumers reported a lack of motivation to utilise the resources and had only brief encounters with materials. Our findings extend on these qualitative reports, and in alignment with self-determination theory, provide support that external motivations for treatment are insufficient for continued engagement with online interventions for psychosis. Individuals with more autonomous motivations for using the website, were more likely to be more active users of the website. This may suggest that individuals who are internally driven to utilise an online resource, may participate more actively in accordance with personal interest and intrinsic factors.

In contrast to our hypotheses, when controlled motivation were low, receiving the email was associated with higher levels of overall use and broader exposure to the website Our findings suggest that individuals who use a resource based on external motivating factors are likely to have limited engagement with an online resource regardless of support provided. As such, mere recommendations are unlikely to result in engagement with online interventions. Our findings suggest that self-guided online interventions will not be optimally used when consumers need to be persuaded to utilise them. This may be relevant when considering the role of digital interventions in mental health care for individuals with psychosis more generally. Recent research highlighted variance in awareness of and interest in technology across individuals with mental illness, including psychosis (Carpenter-Song et al., 2018). There may be a limited role in mental health treatment for online interventions amongst individuals with low interest. Alternatively, these individuals may represent a subgroup requiring initial face-to-face support to identify personally relevant goals, values or opportunities associated with these interventions. A potential model may be blended care, augmenting face-to-face treatment with use of digital interventions. Recent research has demonstrated that this form of therapy is feasible, and effective in both common mental health disorders and more serious mental illness (Depp et al., 2018; Erbe et al., 2017). In considering our findings, preliminary face-to-face sessions might benefit from focusing on cultivating personally relevant motivations for using a resource.

4.2.3. Demographic variables

Our finding that participants with a tertiary education were more engaged, is consistent with recent research where education has been positively associated with general Internet use and utilising the Internet for medical information (Baup and Verdoux, 2017; Välimäki et al., 2017). Further education may reflect increased cognitive capacities, better equipping individuals to engage with content of online interventions. Conversely, amongst individuals with psychosis, tertiary educated individuals may represent a group of individuals with potentially higher socio-economic status and more stable access to Internet. Future research may seek to investigate explanations for this finding.

Older age was positively associated with greater and broader engagement with the website in this study. This finding is consistent with previous research finding higher levels of engagement with an illness self-management mobile app amongst individuals aged 30–60 compared to those aged 18–29 (Ben-Zeev et al., 2016). It has previously been assumed that utilisation of digital technology was exclusive to younger generations. Our findings directly negate this assumption and suggest that older demographic age groups should not be excluded from online interventions. As smartphone ownership continues to increase and Internet literacy and access become ubiquitous it is likely that these assumptions will further dissipate.

4.2.4. Recovery style

In contrast to hypotheses based on findings in face-to-face treatments (Startup et al., 2006; Tait et al., 2003), recovery style did not predict engagement with the SMART website. This may be the result of the average reliability of the measure in this study (Cronbach's α = 0.67). Alternatively, this finding may be explained by the absence of any participants with a strong ‘sealing over’ recovery style in this study. It may be that individuals with a sealing over recovery style are inherently unlikely to participate in research examining a recovery-oriented digital resource – potentially due to a lack of interest. This finding may itself be representative of a lack of engagement. If possible, further research may seek to include more participants with higher levels of sealing over recovery styles.

4.3. Limitations

We relied on website log data to measure participants engagement with the website. Log data has the advantage of being objective and can provide unique insights about patterns and predictors of use. However, log data cannot capture an important element of engagement: user experience and perceptions of the intervention (Short et al., 2018). Analysis of log data is also unable to measure users' engagement with behavioural or attitudinal aspects of the intervention outside of website use (Yardley et al., 2016). In utilising log data to operationalise engagement, we also assumed that higher levels of use reflected more positive engagement. Whilst some level of website use is a requisite to obtain clinical benefit, the relationship between amount of use and associated benefit may not necessarily be linear (Donkin et al., 2013). The level of engagement required to obtain any clinical benefit may vary between individuals (Yardley et al., 2016) and thus assuming a positive linear relationship may be insufficiently specific. Future research may aim to overcome this limitation by investigating the mediating effect of intervention use on positive outcomes and utilising mixed-methods approaches to provide a complete picture of engagement with online interventions in this population.

In this study, we aimed to minimise the impact of confounding factors associated with the research project. For example, participants were reimbursed for completion of baseline questionnaires prior to accessing the website, but not thereafter. However, participants were contacted during the intervention period by telephone or email as reminders for questionnaire completion. This may have had an unintended impact on engagement (Gerhards et al., 2011). Participation in this study required online registration, eligibility screening, and questionnaire completion before accessing website content. It may be that, as a result of participation, individuals in the current study were more highly engaged than individuals accessing online interventions in a naturalistic setting. Engagement with online interventions has been consistently higher in the context of research participation, than in the real world environment (Baumel and Kane, 2018; Christensen et al., 2009; Torous et al., 2017). Further research in real-world context is required to examine if the currently identified predictors remain consistent.

4.4. Conclusion

This study revealed both intervention-and individual- related variables associated with engagement with a self-guided online intervention for psychosis. These findings are novel: To our knowledge, no study to date has undertaken specific investigation of what factors may promote engagement with self-guided online interventions in this population. Our findings suggest that the provision of asynchronous email support and a lack of external motivations for treatment are import predictors of engagement. Asynchronous email presents as a cost-effective and low intensity adjunct to online interventions with potential to improve engagement. However, regardless of additional support such as emails, consumers may be unlikely to engage with these resources when motivated by external reasons such as pleasing others. Attention may instead turn to invoking internal motivations or interests to promote use before beginning these potentially adjunct treatments in those who lack intrinsic motivations for treatment.

Our study adds to the growing body of literature demonstrating that individuals with psychosis are willing and able to utilise digital mental health resources. As interest in how digital technology can be utilised for mental health in psychosis increases, understanding who is likely to utilise interventions becomes an important research target. This examination of specific predictors in a large sample size highlights various targets that can inform future design of digital interventions for individuals with psychosis. Future research should aim to examine whether the predictor variables identified in this study apply to other online interventions for psychosis.

Declaration of competing interest

None.

Acknowledgments

Acknowledgements

The authors gratefully acknowledge the participants, Tara Smark, Rosalie Frankish, Wellways Australia volunteers and staff, members of the PERISCOPE Lived Experience Advisory Panel, investigators, research staff, and service partners of the SMART research program.

Funding

This work was supported by the Victoria Department of Health and Human Services Mental Illness Research Fund (MIRF33) and an Australian Government Research Training Program scholarship awarded to CA.

References

  1. Alfonsson S., Olsson E., Hursti T. Motivation and treatment credibility predicts dropout, treatment adherence, and clinical outcomes in an internet-based cognitive behavioral relaxation program: a randomized controlled trial. J. Med. Internet Res. 2016;18(3) doi: 10.2196/jmir.5352. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Alvarez-Jimenez M., Bendall S., Lederman R., Wadley G., Chinnery G., Vargas S.…Gleeson J.F. On the HORYZON: moderated online social therapy for long-term recovery in first episode psychosis. Schizophr. Res. 2013;143(1):143–149. doi: 10.1016/j.schres.2012.10.009. [DOI] [PubMed] [Google Scholar]
  3. Alvarez-Jimenez M., Alcazar-Corcoles M.A., González-Blanch C., Bendall S., McGorry P.D., Gleeson J.F. Online, social media and mobile technologies for psychosis treatment: a systematic review on novel user-led interventions. Schizophr. Res. 2014;156(1):96–106. doi: 10.1016/j.schres.2014.03.021. [DOI] [PubMed] [Google Scholar]
  4. Anthony W.A. Recovery from mental illness: the guiding vision of the mental health service system in the 1990s. Psychosocial Rehabilitation Journal. 1993;16(4):11. [Google Scholar]
  5. Aref-Adib G., O'Hanlon P., Fullarton K., Morant N., Sommerlad A., Johnson S., Osborn D. A qualitative study of online mental health information seeking behaviour by those with psychosis. BMC Psychiatry. 2016;16(1):232. doi: 10.1186/s12888-016-0952-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Baltierra N.B., Muessig K.E., Pike E.C., LeGrand S., Bull S.S., Hightow-Weidman L.B. More than just tracking time: complex measures of user engagement with an internet-based health promotion intervention. J. Biomed. Inform. 2016;59:299–307. doi: 10.1016/j.jbi.2015.12.015. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Baumeister H., Reichler L., Munzinger M., Lin J. The impact of guidance on internet-based mental health interventions: a systematic review. Internet Interv. 2014;1(4):205–215. [Google Scholar]
  8. Baumel A., Kane J.M. Examining predictors of real-world user engagement with self-guided eHealth interventions: analysis of mobile apps and websites using a novel dataset. J. Med. Internet Res. 2018;20(12) doi: 10.2196/11491. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Baup H., Verdoux H. Frequency and pattern of internet use in patients with schizophrenia or bipolar disorders seeking medical information. Psychiatry Research. 2017;247(Supplement C):152–154. doi: 10.1016/j.psychres.2016.11.028. [DOI] [PubMed] [Google Scholar]
  10. Beatty L., Binnion C. A systematic review of predictors of, and reasons for, adherence to online psychological interventions. International Journal of Behavioral Medicine. 2016;23(6):776–794. doi: 10.1007/s12529-016-9556-9. [DOI] [PubMed] [Google Scholar]
  11. Bell I.H., Lim M.H., Rossell S.L., Thomas N. Ecological momentary assessment and intervention in the treatment of psychotic disorders: a systematic review. Psychiatr. Serv. 2017;68(11):1172–1181. doi: 10.1176/appi.ps.201600523. [DOI] [PubMed] [Google Scholar]
  12. Ben-Zeev D., Brenner C.J., Begale M., Duffecy J., Mohr D.C., Mueser K.T. Feasibility, acceptability, and preliminary efficacy of a smartphone intervention for schizophrenia. Schizophr. Bull. 2014;40(6):1244–1253. doi: 10.1093/schbul/sbu033. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Ben-Zeev D., Scherer E.A., Gottlieb J.D., Rotondi A.J., Brunette M.F., Achtyes E.D.…Begale M. mHealth for schizophrenia: patient engagement with a mobile phone intervention following hospital discharge. JMIR Mental Health. 2016;3(3) doi: 10.2196/mental.6348. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Berger T., Caspar F., Richardson R., Kneubühler B., Sutter D., Andersson G. Internet-based treatment of social phobia: a randomized controlled trial comparing unguided with two types of guided self-help. Behav. Res. Ther. 2011;49(3):158–169. doi: 10.1016/j.brat.2010.12.007. [DOI] [PubMed] [Google Scholar]
  15. Berger T., Hämmerli K., Gubser N., Andersson G., Caspar F. Internet-based treatment of depression: a randomized controlled trial comparing guided with unguided self-help. Cogn. Behav. Ther. 2011;40(4):251–266. doi: 10.1080/16506073.2011.616531. [DOI] [PubMed] [Google Scholar]
  16. Berry N., Lobban F., Emsley R., Bucci S. Acceptability of interventions delivered online and through mobile phones for people who experience severe mental health problems: a systematic review. J. Med. Internet Res. 2016;18(5) doi: 10.2196/jmir.5250. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Bonet L., Izquierdo C., Escartí M.J., Sancho J.V., Arce D., Blanquer I., Sanjuan J. Use of mobile technologies in patients with psychosis: a systematic review. Revista de Psiquiatría y Salud Mental (English Edition) 2017;10(3):168–178. doi: 10.1016/j.rpsm.2017.01.003. [DOI] [PubMed] [Google Scholar]
  18. Carpenter-Song E., Noel V.A., Acquilano S.C., Drake R.E. Real-world technology use among people with mental illnesses: qualitative study. JMIR Mental Health. 2018;5(4) doi: 10.2196/10652. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Christensen H., Griffiths K.M., Farrer L. Adherence in internet interventions for anxiety and depression: systematic review. J. Med. Internet Res. 2009;11(2) doi: 10.2196/jmir.1194. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Couper P.M., Alexander L.G., Zhang N., Little J.A.R., Maddy N., Nowak A.M.…Cole Johnson C. Engagement and retention: measuring breadth and depth of participant use of an online intervention. J. Med. Internet Res. 2010;12(4) doi: 10.2196/jmir.1430. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Deci E., Ryan R.M. Springer Science & Business Media; 1985. Intrinsic Motivation and Self-Determination in Human Behavior. [Google Scholar]
  22. Dennison L., Morrison L., Lloyd S., Phillips D., Stuart B., Williams S.…Yardley L. Does brief telephone support improve engagement with a web-based weight management intervention? Randomized controlled trial. J. Med. Internet Res. 2014;16(3) doi: 10.2196/jmir.3199. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Depp C.A., Perivoliotis D., Holden J., Dorr J., Granholm E.L. Single-session mobile-augmented intervention in serious mental illness: a three-arm randomized controlled trial. Schizophrenia Bulletin. 2018;sby135 doi: 10.1093/schbul/sby135. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Donker T., Blankers M., Hedman E., Ljotsson B., Petrie K., Christensen H. Economic evaluations of internet interventions for mental health: a systematic review. Psychol. Med. 2015;45(16):3357–3376. doi: 10.1017/S0033291715001427. [DOI] [PubMed] [Google Scholar]
  25. Donkin L., Hickie B.I., Christensen H., Naismith L.S., Neal B., Cockayne L.N., Glozier N. Rethinking the dose-response relationship between usage and outcome in an online intervention for depression: randomized controlled trial. J. Med. Internet Res. 2013;15(10) doi: 10.2196/jmir.2771. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Drake R.E., Whitley R. Recovery and severe mental illness: description and analysis. Can. J. Psychiatry. 2014;59(5):236–242. doi: 10.1177/070674371405900502. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Drayton M., Birchwood M., Trower P. Early attachment experience and recovery from psychosis. Br. J. Clin. Psychol. 1998;37(3):269–284. doi: 10.1111/j.2044-8260.1998.tb01385.x. [DOI] [PubMed] [Google Scholar]
  28. Erbe D., Eichert H.-C., Riper H., Ebert D.D. Blending face-to-face and internet-based interventions for the treatment of mental disorders in adults: systematic review. J. Med. Internet Res. 2017;19(9) doi: 10.2196/jmir.6588. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Eysenbach G. The law of attrition. J. Med. Internet Res. 2005;7(1) doi: 10.2196/jmir.7.1.e11. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Farkas M. The vision of recovery today: what it is and what it means for services. World Psychiatry. 2007;6(2):68. [PMC free article] [PubMed] [Google Scholar]
  31. First M.B., Spitzer R.L., Gibbon M., Williams J.B.W. Biometrics Research Department, New York State Psychiatric Institute; New York: 2002. Structured Clinical Interview for DSM-IV-TR Axis I Disorders. [Google Scholar]
  32. Firth J., Cotter J., Touros J., Bucci S., Firth J.A., Young A.R. Mobile phone ownership and endorsement of “mHealth” among people with psychosis: a meta-analysis of cross-sectional studies. Schizophr. Bull. 2016;42(2):448–455. doi: 10.1093/schbul/sbv132. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Gay K., Torous J., Joseph A., Pandya A., Duckworth K. Digital technology use among individuals with schizophrenia: results of an online survey. JMIR Mental Health. 2016;3(2) doi: 10.2196/mental.5379. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Gerhards S.A.H., Abma T.A., Arntz A., de Graaf L.E., Evers S.M.A.A., Huibers M.J.H., Widdershoven G.A.M. Improving adherence and effectiveness of computerised cognitive behavioural therapy without support for depression: a qualitative study on patient experiences. J. Affect. Disord. 2011;129(1–3):117–125. doi: 10.1016/j.jad.2010.09.012. [DOI] [PubMed] [Google Scholar]
  35. Granholm E., Ben-Zeev D., Link P.C., Bradshaw K.R., Holden J.L. Mobile assessment and treatment for schizophrenia (MATS): a pilot trial of an interactive text-messaging intervention for medication adherence, socialization, and auditory hallucinations. Schizophr. Bull. 2012;38(3):414–425. doi: 10.1093/schbul/sbr155. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Killikelly C., He Z., Reeder C., Wykes T. Improving adherence to web-based and mobile technologies for people with psychosis: systematic review of new potential predictors of adherence. JMIR Mhealth Uhealth. 2017;5(7) doi: 10.2196/mhealth.7088. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Klein B., Meyer D., Austin D.W., Kyrios M. Anxiety online—a virtual clinic: preliminary outcomes following completion of five fully automated treatment programs for anxiety disorders and symptoms. J. Med. Internet Res. 2011;13(4) doi: 10.2196/jmir.1918. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Leamy M., Bird V., Le Boutillier C., Williams J., Slade M. Conceptual framework for personal recovery in mental health: systematic review and narrative synthesis. Br. J. Psychiatry. 2011;199(6):445–452. doi: 10.1192/bjp.bp.110.083733. [DOI] [PubMed] [Google Scholar]
  39. Lederman R., Wadley G., Gleeson J., Bendall S., Alvarez-Jimenez M. Moderated online social therapy: designing and evaluating technology for mental health. ACM Transactions on Computer-Human Interaction (TOCHI) 2014;21(1):1–26. [Google Scholar]
  40. Lindner P., Olsson E.L., Johnsson A., Dahlin M., Andersson G., Carlbring P. The impact of telephone versus e-mail therapist guidance on treatment outcomes, therapeutic alliance and treatment engagement in internet-delivered CBT for depression: a randomised pilot trial. Internet Interv. 2014;1(4):182–187. [Google Scholar]
  41. Maijala R., Anttila M., Koivunen M., Pitkänen A., Kuosmanen L., Välimäki M. Internet delivered question and answer column for patients with schizophrenia. Informatics for Health & Social Care. 2015;40(3):267–278. doi: 10.3109/17538157.2014.924946. [DOI] [PubMed] [Google Scholar]
  42. McGlashan T.H. Recovery style from mental illness and long-term outcome. J. Nerv. Ment. Dis. 1987;175 doi: 10.1097/00005053-198711000-00006. [DOI] [PubMed] [Google Scholar]
  43. McGlashan T.H., Levy S.T., Carpenter W.T. Integration and sealing over: clinically distinct recovery styles from schizophrenia. Arch. Gen. Psychiatry. 1975;32(10):1269–1272. doi: 10.1001/archpsyc.1975.01760280067006. [DOI] [PubMed] [Google Scholar]
  44. Mohr D.C., Cuijpers P., Lehman K. Supportive accountability: a model for providing human support to enhance adherence to eHealth interventions. J. Med. Internet Res. 2011;13(1) doi: 10.2196/jmir.1602. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Mohr D.C., Duffecy J., Ho J., Kwasny M., Cai X., Burns M.N., Begale M. A randomized controlled trial evaluating a manualized telecoaching protocol for improving adherence to a web-based intervention for the treatment of depression. PLoS One. 2013;8(8) doi: 10.1371/journal.pone.0070086. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Moitra E., Gaudiano B.A., Davis C.H., Ben-Zeev D. Feasibility and acceptability of post-hospitalization ecological momentary assessment in patients with psychotic-spectrum disorders. Compr. Psychiatry. 2017;74:204–213. doi: 10.1016/j.comppsych.2017.01.018. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. O'Hanlon P., Aref-Adib G., Fonseca A., Lloyd-Evans B., Osborn D., Johnson S. Tomorrow's world: current developments in the therapeutic use of technology for psychosis. BJPsych Advances. 2016;22(5):301–310. [Google Scholar]
  48. Robotham D., Satkunanathan S., Doughty L., Wykes T. Do we still have a digital divide in mental health? A five-year survey follow-up. J. Med. Internet Res. 2016;18(11) doi: 10.2196/jmir.6511. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Rotondi A.J., Sinkule J., Haas G.L., Spring M.B., Litschge C.M., Newhill C.E.…Anderson C.M. Designing websites for persons with cognitive deficits: design and usability of a psychoeducational intervention for persons with severe mental illness. Psychol. Serv. 2007;4(3):202–224. doi: 10.1037/1541-1559.4.3.202. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Rotondi A.J.P., Anderson C.M.P., Haas G.L.P., Eack S.M.P., Spring M.B.P., Ganguli R.M.D.…Rosenstock J.M.D. Web-based psychoeducational intervention for persons with schizophrenia and their supporters: one-year outcomes. Psychiatr. Serv. 2010;61(11):1099–1105. doi: 10.1176/ps.2010.61.11.1099. [DOI] [PubMed] [Google Scholar]
  51. Ryan R.M., Deci E.L. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am. Psychol. 2000;55(1):68–78. doi: 10.1037//0003-066x.55.1.68. [DOI] [PubMed] [Google Scholar]
  52. Ryan R.M., Deci E.L. A self-determination theory approach to psychotherapy: the motivational basis for effective change. Can. Psychol. 2008;49(3):186. [Google Scholar]
  53. Ryan R.M., Patrick H., Deci E.L., Williams G.C. Facilitating health behaviour change and its maintenance: interventions based on self-determination theory. European Health Psychologist. 2008;10(1):2–5. [Google Scholar]
  54. Scherer A.E., Ben-Zeev D., Li Z., Kane M.J. Analyzing mHealth engagement: joint models for intensively collected user engagement data. JMIR Mhealth Uhealth. 2017;5(1) doi: 10.2196/mhealth.6474. [DOI] [PMC free article] [PubMed] [Google Scholar]
  55. Schlosser D., Campellone T., Kim D., Truong B., Vergani S., Ward C., Vinogradov S. Feasibility of PRIME: a cognitive neuroscience-informed mobile app intervention to enhance motivated behavior and improve quality of life in recent onset schizophrenia. JMIR Research Protocols. 2016;5(2) doi: 10.2196/resprot.5450. [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Short C.E., Rebar A.L., Plotnikoff R.C., Vandelanotte C. Designing engaging online behaviour change interventions: a proposed model of user engagement. European Health Psychologist. 2015;17(1):32–38. [Google Scholar]
  57. Short C.E., DeSmet A., Woods C., Williams S.L., Maher C., Middelweerd A.…Crutzen R. Measuring engagement in eHealth and mHealth behavior change interventions: viewpoint of methodologies. J. Med. Internet Res. 2018;20(11) doi: 10.2196/jmir.9397. [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. Simco R., McCusker J., Sewitch M. Adherence to self-care interventions for depression or anxiety: a systematic review. Health Educ. J. 2014;73(6):714–730. [Google Scholar]
  59. Staring A.B.P., van der Gaag M., Mulder C.L. Recovery style predicts remission at one-year follow-up in outpatients with schizophrenia spectrum disorders. J. Nerv. Ment. Dis. 2011;199(5) doi: 10.1097/NMD.0b013e3182174e97. [DOI] [PubMed] [Google Scholar]
  60. Startup M., Wilding N., Startup S. Patient treatment adherence in cognitive behaviour therapy for acute psychosis: the role of recovery style and working alliance. Behavioural and Cognitive Psychotherapy. 2006;34(02):191–199. (doi:10.1017S1352465805002535) [Google Scholar]
  61. Tait L., Birchwood M.A.X., Trower P. Predicting engagement with services for psychosis: insight, symptoms and recovery style. Br. J. Psychiatry. 2003;182(2):123–128. doi: 10.1192/bjp.182.2.123. [DOI] [PubMed] [Google Scholar]
  62. Thomas N., Farhall J., Foley F., Leitan N.D., Villagonzalo K.-A., Ladd E.…Kyrios M. Promoting personal recovery in people with persisting psychotic disorders: development and pilot study of a novel digital intervention. Frontiers in Psychiatry. 2016;7:196. doi: 10.3389/fpsyt.2016.00196. [DOI] [PMC free article] [PubMed] [Google Scholar]
  63. Thomas N., Farhall J., Foley F., Rossell S.L., Castle D., Ladd E.…Kyrios M. Randomised controlled trial of a digitally assisted low intensity intervention to promote personal recovery in persisting psychosis: SMART-therapy study protocol. BMC Psychiatry. 2016;16(1):312. doi: 10.1186/s12888-016-1024-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  64. Thomas N., Foley F., Lindblom K., Lee S. Are people with severe mental illness ready for online interventions? Access and use of the internet in Australian mental health service users. Australasian Psychiatry. 2017;25(3):257–261. doi: 10.1177/1039856217689913. [DOI] [PubMed] [Google Scholar]
  65. Titov N., Andrews G., Davies M., McIntyre K., Robinson E., Solley K. Internet treatment for depression: A randomized controlled trial comparing clinician vs. technician assistance. PLoS One. 2010;5(6) doi: 10.1371/journal.pone.0010939. [DOI] [PMC free article] [PubMed] [Google Scholar]
  66. Torous J., Staples P., Slaters L., Adams J., Sandoval L., Onnela P., J., & Keshavan, M. Characterizing smartphone engagement for schizophrenia: results of a naturalist mobile health study. Clinical Schizophrenia & Related Psychoses. 2017 doi: 10.3371/CSRP.JTPS.071317. [DOI] [PubMed] [Google Scholar]
  67. Välimäki M., Kuosmanen L., Hätönen H., Koivunen M., Pitkänen A., Athanasopoulou C., Anttila M. Connectivity to computers and the internet among patients with schizophrenia spectrum disorders: a cross-sectional study. Neuropsychiatr. Dis. Treat. 2017;13:1201–1209. doi: 10.2147/NDT.S130818. [DOI] [PMC free article] [PubMed] [Google Scholar]
  68. Yardley L., Spring B.J., Riper H., Morrison L.G., Crane D.H., Curtis K.…Blandford A. Understanding and promoting effective engagement with digital behavior change interventions. Am. J. Prev. Med. 2016;51(5):833–842. doi: 10.1016/j.amepre.2016.06.015. [DOI] [PubMed] [Google Scholar]
  69. Zarski A.-C., Lehr D., Berking M., Riper H., Cuijpers P., Ebert D.D. Adherence to internet-based mobile-supported stress management: a pooled analysis of individual participant data from three randomized controlled trials. J. Med. Internet Res. 2016;18(6) doi: 10.2196/jmir.4493. [DOI] [PMC free article] [PubMed] [Google Scholar]
  70. Zuroff D.C., Koestner R., Moskowitz D.S., McBride C., Marshall M., Bagby M.R. Autonomous motivation for therapy: a new common factor in brief treatments for depression. Psychother. Res. 2007;17(2):137–147. [Google Scholar]

Articles from Internet Interventions are provided here courtesy of Elsevier

RESOURCES