Effectiveness and cost-effectiveness of a fully self-guided internet-based intervention for sub-clinical social anxiety symptoms: Protocol for a randomised controlled trial

Abstract

Design and objective

This paper describes the protocol for a large-scale pragmatic, randomised controlled trial and economic evaluation to investigate the effectiveness and cost-effectiveness of the self-directed E-Couch social anxiety module versus a waiting list control condition, for reducing sub-clinical social anxiety symptoms in the general population.

Study population

Community-based adults (aged 18+) with social anxiety symptoms that do not meet the criteria for social anxiety disorder recruited via a direct-to-consumer advertisement on national websites.

Intervention and control

Intervention is the self-guided E-Couch social anxiety module. Control group participants are placed on a waiting list to receive the intervention at the end of the trial. Both groups receive email and text message reminders.

Outcome measures

The primary outcome will be change in self-reported social anxiety score using the Social Phobia Inventory (SPIN). Secondary outcomes will be the changes in the following self-report measures: Brief Fear of Negative Evaluation scale (BFNE-S); depression (CES-D); mental wellbeing (SWEMWEBS); health status (SF36); use of health services; safety events; and adherence, retention, and attrition rates. All measures will be administered at baseline, 6 weeks, and 3, 6 and 12 months.

Analysis

A mixed effects model will be used to analyse the effect of the intervention on the primary and secondary outcomes (intention to treat analysis). Secondary analyses will explore moderators and mediators of effect. A prospective economic evaluation, conducted from a NHS and social care perspective, will provide estimates of cost utility and cost-effectiveness. An interview study will be conducted with 20 participants to explore issues including acceptability, adherence, retention and attrition.

Trial registration numbers

NCT02451878 and ISRCTN15819951

Introduction

The internet is playing an ever more important role in our social and emotional lives. The proliferation and widespread adoption of connected devices such as smartphones and tablet computers are leading to an ‘always on’ culture where people are becoming used to accessing online content for many public and private services, from any location. People go online in large numbers to find self-help solutions for a range of physical and mental health problems, and for broader health-related activities such as diet and exercise.¹ Delivering self-help tools using online platforms offers a low-cost and highly scalable approach that could potentially be harnessed by public health practitioners and policymakers seeking to tackle population-wide problems.² However, the public health community has been slow to capitalise on this opportunity, and in general the pace of innovation has run ahead of both policymakers and evaluators. Thus, for many of these online self-help approaches, we do not have good evidence as to whether they are helpful or not, despite their wide use.

Social anxiety is a major public health problem that often goes unrecognised and untreated. Epidemiological studies indicate that social anxiety disorder is one of the more common mental disorders, with a 1-year prevalence of 4%–6%³ and a lifetime prevalence of up to 12%.⁴ It is characterised by an intense and persistent fear of being negatively evaluated in social or performance situations. Effective treatments exist for social anxiety disorder. The National Institute for Health and Care Excellence (NICE) guideline on the assessment and treatment of social anxiety disorder⁵ recommends that all NHS patients diagnosed with social anxiety disorder should be offered a course of individual cognitive behavioural therapy based on either the Clark and Wells or the Heimberg model of treatment.⁵

Substantially less attention has been directed to sub-clinical social anxiety, in which symptoms do not reach the criteria for a diagnosis. Nevertheless, such social anxiety symptoms cause significant problems and cost at the individual and societal levels, with many people not fulfilling their potential due to avoidance of social situations, which affects their relationships and educational and occupational achievements. Accordingly, the current study is focused on sub-clinical social anxiety symptoms that do not reach the criteria for a diagnosis of the disorder.

There are currently no official diagnostic criteria for sub-clinical anxiety. At present, a score of 20 or above on the validated Social Phobia Inventory (SPIN) self-report questionnaire indicates clinical social anxiety. The population mean for SPIN is 11 or 12 (in previous studies). Based on this general population data, a range of 13–19 on the SPIN might be appropriate to capture people with sub-clinical social anxiety. The prevalence and impact of falling into the sub-clinical range are not currently known.

In this study we will test the benefit of a novel public health approach of online self-help to reduce the level of sub-clinical symptoms in the general population. A public health approach is appropriate, as sub-clinical social anxiety symptoms are very common and are continuously distributed throughout the population, and many people could potentially benefit from an accessible intervention to reduce these sub-clinical symptoms. Also, while there is evidence that therapist-guided internet-based social anxiety applications are effective as treatments, little is known about the effectiveness of fully self-directed interventions, which are becoming ever more available. This study will contribute to the evidence on the self-help delivery model, at least for a sub-clinical population.

Further aims of our work will be to learn more about the benefits and challenges of harnessing the internet to deliver public health interventions, and to explore methodological issues in undertaking large-scale online public health trials. This will include exploring issues of recruitment, acceptability, retention and attrition.

Background

E-Couch (https://ecouch.anu.edu.au) is an online toolkit of self-directed modules covering common mental health problems including social anxiety, generalised anxiety, depression, relationship breakdown, and loss and grief. It was developed by researchers at the Australian National University (ANU). The social anxiety module is based on cognitive behavioural therapy principles and includes components of known effectiveness in face-to-face therapy. It contains a literacy section and five toolkits comprising exposure practice, cognitive restructuring (modifying your thinking), attention practice, social skills training and relaxation. E-Couch is designed to be completed at the participant’s own pace. It is free to use, browser-based and widely accessible on a range of connected devices. Analysis of unpublished data collected from spontaneous visitors from around the world to the publicly accessible E-Couch site shows a statistically significant reduction in social anxiety symptoms among visitors to the social anxiety toolkit. For unguided self-help, higher credibility ratings of a treatment program have been shown to be associated with increased treatment adherence in patients with social anxiety disorder.⁶ We are interested in whether the social anxiety module is effective in people with sub-clinical social anxiety symptoms.

We are not aware of any previous studies of online self-help for sub-clinical social anxiety symptoms. In a small (n = 63) laboratory-based randomised controlled trial with individuals with high levels of social anxiety, Bowler et al. (2012) showed that the social phobia modules of the E-Couch toolkit resulted in a significantly greater reduction in social anxiety than control (effect size d = 1.0).⁷ Although there was no formal face-to-face therapeutic input in this context, the study did ensure that participants completed the set modules. Thus, to date, there have been no controlled studies of the intervention with a fully self-guided approach. To date, there have only been two trials comparing therapist-guided with unguided (or self-guided) internet-based treatment of social anxiety disorder. Undertaken by Titov et al. (2008), an intention-to-treat (ITT) analysis found that the unguided intervention was not effective relative to control. However, there was evidence that the program was helpful for the participants who completed the intervention.⁸ Berger et al.⁹ found that the internet therapy led to symptom reduction in both the therapist-guided and self-guided groups. Both studies highlighted the scope for internet-delivered treatment for social phobia.

A Cochrane review of media-delivered cognitive behavioural therapy (CBT) and behavioural therapy (BT) interventions for anxiety¹⁰ identified 15 studies of social anxiety disorder compared with no treatment. The meta-analysis of these studies found a significant benefit of the interventions, with a standard mean difference of 0.73 (95% confidence interval 0.59 to 0.87) favouring the media-delivered therapy. The Cochrane review identified four studies that compared a media-delivered intervention with face-to-face therapy, finding no significant difference in treatment effect (standard mean difference 0.02, 95% confidence interval −0.18 to 0.22). The authors noted that the effect sizes (for all anxiety interventions combined, not only social anxiety disorder) for these media-based interventions were greatest for internet-delivered CBT. The authors further noted that studies varied greatly in the level of therapist contact (if any) provided to the intervention group, and noted that while therapist contact was beneficial in a trial setting, it may not be practical for a pragmatic population-based approach to service delivery. The authors suggest that fully self-directed tools may have a role for those at the beginning of a stepped care pathway.

In summary, there is evidence that internet-based cognitive behavioural therapy for social anxiety disorder is effective; however, there is a lack of evidence on the value of self-guided internet interventions such as E-Couch with no therapist contact, and no evidence in sub-clinical populations. The self-guided approach requires further research, particularly given the potential resource savings over the therapist-guided approach if widely disseminated across the population, and the fact that many fully self-guided interventions, particularly in the form of smartphone apps, are being promoted without evidence.

Methods

Summary

A large pragmatic randomised controlled trial of the internet-based self-directed E-Couch social anxiety module versus a waiting list control condition, for the treatment of sub-clinical social anxiety symptoms, among a general population sample recruited from the UK population using direct-to-consumer advertisements placed on national websites.

Intervention

The E-Couch toolkit is available online for anyone who registers to use it (https://ecouch.anu.edu.au). For this study, a social anxiety-specific tool will be packaged using the E-Couch social anxiety content to create the intervention. We do not anticipate high levels of contamination in the control arm – we will not name the intervention ‘E-Couch’ in the study information (it will be referred to as an ‘online self-help program’), and E-Couch (and the social anxiety elements within the toolkit) are not widely known or used by the UK population. We will measure contamination through participant self-report at the time of final follow-up. This is a pragmatic study, and we would expect some participants to be also seeking other self-help, in both arms of the study.

Control group

Participants in the control group receive no intervention and are placed on a waiting list to receive the intervention at the end of the trial.

Study population

Internet users who consent to take part in a study for self-help with sub-clinical social anxiety symptoms. Individuals who are likely to have social anxiety disorder (as judged by scoring >19 on the self-completion 17-item SPIN screening measure)¹¹ will be given advice to seek help from health services for social anxiety disorder, for which evidence-based treatments exist.

Recruitment

We will recruit participants through advertisements placed on public websites providing health information, including the NHS website. This is a widely trusted source of health information, and the NHS branding is known to facilitate recruitment to research studies.¹²

Eligibility screening and consent

Individuals expressing an interest in the online advertisement by clicking on a web link will be directed to a study website containing full information on the trial and the opportunity to ask questions of the study team by email or telephone.

In line with our public health approach and the pragmatic nature of this study, our inclusion criteria will be kept very broad. Potential participants will be invited to complete screening questions that will determine whether they are eligible for participation. In the preliminary screening phase they will need to confirm the following: availability of a working email address (to respond to a confirmation email and receive reminders), availability of a mobile phone number (to receive text reminders), and that they are aged 18 or over and resident in the UK. We will exclude people who are currently receiving therapist-guided treatment for social anxiety disorder. Those who meet these preliminary screening criteria will be asked to provide online consent for the next phases of the study, including consent to complete screening measures and to be randomised to one of two conditions. Participants will be advised that they are free to withdraw at any time without giving a reason and that this will not affect their care. There are no discontinuation criteria apart from participant request.

Once individuals have completed the preliminary screening and consent process, they will be asked to complete the SPIN. If they score in the range 13–19 on the SPIN, they will be eligible for the trial as having sub-clinical social anxiety symptoms, and will be invited to complete the baseline measures described below. Based on the general population data available to us, the range of SPIN 13–19 was chosen to capture those people who score above the population mean (usually 11 or 12 in previous studies). If they score 20 or more, they will be excluded from the trial and given advice on seeking help. We will recontact these people (with their consent, by email) 1 month later to determine whether they have acted on our advice and sought other help. The follow-up email will ask them whether they have, and what help they have sought. At the time we send this email, we will also include repeat advice on how to seek help and what effective treatments are available. As there is no accepted screening approach for identifying people with sub-clinical symptoms, we will monitor recruitment and the performance of the SPIN tool. Recruitment was planned to start in April 2016 and last for 6 months.

Initial trial flow and collection of baseline measures

In addition to the SPIN, eligible participants will be asked to complete a battery of self-report baseline questionnaires, and in line with all other trial procedures, these will be collected online through the trial portal. These questionnaires will collect demographic characteristics (age, gender, marital status, education, employment, income, ethnicity and country of residence), and we will measure social anxiety symptoms (SPIN and Brief Fear of Negative Evaluation (BFNE-S) scale), depression (Center for Epidemiologic Studies Depression (CES-D)), mental wellbeing (Short Warwick-Edinburgh Mental Well-being Scale (SWEMWBS)), health status (Short Form Health Survey (SF-36)), and current use of health services and psychological interventions (including self-help).

Once baseline measures are completed, eligible participants will be randomised in a 1:1 ratio with no stratification, using a computer-generated random number sequence run through an automatic online program. Participants in the intervention arm will be given access to a password-protected website and will be encouraged to access and use the self-guided intervention over a period of up to 6 weeks (although they can work through the intervention at their own pace). A major lesson from our previous work on fully self-directed online interventions¹³ is that internet trial recruitment tends to be ‘easy in, easy out’ with high levels of attrition. We will use automated text (SMS) message and email reminders to reduce attrition. Participants will receive one text message within 24 hours of randomisation to thank them for participating and to remind them to access the intervention, and one further text message towards the end of the 6-week period to thank them again and to remind them that they will shortly be asked to complete follow-up measures. They will also receive three email reminders during the 6-week period (both intervention and control groups will receive the reminders at 1 week, 3 weeks and 5 weeks). Participants in the control arm will receive the same reminders for follow-up measures and will be given access to the intervention after final follow-up (12 months). Participants in the intervention arm will be offered continued access to the intervention once the trial is completed. Given that participants either receive the intervention or are on the waiting list, participants will not be blind to allocation. Researchers undertaking the analysis will be blind to allocation.

Follow-up measures

Self-report follow-up measures of social anxiety symptoms (SPIN and BFNE-S scales), depression (Center for Epidemiologic Studies Depression Scale Revised (CESD-R)), mental wellbeing (Short Warwick-Edinburgh Mental Well-Being Scale (SWEMWBS)), health status (SF-36), current use of health services and psychological interventions, and time off from work or study due to psychological problems will be taken at 6 weeks (following completion of the self-guided intervention), 3 months, 6 months and 12 months. At each follow-up point they will receive notification of follow-up measures by email and by text message, and if measures are not completed they will receive up to three follow-up emails. We will also record website usage data and measures of adherence to the intervention (numbers of logins and modules completed), and attrition from the trial (loss to follow-up at each time point). Weekly reports will be generated to track these factors. We will also record any safety (adverse) events – in this study, these are likely to be self-reported and captured in correspondence from participants. Since participant information and measures are collected online, they are automatically entered into a coded spreadsheet. This dataset will be transferred to a statistical software package for analysis, and data management checks will be carried out to ensure data quality. All participant information and measures will be stored securely, all data transfers will be encrypted, only members of the research team will have access to data, and all analyses will only use anonymised datasets.

Primary outcome

The primary outcome is the change from baseline in self-reported social anxiety as measured by the SPIN-17. The SPIN is a self-report questionnaire that includes 17 items assessing symptom domains of social anxiety. A high score indicates greater severity of social anxiety. This measure has been shown to have adequate internal consistency, test–retest reliability, construct validity and sensitivity to change following intervention.¹⁴ Change in SPIN-17 from baseline will be computed for each post-randomisation time point.

Secondary outcomes

The secondary outcomes are fear of negative evaluation measured using the eight-item BFNE-S scale;¹⁵ depression using the 20-item CESD-R;¹⁶ mental wellbeing using the seven-item version of the WEMWBS tool (SWEMWBS);¹⁷ general health status using the SF36, including both mental and physical component scores (MCS and PCS);¹⁸ and safety events.

Sample size

Although previous studies have reported a large treatment effect for supported internet-delivered interventions, we believe this treatment effect is too optimistic for a self-guided treatment in a sub-clinical population group, and is also likely to be smaller in pragmatic settings. For this reason, we intend to recruit 2104 participants (1052 per group) to this trial, which will provide 90% power to detect a small between group effect size of Cohen’s d = 0.2 at the 5% two-sided significance level. Although small at the level of the individual, such effects can translate into potentially large benefits across a population. This sample size has also accounted for a high level of potential attrition of up to 50% (although we aim to mitigate this, we need to be realistic about self-guided internet interventions and the likely level of attrition).

Statistical methods

The primary analysis will follow an ITT analysis. The researchers will endeavour to obtain full follow-up data on all participants to allow a full ITT analysis, but it is very likely that a modified ITT analysis will be carried out where data is missing due to withdrawal, loss to follow up or non-response to some questionnaire items. A modified ITT will include all participants with at least one outcome assessment. We will explore the mechanism of missing data by looking for associations between participant characteristics and the likelihood of non-response to questionnaires at different time points. This can be done using a regression model for binary outcomes (1 = response; 0 = non-response). If significant associations are observed, this lends weight to a missing at random (MAR) assumption. The primary outcome analysis will utilise a mixed effects model, which implicitly accounts for data under a MAR assumption.

We will perform sensitivity analysis for the primary outcome using methods that do not assume a MAR mechanism, such as pattern-mixture models, to assess the robustness of this assumption. If different results are obtained from a pattern-mixture model compared with the mixed effects model, it is likely that the MAR assumption is not valid. As this is a public health intervention for people who are not ill, a per-protocol analysis will also be undertaken to explore the effect of the intervention on participants who complete at least one module of the intervention and have at least one outcome assessment.

Two-sided significance tests will be carried out and 95% confidence intervals will be reported for estimates of treatment effect. The trial results will be reported in accordance with the Consolidated Standards of Reporting Trials (CONSORT) statement.¹⁹ A CONSORT flow chart detailing the number of participants screened, randomised and available for the analysis of the primary outcome will be presented. Reasons for withdrawal, exclusions or losses to follow-up will be shown at each stage of the study by randomised group.

A mixed effects linear model will be fitted to the primary outcome data. This model will utilise data collected at baseline, 6 weeks, and 3, 6 and 12 months following randomisation. The secondary outcomes (fear of negative evaluation, depression, mental well-being and general health) will follow the analysis for the primary outcome. Potential adverse events self-reported by the participants will be presented by randomised group.

Further analyses will include sensitivity analyses to explore the impact of missing data, and subgroup analyses to investigate factors that may moderate the effect of the intervention or influence adherence. For both primary and secondary outcomes, we will include subgroup analyses of baseline factors that may influence the intervention effect (moderators). Baseline subgroups will be compared by including interaction terms in the model (baseline subgroup by treatment group interaction). Baseline factors to be compared include age, gender, educational level and baseline level of sub-clinical social anxiety symptoms

Mediation analyses will also be undertaken to determine whether there is a dose–response effect of the intervention, with more interaction with the website equating to higher levels of treatment effect (examining ‘dose’ in terms of total usage of site and numbers of modules completed). For both groups, their usage of the intervention will be recorded (total number of logins to site, total time on site and total number of modules completed). These will be tallied so that the total at 6 weeks and 3, 6 and 12 months can be computed and used in the analysis. We will also undertake mediation analyses to explore the effect of receiving other help for sub-clinical social anxiety symptoms during the trial on the treatment effect observed. If the program is effective, those randomised to the control condition may be more likely to seek additional help elsewhere – a possible source of confounding that has the potential to reduce the observed effect. In addition, we will use a mediation model^20,21 to assess the influence of adherence/compliance to treatment outcome.

Economic evaluation

A prospective economic evaluation, conducted from an NHS and social care perspective, will be integrated into the trial to assess cost and cost-effectiveness of the self-guided internet-based intervention for people with sub-clinical social anxiety symptoms.

People currently receiving treatment for social anxiety are excluded from the study, and the intervention targets people who are unlikely to pursue help from NHS or social care for their anxiety. Therefore, we do not expect to see a significant direct impact on health and social care costs. We are particularly interested in whether suffering from these symptoms may impact on people’s productivity due to, for example, absence from work or study.

We intend to estimate the costs of developing, delivering and maintaining the intervention. Health service and social care utilisation data will be collected through participant self-report. Unit costs for service utilisation will be derived from standard national sources. Costs will be standardised to current prices where possible.

The effectiveness of the intervention will be measured using quality-adjusted life years (QALYs) with the under the curve approach. Health status will be measured at all time points using the SF-36, converted into SF-6D health utilities using established UK-based utility algorithms and combined with time duration data. The results of the economic evaluation will primarily be expressed in terms of incremental cost per QALY gained between the intervention and control groups. Non-parametric bootstrap estimation will be used to calculate 95% confidence intervals for mean difference of cost and QALYs between the trial groups, and incremental cost-effectiveness ratios. Sensitivity analyses will explore the implications of uncertainty on the incremental cost-effectiveness ratios and will consider the broader issue of the generalisability of the results.

Qualitative study

We will collect data on initial views and experiences of the intervention at 6 weeks for all participants in the intervention arm by providing a free text box at the end of collecting outcomes measures. An in-depth qualitative interview study will also be conducted with a subset of approximately 20 participants in the intervention arm at the 12-month follow-up. A maximum-variation sampling technique will be used to select a diverse sample of potential interviewees based on demographic information and degree of completion of the intervention. Interviews will be audio recorded, will last approximately 1 hour and will be conducted in the participant’s home or (if it is inconvenient or impractical to conduct a home visit) by telephone. They will follow a semi-structured topic guide, which will be piloted. Interview transcripts will be analysed iteratively and thematically using a constant comparative approach, to explore issues related to acceptability, usability, adherence, attrition, and perceived value and impact of the intervention. In addition, we will invite participants who withdraw from the study to take part in a short interview about reasons for withdrawal to explore barriers to taking part in an online-only study and the trial process as well as intervention-specific issues.

Data sharing

After analyses are complete, the datasets will be made available from the corresponding author on reasonable request.

Dissemination

All trial results will be written up by the research team in academic publications and also made available in lay summary format on the trial webpages where participants can see them. The funders and the ethics committees will receive a final report. Authorship will comply with International Committee of Medical Journal Editors guidelines.

Contributorship

JP and KG conceived the study and are co-principal investigators. All other authors contributed to writing the protocol and revising the manuscript. The trial sponsor is the University of Oxford, Joint Research Office, Block 60, Churchill Hospital, Old Road, Headington, Oxford, OX3 7LE. The Oxford Primary Care Clinical Trials Unit standard operating procedures apply to the conduct of the trial. The research team have responsibility for the study design; collection, management, analysis, and interpretation of data; writing of the report; and the decision to submit the report for publication. The funder and sponsor do not have authority over these activities. The study is overseen by an independent Trial Steering Committee (TSC). In this low-risk population-based study of a self-help intervention, the TSC also has the role of data monitoring. The TSC is chaired by Dr Peter Davidson of the University of Southampton and includes an independent trial statistician.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: KG, KB and AB were involved in the development of the E-Couch program and are employed by the Australian National University, which delivers the program to users worldwide.

Ethical approval

We received ethics committee approvals from both the University of Oxford and the Australian National University. The trial protocol is registered with ClinicalTrials.gov (NCT02451878) and with ISRCTN (ISRCTN15819951). This paper describes version 2 of the protocol, dated 18 February 2016.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the MQ charity under its PsyIMPACT funding call (www.joinMQ.org). KG is supported by NHMRC Fellowship 1059620. The ongoing delivery of E-Couch is enabled by funding from the Australian Commonwealth Department of Health. HA was funded by a NIHR National School for Primary Care Research Fellowship (up to September 2015). LL and YY are funded by the NIHR Oxford Biomedical Research Centre (Oxford BRC).

Guarantor

JP is guarantor.

Peer review

The authors have chosen for the 2 individuals who reviewed this manuscript to remain anonymous.