To tweet or not to tweet about schizophrenia systematic reviews (TweetSz): study protocol for a randomised controlled trial

Abstract

Introduction

The Cochrane Schizophrenia Group (CSzG) has produced and maintained systematic reviews of effects of interventions for schizophrenia and related illness. Each review has a Plain Language Summary (PLS), for those without specialised knowledge, and an abstract, which are freely available from The Cochrane Library (https://summaries.cochrane.org). Increasingly, evidence is being distributed using social media such as Twitter and Weibo (in China) alongside traditional publications.

Methods and analysis

In a prospective two-arm, parallel, open randomised controlled trial with a 1:1 allocation ratio, we will allocate 170 published systematic reviews into the intervention group (tweeting arm/Weibo arm) versus the control group (non-tweeting arm). Reviews will be stratified by baseline access activity, defined as high (≥19 views per week, n=14), medium (4.3 to 18.99 views per week, n=72) or low (<4.3 views per week, n=84), based on Google Analytics, which will also be used for evaluating outcomes. The intervention group will have three tweets daily using Hootsuite with a slightly different accompanying text (written by CEA and AB) and a shortened Uniform Resource Locator (URL) to the PLS: a) The review title as it appears in summaries.cochrane.org, b) A pertinent extract from results or discussion sections of the abstract and c) An intriguing question or pithy statement related to the evidence in the abstract. The primary outcome will be: total number of visits to a PLS in 7 days following the tweet. Secondary outcomes will include % new visits, bounce rate, pages per visit, visit duration, page views, unique page views, time on page, entrances, exiting behaviour and country distribution.

Ethics and dissemination

This study does not involve living participants, and uses information available in the public domain. Participants are published systematic reviews, hence, no ethical approval is required. Dissemination will be via Twitter, Weibo and traditional academic means.

Trial registration number

ISRCTN84658943.

Strengths and limitations of this study.

• This is the first randomised controlled trial that we are aware of to evaluate the impact of tweeting health-related web links versus not tweeting on access to the target webpage and/or related webpages.

• This study will provide information to help quantify the effects of tweeting evidence, and will generate many questions for future research.

• Those interested in best evidence for people with schizophrenia may be similar or different to others interested in different areas of medicine, however, in any given area of healthcare there may be a critical mass of followers required to gain traction in the wider community. We are unable to estimate at this time, what the critical number would be.

• We are using free-to-use software and may be able to detect other meaningful effects using more sophisticated tools, which are presently inaccessible to us.

Introduction

For two decades, the Cochrane Schizophrenia Group (CSzG) has been producing and maintaining high quality systematic reviews and meta-analyses of randomised controlled trials evaluating the effects of interventions for schizophrenia and related psychotic illnesses. A systematic review “is a high-level overview of primary research on a particular research question that tries to identify, select, synthesize and appraise all high quality research evidence relevant to that question in order to answer it,”¹ and the well conducted ones tend to be carried out by teams of experts. All published versions of Cochrane systematic reviews and protocols are available from the Cochrane library (http://www.thecochranelibrary.com). The full-text versions of these can be accessed and downloaded freely (in some high-income and most low-income countries) or at a cost to others.² Each review also has an award winning Plain Language Summary (PLS) section to make it more accessible to people without specialised knowledge,³ and an abstract, both of which are freely available from the Cochrane PLS website (https://summaries.cochrane.org).

Twitter is a free to use social media platform that allows users to send a 140-character message called a ‘tweet’. These tweets may contain ‘hashtags’ (#) and/or a Twitter handle (@). # is the means to enable searching for a topic and @ denotes a username for either a person, a company or an entity. Presently, there are 284 million monthly active users sending out 500 million tweets a day. Seventy-seven per cent of accounts are outside the USA and 80% of tweets are sent from mobile devices.⁴ Over the years, the use of Twitter in healthcare has increased, encompassing issues relating to public health surveillance, tracking disease activity of the H1N1 pandemic and isolating the source of a cholera outbreak in Haiti, among other tweets.^5–7 The promotion of systematic reviews via Twitter, however, is a relatively recent development for most Cochrane Review Groups,⁸ despite health-related Twitter messages comprising of a not so insignificant proportion of all Twitter traffic and even predicting geographic regions and trends of illness, based on the tweets.⁹ The year 2011 marked the beginning of the CSzG using various social media platforms and since early 2013, the CSzG invested resources into use of social media as a way of raising awareness of systematic reviews. Twitter, the most active of platforms, is now frequently used by both the group and its followers. The number of followers of the CSzG on Twitter has risen from 296 (in March 2013) to 734 (as of 15 January 2015).

Although Twitter, Facebook and some other platforms are not available in China, 91% of China's population use social media compared to about 67% of America's.¹⁰ The CSzG has been working with a Chinese company, Systematic Review Solutions Ltd,¹¹ to disseminate parallel messages on Weibo, a Twitter-like system, to their followers, who currently number more than 6000. Weibo is in the top 10 social media sites used in China, with over 600 million registered users, of which about 140 million use it regularly (as of March 2014).¹²

Given the increasing use of social media and, in particular, Twitter, in healthcare, we propose to evaluate the impact of tweeting a précis of CSzG's systematic reviews in a randomised controlled trial in most of the rest of the world and mirroring this in China on Weibo. The impact of this social media dissemination, however, is unclear. As Twitter does not provide data to enable assessment of impact, Google Analytics (GA) is an alternative source of data. GA is easy to use and has a wide range of data accessible with the standard (free) account.¹³

This trial aims to evaluate the impact of tweeting health-related web links (freely available on summaries.cochrane.org), versus not tweeting, on access to the target webpage and/or related web pages.

Methods and analysis

Study design

Prospective two-arm, parallel, open randomised controlled trial with a 1:1 allocation ratio as outlined below in figure 1.

Figure 1.

Flow diagram of the study. CSzG, Cochrane Schizophrenia Group; PLS, Plain Language Summary

Eligibility criteria

Inclusion criteria

Published full text CSzG reviews in the Cochrane Library and Plain Language. Summary (PLS) in summaries.cochrane.org (N=170).

Exclusion criteria

Unpublished and withdrawn CSzG reviews.

Published protocol CSzG reviews that appear in The Cochrane Library.

Any CSzG review not relevant to schizophrenia.

Randomisation

Cochrane Schizophrenia Group systematic review baseline access activity was defined as high (≥19 views per week, n=14), medium (4.3 to 18.99 views per week, n=72) or low (<4.3 views per week, n=84) based on GA data for the period 21 September 2013 to 28 February 2014. This classification was based on data exported from GA on number of views within the pilot period (about 3 months) and deciding on a suitable division into categories based on a reasonable definition of high, medium and low ‘popularity’, in terms of this data. This categorisation was then checked by CEA and AAM to ensure there was an even number of reviews within each category in order to carry out the pairing. Reviews were given a unique code, which, along with access activity stratum, was supplied to one of the authors (AAM) who performed the randomisation. Stratifying by baseline access activity, and using a computer generated random number sequence, reviews were first allocated to Tweet and non-Tweet arms, then into pairs of reviews that would have the same reference period for outcome data collection, then to day of the week (Tuesday, Wednesday or Friday) and week number (1 to 29) that tweeting would begin for reviews in the intervention arm. Finally, the sequence of the three tweets for each review (the tweet package) in the intervention arm was also randomised.

Interventions

Intervention group

Reviews in the intervention group will be tweeted three times on the same day at 10:30, 13:00 and 15:00 GMT as guided by the SocialBro web tool, since there is some evidence that multiple postings, three to four times a day, of the same or similar tweet, can be useful for an international following. Days for tweeting are prespecified as Tuesday, Wednesday and Friday, as these are considered to have the heaviest traffic.¹⁴ Each of the three tweets has slightly different accompanying text:

• The review title as it appears in summaries.cochrane.org—and a shortened Uniform Resource Locator (URL) to the PLS.

• A pertinent extract from the results or discussion sections of the abstract—and a shortened URL to the PLS.

• An intriguing question or pithy statement directly related to the evidence presented in the abstract—and a shortened URL to the PLS.

An example of this is outlined in figure 2.

Figure 2.

Example of the three tweets relating to the same review.

We are not testing the impact of the different types of accompanying text. These have been formulated in order to appeal to various followers of the CSzG Twitter page and searchers. We are testing the impact of the package of tweets. To assist the logistics of timing the various tweets, we used Hootsuite, a social media management system. This free package allows formulation and scheduling of tweets. Hootsuite is now available in China and integrates with Weibo (http://blog.hootsuite.com/chinese-localization-weibo/).

Control group

Reviews in the control group are those not tweeted by the CSzG.

Outcomes and data collection

The primary outcome is the total number of visits to a PLS in the 7 days following start of the tweeting intervention. For reviews in the control arm, the outcome period is the same within intervention–control paired reviews. This includes all traffic to the PLS and traffic directly from Twitter. The half-life of a tweet (with a web link), defined as “the amount of time at which this link will receive half of the clicks it will ever receive after it's reached its peak”, has been calculated as 2.8 h.¹⁵ However, to capture any possible cascade effect of tweeting, we extend the monitoring period to 7 days.¹⁶ Secondary outcomes provide other measures of incoming activity (% new visits, pages per visit, visit duration, page views, unique page views, time on page, entrances, bounce rate) and exiting behaviour (events, total events, unique events).

In addition, we will report country distribution of users clicking on PLSs in a separate table.

Source of data

GA, originally called Urchin, before it was signed over to Google in March 2005, will be used as data source for outcomes. GA is mainly used by businesses to identify customers’ needs and how those needs are being met. With the production of various data reports in real-time, GA can answer questions about who has visited, and when and where they had visited, a site, as well as how they ‘arrived’ at that site. GA is a good source of user-relevant data accessible with the standard (free) account.^{17 18} Table 1 outlines the glossary of GA terms.

Table 1.

Glossary of Google Analytic terms

Google term			Explanation
Number of clicks	Direct clicks traffic that does not originate from search-engine results or a referring link in a domain is identified as ‘direct’19	Visits	Number of times people viewed the site
		% New visits	An estimate of the percentage of first time visits
		New visits	Number of first-time visits (from people who had never visited your site before)
		Bounce rate	Percentage of single-page visits (ie, visits in which the person left the site from the entrance page without interacting with the page).
		Pages/visit	The average number of pages viewed during a visit to the site. Repeated views of a single page are counted (also called Average Page Depth)
		Average visit duration	Average actual length of time a visitor spends on the site.20 Google Analytics will record visit duration for a maximum of 30 min, after which it will time-out. If the tab is kept open the duration will continue to be monitored until this point21
	All clicks the overview of all clicks onto the website of interest19	Page views	Number of pages viewed. Repeated views of a single page are counted
		Unique page views	Number of visits during which the specified page was viewed at least once. A unique page view is counted for each page URL + page Title combination
		Average time on page	Average amount of time visitors spend viewing a specified page or set of pages
		Entrances	Number of times visitors entered your site through a specified page or set of pages
		Bounce rate	Percentage of single-page visits (ie, visits in which the person left the site from the entrance page without interacting with the page)
	Twitter referrals clicks that originate from a third-party website where a web link has been provided to the page of interest21 22	Sessions	Same as Unique page views
		Page views	Number of pages viewed. Repeated views of a single page are counted.
		Average session duration	Same as Average time on page (Only data for the CSzG PLS pages will be recorded)
Outbound clicks		Events	An action tracked on the website—for example, exit to Cochrane Library
		Total events	Total events is the number of times events occurred
		Unique events	Unique events is the number of visits during which one or more events occurred

URL, Uniform Resource Locator.

Statistical considerations: power

The sample size for this study is fixed by the number of published Cochrane reviews under the jurisdiction within the CSzG (n=170). Therefore, we can estimate the magnitude of the detectable between-group difference in the primary outcome. With 5% two-sided α and a sample size of 85 per arm, an effect size in the range 0.43–0.5 SDs is detectable with 80–90% power. This is equivalent to a between-group difference in means of 2.8–3.3 visits per week.

Data analysis

We will compare characteristics of the reviews in the intervention and control arms, including baseline access activity, using appropriate descriptive statistics. The primary between-group comparison will analyse reviews as randomised, regardless of how much of the tweeting intervention was actually employed, and will estimate the difference in mean number of visits per week and 95% CI, using analysis of covariance. This will be implemented using multivariable linear regression, and will include in the model baseline number of visits and day of the week that tweeting activity started. Secondary outcomes will be analysed similarly.

As a secondary analysis of the primary outcome, we will conduct a prespecified subgroup analysis to investigate whether any effect of the intervention differs according to baseline activity. We will do this by including an appropriate interaction term in the primary regression model.

We anticipate that the intervention will be implemented fully as planned, and that there will not be any missing primary outcome data. However, in the event that either of these assumptions is untrue, we will consider sensitivity analyses to investigate the effect of the receiving of the intervention as intended, and of imputing missing outcome data.

Ethics and dissemination

This study does not involve any living participants and uses information that is available in the public domain. Participants in this study are systematic reviews rather than people. The summaries.cochrane.org and The Cochrane Library websites will be the participants, and routine data will be extracted and recorded through GA. As a result, no ethical approval is required.^{23 24} Results will be disseminated via Twitter, Weibo and traditional academic means.

Trial organisation

The trial is sponsored by the Nottinghamshire Mental Health Trust. We have no support or clear reasons to establish a Data Monitoring Committee or a Steering Committee.