Crowdsourcing for Conducting Randomized Trials of Internet Delivered Interventions in People with Serious Mental Illness: A Systematic Review

Abstract

Objective

Online crowdsourcing refers to the process of obtaining needed services, ideas, or content by soliciting contributions from a large group of people over the Internet. We examined the potential for using online crowdsourcing methods for conducting behavioral health intervention research among people with serious mental illness (SMI).

Methods

Systematic review of randomized trials using online crowdsourcing methods for recruitment, intervention delivery, and data collection in people with SMI, including schizophrenia spectrum disorders and mood disorders. Included studies were completed entirely over the Internet without any face-to-face contact between participants and researchers.

Databases and sources

Medline, Cochrane Library, Web of Science, CINAHL, Scopus, PsychINFO, Google Scholar, and reference lists of relevant articles.

Results

We identified 7 randomized trials that enrolled N=1,214 participants (range: 39 to 419) with SMI. Participants were mostly female (72%) and had mood disorders (94%). Attrition ranged from 14% to 81%. Three studies had attrition rates below 25%. Most interventions were adapted from existing evidence-based programs, and consisted of self-directed education, psychoeducation, self-help, and illness self-management. Six studies collected self-reported mental health symptoms, quality of life, and illness severity. Three studies supported intervention effectiveness and two studies showed improvements in the intervention and comparison conditions over time. Peer support emerged as an important component of several interventions. Overall, studies were of medium to high methodological quality.

Conclusion

Online crowdsourcing methods appear feasible for conducting intervention research in people with SMI. Future efforts are needed to improve retention rates, collect objective outcome measures, and reach a broader demographic.

1. Introduction

Evaluating and implementing behavioral health interventions to effectively manage serious mental illnesses (SMI), including schizophrenia spectrum disorders or mood disorders, is a major public health priority [1]. SMI is one of the leading causes of disability worldwide [2], and results in symptoms such as anxiety, depression, and low motivation [3], and is associated with low quality of life [4] and increased risk of substance use [5], homelessness [5], hospitalization [6], and suicide [7]. Due to a combination of these factors, widespread societal stigma associated with being labeled mentally ill [8], as well as elevated burden of comorbid medical conditions [9], increased cardiovascular risk and poor health behaviors [10], and poverty [11], people with SMI represent one of the most vulnerable and high-risk patient groups in the United States [12, 13]. In fact, people with SMI experience significantly reduced life expectancy, up to 30 years earlier than the general population [12, 13]. Novel strategies are urgently needed to reach and engage this at-risk group in behavioral health interventions targeting illness self-management, skill building, psychoeducation, health promotion, and social support.

There has been growing interest surrounding the use of online crowdsourcing techniques for conducting health research studies [14]. Crowdsourcing refers to the process of obtaining needed services, ideas, or content by soliciting contributions from a large group of people. In particular, online crowdsourcing involves the use of the Internet through social media websites, forums, health social networks such as PatientsLikeMe, online platforms such as Amazon Mechanical Turk, or other websites or online portals to find people who can accomplish specific tasks, provide ideas or services, or volunteer as participants in a research study [14]. Online crowdsourcing involves soliciting a large number of dispersed people through online posts and open calls to participate, resulting in the self-selection of interested individuals [14]. In sum, online crowdsourcing uses the power of many to solve diverse problems, and has been identified as an approach that can improve quality, cost, volume and speed, as well as an efficient method to engage individuals in health research [15].

In a recent review of 21 health studies, crowdsourcing was successfully used to recruit people online to complete tasks such as problem solving, data processing, surveillance and monitoring, and answering surveys [16]. The tasks ranged from finding solutions to protein structure prediction problems, classifying polyps on CT colonography images, to tracking influenza outbreak through data collection, and soliciting feedback on oral health promotional materials [16]. However, all of the included health studies used observational or survey methods, and many of the tasks completed by participants were short in duration (days or even hours) and generally appealed to the specific interests of the individuals who agreed to complete them [16].

Little is known about whether online crowdsourcing can serve as an appropriate method for conducting behavioral health research involving long-term participation among specific high-risk patient populations such as people with SMI. For example, using online crowdsourcing to conduct a randomized trial of an Internet-delivered behavioral health intervention targeting people with SMI might involve no face-to-face contact between participants and researchers throughout participant recruitment, intervention delivery, and data collection. Using online crowdsourcing for health intervention research has been identified as an area of important opportunity with the potential to deliver interventions to a larger number of patients, and evaluate intervention effectiveness more quickly and at lower costs compared to traditional face-to-face methods [14].

In contrast to most current approaches requiring intensive in-person contact in clinical or community settings, online crowdsourcing could potentially increase intervention scalability and may be effective for expanding current efforts to reach and engage people with SMI, a patient group that is often difficult to enroll in general medical care [17, 18] or recruit and retain as participants in health research studies [19, 20]. For example, there are barriers to participation such as symptom severity, comorbid substance use disorders, recurring hospitalizations, and social factors like poverty, homelessness, and social stigma associated with having a mental illness [17, 18, 20]. Online methods also afford opportunities to deliver targeted interventions to individuals with SMI who may not want to be identified publicly as having a mental illness due to stigma, or who may not be interested in participating in time-consuming face-to-face programs that will take them away from work or school commitments [21]. The success of online crowdsourcing methods for conducting behavioral health research is dependent upon whether individuals with SMI can be reached online.

Emerging evidence highlights that individuals with diverse health conditions are increasingly sharing their illness experiences or seeking advice from others with similar health conditions through popular social media websites and online forums [22, 23].

Studies have shown that even highly stigmatized patient populations such as people with mental health conditions share personal views through blogging, build friendships on social media [24], and use the Internet for accessing health information [25]. Specifically, recent studies have found that individuals with SMI create online relationships at the same rate as individuals without mental illness [26], use popular social media websites to share personal illness stories and seek support [27], and rely on the Internet as an important resource for health-related information [28]. Increasing online connectivity among patient groups such as people with SMI present opportunities for delivering online behavioral health interventions to this population [29].

It is not clear whether online crowdsourcing is a viable approach for participant recruitment, intervention delivery, and data collection among people with SMI. Prior reviews have examined the effectiveness of Internet-delivered interventions for various mental health conditions [30–33], though many studies included in these reviews involved some form of face-to-face contact either during participant recruitment or baseline interviews, or for completing data collection. To our knowledge, there are no prior reviews of randomized trials conducted entirely online without any face-to-face contact between participants with SMI and research staff.

In the current review, we systematically searched the literature to identify randomized trials where online crowdsourcing methods were used to recruit people with SMI as well as deliver an online intervention and collect data online from these individuals. We were interested in studies that did not involve any face-to-face or in-person contact between participants and research staff, where recruitment, intervention delivery, and data collection were completed entirely online through social media, patient forums, online health social networks, or other websites or online portals. Conducting intervention research online with a complex patient group such as people with SMI is a potentially novel application of online crowdsourcing methods.

While the studies included in this review do not explicitly state that they used online crowdsourcing methods, all of the methods employed are consistent with the definition of crowdsourcing as a research method listed above. Classifying these studies as using online crowdsourcing methods has the potential to advance the field of online intervention research by providing a common language for describing efforts to reach out and solicit the power of many through online forums and online communities to participate in a behavioral health intervention study. There is great potential to conduct trials of behavioral health interventions entirely online, and our review is intended to highlight the ways in which such methods have been used to date for reaching the specific patient population of people with SMI online.

We examined the strategies that were used to recruit participants and validate their mental health diagnoses, the types and characteristics of the interventions that were delivered, the types of health outcomes that were collected using these online methods, and whether the interventions were effective. Our objective was to determine whether online crowdsourcing methods are feasible, acceptable, and potentially effective for conducting behavioral health intervention research among people with SMI, and to identify future directions for research in this emerging field.

2. Methods

2.1. Search Strategy

We systematically searched the literature using the following electronic databases in October 2014: Medline, PsychINFO, CINAHL, Scopus, Cochrane Central, and Web of Science. Key words and medical subject heading (MeSH) terms related to mental health, (e.g., schizophrenia, bipolar disorder), were crossed with online technology terms (e.g., crowdsourcing, ehealth, mobile, Internet). The final search strategy used to search Medline is listed in Table 1. No language limits were applied. We also searched Google Scholar and screened reference lists of included studies for additional relevant articles.

Table 1.

Search strategy for Medline

Search	Search Terms
#1	"Schizophrenia"[Mesh] OR "Schizophrenia and Disorders with Psychotic Features"[Mesh] OR "Bipolar Disorder"[Mesh] OR "Psychotic Disorders"[Mesh] OR "Mood Disorders"[Mesh]
#2	schizophrenia OR "psychotic disorder*" OR psychosis OR "bipolar disorder" OR "mood disorder*" OR bipolar OR schizoaffective OR "severe mental illness" OR "serious mental illness"
#3	#1 OR #2
#4	"Telemetry"[Mesh] OR "Telemedicine"[Mesh] OR "Internet"[Mesh] OR "Web Browser"[Mesh] OR "Social Media"[Mesh] OR "Cellular Phone"[Mesh] OR "Text Messaging"[Mesh] OR "Artificial Intelligence"[Mesh] OR "Game Theory"[Mesh] OR "User-Computer Interface"[Mesh] OR "Computer Simulation"[Mesh] OR "Speech Recognition Software"[Mesh] OR "Therapy, Computer-Assisted"[Mesh] OR "Wireless Technology"[Mesh] OR "Remote Sensing Technology"[Mesh]
#5	telemetry OR telemedicine OR telepsychiatry OR telehealth OR telecare OR "telemental health" OR ehealth OR mhealth OR "Mobile Health" OR mobile OR "mobile technology" OR “mobile phone” OR smartphone OR “cellular phone” OR cellphone OR “text messaging” OR “text message” OR SMS OR “internet health” OR internet OR online OR “online recruitment” OR “social media” OR website OR web-based OR “web browser” OR “remote consultation” OR “remote sens*” OR “artificial intelligence” OR “game theory” OR “user-computer interface” OR “computer simulation” OR “speech recognition software” OR “Computer-assisted therapy” OR “wireless technology” OR “remote sensing technology” OR crowdsource OR crowdsourcing OR crowdsourced
#6	#4 OR #5
#7 (Final Search)	#3 AND #6

2.2. Study Selection Criteria

We restricted the current review to: randomized controlled trials that used online crowdsourcing methods through social media, patient forums, online health social networks, or other websites, to conduct the entire trial online; and included participants with SMI, defined in this review as schizophrenia spectrum disorders (e.g., schizophrenia or schizoaffective disorder) or mood disorders (e.g., bipolar disorder or affective disorders). This meant that we only included studies where recruitment of participants with SMI, delivery of the intervention, and data collection were conducted entirely online without face-to-face or in-person contact between researchers or clinicians and participants.

We excluded studies that involved any face-to-face contact between participants and researchers at any point during the study, which included interventions delivered in clinical or research settings, interventions delivered by mental health providers, interventions directly supervised by a clinician or researcher, or interventions that involved a combination of online support and in-person contact. For example, we excluded studies even with minimal contact between study staff and participants [34], because we were interested only in studies that were conducted entirely remotely. In such cases, participants could conceivably be recruited from multiple countries or different geographic areas, and the intervention could be delivered through an online server located at the institution where the research was conducted thereby eliminating the need for costly face-to-face encounters between participants and study staff. Included studies must also have reported outcomes, which broadly included mental or physical health symptoms and functioning, as well as feasibility and acceptability of the intervention or study design. Three of the study authors (JAN, LAM, & GJM) defined the final study inclusion criteria through discussion and review of prior relevant studies in this field [14, 16].

The lead author checked the titles for all retrieved articles, and then reviewed the abstracts for all articles that described relevant topics. The full text was retrieved for articles that appeared to meet our study selection criteria based on a preliminary review of the abstract or for which additional details were needed. The lead author then reviewed the full text for all of the potentially relevant articles. After identifying articles that met the study inclusion criteria or that appeared most relevant, the lead author abstracted the data from these studies into a summary table. All of the study authors then independently reviewed the summary table of included studies. This provided an opportunity to discuss studies that appeared most relevant or studies that met most but not all of the study inclusion criteria. Following review of the summary table, final decisions regarding inclusion were reached through discussion among all study authors, and disagreements were resolved by consensus.

2.3. Data Extraction

We extracted the following characteristics from included studies: country of origin, study design, sample size, sample diagnoses, how the sample diagnoses were validated, recruitment strategy, number of participants who were assessed for eligibility, study duration, intervention description, comparison or control arm description, main outcomes, and attrition rate and reasons for attrition. After the lead author extracted the data from the included studies, all of the authors checked each entry and then reviewed the findings from all of the included studies independently and as a group in order to summarize the evidence and to reach consensus.

2.4. Quality Assessment

To assess the methodological quality of the included studies, we used a 13-item quality assessment scale adapted from an assessment tool that has been used in prior systematic reviews [35]. The scale covers four broad domains related to the quality of the methods: 1) Study Sample; 2) Follow-up and Attrition; 3) Data Collection; and 4) Data Analysis.

Each domain has 2–4 items that have yes/no (+/−) criteria as outlined in Table 2. When the criteria were met the studies received positive scores, and when the criteria were not met the studies received negative scores. The lead author completed the quality assessments for all of the included studies, and then all authors reviewed the ratings to resolve disagreements and reach consensus.

Table 2.

Description of the adapted criteria used to assess the methodological quality of the included studies^a

Criteria		Description
Study Sample
1.	Recruitment	Sufficient details provided about the recruitment methods to allow replication
2.	Participant validation	Strategy was used to validate the self-reported serious mental illness diagnosis of the sample
3.	Participation rate	Participation rate of at least 80% among individuals who met eligibility criteria. This helps to demonstrate that the sample is representative of the target population.
4.	Baseline characteristics	Description of baseline study sample provided. Must include the following key characteristics: age, gender, and mental illness diagnosis
Follow-up and Attrition
5.	Follow-up	Number of participants listed for each follow-up measurement (CONSORT diagram included)
6.	Duration	Follow-up is a minimum of 6 months
7.	Attrition	Response at final follow-up measurement was at least 75%
8.	Non-response	Non-response is not selective at follow-up measurement(s) and attrition is the same across all study arms
Data Collection
9.	Outcomes	Clinical outcomes collected (either objective or subjective)
10.	Objective measures	Objective clinical outcomes collected
Data Analyses
11.	Statistical analyses	Appropriate statistical model was used
12.	Statistical model	The number of cases was at least 10 times the number of independent variables
13.	Interpretation	Presentation of confidence intervals, standard error, or effect size to assist with interpretation of clinically meaningful differences in outcomes.

^aCriteria were adapted from methodological quality assessment tool used by te Velde et al. (2012) [35]

2.5. Assessing the Evidence

Given the heterogeneous nature of the outcomes reported in the included studies, such as various measures of mental health symptoms and functioning, it was not possible to pool the findings for statistical analysis or conduct a meta-analysis. Instead, we used the harvest plot method to synthesize and present the results [36]. The harvest plot is considered an effective approach for summarizing the distribution of evidence across a diverse group of studies [35]. The harvest plot provides a visual representation of the included studies and illustrates whether the results favor the intervention or comparison conditions, the relative attrition rate within each study, as well as the study sample size. Additionally, because our goal was to examine the use of online crowdsourcing methods for conducting behavioral health intervention research including recruitment, intervention delivery and data collection in people with SMI rather than reach conclusions regarding the effectiveness of a particular type of intervention, we determined that the harvest plot was an appropriate approach for illustrating our findings.

3. Results

Our initial search of the electronic databases yielded 9,481 articles after removal of duplicates, of which 849 were relevant and required a review of the abstracts. The full-text was screened for 225 articles, and a total of 7 articles met our study inclusion criteria. Two studies were from Australia [37, 38], 2 from the United Kingdom [39, 40], and 3 from the United States [41–43]. The search results, number of articles screened, and reasons for exclusion are illustrated in Figure 1. Characteristics of the included studies and key feasibility outcomes related to recruitment, intervention delivery, and data collection are listed in Table 3.

Figure 1.

Flow chart of included studies

Table 3.

Summary of randomized trials using online crowdsourcing methods for recruitment, intervention delivery, and data collection in people with serious mental illness

Studya	Origin	Designa	Sample		Recruitment		How was diagnosis confirmed?c	Study Duration	Intervention	Comparison	Outcome Measuresd	Results	Attritione
			Nb	% Male	Strategy	Assessed for Eligibility (N)							%	Reasons
Jones et al (2014) [39]	UK	RCT	39	3	Online advertisements through a national bipolar foundation, foundation meetings, newsletters, email blasts	54	Self-reported diagnosis of bipolar disorder (100%); score >9 on MDQ	10 weeks	Web-based self- directed parenting intervention for bipolar parents (n=19)	Waitlist control (n=20)	Symptom and parenting measures: MDQ; CES-D; ISS; SDQ; PS	Significant improvement in child behavior and perceived parenting measures	39	Lost contact; dropout significantly greater in control group
Kaplan et al (2014) [41]	US	RCT	60	0	Websites, e- news lists, organizational listservs	378	Mood disorder (87%) or schizophrenia spectrum disorder (13%) confirmed by doctor or therapist through release of information form	3 months	Web-based parenting education and peer support listserv for mothers with SMI (n=31)	Online healthy lifestyle facts and education (n=29)	Symptom and parenting measures: PSCS; HFPI; FCI; MOS	Improved parenting and coping skills; decreased parental stress; no improvement in efficacy or support	22	Other illness, preference for face- to-face support, too busy
Kaplan et al (2011) [42]	US	RCT	300	34	Websites targeting people with mental illness, e-news lists	952	Schizophrenia spectrum disorder (22%); mood disorders (78%) confirmed by doctor or therapist through release of information form	12 months	Internet peer support via a listserv (n=101) or experimental Internet peer support via a bulletin board (n=99)	Waitlist control (n=100)	Measures of recovery, quality of life, empowerment, social support and distress: RAS; QOL; ES; MOS; HSCL; OGQ	No differences between groups. Greater participation in Internet peer support resulted in higher levels of distress	17	Too many emails; content of the postings; poor fit; preference for face- to-face support; difficulty using the online format; lost website address; forgot login
Lauder et al (2015) [37]	AU	RCT	156	25	Clinician referral, advertising via conferences and consumer and professional forums, online optimization strategies	158	Bipolar disorder (100%) confirmed by telephone clinical interview	12 months	MoodSwings online program with moderated discussion board for coping and relapse prevention (n=78)	MoodSwings- Plus online program with moderated discussion board and additional CBT-based elements (n=78)	Measures of characteristics of bipolar disorder: ASRM; MADRS; Relapse; SCID; MOS; IPC; MARS; EGAM; GSEVDEP; GSEVMANIA; GPF; GQOL	Reduced mood symptoms, improvements in functionality, quality of life and medication adherence in both groups; Moodswings- Plus had greater improvements	81	Did not commence the program; computer problems; too busy; distressed by questionnaires; unknown.
Proudfoot et al (2012) [38]	AU	RCT	419	30	Advertisements posted in online depression and bipolar disorder community, mental health organizations, and print media advertisements	660	Diagnosis of bipolar disorder (100%) within past 12 months confirmed using cut-off score on MSQ-27; current treated for bipolar disorder	6 months	Online Bipolar Education Program (BEP) (n=139) or BEP enhanced with email support from peers who are successfully managing their illness (n=134)	Attentional control with basic facts about bipolar disorder delivered online (n=134)	Brief IPQ; GADS; WSAS; RSE; SWLS; MHLC; perceived stigma; daily mood ratings	Increased perceptions of control, decreased perceptions of stigmatization, improvements in anxiety and depression across both groups	36	Did not complete intervention; lost contact. Higher adherence to BEP enhanced with peer support compared to BEP alone. Females and participants over age 30 showed greatest adherence
Simon et al (2011) [43]	US	RCT	118	28	Email announcements through the Depression and Bipolar Support Alliance, advertisements online and in mental health clinics	118 (No participants who chose to enroll were excluded)	Bipolar disorder (100%)	3 weeks	MyRecoveryPlan online education program supported with online peer coaching (n=64)	Control group received MyRecovery Plan only (n=54)	Program retention	Peer coaching increased program use (38% vs. 9% in control group)	75	Not specified
Todd et al (2014) [40]	UK	RCT	122	28	Online advertisements and presentations at mental health organizations	240	Self-reported diagnosis of bipolar disorder (100%) confirmed using MDQ, and described using SCID	6 months	Living with Bipolar (LWB) web-based self-management intervention and motivational email support (n=61)	Wait-list control group (n=61)	QoLBD-Brief; WHOQoL- BREF; BRQ; ISS; SASS; SMAI	Feasibility demonstrated by high retention. LWB improved quality of life, wellbeing, depression, recovery, and social function compared to control	14	Lack of time; bereavement; divorce; holiday; physical or mental health concerns

^aRCT, randomized controlled trial

^bAll participants were adults with SMI, defined as schizophrenia spectrum disorders (e.g., schizophrenia or schizoaffective disorder) or mood disorders (e.g., bipolar disorder or affective disorders).

^cMDQ, Mood Disorder Questionnaire; MSQ-27, Mood Swings Questionnaire; SCID, Structured Clinical Interview for DSM;

^dMDQ, Mood Disorder Questionnaire; CES-D, Center for Epidemiological Studies – Depression Scale; ISS, Internal States Scale; SDQ, Strengths and Difficulties Questionnaire; PS, Parenting Scale; PSCS, Parenting Sense of Competence Scale; HFPI, Healthy Families Parenting Inventory; FCI, Family Coping Inventory; MOS, Medical Outcomes Study Social Support Survey; RAS, Recovery Assessment Scale; QOL, Lehmans’ Quality of Life Interview; ES, Empowerment Scale; HSCL, Hopkins Symptoms Checklist; OGQ, Online Group Questionnaire; ASRM, Altman Self-Rating Mania Scale; MADRS, Montgomery-Asberg Depression Rating Scale Self-Assessment; SCID, Structured Clinical Interview for DSM; IPC, Levenson’s Internal, Powerful Others and Chance Locus of Control scale; MARS, Medication Adherence Rating Scale; EGAM, Exploratory Global Assessment Measures; GSEVDEP, Global measure of Severity of Depression; GSEVMANIA, Global measure of Severity of Mania; GPF, Global Measure of Psychosocial Functioning; GQOL, Global measure of Quality of Life; Brief IPQ, Brief Illness Perception Questionnaire; GADS, Goldberg Anxiety and Depression Scale; WSAS, Work and Social Adjustment Scale; RSE, Rosenberg Self-Esteem Scale; SWLS, Satisfaction With Life Scale; MHLC, Multidimensional Health Locus of Control; QoLBD-Brief, Quality of Life in Bipolar Disorder Scale Brief Version; WHOQoL-BREF, World Health Organisation Quality of Life assessment tool brief version; BRQ, Bipolar Recovery Questionnaire; ISS, Internal States Scale; SASS, Social Adaptation Self-Evaluation Scale;

^eAttrition rates are based on the number of enrolled participants who dropped out during the study. We do not report attrition based on the number of invited participants.

3.1. Participant Recruitment Strategies

All participants were recruited without any face-to-face contact with researchers using a variety of methods. For example, online advertisements were widely used, and included advertisements posted in online mental health communities or on popular health websites targeting individuals with mental illness, or posted in patient forums [37–40, 42, 43]. Several studies targeted mental health organizations for participant recruitment, through e-mail blasts to organizational listservs [39, 41], newsletters [39], or advertisements posted in mental health clinics [43]. One study also used print media advertisements in combination with online recruitment methods [38]. These different types of advertisements typically included a link to the study website and details about how to sign up. Study recruitment frequently involved a combination of the above strategies.

3.2. Participant Characteristics and Confirming Diagnosis

There were a combined total of 1,214 participants across all 7 trials, with a range of 39 to 419 participants in each trial (median = 122 participants). Participants were mostly female (72%) and had mood disorders (94%). Only 6% had schizophrenia spectrum disorders such as schizophrenia or schizoaffective disorder. To confirm mental health diagnoses, three of the trials used cutoff scores on the Mood Disorder Questionnaire (MDQ), the Mood Swings Questionnaire (MSQ), or the Structured Clinical Interview (SCID) for DSM [38–40]; two trials received permission from participants through a release of information form to contact their doctor or therapist [41, 42]; one study used telephone clinical interviews [37]; and one study relied on participants’ self-reported diagnoses and other demographic characteristics [43] (see Table 3).

The two studies that confirmed participants’ mental health diagnoses by contacting their doctor or therapist were also the only two studies that reported compensating participants for completing online assessments [41, 42]. Mean age of participants was reported in 5 of the trials and ranged from 37 to 47 years [37, 39–42]. One trial reported that close to one third of participants (31%) were under 30 years of age [38].

3.3. Participant Attrition

Participant attrition was variable between trials, ranging from 14% to 81% (see Table 3). Commonly reported reasons for dropping out included too busy or other commitments [37, 40, 41], technical difficulties or challenges with using the online format [37, 42], difficulty engaging in the intervention [42], and preference for face-to-face programs [41, 42]. Dropouts appeared to be younger in one study [38]. The three studies with the lowest attrition rates (14%, 17%, and 22%, respectively) were also the only studies that reported using strategies to retain participants [40–42]. These strategies included reminder emails and phone calls to prompt participants to complete data collection questionnaires [40, 42], and payment for completing online questionnaires [41, 42]. Also, in two studies retention appeared higher with the addition of an online peer support [38] or peer coaching [43] component.

3.4. Types of Interventions

All of the interventions were delivered online, and ranged in duration from 3 weeks to 12 months. Seven of the eight interventions were adapted from existing evidence-based practices including previously evaluated face-to-face or online programs. There were two online parenting education interventions for parents living with SMI involving self-directed learning through written and video or audio content [39, 41]. One of the parenting interventions was supplemented by a peer support email listserv moderated by a provider and a mother with lived experience of mental illness to provide feedback, answer participants’ questions, and reinforce the educational curriculum [41]. Online peer support served as a key component of several interventions. For example, one intervention involved unmoderated and unstructured online peer support delivered through a bulletin board or listserv that was entirely peer directed [42]. This was an exploratory trial, and as such was the only intervention not adapted from an existing evidence-based program, though development of this online peer support intervention was informed by prior research [42].

Peer support was also used to complement, supplement, and extend formal online interventions, such as an online psychoeducation program for bipolar disorder enhanced with email coaching from individuals who were successful in managing their disorder for at least two years [38], or an intervention involving the development of personal recovery plans and educational modules for bipolar disorder supported with peer coaching from individuals with lived experience through online discussion groups and chats [43]. Two interventions targeted self-management of bipolar disorder by incorporating elements of psychoeducation and cognitive behavioral therapy, supplemented by online peer-to-peer support [37, 40]. These programs included interactive modules focused on understanding bipolar disorder, developing self-esteem and self-efficacy, and strategies for coping with and monitoring symptoms, as well as moderated discussion boards to allow participants to communicate and share their experiences and connect with peers [37, 40].

3.5. Types of Outcomes

Six studies reported mental health or other health-related outcomes, all of which were collected online without face-to-face contact with participants (see Table 3). Various self-reported scales and questionnaires were used to measure depressive symptoms [37–39, 42] and other mental health symptoms such as mood or manic states [37, 39, 40]; anxiety [38, 42]; cognitive and emotional illness representations [38]; functional impairment [38]; self-efficacy and locus of control [37, 38, 42]; self-esteem [38]; mental health stigma [38]; social support [37, 41, 42]; social functioning [40]; mental health recovery [40, 42]; coping strategies and skills [41]; medication adherence [37, 40]; and quality or satisfaction of life [37, 38, 40, 42]. The two parenting education interventions had measures of child behavior [39], parental efficacy [41], parenting skills [39, 41], and parental stress [41].

Three studies reported outcomes related to intervention acceptability or feasibility [40, 42, 43]. These included measures about participation in the intervention and experiences using the intervention [42], feasibility of participant recruitment and retention [40], frequency of use of the intervention [40], use of specific intervention components such as the online forum [40], peer-to-peer messaging [43], discussion groups [43], or self-monitoring tools [43], completion rate of different intervention modules [40], time spent using the intervention [40], and number of questions asked by participants [40].

The health outcomes across studies were generally positive, as highlighted in Figure 2. Three studies showed positive outcomes compared to the comparison conditions, including improved child behavior and measures of perceived parenting [39], enhanced coping skills and decreased parental stress [41], and improved quality of life, wellbeing, depression, recovery and social function [40]. Two studies also showed positive outcomes over time across both the intervention and comparison groups, such as reductions in mood symptoms, and improved functionality, quality of life, and medication adherence [37]; and increased perceptions of control, decreased perceptions of stigmatization, and improvements in levels of anxiety and depression [38]. Two studies included an active comparison condition with comparable intensity to the intervention group [37, 43]. Lastly, the online peer support intervention showed no differences over time for both groups on measures of recovery, quality of life, empowerment, social support, or depression and anxiety [42]. However, the findings were somewhat mixed because greater use of the online peer support intervention was associated with higher levels of distress but more positive experiences with the intervention [42]. Outcomes were consistent across different mental health diagnoses, though the vast majority of participants had mood disorders making it difficult to reach conclusions about the benefits for individuals with schizophrenia spectrum disorders.

Figure 2.

Harvest plot illustrating whether the mental health outcomes of included trials favored the intervention or comparison conditionsa,b

^a Figure Legend: Height of the bar (y-axis) represents the methodological quality rating (%) of the study. The x-axis shows whether the intervention or the comparison condition, both, or neither were favored (positive mental health outcomes). The color of the bars represents the attrition rate, where black represents attrition lower than 25% (3 studies) and grey represents attrition greater than or equal to 25% (3 studies). The patterned bar (1 study) indicates that there was an active control condition with comparable intensity to the intervention group. The number above each bar represents the size of the study sample.

^b One study was excluded (Simon et al, 2011) because no mental health outcomes were reported

3.6. Methodological Quality

The methodological quality of the included studies was generally moderate to high (see Table 4). Each of the studies randomized participants to clearly defined interventions and comparison conditions. Scores were lowest for the quality assessment items related to response rate because only three studies had follow-up greater than or equal to 75%, and also because non-response may have been selective where non-responders were younger or had mental or physical health concerns. None of the studies used objective outcome measures. This was likely because of the online format of the interventions, where the collection of self-reported measures of health outcomes remains most feasible at present.

Table 4.

Quality assessment of randomized trials using online crowdsourcing methods for recruitment, intervention delivery, and data collection in people with serious mental illness^a

Quality Criteria	1	2	3	4	5	6	7	8	9	10	11	12	13	Quality Score (%)
Author (year)	Recruitment	Diagnosis Validation	Participation Rate	Detailed description of baseline sample	Numbers at follow- up (include CONSORT diagram)	Follow- up duration (min 6- months)	Response rate at last follow-up ≥75%	Not selective non- response	Measure mental health or other health related clinical outcomes	Use objective outcome measures	Statistical model is appropriate	Number of cases 10 times number of independent variables	Presentation of confidence intervals, standard error or effect size
Jones et al (2014) [39]	+	+	+	+	+				+		+		+	62
Kaplan et al (2014) [41]	+	+		+	+		+	+	+				+	62
Kaplan et al (2011) [42]	+	+	+	+	+	+	+	+	+		+	+		85
Lauder et al (2015) [37]	+	+		+	+	+			+		+	+	+	69
Proudfoot et al (2012) [38]	+	+	+	+	+	+			+		+	+	+	77
Simon et al (2011) [43]	+													8
Todd et al (2014) [40]	+	+		+	+	+	+	+	+		+	+	+	85

^aThese quality criteria were adapted from the methodological quality assessment tool used by te Velde et al. (2012) [35]. See Table 2 for a detailed description of the quality criteria.

3.7. Ethical Considerations and Privacy

We examined whether there were any concerns related to privacy from recruiting and consenting participants online, collecting participant data online, and delivering interventions using the Internet. All of the included studies consented participants online. Privacy or online safety procedures were described in three studies, and included maintaining password protected secure access [37], ensuring that all communication with participants occurred through a secure website [43], not collecting personally identifying or clinical information [43], and monitoring forum and discussion board posts for risk issues [40]. The only concerns that seemed to arise were participants posting hostile messages to the discussion boards. This happened in two studies, and in both cases the participant who posted the hostile content was removed from the listserv [41, 42], however the hostile messages were unsettling for other participants. In the study of online peer support [42], there were five instances where participants posted comments that raised concerns about potential threats to self or others, but the researchers determined that these were not caused by the intervention itself. Given that there were a combined 11,105 posts from the two hundred participants allocated to either the bulletin board or listerv study conditions, the number of hostile posts or concerns was minimal [42].

4. Discussion

We identified 7 recent studies reporting the use of online crowdsourcing methods for conducting randomized trials of behavioral health interventions among people with SMI. In each of the included studies, participant recruitment, intervention delivery, and data collection were entirely completed online without any face-to-face contact between participants and researchers. To our knowledge, this is the first systematic review examining the use of these emerging online crowdsourcing methods for conducting behavioral health research in this at-risk patient group. These online methods appear feasible and acceptable for reaching people with SMI, for confirming the mental health diagnoses of this patient population, and for effectively delivering a variety of interventions adapted from existing evidence-based programs. The interventions consisted of self-directed programs for education, psychoeducation, self-help, illness self-management, and peer support.

Health outcome measures were heterogeneous and were entirely collected online through participant self-report, among which there were questionnaires assessing mental health symptoms, social support, self-efficacy, quality of life, and medication adherence. In three studies, the results supported the interventions compared to the control groups [39–41]. In one study, health outcomes improved over time for participants across both the intervention and comparison groups [38], and in another study with an active comparison condition there were improvements over time in both groups [37]. Improvements were observed for mental health symptoms, depression, anxiety, and quality of life, as well as parental coping skills and parental stress. These positive findings contribute to growing evidence highlighting the promise of using online interventions for improving the mental health and wellbeing of people with SMI [44–46], and extend current evidence by demonstrating that it is feasible to conduct behavioral health intervention research with people with SMI without requiring costly face-to-face contact. However, future research is needed to established effectiveness of specific types of interventions.

Various methods were used across the included studies to confirm participants’ mental health diagnoses. These included the use of symptom questionnaires, release of information forms to contact participants’ doctors or therapists, a telephone clinical interview, and self-report. Two studies confirmed diagnoses by contacting participants’ doctors or therapists, which appeared to be the most rigorous approaches documented in this review [41, 42]. These were also the only two studies that reported providing monetary compensation to participants. Therefore, among the studies included in this review, it is unlikely that monetary compensation for completing study assessments could have incentivized dishonesty among individuals attempting to meet the inclusionary criteria.

While confirmation of psychiatric diagnosis by a healthcare professional may be an optimal approach to recruiting participants with SMI to participate in online interventions, applying strict criteria to confirm mental health diagnoses may impose further limits to generalizability of online research in this vulnerable population. There is sufficient evidence that people with SMI often lack access to health care and when they do see a provider, their care is often substandard [17, 18]. Confirming the psychiatric diagnoses of individuals recruited using online crowdsourcing methods remains an important concern for future online intervention studies, even though it is unlikely that individuals would be dishonest about having a highly stigmatizing illness such as schizophrenia or bipolar disorder just to enroll in a research study requiring long-term participation without monetary compensation. This presents yet another opportunity to advance the field by developing novel approaches to validating self-reported psychiatric diagnosis in the rapidly growing area of online mental health intervention research.

Several concerns emerged across the included studies. First, participant retention was highly variable with attrition rates ranging from 14% to 81%. Similarly high rates of attrition have been reported in prior studies of Internet-delivered interventions for various mental health conditions [30, 31]. This shows that using online crowdsourcing methods to conduct behavioral health research in individuals with SMI results in attrition rates that are consistent with prior online intervention studies in patient groups with differing mental health conditions. Importantly, three studies included in our review reported attrition rates lower than 25% (14%, 17%, and 22% respectively) [40–42], which is comparable to attrition rates reported in face-to-face trials of behavioral health interventions targeting this at-risk group. For example, a recent review of in-person illness self-management in people with SMI reported a median attrition rate of 24% across nine studies with a range of 18–30% [47], another face-to-face intervention for mental and physical illness management in people with SMI reported a 33% attrition rate [48], and a text messaging intervention for assessment and treatment in people with schizophrenia spectrum disorders involving minimal face-to-face contact between participants and researchers reported a 19% attrition rate [49]. This demonstrates that while there may be greater risk of incurring high attrition rates in online research, it is possible to obtain attrition rates that are comparable to face-to-face intervention studies in people with SMI.

Among the included studies, we observed that the reasons for dropout included being too busy or having other commitments, technical difficulties or challenges with using the online format, and preference for face-to-face programs, which are also consistent with the reasons reported in prior studies of online interventions delivered to people without SMI [31]. Interestingly, the three studies with the lowest attrition rates were also the only studies that described strategies used to promote participant retention [40–42]. This suggests that efforts to increase retention can be successful and are necessary to increase the generalizability of study findings, reduce bias, and improve external validity. Future online research should specifically incorporate retention strategies into the study design, and should consider additional efforts to address potential issues or challenges that may arise with using new technologies. For example, a more extensive orientation period at the start of an intervention could help participants feel more comfortable using and accessing an online program, or providing ongoing technical assistance throughout intervention delivery could improve participant retention and contribute to sustained use of the intervention over time. Individuals with SMI frequently experience cognitive limitations, and prior studies have shown that when introducing new technologies to this patient group it is necessary to provide sufficient opportunities to learn to use the new technology through tailored instruction and training, as well as ongoing assistance to improve intervention usability and acceptability over time [34, 50, 51].

Additionally, there was limited diversity within the included samples. Most participants were female (72%), had mood disorders (94%), and were in their mid-thirties to mid-forties. This is likely partly due to the nature of the interventions, such as interventions targeting mothers with SMI or only people with bipolar disorder. Alternatively, this lack of diversity may indicate that the recruitment methods employed or the interventions delivered across these studies did not appeal to a wider demographic or were not suitable for a wide range of individuals with SMI. For example, people with schizophrenia spectrum disorders may have been less likely to participate in these online interventions due to greater illness severity, difficulty with concentration, symptom provocation, or because the web platforms were not appropriately designed or catered to their interests [28, 51]. Similar concerns related to recruiting demographically diverse and younger samples have been discussed in prior studies targeting vulnerable patient populations [52].

The peer support component in many of the studies highlights one of the most promising aspects of online interventions for reaching and engaging high-risk population groups. While the one trial of an online peer support intervention produced mixed results [42], making it difficult to reach conclusions regarding the direct benefit of online peer support for this population, several of the other interventions included peer support or peer coaching components [38, 43] or listservs, forums or discussion boards to facilitate peer-to-peer interaction [37, 40, 41]. The ability to allow participants to connect with each other through forums or discussion boards, referred to as online peer-to-peer support, is an exciting feature of online platforms [53]. This approach brings together participants’ shared experiences, and may be an effective strategy for improving retention, appealing to the interests of a wider demographic, and enhancing the effectiveness of online behavioral health interventions. This is also consistent with emerging research documenting the increasing use of social networking websites among people with SMI [25, 26], as well as the potential benefits from engaging with others through these websites, as reported in a recent study exploring naturally occurring peer support among individuals with SMI on YouTube [27].

There are also potential risks associated with the use of online peer support. For example, in two studies included in this review there were cases where participants posted hostile messages to the discussion boards that were unsettling for others, and that required the removal of these individuals from the listservs altogether [41, 42]. In such cases it may be necessary for mental health professionals to monitor the content of online peer-to-peer interactions among people with SMI. This may also present opportunities for automated computerized techniques to streamline monitoring in large-scale online mental health interventions, or to rely on feedback from the entire online pool of participants to flag comments as inappropriate. Similar techniques that rely on feedback from the wider community of users can be observed on popular social media websites such as YouTube [27], where a comment is concealed after enough users flag it as inappropriate. Interestingly, to date most of the research surrounding online peer-to-peer interactions across diverse patient groups has determined that the benefits appear to greatly outweigh the risks [54]. Regardless, there are significant opportunities for future research to explore both the potential health benefits and to weigh these benefits against the possible risks of incorporating peer support or facilitating peer-to-peer interaction in online interventions targeting this high-risk group.

Several limitations should also be considered. Because of the diverse types of interventions and the heterogeneous health outcomes collected across the included studies, we were unable to conduct a meta-analysis to establish intervention effectiveness and to reliably assess the effect sizes of the different interventions. With regards to methods employed in this review, the lead author was responsible for screening all titles, abstracts and full text articles for potentially eligible studies. This may have introduced bias in the study selection process, however all of the authors independently reviewed the final included studies and reached consensus through discussion. Further, the final inclusion criteria were defined through discussion among several of the study authors. Additionally, we did not formally search the gray literature, and it is possible that there may be unpublished findings [14]. However, given that the method of using online crowdsourcing to conduct randomized trials of behavioral health interventions is in its infancy, the primary aim of our review was to provide a comprehensive overview and summary of published randomized trials that have employed these practices without any face-to-face contact among people with SMI. It is also important to note that the authors of the studies included in this review did not explicitly state that they were using online crowdsourcing, though all of the methods employed are consistent with the definition of crowdsourcing as a research method. Further, all of the included studies were published since 2011, and over half were published since 2014, indicating that there will likely be tremendous growth in this field. This review should help inform future research using crowdsourcing and online methods to target individuals with SMI.

5. Conclusion

Online crowdsourcing methods appear highly promising for conducting randomized trials of behavioral health interventions targeting people with SMI. As this field advances rapidly, we draw four important considerations from our review that can help to inform future research. First, online crowdsourcing methods appear feasible and acceptable for recruiting participants with SMI, confirming participants’ mental health diagnoses, delivering evidence-based behavioral health interventions, and collecting self-reported health outcomes, all without face-to-face contact between participants and researchers.

While there may be unforeseen risks in conducting behavioral health intervention research without face-to-face contact with study staff, few risks were reported across the included studies and many of the interventions in this review demonstrated preliminary effectiveness in improving mental health outcomes. Nevertheless, future research using these methods must focus on establishing intervention effectiveness, measuring safety, and collecting more robust health outcome measures. For example, as self-tracking tools including wearable devices, remote sensors or accelerometers that sync wirelessly to smartphones or web applications increase in popularity and become more widely available [55], there will be exciting opportunities to collect objective health measures through online crowdsourcing methods [14]. Second, efforts to recruit individuals with SMI online should be carefully designed to appeal to a wider demographic, including younger individuals, more people with schizophrenia spectrum disorders, and more men. Efforts to reach a more representative population of individuals with SMI online will improve generalizability and external validity of these emerging methods. Third, strategies must be employed to promote greater participant retention. We observed that simple strategies such as email and telephone reminders appear effective at promoting retention, but additional efforts to help train and orient participants to the online intervention and provide ongoing assistance to overcome technical challenges should also be considered. Finally, future interventions should leverage the connectivity of online platforms to facilitate peer-to-peer interaction between participants as a way to enhance program effectiveness and support participants. Future research must also evaluate the potential benefits and risks of online peer-to-peer interaction between study participants.