Harnessing Mobile Technology to Reduce Mental Health Disorders in College Populations: A Randomized Controlled Trial Study Protocol

Abstract

About a third of college students struggle with anxiety, depression, or an eating disorder, and only 20–40% of college students with mental disorders receive treatment. Inadequacies in mental health care delivery result in prolonged illness, disease progression, poorer prognosis, and greater likelihood of relapse, highlighting the need for a new approach to detect mental health problems and engage college students in services. We have developed a transdiagnostic, low-cost mobile mental health targeted prevention and intervention platform that uses population-level screening to engage college students in tailored services that address common mental health problems. We will test the impact of this mobile mental health platform for service delivery in a large-scale trial across 20+ colleges. Students who screen positive or at high-risk for clinical anxiety, depression, or an eating disorder and who are not currently engaged in mental health services (N=7,884) will be randomly assigned to: 1) intervention via the mobile mental health platform; or 2) referral to usual care (i.e., campus health or counseling center). We will test whether the mobile mental health platform, compared to referral, is associated with improved uptake, reduced clinical cases and disorder-specific symptoms, and improved quality of life and functioning. We will also test mediators, predictors, and moderators of improved mental health outcomes, as well as stakeholder-relevant outcomes, including cost-effectiveness and academic performance. This population-level approach to service engagement has the potential to improve mental health outcomes for the millions of students enrolled in U.S. colleges and universities.

Introduction

Prevalence of mental health problems among college students is high. In national epidemiologic studies, past-year prevalence of mood and anxiety disorders was 11% and 12%, respectively [1], and 9% of college students screened positively for an eating disorder (ED) [2]. Problematically, such prevalence has also risen steadily in recent years [3–5]. Among students with probable disorders, only 20–40% receive treatment [1, 6], with this treatment gap being even wider among students of color and those from low-income families [6, 7].

Early intervention is particularly important during the traditional college ages of 18–24, when mental illnesses account for the largest burden of any disease [8]. Untreated symptoms become more frequent, severe, and persistent over time [9–11]. Students often have difficulty getting appropriate health care [12], which can have lasting consequences on functioning, physical health, suicidality, social relationships, and educational attainment [13, 14].

Although many campus counseling and health centers have increased their capacity to provide services [15], there remain major limitations to access and delivery of mental health services in college populations. For students, important barriers include lack of problem recognition, lack of time, lack of urgency to seek services, and stigma [16]. Increasing the number of providers, by itself, would neither address the large number who wait until they reach a crisis level to seek mental health support, nor alleviate stigma or campus leaders’ concerns about continuing to pour more resources into services without appreciable slowing in demand [15]. These factors highlight the need for a more efficient, proactive, and accessible service delivery model for managing mental health in college student populations through combined targeted prevention and intervention.

Mobile health (mHealth) technologies have potential to improve mental health care on college campuses by overcoming treatment barriers and increasing efficiency. Such interventions can offset in-person clinical demands, increase access, enhance treatment precision, and reduce costs [17]. Mobile technologies are efficacious for screening and treatment of anxiety, depression, and EDs across settings and populations, including in college students [18–24], for whom smartphone use is ubiquitous [25], with high acceptability given their convenience and anonymity. However, these interventions have been primarily delivered independently, as stand-alone interventions to treat one specific mental health problem. A key challenge is delivering these technologies effectively and combining these different prevention and treatment targets at a population level. Such an approach would link screening with prevention and intervention, maximize engagement, and address comorbid problems.

Methods and Design

Specific Aims

Our team has developed a transdiagnostic, low-cost mobile mental health targeted prevention and intervention platform that uses population-level screening to engage college students in tailored services that address their most common mental health problems. This platform represents an ideal model for service delivery among college populations given a number of key features. First, the platform harnesses an established national infrastructure for screening college populations—the Healthy Minds Network (https://healthymindsnetwork.org). Second, the platform uses promising, evidence-based mobile programs available through a commercial vendor, SilverCloud Health (https://www.silvercloudhealth.com), to increase access to services, as they are low-cost, convenient, and acceptable to students. Third, because a variety of disorders emerge or worsen during young adulthood, the platform is transdiagnostic, addresses comorbid mental health issues, and is designed for both prevention and treatment. The platform screens students for common mental health conditions (i.e., anxiety, depression, EDs) and then engages those with these problems or at high risk for their onset in tailored programs. Finally, the platform uses personalized screening and intervention to increase service uptake, enhance engagement, and improve outcomes.

To date, a mobile mental health platform comprised of a suite of evidence-based, cognitive-behavioral therapy (CBT) interventions has not been tested at a large scale. In the current study, we are testing the impact of such a platform in a trial across at least 20 diverse colleges. Students who screen positive or at high-risk for anxiety, depression, or EDs (with the exception of anorexia nervosa [AN], which often requires more intensive medical monitoring), and who are not currently engaged in mental health services (N=7,884) will be randomly assigned to: 1) intervention via the mobile mental health platform; or 2) referral to usual care (i.e., campus health or counseling center). We targeted these disorders as they account for a substantial proportion of the mental health burden on college campuses [1, 26–28]. We have the following specific aims:

• Aim 1 (uptake): Compare uptake (i.e., individuals beginning treatment) in the mobile mental health platform arm to referral to usual care arm. Hypothesis 1: The mobile mental health platform will yield substantially higher uptake of services compared to referral to usual care.

• Aim 2 (effectiveness): Compare the effectiveness of the mobile mental health platform to referral to usual care based on (2a) presence of mental health disorders (primary outcome); (2b) disorder-specific symptoms (secondary outcome); and (2c) quality of life and functioning (secondary outcome). The mobile mental health platform will be superior to referral to usual care in (Hypothesis 2a) reducing clinical cases; (Hypothesis 2b) reducing disorder-specific symptoms; and (Hypothesis 2c) improving quality of life and functioning.

• Aim 3 (targets, mediators, predictors, and moderators): (3a) Examine if the mobile intervention changes theoretically-informed targets, both transdiagnostic (i.e., decreased dysfunctional cognitions, increased use of CBT skills) and disorder-specific (i.e., reduced avoidance for anxiety, increased behavioral activation for depression, reduced dietary restraint and weight/shape concerns for EDs); (3b) determine if changes in targets are associated with clinical benefit; and (3c; exploratory) identify other putative mediators of change (e.g., early engagement in help services, rapid response), (3d; exploratory) within-mobile program predictors of outcome (e.g., sessions completed), and (3e; exploratory) treatment moderators. Hypothesis 3a: The mobile intervention will result in significant change in both transdiagnostic and disorder-specific targets. Hypothesis 3b: Change in targets will be associated with clinical benefit (primary and secondary outcome variables).

• Aim 4 (stakeholder outcomes, exploratory): Evaluate stakeholder-relevant outcomes: (4a) cost-effectiveness; (4b) students’ academic performance; and (4c) attitudes. Relative to referral to usual care, the mobile mental health platform (Hypothesis 4a) will reduce downstream costs and have a higher net-benefit from a payer’s perspective and (Hypothesis 4b) be superior in improving academic performance. Hypothesis 4c: Stakeholder attitudes towards mobile platforms will significantly vary and will correlate with intention to adopt a mobile platform permanently.

Experimental Design

This study will involve at least 20 campuses, with a projected 7,884 undergraduate students enrolled across campuses. An initial sample of at least 146,000 students will be invited to take the baseline screen. As seen in Figure 1, we will use evidence-based mobile screening to screen the sample for anxiety, depression, and EDs, though other problems will also be screened (i.e., post-traumatic stress disorder, insomnia, substance abuse). This is both to provide comprehensive data on college mental health, as well as to allow tailoring of the intervention based on presence of these additional issues. Students identified either to be at high risk for or to screen positive for clinical anxiety, depression or an ED will be eligible and invited to participate in the full study and followed for two years. Enrollment will be limited to undergraduate students (who can be of any age ≥18 years old). Recruiting from varying sized campuses will allow us to approximate the diversity of colleges nationwide and to examine potential variations in intervention effectiveness across settings.

Figure 1.

Symptom Stratification Within Our Population Mental Health Management Program and Study Flow

Site recruitment and retention.

Campus selection will be a dynamic process that considers both support from campus leadership, as well as our aim to achieve a final sample of campuses representative of the full spectrum of U.S. higher education concerning the following factors: geographic region, enrollment, sector (public vs. private), race/ethnicity, admissions selectivity, and institutional type (e.g., research-intensive, community college).

Participant recruitment and retention.

The Registrar (or campus unit) at each site will provide a database with contact and basic information for a random sample of undergraduates (or all of them, for smaller schools). Students who are <18 years and/or studying abroad will not be eligible.

Students will be recruited via campus email for the study screen. They will first receive a brief “pre-notification” email letting them know an invitation to the study is coming—this strategy has been shown to boost survey response [29]. Two to three days later, we will send the recruitment email with a link to the screen, and non-responders will receive reminder emails. As a modest incentive for participation in the screen, participants will be entered into a cash sweepstakes for one of 40 $100 prizes. Based on our research, we anticipate students with mental health concerns will be overrepresented in the sample given that the survey is described as one about mental health [30]. This is a desirable feature for an intervention intended to reach this population.

Participants

Participants eligible for the full study (i.e., the trial) will be undergraduates who are not currently (i.e., within the past month) engaged in mental health services and who screen positive for clinical anxiety, depression, or EDs (with the exception of AN) or are at high risk for one or more of these problems. We will aim to enroll a diverse sample of students (in terms of race/ethnicity, etc.). Students will be required to have access to a smartphone (86% of college students have a smartphone) [25]. Students already currently engaged in mental health treatment will be excluded given the goal to increase access to services for those not receiving help. Students who endorse suicidality will receive additional, immediate follow-up (including contact information for the National Suicide Prevention Hotline) but will not be excluded.

Procedure

Immediately after completing the initial online screen, students will receive personalized feedback about their results. Feedback will be followed by a brief statement of eligibility for and description of the next phase of the study (e.g., study purpose and duration, activities involved, assessment time points), and students will be asked to confirm their willingness to continue and provide their informed consent. Upon indicating their willingness to continue, they will be asked to complete a brief baseline questionnaire and then will be randomized (automatically through the survey platform [Qualtrics] and without stratification) and provided either a link to the mobile intervention or information on how to access clinical services, based on their randomized condition. Eligible individuals who choose not to continue in the study will receive a recommendation to seek care and a campus-specific list of resources.

Intervention Condition

The platform, hosted by SilverCloud Health, includes three guided interventions, one for each primary problem. The interventions were derived from evidence-based, CBT guided self-help treatments that were efficacious in traditional efficacy trials, specifically for anxiety depression, and EDs [31–35]. Six to eight core modules are offered for each primary problem, which take about 20 minutes each. Modules include a combination psychoeducational content and quizzes, interactive exercises, videos, and personal stories. To maximize efficacy, key treatment targets (i.e., decreased avoidance for anxiety disorders; increased behavioral activation for depression; decreased dietary restraint and weight/shape concerns for EDs; and decreased dysfunctional thoughts and increased use of CBT skills for all three disorders) are addressed early in the programs.

Tailoring.

Individuals who screen positive for more than one of the three mental health disorders at a clinical level (anxiety, depression, EDs) will be asked to select which problem they would like to primarily focus on, at least to begin with, and will begin by working on that problem in SilverCloud. If users do not screen positive for any clinical problems but are at high risk for multiple problems, they will also be asked to select which problem they would like to begin with. This feature is designed to get students involved as stakeholders and to increase engagement. If users screen positive for a single clinical problem and as high risk for others, they will begin with focusing on the problem for which they screened at a clinical level. For users with multiple problems, key modules to address comorbidities are offered after the core content from the primary program is offered. In this way, the program is personalized to address each user’s unique needs. Further, coaches also have information about comorbidities and tailor coaching throughout the user’s time in the intervention accordingly. In addition, users complete brief assessments of depression, anxiety, and ED symptoms each week within the platform. Furthermore, if participants with clinical problems have not demonstrated improvement in their symptoms by six weeks into the intervention, as determined by their responses on the weekly symptom measures within the platform, they are provided with a referral to usual care on campus. Users are provided access to the platform for six months.

Coaching.

The role of the coach in the offered guided self-help interventions includes supporting and enhancing user motivation; monitoring progress; facilitating goal setting and offering accountability; providing feedback on technique usage and encouraging practice; answering user questions; and monitoring for/responding to clinical risk. Communication with users is primarily via two-way, asynchronous messaging, with an optional, supplementary phone call at the beginning of the user’s time in the program to enhance goal setting. Coach messaging is done via a web-based “dashboard,” and delivered to users within the environment of the program (app or web browser access). The dashboard allows coaches to aggregate across population-level data to efficiently monitor multiple users at one time. Messages are mostly unscripted to allow for personalization, although some common situations (especially risk-related) are guided by editable templates. Messaging is guided by the goals of reinforcing key intervention messages, supporting individualized application of techniques, and promoting meaningful engagement with content. Messages may include: providing feedback on technique completion; helping users apply program content to personal goals; commenting on observed symptom changes; and engaging motivational interviewing techniques such open-ended questioning. Coaches have participants’ baseline screen results so they can personalize treatment and help the user apply skills learned to comorbid problems. If the coach feels the mobile program cannot adequately address comorbidity, participants are referred to appropriate campus/community resources.

Coaches will be selected by the study team and have various educational backgrounds, including clinical psychology, social work, and counseling. All will have at least a bachelor’s degree and most will be pursuing an advanced degree. Coaches complete training in CBT, motivational interviewing, risk management, and the mobile platform. Each coach will be paired with a licensed psychologist supervisor who monitors their coaching and provides weekly supervision. The study’s Fidelity Monitoring Center will conduct regular fidelity checks, using a fidelity checklist to monitor 10% of all coach messages, and offer constructive feedback to ensure coaches follow protocol (e.g., referrals provided if symptoms do not improve).

Referral to Usual Care Condition

All participants in the referral to usual care condition will receive a recommendation to seek care and will be given information on resources available through their respective health or counseling center, including contact information to make an appointment.

Safety Plan

National data show that 9% of students have seriously considered suicide in the past year [36]. Thus, suicidality will be consistently monitored throughout the mobile intervention in intervention arm and during follow-up assessments in both arms. Participants who endorse any suicidality on the study assessments will receive immediate follow-up, including a referral to the National Suicide Prevention Lifeline. Despite the perception that assessing suicidality will increase risk, asking about this does not increase risk and, among those who endorse active suicidal plans or intent, may lead to increased help-seeking behaviors and safety [37]. We decided to include students with suicidality, given data that only 20–40% of college students with mental health problems are treated [1, 38].

Several additional safety precautions and procedures will be implemented. Coaches are notified when a user indicates suicidality on the initial screen. Coaches also monitor the platform weekly to assess for significant changes in symptoms (e.g., increased purging). Any participant deemed unsafe or needing more intensive clinical intervention will be given a referral by their coach. Appropriate follow-up (e.g., contacting the university’s on-call clinical staff or local police) will be implemented if participants are deemed to be at imminent risk and no response is provided by the participant indicating their safety. This protocol is to safeguard against concerns with keeping individuals in guided self-help care only when more intensive services may be warranted. Further, individuals who remain in the clinical symptom range, in both the intervention and control conditions, at the 6-month or 2-year follow-up assessment, will be offered a referral for in-person treatment. For individuals in the intervention condition, in line with a stepped care model, those with clinical symptoms who do not demonstrate symptom improvement by 6 weeks will be referred to in-person care by their coaches but will still be encouraged to engage in the program. We note that we will capture engagement in in-person care in both conditions in all follow-up assessments and will be able to account for this in analyses, including running a sensitivity analysis to determine the effect of including versus not including participants in the intervention condition who accessed additional care beyond the mobile platform.

Assessments

Table 1 lists the assessments and time points. Participants will be assessed at screening, and for participants who meet eligibility criteria and agree to participate in the randomized portion of the trial, at baseline, 6 weeks, 6 months, and 2 years. The screen will be incentivized with cash sweepstakes, the baseline and 6-week assessments with $5 electronic gift cards, and the 6-month and 2-year assessments with $10 electronic gift cards. Participants will also be entered into sweepstakes to win a gift card upon completing each of the baseline, 6-week, 6-month, and 2-year surveys. In addition, if participants complete all four of those surveys, they will also be entered into an additional sweepstakes.

Table 1.

Assessments and Time Points

	Screen	Baseline	Inter-vention Period	Post-Baseline Assessments			Primary Purpose
				6 wks	6 mos	2 yrs
PARTICIPANTS
Demographics	X						Mod
Expectancy/credibility		X					Mod
Degree of cognitive distortion	X						Mod
Anxiety: Generalized anxiety; social phobia; panic	X			X	X	X	TO/Mod (symptom severity; comorbidity)
Depression	X			X	X	X	TO/Mod (symptom severity; comorbidity)
Eating disorders (SWED)	X			X	X	X
Eating disorders (EDE-Q)		X		X	X	X	TO/Mod (symptom severity; comorbidity)
Chronicity	X						Mod
Comorbidities: Trauma; insomnia; substance use	X				X	X	Mod
Impairment/quality of life		X			X	X	TO/Mod
Motivation for treatment		X		X	X	X	Mod
Mental health treatment utilization	X			X	X	X	TO
Academic impairment (obtained from colleges)		X			X	X	TO
Self-reported GPA					X	X
Target engagement Transdiagnostic: dysfunctional cognitions; CBT skills Disorder-specific: avoidance; behavioral activation; restraint; weight/shape concerns		X		X	X	X	Med
Early engagement				X			Med
Rapid response				X			Med
Screening and treatment feedback					X	X	--
TREATMENT PROCESS (INTERVENTION CONDITION)
Mobile treatment process (e.g., sessions completed, messages with coach)			X				Pred
Fidelity/personalization of coaching (i.e., user perception and coder ratings)			X	X	X		Pred
STUDENT MENTAL HEALTH SERVICES CENTERS
Attitudes toward mobile mental health platform	X				X		--
Perceptions of mobile mental health platform					X		--
Intended adoption					X	X	--
Mental health service utilization	X				X	X	TO

Notes: Light gray shading = occurring during intervention period. Mod = moderator; TO = treatment outcome; Med = mediator; Pred = predictor.

The screen will assess demographics, including age, gender, year in school, race/ethnicity, and socioeconomic status (i.e., highest level of parental education). Next, both the screen and follow-up assessments will include brief, validated mental health screens. For anxiety, we will assess generalized anxiety disorder (GAD), social phobia, and panic disorder as these are the most common anxiety disorders found in college students [1], using the GAD-Questionnaire-IV (GADQ-IV) [39], the Social Phobia Diagnostic Questionnaire (SPDQ) [40], and the Panic Disorder Self-Report (PDSR) [41]. All have been shown to have excellent 2-week retest reliability in college samples, strong inter-rater agreement with diagnostic interviews, and high sensitivity and specificity [39–43]. For depression, we will use the Patient Health Questionnaire, a 9-item measure that is widely used to assess presence of possible depression and severity, as well as suicidality, with sensitivity 88% and specificity 88% [44–46]. For EDs, we will use the Stanford-Washington University EDs Screen (SWED), which includes the Weight Concerns Scale [47, 48], to screen for EDs at the screen and follow-up assessments. The SWED has been validated and used in past research, with sensitivity and specificity being high for ED cases compared to a face-to-face interview [49]. The baseline and follow-up assessments will also include the ED Examination-Questionnaire (EDE-Q) [50], a 28-item measure derived from a diagnostic interview, the standard measure of ED psychopathology, to provide a continuous measure of ED psychopathology. The EDE-Q demonstrates reliability of scores, as well as construct validity [51]. Table 2 shows established cut-off scores on the screening measures to determine high risk and clinical status. We also note that we will assess chronicity (self-reported onset) at baseline using a one-item measure.

Table 2.

Screening Measures and Cut-Offs for Anxiety, Depression, and Eating Disorders

Construct	Measure	Cut-off score
Anxiety disorders
Generalized anxiety disorder, at high risk	GAD-Q-IV	5.7
Generalized anxiety, clinical	GAD-Q-IV	diagnostic criteria
Social anxiety disorder, at high risk	SPDQ	7.4
Social anxiety disorder, clinical	SPDQ	diagnostic criteria
Panic disorder, at high risk	PDSR	8.8
Panic disorder, clinical	PDSR	diagnostic criteria
Depression
Depression, at high risk	PHQ-9	5≤PHQ-9<10
Depression, clinical	PHQ-9	PHQ-9≥10
Eating disorders
Eating disorder, at high risk	SWED	WCS>47 + at least 1 purging episode (vomiting or diuretic/laxative use) in past 3 months
Eating disorder, clinical	SWED	Meet at least one of the following (in past 3 months): 6+ binge eating episodes 6+ vomiting episodes 6+ diuretic/laxative use episodes

Notes: GAD-Q-IV=Generalized Anxiety Disorder Questionnaire-IV; SPDQ=Social Phobia Diagnostic Questionnaire; PDSR=Panic Disorder Self-report; PHQ-9=Patient Health Questionnaire; SWED = Stanford-Washington University Eating Disorders Screen.

We will screen for comorbidities. Comorbidities assessed will include post-traumatic stress disorder (PTSD), insomnia, and substance use. For PTSD, we will use the Primary Care PTSD Screen [52], a 4-item screen with a sensitivity of .78 and a specificity of .78. Insomnia will be measured by the Insomnia Severity Index [53], an 8-item measure with good internal consistency and 86% sensitivity and 88% specificity for detecting insomnia. Substance abuse will be assessed as alcohol use and illegal drug use. For alcohol use, we will use the AUDIT-C Alcohol Consumption Questions [54, 55], a 3-item scale that reliably identifies hazardous drinkers and those with alcohol use disorders. For illegal drug use, we will assess past 30 day use for each of the most common drugs, as is done in the Healthy Minds Study [56]. Impairment/quality of life will be assessed by the Short Form-12 (SF-12) [57], which has strong 2-week retest reliability.

We will assess motivation for treatment, including students’ interest in treatment, perceived need, barriers, and plans to engage, and mental health treatment utilization, including receipt of therapy, medication, self-help, and informal support, using measures from the Healthy Minds Study [56], which were adapted from the validated Healthcare for Communities survey [58]. Academic outcomes, including term-by-term GPA and enrollment status (to assess drop-out), will be obtained via self-report.

Treatment expectancy/credibility, a potential moderator, will be assessed using 4 items from the Credibility and Expectancy Questionnaire [59, 60], which demonstrates high internal consistency and test-retest reliability. Cognitive distortion, another potential moderator, will be assessed using the Dysfunctional Attitude Scale Short-Form 2 [61]. Both will be assessed at baseline.

We will assess several mediators, which fall into two categories. First, we will assess target engagement in both the control and treatment conditions at baseline and 6 weeks. For change in dysfunctional cognitions, we will use the Dysfunctional Attitude Scale Short-Form 2. For change in CBT skills, we will use the self-report version of the Skills of Cognitive Therapy Scale (revised to be relevant for all primary disorders), which has been shown to be correlated with observer ratings and to be more strongly associated with end-of-treatment outcomes than therapist ratings [62–64]. For change in avoidance, we will use item 3 from the Overall Anxiety and Impairment Scale [65–67], which has demonstrated convergent validity with a clinician-rated measure [68, 69]. For change in behavioral activation, we will use the Behavioral Activation for Depression Scale-Short Form [70], which has been shown to correlate with activity tracking data. For change in dietary restraint, we will use the ED Examination-Questionnaire Restraint subscale [50], which has strong convergence with the Eating Disorder Examination interview version [51]. For change in weight/shape concerns, we will use the Weight Concerns Scale [48]. All of these measures have demonstrated internal consistency, reliability, construct and predictive validity, and as discussed, convergent validity with observer ratings.

Second, other mediators will be assessed. Early engagement will be assessed at 6 weeks based on self-report of mental health services utilized in the past 6 weeks (e.g., number/time spent in mobile and in-person sessions). Rapid response will be assessed at 6 weeks and defined as ≥50% reduction in symptoms by that point [71, 72].

Other measures.

We will collect information on treatment process (intervention condition only), including sessions completed, number of messages sent by user and coach, program techniques completed, and time spent with the program. We will evaluate fidelity of coaching, evaluating coaches on metrics that include tone and style (authentic, personalized) of their interactions with users, content (depth, goal orientation), efficiency (response time), and user engagement and outcomes, using a measure developed for this study. User perceptions of coach support and personalization will also be measured at 6 weeks and the end of treatment using a scale designed for this study. Attitudes of stakeholders (e.g., counseling/health center directors, as well as upper-level administrators involved in study execution) regarding using an mHealth platform, will be assessed at baseline. Stakeholder perceptions of the online platform will also be assessed using 11 items that relate to relative advantage, compatibility, complexity, trialability, and observability of an innovation using an adapted scale from Scott et al. [73]. Stakeholders’ intended adoption as a result of using the platform offered as part of the present study will be assessed with an adapted item [73] at the end of the study. A cost analysis will quantify the cost of mental health-related resources required in intervention as compared to the control condition. The cost of standard mental health service use will be quantified based on students’ reports of using counseling/therapy and other mental health care. For intervention-specific costs, direct labor costs will account for cost to both maintain the online platform and provide coaching. Time spent in supervision will be included. These costs can be ascertained by coaches’ hourly wage (inclusive of fringe) and will account for training level and type of professional. Non-labor costs include depreciation costs of equipment (e.g., servers), which will be obtained from the technology partner and from websites of office supply retailers. As an alternative perspective on intervention-specific costs, we will simply examine the current prices for access to the SilverCloud platform. For use of standard mental health services (in both the intervention and control group), the cost of health care utilization will be calculated using direct labor costs. Provider time is the dominant source of direct labor costs, obtained by estimating providers’ hourly wage (inclusive of fringe), training level, and type of professional, using student-reported estimates of health care utilization. Indirect labor costs will be estimated based on standard clinical rates. Use of additional services (e.g., hospitalizations) will be obtained via self-report and translated to monetary costs using national average costs from the Medical Expenditure Panel Survey.

Data Analysis

Power Analysis.

Sample size was based on achieving ≥80% power to detect minimally meaningful detectable differences in primary and secondary outcomes between arms using 0.05 level two-sided tests within each cohort; the outcomes were substantively independent based on theoretically distinct concepts. We will include at least 20 campuses, with a projected 3,154 high risk and 3,154 clinical participants providing two-year follow-up data. We accounted for 20% missing data for the two-year follow-up assessment and expect to enroll about 3,942 participants in each cohort (total N = 7,884). This long-term follow-up rate is similar to rates obtained by our team in other studies of digital interventions [31, 74]. All power estimates were done assuming a 0.005 intra-school correlation with approximately equal number of students per campus and with two-year outcome as the primary endpoint. For service uptake, the study is expected to have 85% power to detect a difference in two-year uptake of 26% in intervention arm vs. 20% in control arm in high-risk participants and 98% power to detect a difference in uptake of 60% vs. 50% in clinical participants. For effectiveness, the study is expected to have 87% power to detect a difference in positive screen for clinical level disorders at two-years of 24% vs. 18% in high-risk participants and 99% power to detect a difference of 50% vs. 37.5% in clinical participants. For continuous outcomes, separately within each disorder subgroup, the projected sample size will have 85% power to detect a standardized difference in mean between arms of 0.25.

Methods for handling missing data to follow-up.

All analyses will be intent-to-treat analyses, using full-information maximum likelihood to handle missing data.

Analyses for Aims 1–2 (uptake, effectiveness).

To evaluate efficacy of the mHealth platform vs. referral to usual care, in engaging participants (Aim 1), we will examine uptake defined as at least one session of mental health services use (either online or in-person of any sort of treatment) assessed at 6 weeks, 6 months, and 2 years. We will also examine effectiveness in reducing a positive screen in clinical participants (all of whom have a clinical level disorder at baseline) or preventing a positive screen in high risk participants (none of whom have a clinical level disorder at baseline) for mental health disorders (any of anxiety, depression, or ED as the outcome reflecting a “clinical case”) (2a). This allows for a single, universal outcome applicable to everyone in the study. Primary endpoint will be at 2 years, and analyses will be carried out using both primary endpoint as well as repeatedly assessed outcomes at 6 weeks, 6 months, and 2 years; thus, an advantage of our analytic approach is that we will take advantage of all available follow-up data. Both uptake and effectiveness will be analyzed using hierarchical logistic regression models with random intercepts for colleges with students nested within colleges (to adjust for potential clustering of outcomes within campuses), and separate models will be fit for high-risk participants and for clinical participants. For the repeated data, the model will also include students as random intercepts to account for repeated assessments within students, and the analytic approach will be able to capitalize on all available data. The models will include fixed effects for treatment condition, time, and interactions between time and treatment condition. With assessments at 6-week, 6 months and 2 years, time will have indicators for 6-months and 2 years. To evaluate reduction in disorder-specific symptoms (2b) and improvement in quality of life (2c), hierarchical linear models will be used. Analyses of symptom change will be conducted within each clinical disorder subgroup separately with relevant symptom scores at all four study times as the response variables. We will also use a z-score to standardize primary symptom measures to obtain an estimate for a unified intervention effect across the three clinical disorders. All symptom models will include random intercepts for colleges and for students nested within colleges to account for repeated assessments within students, and will estimate the following fixed effects: main effect of time, treatment condition, and two-way interaction between time and treatment condition. Parameter estimates from these models will allow separately comparing change between arms from pre to 6 weeks, pre to 6 months, and from pre to 2-year follow-up. Each model will use dummy-coded factor terms as a stringent test of the treatment to ensure that: 1) treatment was associated with significantly greater change from pre to post than the control group; and 2) gains made in treatment at 6 weeks are maintained and significantly greater than baseline at each follow-up. For all analyses, we will also run sensitivity analyses after identifying and removing multivariate outliers based on the Minimum Covariance Determinant [75].

Analyses for Aim 3 (targets, other mediators, within-mobile program predictors, moderators).

Potential mediators, are categorized into three groups, as described earlier: 1) clinical targets (3a and 3b); 2) early engagement with services (3c); and 3) early clinical response/rapid response (3c). We will test for mediation using structural equation models (SEM) with confidence intervals derived from bootstrapping of indirect effects and total effects. We will examine targets and mediators (e.g., changes in distorted cognition score at 6 weeks or number of treatment sessions during first 6 weeks) preceding changes in outcomes (e.g., change in standardized symptom score at 6 months). To draw more informative and coherent conclusions across the three disorders, we will explore meaningful transdiagnostic clinical targets as well as disorder-specific targets. For transdiagnostic mediator outcomes, binary clinical case status and quality of life measures (e.g., SF-12 score) are natural outcome measures, but for symptom outcomes, we will use standardized symptom scores (for transdiagnostic outcomes) and non-standardized symptoms scores (for disorder-specific outcomes). Moderated mediation (with treatment vs control as the moderator) will be tested by constraining the indirect effect to be equal across treatment and control groups and testing differences in chi-square fit. Any significant misfit caused by specifying the indirect effect of each mediator to be equal suggests that there are significant differences in this mediational path between the treatment and control. All mediational paths between the primary predictor and outcome will be estimated simultaneously to account for any and all interdependence between paths and to ensure these interdependencies do not bias estimation. Note that in applying moderated mediation analyses to both transdiagnostic and disorder-specific targets, we will be simultaneously testing whether the intervention engages the targets (3a) and whether intervention-induced changes in targets are associated with clinical benefit (3b). Putative moderators (3e) will examine interaction terms with time point and random assignment in regressions to identify subgroups that benefit most from treatment, and others for whom additional tailoring may be necessary in future refinements of the delivery model. In cases where moderators are significant, we will then conduct mediator analyses separately by subgroups defined by these moderators, because different intervention effects often imply different mechanisms by which effects are occurring [76]. In moderation models, results will estimate the following fixed effects: 1) main effect of time, treatment condition, and each proposed moderator (i.e., gender, race/ethnicity [White, non-White], expectancy/credibility, the baseline symptom severity for anxiety, depression, and EDs, chronicity, psychiatric comorbidity, disorder for which the person sought treatment, cognitive distortion, impairment, family income, motivation, and clinical/at-risk); 2) two-way interactions between time and treatment condition (i.e., the primary outcome), treatment condition and each moderator, and time and each moderator; and 3) three-way interactions between time, treatment condition, and each moderator. The models will also estimate random effects of time nested within individuals, and intercepts nested within campuses. Within-mobile treatment predictors of change (3d) (i.e., number of sessions completed, number of messages sent by user and coach, program techniques completed, time spent with the program, and coach personalization) will use multilevel models using the fixed effects of main effects of time, and each within-mobile program predictor, and the interaction between within-mobile program predictors and time estimating random effect of time nested within individuals and intercepts nested within campuses.

Analyses for Aim 4 (stakeholder outcomes):

Economic evaluation analyses (4a) were designed to provide information for stakeholders considering implementing the mobile platform [77]. We define net benefit as the difference between cost of providing care (from the payer perspective) and cost of the mobile platform. As we do not yet know the value that payers would place on the platform, we will estimate the maximum a payer would pay as the difference in the cost of care between the platform and usual care. If this estimate is greater than the cost of providing the mobile platform, we will know the platform is viable. We will also calculate incremental cost-effectiveness ratios, corresponding to the incremental difference (intervention vs. control) in costs divided by the incremental difference in the primary outcome, the number of clinical cases at 2-year follow-up. Academic outcomes (4b) will be analyzed using the methods described for the primary outcomes under Aim 1. Academic outcomes will also be included into cost analyses, by translating drop-outs from college into losses in tuition (for institutions) and losses in lifetime expected productivity (for students and society generally), as in our previous work [78]. Stakeholder attitudes (4c) will be assessed for variation using descriptive statistics, and will be used to predict intention to adopt the platform using linear regression.

Discussion

The goal of the current study is to evaluate the utility of a mobile mental health platform comprised of a suite of evidence-based, CBT interventions versus referral to usual care for improving uptake of services as well as clinical effectiveness, including reducing the number of individuals with mental health disorders, reducing disorder-specific symptoms, and improving quality of life and functioning. Achieving our study aims will provide key data that may help colleges and universities adopt digital technology for mental health service delivery. If successful, we anticipate the platform could have clinical benefit to students by providing symptom monitoring and early intervention, consolidating the number of mental health programs available on campus, and decreasing barriers to treatment-seeking (e.g., convenience, lack of problem recognition, stigma). We anticipate the platform could also appeal to university stakeholders (e.g., clinicians, administrators) and minimize barriers to implementation sustainability on campuses by engaging users in multiple interventions within one platform, thus improving university buy-in and decreasing costs (versus purchasing multiple interventions), and implementing one accessible, population-based, comprehensive model that treats and prevents mental health problems while conserving in-person resources for students most in need. We will enhance potential for stakeholder support by evaluating the economic return on investment through a cost-effectiveness analysis. Finally, the online delivery model enables rapid dissemination, tailoring to individual campuses, and remote delivery, which could expand capacity for settings with limited on-campus services. Study data will also be used to identify subpopulations for whom further tailoring adapted services are needed (e.g., underrepresented minority groups), to optimize this system of care over time, as well as to confirm mechanisms of the intervention. Findings will set the stage for expanding the model to intervene with other problems as new programs are established, inform translation into other care systems, and stimulate policy for improved population health management.

There are many strengths of the current study. First, we are conducting a rigorously-designed randomized controlled trial that will engage 20+ colleges to intervene with a projected 7,884 students (i.e., unprecedented scope), which will yield data of relevance to transforming college student mental health services and address the wide treatment gap for college students with or at risk for mental disorders. Second, we are applying established screening practices based on our work deploying a national mental health screening program to engage large populations of college students into services. Third, we are delivering targeted prevention and treatment to address the mental health continuum among college students. Finally, we are providing interventions for multiple problems using one comprehensive mobile platform to address implementation barriers and mimic the flexibility of in-person care.

Outcomes will inform whether this population-level approach to service engagement yields clinical benefit to students, appeals to university stakeholders, and produces an economic return on investment compared to usual care. If successful, this population-level approach to service engagement has the potential to improve mental health outcomes for the millions of students enrolled in U.S. colleges and universities. The model could also be adapted, such as for employees and secondary school students, increasing reach and impact. Finally, we note that our proposed approach is also highly relevant given COVID-19 pandemic, when need for digital approaches for to intervention are greater than ever. Even following the pandemic, the face of mental health services will likely be changed forever, with an increased focus on digital technology [79, 80]. As such, findings from the current study will be extremely timely.

List of Abbreviations

AN

Anorexia Nervosa

CBT

Cognitive Behavioral Therapy

ED

Eating Disorders

EDE-Q

Eating Disorders Examination-Questionnaire

GAD

Generalized Anxiety Disorder

GADQ-IV

Generalized Anxiety Disorder Questionnaire IV

mHealth

Mobile Health

PDSR

Panic Disorder Self-Report

PTSD

Post-Traumatic Stress Disorder

SEM

Structural Equation Models

SF-12

Short Form 12

SPDQ

Social Phobia Diagnostic Questionnaire

SWED

Stanford-Washington University Eating Disorders Screen