Uptake and effectiveness of a self-guided mobile app platform for college student mental health

Abstract

Background

College students endorse high rates of mental health problems. While many colleges offer on-campus services, many students who could benefit from mental health services do not receive care. Indeed, nearly half of students who screen positive for depression, for example, do not receive treatment. Digital mental health programs, such as those delivered via mobile apps, may help expand access to mental health care and resources. This mixed-methods study aims to examine the uptake and effectiveness of an implementation of IntelliCare for College Students, a self-guided app-based mental health platform, on two university campuses.

Methods

Data on counseling center utilization was collected prior to the implementation of the app (pre-implementation phase) and while the app was available on campus (implementation phase). Data on app usage was collected throughout the implementation phase. A subset of participants (n = 20), along with counseling center staff members (n = 10), completed feedback interviews.

Results

Overall, uptake of the app platform was low. A total of 117 participants downloaded the app and registered their study ID during the implementation phase. Approximately 24% (28/117) of participants used the app only once. The number of days between the first and last day of app use ranged from 0 to 299, with a mean of 35.01 days and a median of 14 days. A relatively small portion of the sample (26.5%; 31/117) downloaded one or more of the IntelliCare interactive apps. In examining counseling center utilization, there were no significant changes in intake appointments, individual therapy sessions, or crisis appointments observed from the pre-implementation phase to the implementation phase of the study. Feedback interviews highlighted the significant level of disruption caused by the COVID-19 pandemic and shift to remote learning, including challenges disseminating information to students and a preference to spend less time with digital devices outside of class time.

Conclusions

Findings from this study indicate that there is an ongoing need to identify ways to reach college students and support student mental health and wellness for the remainder of the COVID-19 pandemic and beyond.

Highlights

• •The design of and implementation plan for an app for college students was developed.
• •The app was then implemented during the COVID-19 pandemic.
• •Overall, uptake of the app was low.
• •Feedback interviews highlighted disruption caused by the COVID-19 pandemic.
• •During remote learning, students wanted to spend less time with digital devices.

1. Introduction

More than 19 million individuals are enrolled in colleges across the United States (Hussar et al., 2020). Students at these institutions take on rigorous academic responsibilities while balancing peer relations, financial pressure, and rapid changes in their environment. Emerging adult students (ages 18–25) navigate a particularly high-risk developmental period associated with mental health disorder onset (Jones, 2013; Conley et al., 2014). However, entering college at any age is a critical and often stressful point in an individual's life as they begin a new career and may accrue large amounts of debt, and thus, older students are also at risk for heightened stress (Acharya et al., 2018; Walsemann et al., 2015). At the same time, many college students report mental health concerns. For example, in a large sample of the general adult population, approximately 9% screened positive for depression (PHQ-2 ≥ 3) (Fleishman et al., n.d.). Using the same cutoff, approximately 26.9% of individuals in a large sample of college students screened positive for depression (Lipson et al., 2019). Not only do clinical disorders such as depression impact college students, but subthreshold symptoms of anxiety and low mood are common (Duffy et al., 2019).

Despite many colleges offering on-campus mental health resources, many students who could benefit from mental health services do not receive care. Indeed, almost half of students who screen positive for depression do not receive treatment (Lipson et al., 2019). This may be attributed to a variety of structural (e.g., inaccessible location) and psychological (e.g., stigma) barriers (Cohen et al., 2020). Among the students who do initiate treatment, they are often placed on long waitlists due to limited availability of counseling center clinicians (Cohen et al., 2020). One method of addressing these concerns is the introduction of low-resource and on-demand digital support tools, such as mental health apps. Relative to other digital mental health tools, such as websites, mobile apps offer potential advantages, including greater convenience for students and the ability to easily send push notifications to engage a user and direct them back to the app. Mental health apps can be offered on college campuses as a supplement to traditional services or as an early step in a stepped-care model for students with less intensive needs (Cohen et al., 2021). However, there are few apps designed specifically for the mental health needs of college students that have also been rigorously studied (Duffy et al., 2019).

The IntelliCare for College Students app platform was developed to fill this gap, addressing students' barriers to receiving mental health care and capitalizing on the accessibility of digital resources. It was designed as a self-guided modified version of the IntelliCare Hub, and provides users with the opportunity to download and engage with other interactive IntelliCare apps which guide users through evidence-based cognitive and behavioral skill-building exercises. The IntelliCare platform has demonstrated efficacy in reducing depression and anxiety symptoms in adult populations (Graham et al., 2020; Ford & Norrie, 2016, Graham et al., 2020).

To develop and test the IntelliCare for College Student app, partnerships were formed with two large, public universities in the same Midwestern state. We refer to the universities henceforth as University A and University B. Interviews, surveys, focus groups, and co-design workshops were conducted with students, administrators, and counseling staff at these two institutions over the course of a year to adapt the content in the IntelliCare Hub for their student populations. The goal of the activities was to understand students' barriers to receiving mental health care, interest in digital mental health tools, and ideas for building a mental health app for college students (Cohen et al., 2020; Lattie et al., 2020a). The feedback provided through these activities directly contributed to the development of the program. After the program was developed, a single-arm eight-week usability trial was conducted with 20 students endorsing both high and low symptoms of depression and anxiety to identify areas for improvement, test feasibility, and examine preliminary effects on clinical targets (Lattie et al., 2020b). In this trial, the majority of participants (18/20, 90%) continued to use the app beyond the eight-week study period. While there were no observed changes in depression or anxiety symptoms, significant improvements were observed in the frequency with which participants used both cognitive and behavioral coping skills.

The current paper reports the results of a pragmatic, open trial of the IntelliCare for College Students platform as a self-guided program on the two college campuses. The original aims were to examine the impact of implementing IntelliCare for College Students on the utilization of counseling services on the university campuses, and to examine the impact of program use on symptoms of depression and anxiety, mental health literacy, knowledge of mental health care services, and the perceived usefulness and frequency with which participants used cognitive and behavioral coping skills. In the original trial design, we planned to collect counseling center utilization data as a baseline prior to program implementation. Then, we planned to randomly select one campus for program implementation during the first semester, while the other site would continue to collect baseline data. During this first semester, we aimed to identify additional implementation strategies to be incorporated for an enhanced implementation plan, which would be applied on both campuses during the second semester.

However, approximately one month after the program was implemented at the first campus, the COVID-19 pandemic resulted in a shutdown of on-campus activities. Mental health service delivery transformed and in-person counseling services halted, resulting in large changes in how students utilized counseling centers. Due to COVID-related changes and ongoing uncertainty surrounding operations on the university campuses with which we partnered, many of the program implementation plans were unable to be enacted. Although the study was drastically altered, the introduction of a digital mental health support tool may have been unexpectedly timely given that the pandemic exacerbated already high levels of distress among students and in-person services were unavailable (Son et al., 2020; Wang et al., 2020).

During COVID-19, both remote-delivered mental health services and self-guided online mental health resources became particularly important as students faced new financial, academic, and social stressors. Students' wellbeing during COVID-19 suffered as students reported experiencing health concerns, concentration difficulties, social isolation, difficulties with academics, financial problems, changes in eating and sleeping habits, and new living environments (Son et al., 2020). Studies have found an increase in mental health concerns in college students during the COVID-19 pandemic. According to an Active Minds Survey, one in every five students reported they experienced significantly worsened mental health due to COVID-19, 80% said their mental health had been impacted by the COVID-19 pandemic, and 55% reported that they did not know where to access mental health services if they, or someone else, needed mental health support (https://www.activeminds.org/studentsurvey/, n.d.). Additional studies show that college students experienced increases in mental health symptoms during the pandemic (Lattie et al., 2020b; Son et al., 2020). One study reported that 75% of students experienced increased stress and anxiety symptoms (Son et al., 2020). Similarly, another study found that almost 50% of students experienced moderate-severe depression symptoms, approximately 38% of students experienced moderate-severe anxiety symptoms, and 18% of students experienced suicidal thoughts. In addition to these alarming rates of mental health symptoms, less than half of the students reported being able to cope with the situation (Wang et al., 2020). There remains a need for college campuses to provide accessible and effective mental health resources for their students. Mental health apps have significant potential as a scalable, convenient, and accessible resource for students to promote mental wellness, yet engagement and use of these apps remain low.

The original aims of this study were to 1) determine the effect that implementing IntelliCare for College Students for two academic semesters had on the utilization of campus counseling services; and 2) examine the impact of program use on users' symptoms of depression and anxiety, mental health literacy, knowledge of mental health care services, and the perceived usefulness and frequency with which participants used cognitive and behavioral coping skills. It was hypothesized that 1) the number of appointments for crisis and follow-up would be reduced during implementation relative to pre- implementation and that 2) sustained use of the program would result in reductions in depression and anxiety, and these reductions would be mediated by increases in mental health literacy, knowledge of mental health care services, and cognitive and behavioral coping skills. In order to implement the program properly, we planned to test an enhanced implementation plan in the second semester of the trial. However, due to conditions related to the pandemic, we were unable to enact enhanced implementation strategies, and uptake of the program and questionnaire completion rates were too low to meaningfully examine the original aims. As such, this paper reports on an initial, single arm evaluation of an implementation plan, assessing app uptake (defined as downloads and app use), and impact on counseling center utilization. We also report on the results of a series of feedback interviews conducted with program users and counseling center staff to inform future program implementations. Although this paper cannot examine the original aims of the study as intended, it provides information on program implementation and program use that may be helpful for designing future iterations of the program or designing other digital mental health interventions for college students.

2. Methods

2.1. Setting

University A is located in a suburban area approximately 50 miles outside of a large city. As of Fall 2020, nearly 16,800 students were enrolled across the undergraduate, graduate, and professional schools. The undergraduate student population is 48.2% White, 19.2% Black/African American, 21.1% Latinx, 5.9% Asian, 3.9% Multiracial, 0.1% Native American, 0.1% Native Hawaiian, and 0.2% of unknown ethnic/racial category. Approximately 21% of students live in on-campus housing.

University B is located within a large city and had over 33,500 students enrolled across the undergraduate, graduate, and professional schools in Fall 2020. The undergraduate student body is 34% Latinx, 26% White, 20% Asian, 8% Black/African American, and 5% of other ethnic/racial categories. Approximately 11% of students live in on-campus housing.

2.2. Intervention

The IntelliCare for College Students program included multiple features designed to promote mental wellbeing, build stress management skills, and connect students with campus resources. Specific features included: (1) mood recording tools that allowed participants to document their mood using an emoji-based rating scale and/or journal entry; (2) in-app symptom assessments with personalized suggestions for strategies to improve mood; (3) mood tracking tools which allowed participants to view fluctuations in mood recordings and symptom assessments over time; (4) a list of available on-campus resources at the student's university along with their contact information; (5) a list of emergency contacts in the event of a mental health emergency; and (6) psychoeducational lessons on mental health and stress management topics (e.g., Self Care 101, Activities & Emotions). Participants were not required to complete questionnaires to gain access to the above-listed resources – they were able to access the lists of on-campus resources, emergency contacts, and psychoeducational lessons directly from the app's homepage. Previous versions of the IntelliCare Hub did not include the mood recording tools, the personalized suggestions following symptom assessments, the lists of on-campus resources, or the list of emergency contacts. These were additions to the Hub based on initial design work with college students. Additionally, existing psychoeducational lessons on mental health were revised to include more college student-specific examples, and lessons were added on topics requested from students in the initial design (such as lessons on how to help a friend struggling with mental health issues, on navigating college life, and healthy and unhealthy relationships). Students also had access to additional apps in the IntelliCare suite. These apps were developed at Northwestern University, have been tested in previous trials, and are focused on supporting the practice of evidence-based cognitive and behavioral skills (see Supplemental File 1 for descriptions of these apps) (Graham et al., 2020; Ford & Norrie, 2016, Graham et al., 2020).

2.3. Dissemination and implementation strategies

The app was made publicly available to students at University A in February 2020 and students at University B in July 2020. Data were collected through the end of the Fall semester in December 2020. Multiple approaches were used to share information about the IntelliCare for College Students app and encourage students to join the study. In the month prior to the COVID-19 pandemic, both in-person strategies and digital strategies were used, which had been developed through a careful user-centered design process to develop a detailed dissemination and implementation plan. Posters and digital monitor advertisements were displayed in high-traffic areas on campus, and study staff met with campus leadership (both students and staff) to gain buy-in and have leadership share information about the program's availability. Our research team was able to meet with leadership from residence life, campus student cultural centers, and counseling center staff to ensure that all student resident advisors were notified of the program, and that flyers would be displayed in campus dormitories, student centers, and within the counseling center.

Following the onset of the pandemic, digital strategies were more heavily relied upon to reach students. Digital strategies included 1) emailing student organizations, 2) posting to campus-specific social media pages (i.e. Facebook, Instagram, Twitter, and Reddit), and 3) directly contacting individuals on campus who could be interested in sharing information about wellness resources, such as professors and student affairs staff members. On both campuses, information about the availability of the program was included in emails that went out to the full student body, but we were unable to access data on the percentage of students who opened those emails. To email student organizations, our research team searched through public listings of student organizations on each campus and sent emails with study advertisements to the designated organization contact person, who was typically a student leader. In these emails, we encouraged student leadership to share information about the program's availability with their membership, and invited them to further discuss the program with us if they were interested. As seen in Supplemental File 2, which provides examples of research study advertisements, the program was promoted as a self-care tool. In posting to campus-specific social media pages, our research team was granted access to and permission to post in several campus-specific online communities and also shared IRB-approved social media advertisements with campus leadership (both students and staff) to prompt them to share advertisements within online communities to which our research staff did not have access. In directly contacting other individuals on campus, our research team was introduced to several individuals in student affairs leadership positions through our relationships with counseling center staff, and sent information about the program's availability to them to be shared in campus-specific emails and on campus-specific websites. Because students in our design workshops indicated that they are more likely to engage with campus resources that were recommended to them based on their academic program, we also directly emailed all department chairs at both universities to notify them of the program's availability and to encourage them to share the study information with their student bodies. Unfortunately, we were not able to capture definitive metrics regarding who, in terms of student, staff, and faculty leaders, shared the message with their relevant student bodies. For example, at University A, 7 of the 44 department leaders replied to our message indicating that they would share it with students, yet students who used the app reported to our study team that they had been sent information about the app from faculty leaders who had not responded to our study team and thus, 7/44 (15.9%) is likely an underestimate. Additionally, information regarding the app was posted on the Counseling Center websites at both universities and counselors were made aware that the app was available as a resource they could recommend to their clients.

All recruitment materials directed students to complete an online screener if they were interested in joining the study. The screener assessed students' university affiliation, age, and smartphone ownership. To be eligible for participation, students were required to: (1) be enrolled at least part time at University A or University B, (2) be over 18 years of age, and (3) own a smartphone. Upon completion of the screening process, eligible students were instructed to complete an online informed consent form. All study procedures were approved by the institutional review board at the author's institution.

2.4. Quantitative data collection

2.4.1. App usage

Data were collected from the app regarding each participant's app interactions. The primary app usage metrics reported in this analysis include: number of assessments completed, number of unique days the participant opened the app, and number of days in-between the participant's first and last day of use. We also report on the number of participants who downloaded other IntelliCare interactive apps, the frequency with which participants accessed resources and lesson content within the Hub app, and the frequency with which participants completed symptom assessments.

Participants were able to complete mental health assessments within the app at any point during study participation, and all assessments were optional. Mental health symptom assessments, including the Patient Health Questionnaire-8 (PHQ-8) (Kroenke et al., 2009) and Generalized Anxiety Disorder-7 (GAD-7) (Spitzer et al., 2006), were available to students on a weekly basis.

Patient Health Questionnaire-8 (PHQ-8) (Kroenke et al., 2009). The PHQ-8 is an 8-item assessment that measures symptoms of depression. Participants are asked to rate how often they have been bothered by symptoms in the last two weeks on a 4-point unipolar scale (Not at all - Nearly every day). In this study, the PHQ-8 demonstrated reliability (α = 0.88). In past studies, a cut-off score of 10 on the PHQ-8 has demonstrated 88% sensitivity and 88% specificity for major depression (Kroenke et al., 2009; Kroenke & Spitzer, 2002).

Generalized Anxiety Disorder-7 (GAD-7) (Spitzer et al., 2006). The GAD-7 is a 7-item assessment that measures symptoms of anxiety. Participants are asked to rate how often they have been bothered by symptoms in the last two weeks on a 4-point unipolar scale (Not at all - Nearly every day). In this study, theGAD-7 demonstrated reliability (α = 0.88). In past studies, a cut-off score of 8 the GAD-7 has demonstrated 83% sensitivity and 84% specificity for generalized anxiety disorder (Plummer et al., 2016).

Following the completion of a mental health symptom assessment, participants received personalized feedback on their symptoms, as seen in Fig. 1. To maximize the generalizability of how students used the app under “real world” conditions, completing mental health assessments was optional to study participation and we did not contact participants to prompt mental health assessment completion outside of the built-in app notifications.

Fig. 1.

Screenshot of symptom feedback.

2.4.2. Counseling center utilization

Data were collected from the counseling centers at both universities regarding how many intake appointments, individual counseling and crisis management/triage appointments were scheduled each month from August 2018 to December 2020. For University A, August 2018 to January 2020 was labeled as the pre-implementation phase, and February 2020 to December 2020 was labeled as the implementation phase. For University B, August 2018 to July 2020 was labeled as the pre-implementation phase and August 2020 to December 2020 was labeled as the implementation phase.

2.5. Qualitative data collection

Feedback interviews were conducted with both students who used the IntelliCare for College Students program and counseling center staff members. These semi-structured interviews were conducted by two advanced research assistants (a male with a master's degree and a female enrolled in a master's degree program, both in their 20's), who both had past experience conducting feedback interviews and received training from the principal investigator to ensure that similar probes and follow-up questions were used. We planned to conduct 20 student interviews and 10 staff member interviews. Students who had consented into the main trial were contacted by email and invited to participate in a supplementary 30-minute semi-structured, audio-recorded user feedback interview. All counseling center staff members from both universities were also invited by email to participate in a 30-minute semi-structured feedback interview. Questions included in these interviews were selected to elicit information on how this study was perceived and on potential improvements that could be made for future implementations. Student participants provided feedback on their experiences using the IntelliCare for College Students program, shared information about preferences for receiving information about programs and services available on their campus, and discussed how the COVID-19 pandemic impacted their preferences for learning about and using digital tools. Counseling center staff participants provided feedback on how they observed the IntelliCare for College Students program and other digital mental health tools being used, how the program's availability impacted the counseling center, and how the COVID-19 pandemic impacted their work with students. Both students and counseling center staff members were provided with a $20 gift card for participating in the feedback interviews.

2.6. Data analysis

We used a mixed-methods approach to analyzing the impact of the IntelliCare for College Students program, in that we analyzed quantitative data on app usage and counseling center utilization alongside qualitative data generated through interviews with students and staff. Demographic data were examined primarily in the form of descriptive statistics, and chi-square tests and a student's t-test were used to examine potential demographic differences between the participants who consented to the study that did (n = 117) and did not (n = 65) download and register their participant ID to the IntelliCare for College Students app. App usage data were examined in the form of descriptive statistics. See Supplemental File 3 for data analyses conducted on mental health symptom data and intervention target data. These data analyses are in line with the original plan for this project and are not included in the main body of this paper due to a high degree of data missingness.

As originally planned for this project, counseling center utilization data were examined using Simulation Modeling Analysis (SMA) for Time-Series data to determine if there were changes in utilization between the pre-implementation and implementation phases. Because we were unable to enact an enhanced implementation strategy during the second semester of program implementation, we examined implementation as a single phase. SMA evaluates the statistical significance of between-phase changes in data streams and also accounts for the presence of autocorrelation (the non-independence of data points in time-series data streams). A phase effect size (Pearson's R) is produced for each phase-comparison that is then compared to a distribution of data streams, using bootstrapping, resulting in an empirical estimate of the probability of the observed effect occurring by chance (Borckardt et al., 2008).

Interviews were transcribed and double-coded using a thematic analysis approach (Braun & Clarke, 2006). The interview transcripts were first reviewed for thematic content. In creating the codebook, coders met to review their initial codes to reach initial consensus and identified the primary themes that were used to create a codebook. Then, a second round of review took place as the coders reviewed the codebook, refined codes as needed, and completed a final round of coding. The coders met regularly with each other throughout the analytic process and discussed these codes with the study principal investigator, to come to a consensus, and ensure validity.

3. Results

3.1. Participant retention

Between February 1st, 2020 and December 31st, 2020, 310 individuals completed the screener. Of these individuals, 174 were students at University A and 124 were students at University B. A small subset of 12 individuals who completed the screener was ineligible for the study because they were not students at either university. One individual was younger than 18 years old and hence, ineligible. No individuals were excluded based on smartphone ownership.

Of the 297 individuals who completed the screener and were eligible for participation, 182 participants consented to the study by completing an informed consent form on REDCap. After completing the consent procedures, participants were automatically provided with a study ID through REDCap, and provided with instructions to download the IntelliCare for College Students Hub app on their phone and register their study ID in the app. Study staff sent reminder emails to participants to register their study ID in the app. A total of 117 participants downloaded the app and registered their study ID during the study period.

3.2. Participant characteristics

Participant demographics are presented in Table 1. To examine if there were any significant differences in race, ethnicity, gender, and university affiliation between the participants who consented to the study that did (n = 117) and did not (n = 65) download and register their participant ID to the IntelliCare for College Students app, a series of Pearson's Chi-Square tests and a Student's T-Test were conducted. Participants who indicated they were American Indian or Alaska Native (n = 2) and selected “other” as their gender (n = 3) were removed from the chi-square analyses given their small sample size. Chi-square results showed no significant differences in race χ²(3) = 1.58, p = 0.66, ethnicity χ²(1) = 0.004, p = 0.95, gender χ²(1) = 0.176, p = 0.67, and university affiliation χ²(1) = 0.22, p = 0.64 between those that consented that did and did not download the app. The t-test also indicated there was no significant difference in age between the two groups t(180) = −0.78, p = 0.43. Although these tests were not powered to detect any specific differences, there do not appear to be any clinically relevant differences in the demographics. Because the PHQ-8 and GAD-7 were only administered through the app, we were unable to examine if there was a difference in depressive or anxious symptoms between the two groups, and we did not examine symptoms at the time of feedback interviews.

Table 1.

Demographic characteristics.

Characteristic	Consented (n = 182)	Consented and Downloaded App (n = 117)	Students who Participated in a Feedback Interview (n = 20)	Counseling Center Staff who Participated in a Feedback Interview (n = 10)
Age, mean (SD)	23.59 (5.29)	23.82 (4.97)	24.27 (5.93)	41.67 (9.53)
Gender (n, %)
Female	148 (81.3%)	96 (82.1%)	16 (80.0%)	6 (60.0%)
Male	28 (15.4%)	17 (14.5%)	4 (20.0%)	4 (40.0%)
Other	3 (1.6%)	2 (1.7%)	0 (0.0%)	0 (0.0%)
Declined to respond	3 (1.6%)	2 (1.7%)	0 (0.0%)	0 (0.0%)
Race (n, %)
White	82 (45.1%)	56 (47.9%)	6 (30.0%)	8 (80.0%)
African American	20 (11.0%)	12 (10.3%)	4 (20.0%)	0 (0.0%)
Asian	43 (23.6%)	25 (21.4%)	2 (10.0%)	1 (10.0%)
American Indian/Alaska Native	2 (1.1%)	2 (1.7%)	0 (0.0%)	0 (0.0%)
More than one race	22 (12.1%)	15 (12.8%)	7 (35.0%)	0 (0.0%)
Declined to respond	13 (7.1%)	7 (6.0%)	1 (5.0%)	1 (10.0%)
Ethnicity (n, %)
Non-Hispanic	143 (78.6%)	92 (78.6%)	15 (75.0%)	8 (80.0%)
Hispanic	37 (20.3%)	24 (20.5%)	5 (25.0%)	2 (20.0%)
Declined to respond	2 (1.1%)	1 (0.85%)	0 (0.0%)	0 (0.0%)
University (n, %)
University A	105 (57.7%)	69 (59.0%)	14 (70.0%)	4 (40.0%)
University B	77 (42.3%)	48 (41.0%)	6 (30.0%)	6 (60.0%)

3.3. Recruitment sources

Of the participants who consented to the study (n = 182), 66 (36.3%) participants heard about the study through email, 36 (19.8%) heard about the study through student organizations, 17 (9.3%) heard about the study through flyers, 11 (6.0%) heard about the study through word of mouth, 6 (3.3%) heard about the study through Facebook, 4 (2.2%) heard about the study through Instagram, and 42 (23.1%) heard about the study through other sources. Other recruitment sources included university counseling center websites, faculty members, coursework, digital advertisements, and a university Reddit forum.

3.4. App usage

There was considerable variability in the frequency with which participants used the IntelliCare Hub app. The number of unique days that the app was used ranged from 1 to 95, with a mean of 6.29 days (SD = 11) and a median of 2 days. The number of days between the first and last day of app use ranged from 0 to 299, with a mean of 35.01 days (SD = 55.87) and a median of 14 days.

Approximately 24% (28/117) of participants who consented to the study and downloaded the app used the app only once. Fisher's exact tests revealed no significant differences in race (p = 0.59), ethnicity (p = 0.27), gender (p = 0.32), or university (p = 0.52) between participants who used the app only once and participants who used the app more than once.

Because participants primarily used the IntelliCare Hub app, we examined how features within that app were used. First, we examined how students used the app to learn about other wellness resources available to them (e.g., campus counseling center, student services, campus recreation), and found that approximately 25% (29/117) of student users accessed the campus-specific resources within the app. The number of distinct days that these users accessed these resources ranged from 1 to 5 days, with a mean of 1.31 days and a median of 1 day. The number of days between the first and last day users accessed campus resources ranged from 0 to 224 days, with a mean of 12.76 days, and a median of 0 days. A much smaller subset of users (7.7%; 9/117) clicked into the emergency resources section titled “Get Help Now” within the IntelliCare Hub. The number of distinct days that this subset of users’ accessed information about these emergency resources ranged from 1 to 3, with a mean of 1.33 days and a median of 1 day. The number of days between the first and last day users accessed the emergency resources ranged from 0 to 30 days, with a mean of 3.44 days, and a median of 0 days.

Participants also had access to 12 brief psychoeducation lessons on self-care, mental health, and wellbeing topics. We examined participants' use of these lessons and found that 23.1% (27/117) of users accessed the psychoeducation lessons within the app. Users most frequently accessed lessons on self-care (11.11%; 13/117) and depression and anxiety (11.11%; 13/117), followed by navigating college life (6.84%; 8/117), the value of social support (5.98%; 7/117), thoughts and feelings (5.98%; 7/117), and activities and emotions (5.98%; 7/117). The number of lessons this subset of users accessed ranged from 1 to 12, with a mean of 3.04 lessons, and a median of 2 lessons. The number of distinct days that users accessed the lessons ranged from 1 to 3 days, with a mean of 1.33 days and a median of 1 day. The number of days between first and last day of lesson use ranged from 0 to 30 days, with a mean of 2.59 days, and a median of 0 days.

A relatively small portion of the sample (26.5%; 31/117) downloaded one or more of the IntelliCare interactive apps (see Supplemental File 1 for a list and description of these apps). Of participants who downloaded one or more of the interactive apps, the median number of app downloads were 2.23 (range: 1 to 5). Day to Day and Daily Feats were the most commonly downloaded apps, with 16 downloads each, closely followed by Thought Challenger with 15 downloads.

Mental health symptom assessments were available for participants to complete on a weekly basis but none of the mental health symptom assessments were obligatory for participation in this study. We received baseline mental health symptom assessments from 73 participants, the median PHQ-8 score was 10.5 (range: 1–24) and the median GAD-7 score was 10 (range: 0–21). However, follow-up assessments were completed inconsistently, and thus outcomes cannot be reliably calculated. Mean assessment scores are provided in Supplemental File 3.

3.5. Counseling center utilization

No statistically significant changes in monthly intake appointments, individual therapy sessions, or crisis appointments were observed from the pre-implementation phase to the implementation phase of the study (all p's > 0.05). While there were observable reductions in appointments between the pre-implementation and implementation phases, these changes were likely due to the impact of COVID on counseling center operations. See Table 2 for monthly averages of appointment counts during the pre-implementation and implementation phases at both universities.

Table 2.

Mean counseling center appointments by month.

Appointment type	Pre-implementation phase		Implementation phase		AR	Level change	Slope change
	M	SD	M	SD
University A
Intake appointments	25.4	18.2	9.7	13.1	0.439	r = −0.427, p = 0.124	r = −0.319, p = 0.275
Individual therapy appointments	224.6	138.3	149.3	85.1	0.379	r = −0.294, p = 0.294	r = −0.246, p = 0.367
Crisis appointments	3.3	2.9	1.7	2.1	0.142	r = −0.278, p = 0.183	r = −0.343, p = 0.110
University B
Intake appointments	117.1	60.7	99.8	37.4	0.59	r = −0.116, p = 0.731	r = −0.408, p = 0.248
Individual therapy appointments	531.5	166.2	442	122.6	0.401	r = −0.213, p = 0.429	r = −0.264, p = 0.362
Crisis appointments	12.8	10.1	7	4.2	0.587	r = −0.234, p = 0.482	r = −0.229, p = 0.535

Note. AR= Lag-1 autocorrelation, calculated for the entire data stream using the expectation maximization algorithm. Level and slope change results are presented as correlation coefficients (Pearson's r).

3.6. Feedback interviews

Through the quantitative data gathered, we see that uptake of IntelliCare for College Students was low and that there was not a significant impact of the program on counseling center utilization. However, this does not necessarily indicate that programs such as IntelliCare for College Students are not of potential value. In fact, interviews with students who used the IntelliCare for College Students program and with clinicians from the counseling centers noted the perceived value of this resource, and underscored the COVID-19 pandemic as a disruption in what would be “normal” campus life. In the next section, we describe three main themes with implications for planning future implementations of digital mental health programs on college campuses.

3.6.1. Motivations for and use of the program

In exploring motivations for using the program, the most common goal students reported was to keep track of their mood over time. Past research has established that desire to track health-related data is a primary reason for using health apps (Rubanovich et al., 2017). For example, Participant 20A used the tracking features to look through their “mental health progress from week to week and month to month to see just how my mood is over time. Because in the moment, you have all these feelings and emotions but it's good to look back and see if you're making progress or if something needs to change.” Participant 12B expressed that journaling allowed them to identify triggers to their anxiety:

“I think the biggest difference is that I just became more self-aware and a little bit more educated on anxiety and depression because I've always known that I've had really bad anxiety, but I didn't know what exactly triggered it and being able to journal at the end of the night or the next morning about the previous day, I think that helped me become more self-aware.”

Students noted that developing increased awareness of personal mental health through tracking features appeared to lead to improvements in quality of life and wellbeing.

Multiple students suggested that they downloaded the program directly as a result of the COVID-19 pandemic, and noted that “when COVID first started, so it was a more difficult time. We were more so in the house, not really interacting with people, and so I thought it was very, very helpful during that specific time” (Participant 11B). For some students, the decision to use the app was in part a result of the loss of in-person counseling services: “I had counseling at school and since I was like okay, now I'm home and I won't have like that face-to-face contact, maybe it would be good to have sort of like this app to use as reflection to see how I'm doing” (Participant 5A). While the program was not designed specifically for use during the pandemic, it is clear that, for some students, it filled a need for support that was created by new pandemic-related stressors.

An initial goal in developing the IntelliCare for College Students program was to increase accessibility of mental health resources and services on campus, and engage students who may not otherwise be engaged with mental health services. In this study, we observed that some of the students interviewed held leadership roles on campus such as orientation leaders, resident advisors, and members of student organizations focused on mental health and wellbeing. While the study was conducted during an unusual school year in which many of the plans for disseminating program information were disrupted, Participant 8A noted that having the mental health resources in the app could help them in their leadership position:

“But it – it was nice to know that it's something that's there, especially because – because I work as a community advisor, and I do interact with a lot of students. If I was back on campus, and I noticed that one of my residents needed, or I think, could benefit from one of those resources, it's nice that I can be on my phone and then just find it really easily, rather than having to look it up, and try to verify a source. It's already verified.”

Based on this feedback, we believe that future implementations of digital mental health treatments should include strategies such as identifying student leaders to help share information about the app itself, and to use the app to help these leaders identify other appropriate resources for students.

Generally, feedback on the program was positive. Students interviewed reported that the program helped increase their mental health literacy and that the program helped them learn self-management skills. Participant 18B commented, “I found myself being able to just step back and think about more neutral or happier thoughts versus just the constant worry, which sometimes leads to maybe giving up on an assignment. I always come back to it, but it's helped me just pause, think about things, take a few breaths, and then continue once I feel more stable.” Participants were able to apply new cognitive coping skills to their lives, which had positive impact on their well-being.

Notably, many students focused on the IntelliCare Hub app, and did not use the IntelliCare interactive apps that have been the central component of the platform in our past trials. In past trials of IntelliCare, participants were encouraged to use the platform daily for an 8-week period and to focus on exploring the IntelliCare interactive apps. In this study, students were provided with access to the platform as an openly available resource. The difference in expectations set for use of IntelliCare appears to have impacted use of the platform fairly notably. While there were app-specific goals (for example, in Daily Feats, users are encouraged to log in daily to build “streaks” of use and “level up”), some students interviewed reported that they did not understand what the interactive apps were for, while others reported that they did not want to download multiple apps due to limited storage. Of students who used the IntelliCare interactive apps, feedback was generally positive. Students used the interactive apps to build specific stress management and coping skills, as Participant 18B explains:

“It's helped me well, one, identify where I am. I feel like sometimes – if you want to get to a particular point – you really need to know where you are to start and I found myself using ...Worry Knot just because I'm in school and everything and well a nephew with his e-learning. So, it gets hard to juggle everything and concentrate. So, I find myself sometimes worrying way too much about upcoming assignments, things like that. So, using the Worry Knot [app] has definitely – and several of the other apps – has definitely helped me become more aware and find alternatives to going about my thought process or behavior.”

Students had particularly positive feedback for the “Daily Feats” interactive app, which encouraged participants to complete daily activities and celebrate accomplishments. Participant 18B commented, “I think a lot of us wake up every day. Everything is still so uncertain and having those to remind you that, Oh, I did do something one day. I didn't feel like this was a waste of a day because I did eat something healthy. I did do some research and it lets you customize what you want your Daily Feats to be and you can choose from what you did. I think that's really helpful especially during this time.” The strategies taught within Daily Feats felt particularly relevant amidst the uncertainty of the COVID-19 pandemic, and, as one participant commented “helped me account for small victories and move on to bigger ones” (Participant 13B). Given the clinical utility of the interactive apps and their potential for teaching valuable self-management skills, this trial revealed the value in setting expectations for use of the apps, rather than relying on user-led exploration of what parts of IntelliCare may be useful for them.

3.6.2. Disseminating information and engaging potential end users

First, while digital mental health programs are often lauded for their ability to reach users wherever they are, successfully implementing digital mental health programs is in large part dependent on the fit of the program into the sociotechnical ecosystem. From our interviews, we saw the impact of the COVID-19 pandemic on the ability to disseminate information about the availability and utility of IntelliCare for College Students and on students' interest in and willingness to use digital mental health tools. Participant 1A described the shift away from word-of-mouth program dissemination: “I definitely hear about things through email and not, uh, word of mouth anymore. I definitely hear things through different means than I had in the past because I'm not interacting with people anymore as much.” While most students felt comfortable using technologies to support communication during the pandemic, difficulties emerged related to appropriately capturing student attention to build motivation to engage with the app-based program. One participant reflected on this issue,

“For me, because we're in COVID, it's too much – the emails have been coming a little bit more excessive than normal. I know that was usually the best way for me, but since I get emails from everyone now, it's been a little bit harder to kind of siphon through all my emails. … Because I remember when I first saw the flyer, I was like, ‘Oh, that's cool.’ And I was reading it in the elevator, and that's kind of when you have a lot more time, and I had seen the flyers probably three times before I decided to download the app. So, definitely something visually that I can kind of see every so often. I think that was kind of the most helpful, and if we weren't in a pandemic right now, I think emails would be very helpful, too” (Participant 11B).

This message was reflected by other students and clinicians interviewed. Clinicians and students both voiced that levels of emotional distress were generally heightened throughout 2020. While email was a primary method of communication both prior to, and during, the pandemic, many individuals felt overwhelmed by the volume of emails received, and were no longer able (or willing) to devote attention to emails that were not time-sensitive and directly related to their university responsibilities. While flyers could be viewed multiple times to build interest in the program (as reflected by Participant 11B above), emails were often going unread. This is consistent with Smith et al. (2021) research on college students' use of mental health apps during the pandemic, in which they found that students were experiencing digital overload which led to disengagement from apps (Smith et al., 2021).

Many of the dissemination and implementation strategies that were planned for this trial, such as on-campus strategies and close partnerships with student organizations on campus, were unable to be enacted during the pandemic. This was in part due to COVID-19 public health protocols, and in part due to the continued and heightened levels of stress observed on the campuses during the study period. Student organizations and campus offices that had been engaged during the design of the program were often unable to engage with the research team during the study period due to conflicting demands and pressures on the universities brought about by the dual pandemics of COVID-19 and racial injustice throughout 2020. When one clinician was prompted to reflect on the impact of the app on counseling center utilization, they noted “I mean just to be honest I don't know how much it has, which I think is kind of a bummer, but I think it was just such bad timing with COVID and whatnot” (Staff ID 22).

The switch to remote-learning and remote-working was isolating for many, and both students and clinicians talked about the value of getting away from screens and being connected with people in a face-to-face capacity. Participant 3A suggested, “With e-learning starting up I think because I was on my computer a lot more than normal, I was neglecting, like, other things that were also digital, which I feel like also made me use the app less.” Screen fatigue was also a concern, as Participant 13B addressed: “For my class schedule, I'm spending literally five or six hours just looking at my screen, whether it's taking notes or listening to a lecture, so after that... I really just want to go eat, or go to sleep, or relax, or rest my eyes some type of way.” Thus, the results of this study are likely not fully generalizable to implementations in a post-pandemic world in which college campuses once again become hubs of social interaction, and students are attending classes in person rather than on their computer screens.

3.6.3. Opportunities for improving the program

Students and clinicians offered a number of suggestions for better engaging students with the IntelliCare for College Students program. The most common suggestion students offered was to combine the IntelliCare Hub app with the IntelliCare interactive apps. This appears critical, as many students in this study did not download and routinely use the interactive apps. Participant 17A suggested this due to limited storage on their phone: “If there was a way to combine them together it would be really helpful too so I don't have to download as many separate apps” while Participant 6A made the same suggestion for simplicity's sake: “I'd just like them all to be in one place because life is busy and it would make life easier.” This change to the IntelliCare program is currently being made, and should make all of the IntelliCare platform features more accessible to users in future implementations.

While the lessons in the Hub app were generally viewed favorably, students suggested additional lesson topics to add to the program, such as “healthy relationship with food” (Participant 15B), “coping during social justice movements” (Participant 13B), and “bipolar disorder or schizophrenia” (Participant 8A). Students also suggested making the app's colors more lively and gamifying the app functions to encourage engagement. Participant 16A commented that “the color scheme is kind of basic. I don't know if maybe being able to customize it with different colors would be a nice option. Or I guess maybe just having a point system. If you get things done or use the app, because I know it has streaks if you go on it multiple days in a row, but maybe something more motivational.” While the Daily Feats app had a “streak” building feature at the time of this study, there appeared to be an opportunity to use streaks along with dynamic messaging and reminders to build motivation to engage in the program. Although the program offered participants the opportunity to set up daily reminders to use the app, some participants were not aware of this feature or wanted to receive more personalized reminders, suggesting a need for more structured onboarding or guidance.

There were also opportunities identified for improving buy-in, and referrals, from counseling center clinicians. While counseling center clinicians were not intended to be gatekeepers of, or primary referrers to, the program, their role in supporting access to digital mental health programs on campus is valuable. As one clinician who recommended the app to students in a therapy group noted, “I think a lot of it is my own buy-in, which I think is something like with the IntelliCare app that I think we could have improved on with our center – just in our own clinician buy-in – that this can be helpful and this can be useful. Honestly, until I used it with that group, I had this like, ‘Oh, this is great, but I don't know how helpful it might actually be to students.’ And then seeing it in my group I was like, ‘Oh, hell yeah. This is really helpful’” (Staff ID 22).

Research project staff presented the app at counseling center staff meetings and sent follow-up emails to encourage clinicians to test the app themselves, yet many clinicians did not follow-up to download it on their own phones. As one noted “I know that when it was first launching, there was a special code or something you needed to get into it. And I think that I never received that code or got that code. I never maybe requested it. I'm not sure. And so, I had hoped to put it on my phone so that I had it, because that's one of the things I try to do when students tell me about apps that they use. But I just never was able to get it on my phone. And so, I think the ease of trying to download it and get it as an app that I personally have on there would have helped me recommend it, I think, more often” (Staff ID 28). In future program dissemination and implementation efforts on college campuses, counseling center clinicians should likely be provided with more hands-on training with IntelliCare for College Students, more frequent reminders of program functionality, and more accessible program description documents to better share program information with students. As peers have been identified as a valuable referral source, the use of referrals directly from counseling center clinicians to students could be a valuable pathway to support peer-to-peer referrals and demands further attention.

4. Discussion

This study aimed to examine uptake of the IntelliCare for College Students program through two campus-wide implementations, and to examine the impact of the program on counseling center utilization. Overall, the number of students who downloaded the app was lower than we originally expected, given the sizes of the universities included in the trial. Out of a combined student population of approximately 50,000 students, only 117 students downloaded and registered this app. The dissemination and implementation strategies used in this trial were clearly ineffective. As noted in the results, we did not find an observable impact on counseling center utilization. The low uptake is consistent with a recently published systematic review of the impact of COVID-19 on clinical trials (McCloud et al., 2020). The question of whether a well-implemented mobile mental health app could impact counseling center utilizations remains unanswered. Given the difficulties observed in this trial, we are able to offer considerations for future studies of the dissemination and implementation of digital mental health tools on college campuses. Below, we discuss the challenges associated with pragmatic trials of digital mental health programs, potential explanations for the observed low levels of app uptake, and dissemination challenges in light of the COVID-19 pandemic.

4.1. Pragmatic trial design challenges

Of the students who downloaded the app, many stopped using the app early on, with nearly one in four students using the app only once after download. These results contrast with previous findings from an 8-week trial of the original IntelliCare platform, in which participants were randomized to use IntelliCare with the support of a coach, or in self-guided/non-coached manner. Findings from that trial found that 90.7% of coached participants and 83.4% non-coached participants continued to use IntelliCare throughout the full trial period (Lattie et al., 2020b). These results also contrast with previous findings from an 8-week usability trial of self-guided IntelliCare for College Students, where 90% of participants used the app beyond the 8-week trial period. Notably, the 8-week usability trial required that participants complete assessments, and had a clear start and end point, and provided greater monetary incentives to participants in that all participants were eligible to receive a total of $70 for participating in the study, including $10 for completing each of the monthly in-app assessments, and $20 for completing each of the user feedback interviews (Lattie et al., 2020b). The pragmatic design of this trial highlights how research findings may change drastically when interventions are implemented in more pragmatic, real-world contexts (Deaton & Cartwright, 2018; Ford & Norrie, 2016).

When participants did not use the app, we were also unable to collect the assessment data that was desired for examining intervention effects. While in-app assessments worked well for the usability trial, there was a high degree of missing data on the mental health symptom assessments and the assessments of the intervention targets (mental health knowledge and skill use) in this trial, as many students quickly discontinued use of the app. This pattern has previously been observed in research trials of MoodGYM, a widely studied internet-based cognitive behavioral therapy program. For example, one analysis found that only 0.5% of community MoodGYM users completed a noncompulsory final assessment, compared with 22.5% of participants in a trial evaluating the same program (Christensen et al., 2004). Thus, in the present trial, we were unable to meaningfully examine potential relationships between program use, mental health symptoms, and mental health knowledge and skill use.

Because the IntelliCare platform has demonstrated efficacy in reducing symptoms of depression and anxiety in multiple trials (Graham et al., 2020; Mohr et al., 2017; Mohr et al., 2019), these findings point to a need for including better engagement strategies for users of the IntelliCare for College Students program. Including human support in the form of coaching is likely to help engage users, as is the use of varied, personalized push messaging to pull users back into using the program (Carlbring et al., 2018; Karyotaki et al., 2021; Borghouts et al., 2021).

4.2. Potential explanations for low uptake

There are several possible explanations for the low levels of uptake observed in this trial. The context of the COVID-19 pandemic was markedly different from that of the 8-week usability trial, and thus, there were a number of challenges experienced in disseminating information about the availability of IntelliCare for College Students. Students in our design studies suggested that flyers and word-of-mouth communication were preferred methods of learning about new mental health support tools (Lattie et al., 2020a). Students in these studies also emphasized the need to see information about new resources in multiple formats and in multiple places (both on campus, and within their online worlds). However, the delay between design in the pre-pandemic world, and implementation in the COVID-19 pandemic world, resulted in us attempting strategies to engage students that were relatively fruitless. For this study, we hoped to target the same group of end users from design to implementation. The amount of time from design to implementation occurred over a 2–3-year period. During this time, the world dramatically changed with the onset of the COVID-19 pandemic and we were unable to enact previously designed plans.

Alternatively, the low levels of uptake in this trial could reflect students' general hesitation to use a mobile app as their primary source of mental health support, rather than be the result of the upheaval caused by the COVID-19 pandemic. Looking at a more general adult population, researchers found an increase in the self-reported use of digital mental health tools during the early stages of the COVID-19 pandemic (Sorkin et al., 2021). Other recent studies of mental health apps on college and university campuses have either explicitly noted difficulty recruiting and engaging students (Sathian et al., 2020) or describe extensive recruitment strategies used to recruit a requisite number of college students from very large populations of students for randomized controlled trials (McCloud et al., 2020; Newman et al., 2021). In a recent study of adolescents and younger adults, only 1.5% of those surveyed reported that their first choice of treatment for mental health concerns was to use a mobile app, while 31.7% indicated that they used mental health apps in the last 6 months, and 61.6% indicated that they were willing to use mental health apps (Cohen et al., 2021). While apps are generally considered more accessible than traditional, in-person services, college students have previously suggested that they believe apps are impersonal, they are not sure of the efficacy of apps, and they are afraid of privacy breaches in apps (Kern et al., 2018). Additionally, apps have unique barriers to entry compared to other technology-enabled services such as websites or text-message based programs. They require storage space on an individual's phone and must be consistently updated to work with the latest mobile operating systems. Thus, the usability and acceptability of mobile-only apps for mental health relative to mobile-responsive websites and text-message based programs among college students demands further attention.

4.3. Dissemination challenges in light of the COVID-19 pandemic

Dissemination efforts were abruptly altered when students stopped attending in-person classes, and methods such as face-to-face meetings and broad-scale flyer distribution across campus were largely unable to be used. The shift in students' academic and social lives had a significant impact on how students received information about programs available to them on campus and impacted their preferences for how they would like to receive program information in the future. While word-of-mouth was identified as a successful dissemination strategy prior to the COVID-19 pandemic, few students reported hearing about the study through word-of-mouth in our trial. Students reflected on a shift from word-of-mouth to email communication, stating that they have fewer opportunities to share campus information with peers. In the feedback interviews, students reported being inundated with email communications from their university throughout the pandemic, which meant that program information was frequently competing with other campus-related information for students' attention. While email was still the most frequent recruitment source for our study and many students indicated email was still a preferred source for campus information, future studies should carefully consider that communication channels that are frequently used may not be the best way to reach potential participants. Further, ways to break through saturated media such as email should be carefully designed – this includes an email's source, length, and subject line to increase the likelihood that students will engage with study information by email. A systematic review examining factors that affect participants' enrollment in digital health interventions found that effective recruitment strategies include personalizing recruitment materials and ensuring that recruitment messages are understandable, relevant to the participant, and come from a known source (O’Connor et al., 2016). Previous studies have also shown that recruitment materials must be carefully developed to ensure that the content, style, and language used is acceptable and engaging to young adults (Lorimer & McDaid, 2013). As such, it is likely critical to include study participants in the development of recruitment materials.

Many dissemination strategies presented challenges during the pandemic. Campus flyers could not be used as students were not physically present on campus. It became more difficult to develop connections with student organizations and campus offices due to competing priorities and difficulties communicating with these organizations remotely. While telephone and videoconference-based meetings remained an option, many students and staff found themselves in a state of crisis and were unable to engage in activities that were not immediately essential to their schoolwork or job responsibilities. Because we had a high level of unpaid engagement with students and staff throughout the design phase of this study, we did not consider providing compensation for campus partners to assist with dissemination and implementation efforts. As many students and staff found themselves overwhelmed by their core responsibilities, engagement with our project appeared to lessen out of necessity. Due to challenges with in-person dissemination strategies, we shifted to using digital recruitment methods such as social media. While some students preferred using social media to learn about campus programs, few students reported hearing about the app through social media despite advertising the study on Facebook, Reddit, and Instagram. Alternative dissemination strategies should be explored as many students indicated they did not want to engage with email and social media due to the increased amount of time they spent online during remote learning.

Of students who downloaded the program, some suggested that they used it more often as a result of the pandemic, due to increased stressors or the absence of in-person support. However, many students also reported feeling overwhelmed by their frequent use of technology and felt disinterested in using another online tool. This challenge provides opportunities for digital mental health apps to promote in-person wellness activities, encourage in-person connections and social interactions, and offset some of the challenges associated with feeling isolated in a remote learning environment. Providing coaching support to students using the program may also help motivate students to continue using it, and to help students identify personally relevant parts of the program. During the COVID-19 pandemic, University B successfully expanded their peer support program, and the infrastructure of peer support programs could be used to provide support to program users. Additionally, digital mental health apps could offer night modes or audio options that could provide relief from looking at a screen, while still offering the benefits of an online mental health tool. Digital mental health tools must take into consideration students' use of, and attitudes toward, technology and encourage healthy engagement with online mental health resources, especially in the context of an increasingly online society.

These dissemination challenges caused by university life moving online likely contributed to lower-than-expected uptake of the mobile app. When students discussed their preferences for learning about campus programs, students stressed the importance of personal outreach. Developing digital word-of-mouth strategies may be important as having the information come from a known and trusted source may increase the likelihood that students will respond to study advertisements. Receiving digital health intervention recommendations from known sources has been found to be a helpful recruitment strategy across digital health interventions (Sathian et al., 2020). Peer referrals may be particularly useful as many participants reported recommending the app to their peers and many students said they would feel comfortable sharing information about the app with their classmates. Embedding a referral program within the app which allows students to invite their friends to use the app, could be highly effective for college students who are interested in learning about campus resources from their peers, yet to date no study has examined the effectiveness of an in-app recommendation feature. Encouraging counseling center clinicians to share information about the program with their clients may also increase students' uptake of the app given previous studies suggest that patients are more likely to use mental health apps if they are recommended by their provider (Neary & Schueller, 2018).

As hybrid and remote learning models are likely to continue beyond the COVID-19 pandemic, it may also be beneficial to work with both university staff and students to develop innovative and effective ways to share information about programs available on campus which will enable students to stay involved with campus life while studying remotely. Prior to the pandemic, remote learning was already common in many professional degree programs, and remote students who were not geographically close to their institution typically did not have access to mental health services that were offered exclusively in-person and on-campus. Thus, it is crucial to identify effective dissemination strategies in the context of remote and hybrid learning, as we will likely see significant structural changes to how universities operate going forward. Future research should examine dissemination strategies that are most effective for engaging students in online mental health resources on college campuses.

5. Conclusions

From this trial, we learned that relying on primarily digital communication pathways and uncompensated campus partners to engage college students in an app-based mental health and wellness program was not fruitful. Due to the changes to campus life caused by COVID-19 pandemic, we were unable to meaningfully interpret changes in counseling center utilization. College students and staff were largely overwhelmed by the volume of digital communications throughout the study period, and were not interested or able to engage with the study. It appeared that the heavy reliance on virtual services, combined with social isolation of college life, made student reluctant to use another virtual tool. While this implementation of the IntelliCare for College Students program faced substantial challenges in recruiting and retaining students, there is an ongoing need to identify ways to reach college students and support student mental health and wellness for the remainder of the COVID-19 pandemic and beyond. Over the past ten years, student utilization of campus mental health resources has been rising, highlighting an increase in demand for student mental health support (Lipson et al., 2019). Despite this growing need for mental health services for college students, students often face barriers such as long waitlists and inaccessible locations when accessing traditional face-to-face campus counseling services (Cohen et al., 2020). Online mental health resources can help to reduce some of these barriers and provide college students with an accessible alternative to traditional, face-to-face mental health services. Online mental health tools provide many benefits including increased convenience and flexibility for students, reduced stigma, and fewer logistic barriers to accessing mental health care (Hadler et al., 2021). Thus, mental health apps can provide students with additional tools to support their mental health and wellbeing that are easily accessible and offset some of the barriers associated with in-person services. Future work should investigate how to best promote the use of mental health apps on college campuses given the widespread need for student mental health support.

The following are the supplementary data related to this article.

Supplemental File 1

IntelliCare apps.

Supplemental File 2

Recruitment advertisements.

Supplemental File 3

Assessment results.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: DM has accepted honoraria and consulting fees from Apple, Inc., Otsuka Pharmaceuticals, and the One Mind Foundation, royalties from Oxford Press, and has an ownership interest in Adaptive Health, Inc. which has a license from Northwestern University to commercialize IntelliCare. EL has received consulting fees from Actualize Therapy, LLC (last in June 2018), a company that formerly held the license to IntelliCare, and has received honoraria from Streamline Healthcare Solutions, LLC.