Feasibility of a Prototype Web-Based Acceptance and Commitment Therapy Prevention Program for College Students

Abstract

Objective

This study examined the feasibility of a prototype web-based Acceptance and Commitment Therapy (ACT) program for preventing mental health problems among college students.

Participants

Undergraduate first-year students (n = 76) participated between May and November 2011.

Methods

Participants were randomized to ACT or a waitlist with assessments conducted at baseline, post and 3-week follow-up. Waitlist participants accessed the program after the second assessment.

Results

Program usability/usage data indicated high program acceptability. Significant improvements were found for ACT knowledge, education values and depression with ACT relative to waitlist. Subgroup analyses indicated ACT decreased depression and anxiety relative to waitlist among students with at least minimal distress. Within the ACT condition, significant improvements were observed from baseline to 3-week follow-up on all outcome and process measures.

Conclusions

Results provide preliminary support for the feasibility of a web-based ACT prevention program.

Mental health problems are highly prevalent in college with almost 50% of college students having a diagnosable psychiatric disorder (e.g., depressive, anxiety, substance use, eating disorders) within a given year.¹ Nearly one-fifth of college students have suffered from depression in their lifetime and one-eighth from anxiety disorders in the last 12 months.² Severe drinking problems are also common, with over 45% engaging in binge drinking³ and 20% experiencing an alcohol use disorder.¹ Approximately 2 to 4% of young adult females acquire an eating disorder, with the peak age of onset between 16 and 20 years of age.^4,5 Furthermore, suicide is a leading cause of death among college students.⁷ Given the large numbers of students who experience psychiatric disorders, the demand for counseling services on college campuses is high, which has created a challenge for campuses to meet the needs of the student body.^8,9

An infrastructure exists in higher education to provide preventive and ameliorative programming through student orientation and success programs. Effective prevention programs exist for eating disorders,¹⁰ harmful alcohol use,¹¹ and depression and anxiety disorders.¹² Yet, with the exception of alcohol abuse prevention,¹³ the penetration of such prevention programs into the college campus remains generally limited. Part of the problem may be that the majority of prevention programs are targeted towards specific disorders or a relatively narrow range of problems and require multiple sessions with trained professionals. Implementing a full range of problem-specific programs would be complex, costly, and over-burden students.

One promising approach for overcoming the aforementioned barriers to implementation is to develop web-based transdiagnostic programs that can prevent and ameliorate a range of mental health problems within a single program by targeting shared risk factors.¹⁴ The cost of delivery, burden on staff resources, and training needs would all be reduced at the same time that the capacity to disseminate the program across the student body with greater convenience to participants would be increased. Researchers have begun to develop and evaluate such programs, with promising results.^15,16

A powerful possible target for a transdiagnostic prevention approach is psychological flexibility,^17–19 the ability to be mindful of experiences in the present moment, in an accepting and non-judgmental way, while behaving consistently with one’s values, even when one’s thoughts and feelings oppose taking valued action.¹⁹ There is a substantial literature base indicating psychological flexibility as a common protective factor in the development and exacerbation of a variety of psychological problems including depression, anxiety, substance abuse, eating disorders, stress, adjustment to traumatic experiences, adjustment to chronic medical conditions, school performance, employment performance, and burnout.^18–20 A key process that impacts psychological flexibility is experiential avoidance, the unwillingness to experience difficult emotions, thoughts, or sensations, even when avoiding these experiences is harmful. In the absence of stress, people prone to avoiding unpleasant thoughts and feelings may have few problems. However, when encountering stress (such as developmental, social, financial, or academic stressors), they may lock into self-amplifying efforts to suppress unwanted thoughts and feelings, thus exacerbating their problems.¹⁸

Psychological flexibility/experiential avoidance can be targeted through acceptance and mindfulness-based interventions such as Acceptance and Commitment Therapy (ACT)¹⁹ to produce improvements across a range of psychological problems. ACT is a form of cognitive behavior therapy that applies acceptance, mindfulness, and behavior change processes to develop psychological flexibility and reduce experiential avoidance. Medium to large effect sizes have been found in randomized controlled trials evaluating the effectiveness of ACT for depression, anxiety, substance abuse, eating disorders, suicidality, self-harm, psychosis, smoking, prejudice, and worksite stress and burnout²² (for meta-analyses see^{20, 23, 24}). In addition, several studies have shown that the effect of ACT on clinical outcomes is accounted for by its impact on psychological flexibility/experiential avoidance.^20,22

These findings raise the question of whether ACT could similarly be applied to prevent mental health problems by targeting psychological flexibility/experiential avoidance in non-clinical samples.¹⁸ A series of studies have found that ACT can improve mental health and positive functioning in the context of worksite wellness programs^25,26 and two recent randomized trials have found that ACT can reduce depression and anxiety symptoms among at-risk individuals with mild to moderate depression.^27,28 Of particular relevance, a recent study found that an ACT self-help book produced improvements in mental health as well as reduced the incidence of clinical levels of distress at 2-month follow up relative to a waitlist condition among native Japanese students attending higher education in the US. Thus, ACT appears to be effective with non-clinical samples and can potentially reduce the onset of psychological problems.²⁹

Within the college setting, a prevention program might be best launched with college freshmen. The first year in college is fraught with transitions and new stressors.³⁰ In addition, many freshmen dropout for emotional/psychological reasons.³¹ Due in part to these issues, almost all higher education institutions have existing mechanisms to help freshmen with the transition to college, which can provide a framework for implementing a new prevention program.³²

The current study sought to develop and test a prototype web-based program called ACT on College Life (ACT-CL), which is designed to target psychological flexibility to prevent the onset and exacerbation of a broad range of psychological disorders. The prototype program was evaluated in a small randomized controlled trial comparing ACT-CL to waitlist among a general sample of 18 to 20 year old first year college students on measures of psychological distress, psychological flexibility, personal values, ACT knowledge and program usability. Development of a full ACT-CL program may be warranted if the ACT-CL prototype demonstrates feasibility, acceptability, and the potential for efficacy.

METHODS

Participants

The sample consisted of 76 first-year undergraduate students who were 18 to 20 years of age and recruited from the Reno, NV area. The sample was 53.9% female with a mean age of 18.37 (SD = .54, Mode = 18). The sample was 71.1% White, 9.2% Asian, 9.2% American Indian/Alaska Native, 7.9% Black, and 2.6% Native Hawaiian or Other Pacific Islander; 15.8% of the sample described themselves as Hispanic/Latino. In terms of school attended, 86.8% attended the local university, 9.2% attended the local community college, and 3.9% attended a university or 4-year college outside the State.

Procedure

Participants were recruited through flyers posted on two local university and community college campuses as well as advertisements in the local school newspapers and school websites. After providing informed consent, participants completed a web-based survey assessing psychological distress, psychological flexibility, personal values and ACT knowledge. Informed consent and the first assessment were completed in person at the study offices, but all subsequent participation was completed remotely by participants through secure websites. Participants were then randomized to the ACT-CL (n = 37) or waitlist condition (n = 39). Participants in the ACT-CL condition were given 3 weeks to complete the web-based program and those in the waitlist were asked to simply wait 3 weeks before completing the next assessment. All participants completed a second web-based survey 3 weeks after baseline. After completing this second survey, participants in the waitlist condition were given access to the ACT-CL program, followed by a third web-based post survey 3 weeks later. Within the ACT-CL condition, a third follow up survey was provided 3 weeks after completing the post survey (6 weeks after the baseline survey). All participants received $60 at the end of the study (independent of whether they completed all of the study procedures). Ethical approval for the study was provided by a research institute’s Institutional Review Board.

ACT-CL

The prototype ACT-CL program consisted of two web-based multimedia lessons and supplementary tailored emails, which were developed as a “proof of concept.” Program content was adapted from empirically validated ACT treatment and self-help protocols for a variety of mental health problems, including non-clinical worksite and college student wellness programs. The program content, functionality, and look and feel was targeted specifically to incoming college students, based on input from ACT therapists with expertise in college populations as well as formative evaluation procedures conducted with first year college students (i.e., focus groups, usability testing) .

Participants registered for the website with a unique login ID, which was used to coordinate collecting program usage data, tailored program content (i.e., integrating responses to previous exercises into program content), and automated emails. Each lesson was structured in a “tunneled” format such that participants had to complete the program in pre-determined sequence and could not skip around to various pages. Participants were counted as completing the lesson after they viewed the last page. The first lesson targeted values clarification and goal setting (see Table 1). After completing the first lesson participants were instructed to practice the skills they had learned over the next week, during which time they received tailored emails from the program checking in on the goal they set and suggesting additional exercises to work with one’s values (e.g., reflecting on people you admire and what that says about your values). After a week, an automated email informed participants that the next lesson was available. The second lesson targeted acceptance of difficult thoughts and feelings (see Table 1). After completing the second lesson, participants again received tailored emails for the following week checking in on the goal they set and providing additional methods for practicing willingness (e.g., using the NAME technique: Noticing, Acknowledging, Making room for, and Expanding, to work with difficult internal barriers).

Table 1.

ACT-CL Lesson Content Outline

Lesson 1: Exploring your values
Component	Example Content
Defining what values are	Values as a direction animation; Quiz
Clarifying one’s values	Values card sorting exercise
Reflecting on one’s values	Journaling exercise; “Writing your headline” exercise
Defining effective and values-based goals	Reading about SMART and BRAVE goals; Quiz
Goal setting	Goal setting worksheet
Wrap up	Going further content (e.g., FAQ, other media); Lesson summary
Lesson 2: Dealing with barriers
Component	Example Content
Review of last session	Session review animation; Homework check-in
Exploring internal barriers (i.e., thoughts, feelings, urges) to values-based actions	Values bulls eye exercise; Internal barriers checklist
The problem with control strategies	Passengers on the bus animation; Pain vs. suffering exercise
Defining and practicing willingness	And vs. but exercise; Leaning in animation; Quiz; Breath holding to practice willingness
Linking willingness to values-based actions	Exploring reasons for choosing willingness; Setting a goal related to willingness
Wrap up	Going further content (e.g., FAQ, other media); Lesson summary

The web-based multimedia lessons were designed to provide program content through a variety of high quality presentation formats including animations, audio narration, text and graphic elements (see Figure 1 and 2 for example screenshots). For example, in lesson 2 users were provided an animation depicting the “passengers on the bus” metaphor, which is often used in ACT to discuss experiential avoidance, acceptance and values.²⁰ Lessons were also designed to be highly interactive through a combination of worksheets to apply program content to one’s life, interactive metaphors, experiential exercises, and interactive assessments to help refresh and reinforce important information. These multimedia and interactive components were integrated throughout each lesson to maximize efficacy and engagement (e.g., short didactic sections followed by examples and interactive exercises).

Figure 1.

ACT-CL screenshot from a lesson animation

Figure 2.

ACT-CL screenshot from an interactive exercise

Measures

The current pilot study sought to examine the initial feasibility and acceptability of the ACT-CL prototype to determine whether further development of the full program is warranted. Given the prototype nature of the intervention, this Phase I feasibility trial assessed the impact of the ACT-CL on proximal outcomes: program usage and usability, ACT knowledge, stress, depression, anxiety, psychological flexibility, and values. Evaluating the efficacy of ACT-CL in preventing the onset of mental health problems in the long-term will be conducted when the full program is developed.

Program Usage

The ACT-CL program automatically collected data on participants’ use of the program including number of pages viewed, time spent on pages, number of logins, and specific responses to exercises. This data was used to examine program engagement and usage across the two lessons.

System Usability Scale (SUS)

³³ The SUS is a widely used 10-item scale designed to assess program usability and acceptability. Each item is rated on a 5-point scale ranging from 1 (“strongly disagree”) to 5 (“strongly agree”) with a total score range of 20 to 100. Research summarized across 206 studies indicates that the SUS items load onto a single latent factor, have a high level of internal consistency (Cronbach’s alpha = .91), and that the measure can be used to distinguish between more and less usable program.³⁴ Empirically derived cutoff scores have been established for the SUS to identify below and above average program usability ratings.³⁵ The Cronbach’s alpha for the current sample was .76.

ACT Knowledge Questionnaire

A 16-item ACT knowledge questionnaire was used to assess changes in participants’ understanding of ACT core concepts before and after completing the ACT-CL program. The questionnaire involves a series of multiple choice and true/false questions related to ACT concepts presented in the two multimedia lessons (e.g., “True/False: Values are specific goals and outcomes one hopes to achieve”). The total score ranged from 0 to 16. The ACT knowledge questionnaire was based on existing knowledge questionnaires used in ACT bibliotherapy studies.²⁹ The 3 week test-retest reliability for the knowledge test within the waitlist condition was r (36) = .64, p < .001.

Depression, Anxiety and Stress Scale (DASS)

³⁶ The 21-item version of the DASS was used to assess symptoms of depression, anxiety and stress. The DASS is composed of 3 distinct, 7-item subscales assessing depression, anxiety and stress independently. Previous studies have found that these subscales do not load onto a single latent variable and thus cannot be calculated as an overall total score.³⁶ Participants are asked how much each statement applied to them over the past week on a 4-point scale ranging from 0 “did not apply to me at all” to 3 “applied to me very much, or most of the time” with a total score range for each subscale of 0 to 42. The DASS has been found to have good reliability and validity in past research.^36,37 Past research with college students indicates adequate internal consistency with Cronbach’s alpha of .87, .79 and .84 for the depression, anxiety and stress subscales.³⁸ Studies have found that the DASS subscales accurately distinguish depressed and anxious clinical presentations.³⁷ The Cronbach’s alpha in the current sample was .81, .70, and .75 for the depression, anxiety and stress subscales, respectively.

Acceptance and Action Questionnaire-II (AAQ-II)

²¹ The AAQ-II is a 7-item measure of psychological flexibility and experiential avoidance. Participants rate each item on a 7-point scale ranging from 1 (“never true”) to 7 (“always true”) with a total score range of 7 to 49. The AAQ-II has been found to have adequate reliability and validity in past research, including in non-clinical college student samples, with a Cronbach’s alpha of .84 and a test-retest reliability of .81 at 3 month follow up.²¹ Past research has also indicated adequate discriminant and convergent validity with the AAQ-II on measures of psychological distress, functioning, and related variables (i.e., thought suppression).²¹ The Cronbach’s alpha for the current sample was .84.

Personal Values Questionnaire (PVQ)

³⁹ The 10-item relationship and education subscales of the PVQ were used to assess various aspects of values. Participants are asked to provide a brief narrative describing their values within each subscale domain (relationships and education). A series of 3 questions assess intrinsic/positive motivations for this value on a 5-point scale ranging from 1 “Not at all for this reason” to 5 “Entirely for this reason” with mean sum scores ranging between 1 and 5. Intrinsic/positive reasons include valuing education or relationships because they lend meaning to the students’ lives or make their lives richer and more interesting. Participants are also asked to rate success in behaving consistently with their values over the past 3 weeks. The 3 week test-retest reliability within the waitlist condition for the values variables used in the analyses were as follows: intrinsic/positive motivation for relationship values, r (36) = .55, p < .001, intrinsic/positive motivation for education values, r (36) = .57, p < .001, relationship values success, r (36) = .45, p < .001, education values success, r (36) = .73, p < .001.

Data Analysis Plan

Analyses were conducted with outcome, process, and program usage variables to examine the acceptability and feasibility of the ACT-CL program. Descriptive statistics were calculated to examine program usability and usage data. Mixed-model repeated measures (MMRM) analyses examined differences between ACT-CL and waitlist on outcome and process measures from pre to post. Hierarchical linear modeling (HLM) analyses were conducted to examine within group changes in the ACT-CL condition on outcome and process variables from pre to post to follow up. HLM analyses also examined within group changes in the waitlist condition before and after using the ACT-CL program (assessment point 2 and 3). In cases where the HLM analysis failed to converge, MMRM analyses were used to examine within group changes over time in ACT-CL. Unstructured covariance models were used in each MMRM and HLM analysis.

There was some attrition across assessment phases. Two participants (one from each condition) dropped out of the study after completing the baseline survey, reportedly because they were too busy with school. An additional participant in the ACT-CL condition dropped out before completing the third web-based survey also citing time constraints. An intent-to-treat analytic approach was taken such that study dropouts were included in all analyses. MMRM and HLM provide a powerful method for conducting analyses with the full intent-to-treat sample as they use all available data and can model change even with missing data. Outlier analyses identified an outlier in the ACT-CL condition who reported experiencing a major non-research related adverse event (accidental death of a close family member) during the program-testing phase. This participant was excluded from all reported analyses. Of note, 92% of participants in the ACT-CL condition completed the entire ACT-CL program and almost all of the participants who did not finish the program still completed the post and follow up assessments.

Interventions focusing on preventing the exacerbation of mental health symptoms through targeted or indicated prevention efforts have yielded higher effect sizes than universal interventions.⁴⁰ Likewise, recent studies have found that in non-clinical and subthreshold samples, ACT may have a greater impact on psychological distress among individuals with elevated symptoms.^26,28 Subgroup analyses using MMRM were thus conducted to explore pre to post intervention effects on measures of distress separately among students who were or were not currently experiencing at least minimal symptoms of depression, anxiety and/or stress based on recommended DASS cutoff scores (DASS depression ≥ 10 or DASS anxiety ≥ 8 or DASS stress ≥ 15).³⁶

Pilot studies such as the current one tend to be underpowered because resources and time have to be distributed between program development and conducting a feasibility trial to determine “proof of concept.” In order to ensure adequate development time and resources for developing the web-based program in the current study, the recruitment time window and sample size was purposefully limited. To adjust for these factors, and given the goals for examining initial feasibility of the program, significance was determined at a value of p ≤ .10. This provided adequate power of .80 to detect a medium effect size of Cohen’s d = .60 with a sample size of 35 participants per group using a two tailed test and alpha of p ≤ .10.

RESULTS

Preliminary Analyses

Prior to analysis, frequency distributions and plots were examined for unusual data distributions or data points. Square root transformations were employed for DASS depression, anxiety and stress variables at each time point to approximate a normal distribution. No significant between group differences were found at baseline on key demographic, outcome, or process variables using independent samples t-tests (p > .10). Table 2 displays descriptive statistics on outcome and process variables by time point and condition.

Table 2.

Descriptive statistics for each variable and time point by condition

	Waitlist (n = 39)			ACT-CL (n = 37)
Variable	Baseline T1 M (SD)	Baseline T2 M (SD)	Post T3 M (SD)	Baseline T1 M (SD)	Post T2 M (SD)	Follow Up T3 M (SD)
ACT Knowledge *, +	6.60 (2.53)	6.63 (2.42)	10.18 (3.15)	7.35 (2.48)	11.11 (3.23)	10.87 (3.09)
Depression *, +	5.64 (6.76)	7.04 (8.42)	6.15 (7.23)	5.56 (5.56)	4.80 (6.57)	2.82 (4.85)
Anxiety+	6.00 (6.26)	5.11 (7.12)	5.19 (7.06)	5.17 (5.16)	2.86 (3.51)	2.53 (4.76)
Stress+	10.26 (7.37)	9.04 (7.83)	8.70 (6.52)	7.78 (6.30)	6.26 (5.89)	4.53 (6.06)
Experiential Avoidance+	17.50 (7.11)	17.28 (6.94)	16.91 (6.08)	15.56 (5.48)	14.74 (5.43)	13.96 (6.13)
Education Positive Motivat.*, +	4.41 (.57)	4.48 (.58)	4.36 (.72)	4.26 (.54)	4.59 (.41)	4.46 (.57)
Education Success*, +	4.36 (.87)	4.34 (.91)	4.27 (.80)	4.06 (.75)	4.40 (.69)	4.38 (.78)
Relationship Positive Motivat.+	4.44 (.50)	4.55 (.59)	4.54 (.51)	4.44 (.44)	4.57 (.42)	4.62 (.45)
Relationship. Success+	4.31 (.77)	4.46 (.72)	4.65 (.63)	4.42 (.73)	4.66 (.48)	4.68 (.53)

NOTES:

^*Indicates significant (p < .10) between group effects from pre to post;

⁺Indicates significant (p < .10) within-group effects in the ACT-CL condition from pre to follow-up.

The range of possible values for each variable are as follows: ACT Knowledge 0 – 16; Depression 0 – 42; Anxiety 0 – 42; Stress 0 – 42; Experiential Avoidance 7 – 49; Values Positive Motivation 1 – 5; Values Success 1 – 5

Program Usage and Usability

There were no significant differences on program usage and usability variables between ACT-CL and waitlist participants who used the program (p > .10). Thus, program usage data was summarized across both the ACT-CL condition and waitlist condition (after using the ACT-CL program). The vast majority of students (92%) completed both lessons. Participants generally spent an adequate amount of time in the program (M = 81.98 minutes, SD = 22.68, Median = 77.57). The majority logged onto the program twice (58.1%), with most others logging in 3 (21.6%) or 4 to 5 (16.3%) times. Regarding the tailored emails, 85.3% reported reading the emails and 69.0% of those who read the emails reported engaging in the suggested exercises. Although participants took 3.89 days on average to complete each lesson after it was made available (SD = 2.57), 85% completed both lessons within the designated 3-week intervention period.

The mean usability rating of ACT-CL on the SUS combining both ACT-CL and waitlist participants was 84.55 (SD = 10.91). This is above the cutoff for an “A” score (80.3), which represents the top 10% of program scores and those programs which people are likely to recommend to their friends.³⁵ Participants appeared to be split when asked “What was the most important thing you learned from this program?” with 54% describing content from the values lesson and 46% from the acceptance lesson.

Between Group Analyses

Significant time by condition interactions were observed with MMRM on depression, F(1, 72.23) = 2.87, p = .095, Cohen’s d =.40), ACT knowledge, F(1, 71.85) = 38.85, p < .001, Cohen’s d = 1.47, education values success, F(1, 71.94) = 5.31, p = .024, Cohen’s d = .54, and intrinsic/positive motivation for education, F(1, 72.01) = 4.62, p = .035, Cohen’s d = .51. As can be seen in Table 2, participants in ACT-CL improved more on each of these measures relative to waitlist from pre to post. No other significant between-group effects over time were observed.

Subgroup analyses with distressed students (i.e., at least minimal symptoms of depression, anxiety and/or stress based on recommended DASS cutoff scores; depression ≥ 10 or anxiety ≥ 8 or stress ≥ 15)³⁶ indicated significant time by condition interactions with greater reductions in anxiety, F(1, 30) = 4.98, p = .033, Cohen’s d = .81, and depression, F(1, 30) = 6.23, p = .018, Cohen’s d = .91, in the ACT-CL condition relative to waitlist. No significant between group differences were observed on depression or anxiety among nondistressed students (p > .10). There were also no significant between group differences for stress in either subgroup.

Within Group Analyses

HLM analyses were conducted to examine pre to follow up changes in the ACT-CL condition. Significant improvements (p < .10) were observed from pre to follow up on each outcome and process measure (see Table 3). Effect sizes for within group changes ranged from Cohen’s d of .50 to .97. HLM failed to converge for ACT knowledge, but MMRM analysis found a significant improvement over time on ACT knowledge, F(2, 33.28) = 29.56, p < .001, Cohen’s d = 1.88. In each case, results indicated continuing improvement on outcome and process measures over time in the ACT-CL condition.

Table 3.

HLM results for pre to follow up changes within the ACT-CL condition

Variable	Slope Estimate	SE	t	p	Cohen’s d
Depression	−.47	.13	−3.68	.001	.97
Anxiety	−.41	.13	−3.16	.003	.95
Stress	−.48	.14	−3.51	.001	.81
Experiential Avoidance	−.85	.46	−1.85	.073	.52
Education Positive Motivat.	.10	.06	1.79	.085	.50
Education Success	.16	.07	2.23	.033	.92
Relationship Positive Motivat.	.08	.04	2.00	.054	.74
Relationship Success	.13	.06	2.10	.043	.78

HLM analyses were also conducted to examine changes on process and outcome measures within the waitlist condition before and after using the ACT-CL website (assessment point 2 and 3). There was a significant improvement in ACT knowledge from pre to post in the waitlist condition, slope estimate = −3.52, SE = .43, t(36.14) = −8.20, p < .001, Cohen’s d = 1.12. There were no other significant changes after using the ACT-CL program in the waitlist condition (p > .10).

COMMENT

The current study sought to evaluate a prototype version of a web-based transdiagnostic prevention program based on ACT. Two modules targeting values and acceptance were tested. However, the full ACT model is composed of considerably more elements, including a focus on mindfulness, contact with the present moment, perspective taking, cognitive defusion (learning to notice a thought as a thought), and committed action. Thus, this trial was designed to be a proof of concept, to evaluate whether the cost and effort of developing a full program is warranted. As a methodological reflection of that purpose, follow up was short, no follow up was collected in the control group, and there was a focus on proximal programmatic outcomes.

Results suggest that the program may be acceptable to, and impactful with, college freshmen. There was high program utilization and system usability with the prototype program, attesting to the potential feasibility and acceptability of this approach with first year college students. Preliminary results also suggest the program may have an impact on at least some mental health problems given the between group differences on depression symptoms as well as large effect sizes for ACT-CL on depression and anxiety symptoms among those with at least minimal symptoms. At a process level the program appeared to successfully impart ACT concepts, as illustrated by significant increases in ACT knowledge at post relative to waitlist. Although the program did not target academics per se, it impacted students’ attitudes towards school: There were significant between group differences in both intrinsic/positive motivation and success in living out educational values. The within group results in the ACT-CL condition were positive with medium to large effect sizes for all outcome and processes tested showing continued improvement over the three-week follow up. However, within group findings were not replicated in the waitlist condition after using the website, with only ACT knowledge improving significantly from pre to post.

The effects of ACT-CL on education values may be important for dissemination. There are a growing number of studies indicating ACT can improve behaviors relevant to educational outcomes. A recent study with test anxiety found that ACT did as well as traditional cognitive behavioral methods in reducing anxiety but was superior in improving academic test performance.⁴¹ ACT has also been shown to improve innovation at work,²⁵ persistence in difficult tasks,⁴² social functioning,⁴³ and exercise.⁴⁴ Even relatively small changes in positive motivation and perceived success in educational values could have a substantial impact when such a program is implemented at a universal level across the entire student body. Although universal approaches tend to have smaller effect sizes,⁴⁰ the increased reach provided by such an approach can lead to a much greater public health impact across a student body. Given the emphasis at most universities on student retention and academic success, the ability to improve academic functioning may be an important institutional outcome above and beyond mental health prevention and amelioration. These positive behavioral outcomes may also serve to enhance “buy in” from students who feel that they do not need mental health prevention programming per se, but would be interested in improving functioning in areas such as school, relationships, and physical health.

The lack of condition effects on psychological flexibility/experiential avoidance at post raises a concern given extensive research showing that ACT significantly impacts this process.²¹ This may be attributable to the limited treatment components included in the prototype program. Cognitive patterns are known to become overextended in ways that elicit and support experiential avoidance (i.e., rumination, worry, problem solving), and cognitive defusion interventions in ACT are particularly crucial to increasing psychological flexibility (for a recent review of laboratory tests of cognitive defusion see Levin et al.⁴²). The observed trends for increases in psychological flexibility from baseline to 3-week follow up suggest ACT-CL, even in its prototype form, may have some impact on this process over time. It would be important for future research to evaluate the effects of ACT-CL on psychological flexibility with longer between-group follow up assessments and with the more complete intervention package as a comprehensive ACT program may have a greater impact. It may be the case however that a longer program with more modules will result in poor compliance. As the program continues to be developed, further research will be needed to examine how modules can be added and expanded on while maintaining user engagement and program efficacy.

It is unclear why findings did not replicate in the waitlist condition after participants completed the ACT-CL program. Potentially participants may have been less engaged in using the website after having already waited 3 weeks in the study, losing the motivation and time that may have been more readily present if they had started the program sooner. Features of the design may have inflated program engagement for ACT-CL participants (i.e., meeting with investigators in-person when randomized and being introduced to the program during this meeting) and it will be important to examine whether users engage in the program without these features. However, this explanation is unlikely to fully account for the findings given the lack of differences on program usage and usability ratings between waitlist and ACT-CL participants. Another explanation is that the within group data for the ACT-CL condition included 3 time points compared to only 2 time points in the waitlist condition. The longitudinal effects in the ACT-CL condition suggests that outcome and process measures improve more with time, potentially due to continuing to practice the skills taught in the program. If waitlist participants had completed another assessment 3 weeks after using the website, the results may have replicated across more outcome and process measures. Lastly, this may indicate simply a failure to replicate results, which is particularly a concern given the small sample size and less stringent significance cutoff (p < .10). Thus, further research is needed to determine whether positive findings with ACT-CL replicate in a larger sample with adequate statistical power as well as to explore the relationship of program engagement to outcomes.

Limitations

Further development and research will be needed to address the limitations of the present pilot study. The program itself consists at this point of only two modules and the relevance of the findings to college students in general is unclear given the small sample size, inclusion/exclusion criteria, and limited geographic focus of the study. Although a portion of students participated from other schools, including a local community college, the sample size was too small to determine if there were differences across types of institutions. It is worth noting that two of the four study dropouts were from the community college, suggesting the program may be less engaging for these students. Additional unmeasured variables that covary with institution type such as having a full-time job may also have impacted engagement. In addition, it is unclear whether students’ engagement in the program was influenced by the monetary compensation for participating in the study and it would be important to conduct further research without such compensation. This is an important issue, particularly in a prevention context, and some form of contingencies may be needed to ensure student engagement (i.e., course credit, mandatory participation, raffles). Yet this also needs to be balanced with what is cost effective for universities to implement at a large scale.

Due to the pilot nature of the study a less conservative value was used for significance testing (p < .10) and additional research is needed with a larger sample size to determine if findings replicate. This is particularly important given the lack of improvements observed in the waitlist condition after viewing the website. The pilot study was not designed to determine preventative effects per se and focused on a more limited range of mental health problems than befits the broader transdiagnostic goals of the program. Future research is needed to determine the impact of this program in preventing the broad range of mental health problems it is designed to target.

Conclusions

Overall, the present study suggests that even a limited version of ACT targeting values and acceptance can be used in a web-based format with college students to address issues including mental health and educational domains. A full program, with multiple modules targeting all aspects of ACT and supported by mobile applications and email features, holds out promise for a transdiagnostic program that could meet the needs of college students and institutions of higher education alike.