Internet-delivered cognitive behavior therapy versus treatment as usual for anxiety and depression among Latin American university students: A randomized clinical trial

Abstract

Objective:

Untreated mental disorders are important among low- and middle-income country (LMIC) university students in Latin America, where barriers to treatment are high. Scalable interventions are needed. This study compared transdiagnostic self-guided and guided internet-delivered cognitive behavioral therapy (i-CBT) with treatment as usual (TAU) for clinically significant anxiety and depression among undergraduates in Colombia and Mexico.

Method:

1,319 anxious (GAD-7=10+) and/or depressed (PHQ-9=10+) undergraduates (mean [SD] age = 21.4 [3.2]); 78.7% female; 55.9% first-generation university student) from seven universities in Colombia and Mexico were randomized to culturally adapted versions of self-guided i-CBT (n=439), guided i-CBT (n=445), or treatment-as-usual (TAU; n=435). All randomized participants were reassessed 3 months after randomization. The primary outcome was remission of both anxiety (GAD-7=0–4) and depression (PHQ-9=0–4). We hypothesized that remission would be higher with guided i-CBT than the other interventions.

Results:

Intent-to-treat analysis found significantly higher adjusted (for university and loss to follow-up) remission rates (ARD) among participants randomized to guided i-CBT than either self-guided i-CBT (ARD=13.1%, χ²₁=10.4, p=.001) or TAU (ARD=11.2%, χ²₁=8.4, p=.004), but no significant difference between self-guided i-CBT and TAU (ARD=−1.9%, χ²₁=0.2, p=.63). Per-protocol sensitivity analyses and analyses of dimensional outcomes yielded similar results.

Conclusions:

Significant reductions in anxiety and depression among LMIC university students could be achieved with guided i-CBT, although further research is needed to determine which students would most likely benefit from this intervention.

Introduction

In low-and-middle-income countries (LMICs) such as Colombia and Mexico, there is significant unmet need for treatment of mental disorders among university students due to a high prevalence of clinically significant anxiety and depression coupled with low rates of treatment (Benjet et al., 2019; Ochnik et al., 2021). Only about 6.7–11.4% of LMIC college students (including those in Colombia and Mexico) receive minimally adequate treatment, compared to roughly 23.1% of college students in high-income countries in other geographical regions (e.g., North America, Europe; Auerbach et al., 2016). Many university students in Latin America are the first in their families to attend higher education, resulting in significant financial and adjustment burdens that can impact mental health (Covarrubias et al., 2015; Tello & Lonn, 2017). Common barriers to accessing treatment among Latin American college students include attitudinal barriers such as lack of recognition of treatment need, stigma, wanting to handle the problem on one’s own, preference for family support to professional support, and structural barriers such as cost and limited access to treatment (Benjet et al., 2020; Orozco et al., 2022).

Digital or internet-delivered mental health interventions (i-interventions), such as internet-delivered cognitive-behavioral therapy (i-CBT), could address many of these barriers given their low cost and logistical ease of delivery and compatibility with attitudes of self-reliance, reducing barriers due to stigma and shame (Benjet et al., 2020; Ralston et al., 2019; Ramos & Chavira, 2022). I-interventions deliver content using computer programs that either use no human guidance (self-guided i-interventions) or minimize the need for human guidance (guided i-interventions). We do not include in this definition traditional psychotherapy delivered through video conferencing or telemedicine. Guided i-CBT is typically more effective than self-guided i-CBT (Baumeister et al., 2014), partly due to better accountability, higher adherence, and lower attrition (Karyotaki et al., 2021). Guided and self-guided i-CBTs are often as effective as or more effective than treatment-as-usual (TAU) and more effective than waitlist controls (Karyotaki et al., 2021; Karyotaki et al., 2017).

College students are in some ways an ideal population for i-interventions, given that they are required to have some degree of internet access and literacy to complete their coursework. A study in a public university in Mexico, for example, found that virtually all students used the internet, with 90% accessing it in their homes and the rest either at the university, in public spaces, or in the homes of others (Morales Ramírez et al., 2020). Additionally, the COVID-19 pandemic, with the need it created for social distancing and the evidence obtained during the pandemic of substantial adverse effects on university student mental health (Wood et al., 2022), may have made i-interventions even more acceptable to students than previously (Gruber et al., 2021). However, there remains a paucity of comparative effectiveness research on i-CBTs among LMIC college students in Latin America relative to their counterparts in high-income countries (Escobar-Viera et al., 2021; Jiménez-Molina et al., 2019). Further, current literature on the potential advantages of culturally adapting i-interventions is mixed (Ellis et al., 2022; Ramos & Chavira, 2022; Spanhel et al., 2021).

Previous research (mostly in high-income countries) has found moderate-to-large treatment effects of i-interventions in reducing anxiety (Eilert et al., 2020) and depression (Wright et al., 2019) in general population samples and small-to-moderate effects among university students (Harrer et al., 2019). However, despite comorbidity being the rule rather than the exception among anxious and depressed university students (Auerbach et al., 2019) and comorbid anxiety-depression being more severe and persistent than either disorder alone (Hofmeijer-Sevink et al., 2012), many i-interventions fail to consider comorbidity. A meta-analysis found medium-to-large effect sizes of transdiagnostic/tailored internet-delivered cognitive behavioral therapy (i-CBT) in treating comorbid anxiety and depression and stronger effects on depression than disorder-specific interventions (Păsărelu et al., 2017). However, little is known about whether transdiagnostic i-CBT is effective among university students in LMICs (Fu et al., 2020).

This evidence gap is of special importance given that Latin American university students report high rates of stress and greater needs for services than students in high-income countries due to the large proportion of first-generation students with substantial economic burdens (Schendel & McCowan, 2016). Despite high anxiety and depression prevalence among LMIC university students, but campus counseling services, when they exist, are typically under-resourced (Evans-Lacko & Thornicroft, 2019).

Best practice recommendations suggest using a stepped-care approach to treat anxiety and depression in under-resourced settings (Bower & Gilbody, 2005). I-interventions are ideal in this regard because they include both low-intensity self-guided options and higher-intensity guided options that typically require only 10–15 minutes of a BA-level coach per participant per week to motivate the student to continue using the program and to provide feedback. Their scalability and cost-effectiveness make it important to compare self-guided and guided i-CBT to treatment as usual (TAU). Based on these considerations, we conducted a three-arm randomized equal allocation controlled pragmatic trial comparing self-guided and guided transdiagnostic i-CBT to TAU among students with clinically significant anxiety and/or depression at universities in Colombia and Mexico.

The decision to use TAU as the control condition rather than the more typical waiting list or active control conditions (e.g., mood monitoring) was made based on the goal of evaluating the extent to which the scalable interventions under study improve on usual practice. In making this decision, we recognized that usual practice differs widely across universities. As a result, we carried out exploratory subgroup analyses separately in universities with and without student mental health clinics and, within the latter universities, in separate subsamples of students recruited from the general student body (in other words, students that were not seeking out help but rather responded to an email invitation or social media campaign) and from clinic waiting lists (i.e., help-seeking students). We also inquired about other treatments used by participants assigned to TAU as well as by those in the i-CBT arms.

Consistent with previous research (Cuijpers et al., 2019; Wright et al., 2019), our preregistered hypothesis (Benjet et al., 2022; ) was that participants randomized to guided i-CBT would have significantly better aggregate outcomes than those randomized to self-guided i-CBT. This has been found not only for remission but also for symptoms of clinical depression (Berger et al., 2011; Cuijpers et al., 2019) and anxiety disorders (Ciuca et al., 2018; Oliveira et al., 2023). However, as our trial combined novel interventions and methods of intervention delivery, we could not draw on prior theory or research to hypothesize whether the aggregate outcomes associated with guided and self-guided i-CBT would be significantly better than those associated with TAU.

Method

Data Transparency and Openness

Study materials and analysis codes are available upon request by contacting the corresponding author with the proposed use of the data (see details in Supplement A). Deidentified participant-level data can be made available with a signed data use agreement by researchers whose proposed use of the data has been approved. We prospectively registered the trial; see clinicaltrials.gov (Identifier: NCT04780542) and reported in the registration how we determined the sample size, data inclusion and exclusion criteria, the manipulations, and the measures. We followed the journal article reporting standards (JARS; Kazak, 2018) in presenting results.

Participants and Procedures

The trial was an assessor-blinded, multisite, randomized clinical trial (RCT). It received ethical approval from the institutional review boards of the Instituto Nacional de Psiquiatría [National Institute of Psychiatry] Ramón de la Fuente Muñiz in Mexico and Harvard University in the US. Participants were n=1,319 undergraduates at seven universities in Colombia and Mexico (combined enrollment of 422,000 students). The recruitment occurred March-October 2021 (Supplement B). Inclusion criteria were at least 18 years of age, at least moderate anxiety and/or depression as operationalized by Generalized Anxiety Disorder-7 (GAD-7; Spitzer et al., 2006) and/or Patient Health Questionnaire-9 (PHQ-9) scores of 10+ (Kroenke et al., 2001) and providing informed consent. Exclusion criteria were positive screens for either lifetime bipolar disorder or lifetime nonaffective psychosis or recent suicidal ideation with intent. Each university had a clinical liaison who was notified immediately for further evaluation of baseline respondents who screened positive for suicidal ideation with intent. Other ineligible respondents were referred to TAU. Table 1 details the baseline characteristics of participants concerning the country of residence, gender, sexual orientation, age, first-generation university student status, and clinically relevant variables.

Table 1.

Selected baseline characteristics of participants

	Total		Guided		Self-Guided		TAU
	n	(%)	n	(%)	n	(%)	n	(%)
I. Country
Colombia	594	(45.0)	203	(45.6)	191	(43.5)	200	(46.0)
Mexico	725	(55.0)	242	(54.4)	248	(56.5)	235	(54.0)
II. Sex
Male	281	(21.3)	93	(20.9)	98	(22.3)	90	(20.7)
Female	1,038	(78.7)	352	(79.1)	341	(77.7)	345	(79.3)
III. Sexual orientation
Heterosexual	916	(69.4)	308	(69.2)	293	(66.7)	315	(72.4)
Gay or lesbian	61	(4.6)	26	(5.8)	17	(3.9)	18	(4.1)
Bisexual	196	(14.9)	55	(12.4)	80	(18.2)	61	(14.0)
Other (asexual, unsure, “other”)	146	(11.1)	56	(12.6)	49	(11.2)	41	(9.4)
IV. Age
18–19	354	(26.8)	124	(27.9)	98	(22.3)	132	(30.3)
20	251	(19.0)	76	(17.1)	105	(23.9)	70	(16.1)
21–22	397	(30.1)	130	(29.2)	138	(31.4)	129	(29.6)
23 or older	317	(24.0)	115	(25.8)	98	(22.3)	104	(23.9)
V. First generation university student
Yes	738	(55.9)	252	(56.6)	240	(54.7)	246	(56.5)
No	581	(44.0)	193	(43.4)	199	(45.3)	189	(43.4)
VI. Did the university have a mental health clinic?
Yes and student recruited from waiting list	290	(22.0)	97	(21.8)	98	(22.3)	95	(21.8)
Yes but student recruited from student body	607	(46.0)	207	(46.5)	200	(45.6)	200	(46.0)
No (all students recruited from student body)	422	(32.0)	141	(31.7)	141	(32.1)	140	(32.2)
VII. Severity of anxiety (GAD-7)a
Severe	494	(37.4)	164	(36.8)	164	(37.4)	166	(38.2)
Moderate	434	(32.9)	149	(33.5)	146	(33.3)	139	(31.9)
Mild or none	391	(29.6)	132	(29.7)	129	(29.4)	130	(29.9)
VIII. Severity of depression (PHQ-9)b
Severe or moderately severe	488	(37.0)	160	(35.9)	159	(36.2)	169	(38.8)
Moderate	702	(53.2)	240	(53.9)	236	(53.8)	226	(51.9)
Mild or none	129	(9.8)	45	(10.1)	44	(10.0)	40	(9.2)
IX. Comorbidity
Severec on both	282	(21.4)	95	(21.3)	95	(21.6)	92	(21.1)
Severec on one and moderate on the other	284	(21.5)	88	(19.8)	88	(20.0)	108	(24.8)
Severec on one and mild-none on the other	134	(10.2)	46	(10.3)	45	(10.2)	43	(9.9)
Moderate on both	233	(17.7)	85	(19.1)	83	(18.9)	65	(14.9)
Moderate on one and mild-none on the other	386	(29.3)	131	(29.4)	128	(29.2)	127	(29.2)
(n)	(1,319)		(445)		(439)		(435)

Note. Abbreviations: Guided=guided internet-delivered cognitive behavioral therapy (i-CBT); Self-guided=self-guided i-CBT; TAU=the participants randomized to treatment as usual; n=number of participants defined by the row headings who were in the total randomized baseline sample or in intervention arms; %=the percent of all randomized baseline respondents in the column who were in the subgroups defined by the row headings; GAD-7=Generalized Anxiety Disorder-7; PHQ-9=Patient Health Questionnaire-9.

^aSevere refers to GAD-7=15+, moderate to GAD-7=10–14, mild or none to GAD-7=0–9.

^bSevere refers to PHQ-9=20+, moderately severe to PHQ-9=15–19, moderate to PHQ-9=10–14, mild or none to PHQ-9=0–9.

^cIncluding either severe or moderately severe PHQ-9.

In conjunction with social media postings, participants were recruited by emails sent to a weekly random sample of approximately 1,000 students in the general student body of six universities. In the two of these six universities that had student mental health clinics, students on clinic waiting lists were also invited to participate while reserving their place on the waiting list. In the seventh university in the study, we were allowed to recruit only from the clinic waiting list, not from the general student body. Each participant completed a 30-minute baseline online self-administered questionnaire (SAQ) to assess current symptoms along with hypothesized risk and protective factors for treatment response. Randomization occurred only after the baseline SAQ was completed. A 3-month post-randomization online SAQ was then sent to all students randomized into the study. Initial follow-up SAQ nonrespondents were sent email reminders followed by WhatsApp/Telegram messages and phone calls to minimize loss to follow-up.

Randomization

In all, n=65,880 students were invited from the general student body and n=678 from clinic waiting lists (Figure 1). Baseline SAQs were completed by n=1,714 of these students. The n=1,319 eligible students who completed baseline SAQs included n=1,029 recruited from the general student body and n=290 recruited from clinic waiting lists. These students were randomized with equal allocation across the 3 intervention arms (Figure 1). The allocation sequence was automatically generated in the online SAQ platform stratifying by the 3-way cross-classification of sex at birth, anxiety symptom severity, and depression symptom severity. The participants were not blinded to their intervention arm (Supplement B). Interventions

Figure 1.

Study recruitment and retention flow chart

i-CBT.

The i-CBT intervention was a culturally adapted version of SilverCloud Health’s transdiagnostic Space from Anxiety and Depression program. This intervention uses the CBT strategies of behavioral activation and cognitive restructuring in 7 core modules (introduction, psychoeducation, mood monitoring, behavioral activation, thought monitoring, challenging negative automatic thoughts, and relapse prevention) and several additional modules (e.g., anger, grief, sleep, reframing core beliefs) designed to be completed in 8 weeks. The i-CBT intervention uses the same techniques as face-to-face CBT, which are communicated to the user via written text, graphics, videos, and interactive tools. Sessions include texts, testimonials, audio, educational video clips, interactive quizzes, exercises, and homework. The i-CBT can be used with or without guidance, has been shown to be more effective than controls in treating anxiety and depression (Enrique et al., 2021; Richards et al., 2020), and has been reported to have high acceptability and satisfaction among university students (Enrique et al., 2019a; Palacios et al., 2018). Guides (with at least undergraduate degrees in psychology trained in the intervention and in how to deliver feedback) in the guided version helped users create a personalized experience and provided weekly feedback via messages, whereas in the self-guided version users received no personalized feedback. All content and exercises were identical in both versions. This i-CBT intervention is considered transdiagnostic because it focuses on targeting both anxiety and depression, either with or without several other comorbid disorders (such as eating disorders, insomnia, and substance use disorders) (Bisby et al., 2023; Norton & Roberge, 2017). Specific details of the i-CBT content are accessible by contacting the developers (https://www.silvercloudhealth.com/). Cultural adaptation consisted of changing the photos, names and some content of the “personal stories” that are woven throughout the program as examples to students. These changes were based on an iterative process of questionnaires and focus groups of Colombian and Mexican students (Supplement B).

Treatment as usual.

TAU consisted of whatever mental health service each university typically provided students. As noted above, three universities had formal mental health clinics, whereas the others had only informal counseling services provided by faculty with referrals to community treatment providers. Because of the pandemic lockdown at the time of randomization, most university services were provided only online via videoconferencing platforms during the study. No constraints were applied concerning TAU because the current RCT focused on evaluating the incremental benefit of i-CBT over routine care.

Outcome Measures

Anxiety and depression.

As noted above, anxiety was assessed with the GAD-7 (Spitzer et al., 2006) and depression with the PHQ-9 (Kroenke et al., 2001). The combination of the two was assessed with a summation of GAD-7 and PHQ-9 scores known as the Patient Health Questionnaire Anxiety and Depression Scale (PHQ-ADS) (Kroenke et al., 2019). These scales are widely used in clinical trials and have excellent psychometric properties (Kroenke et al., 2019). Previous research found high internal consistency (α ≥ 0.82) and strong convergent validity of the GAD-7 in Mexican (Gutiérrez-Velilla et al., 2022) and Colombian (Monterrosa-Blanco et al., 2021) samples. Likewise, the PHQ-9 has shown high internal consistency (α ≥ 0.80) and strong convergent validity in Mexican (Arrieta et al., 2017) and Colombian (Miranda & Scoppetta, 2018) samples. In the current study, Cronbach’s α was 0.86 for the GAD-7, 0.80 for the PHQ-9, and 0.82 for the PHQ-ADS. The primary outcome was 3-month remission (scores of 0–4) on both the GAD-7 and PHQ-9 in the total sample, but we also examined remission in the two component scales as well as mean change in the dimensional 0–21 GAD-7, 0–27 PHQ-9, and 0–48 PHQ-ADS scores as secondary outcomes.

Covariates.

A total of 282 covariates was included in the baseline SAQ based on prior research investigating predictors of anxiety and depression treatment response (Donker et al., 2013; Maj et al., 2020). See details in Supplement C. As described below in the section on Analysis Methods and in more detail in Supplement B, we used these baseline covariates to adjust for nonrandom loss to follow-up.

Power Analysis

As discussed in more detail elsewhere (Luedtke et al., 2019) and in our preregistered protocol (Benjet et al., 2022; clinicaltrials.gov Identifier: NCT04780542), we used simulation to estimate the required sample size for 0.8 power to detect heterogeneity of treatment effects (the ultimate focus of this research) that would result in a 5% higher aggregate remission rate if all participants were optimized across interventions rather than randomized across interventions. This required sample size was approximately 500 participants per intervention arm initiating treatment based on an assumption of mildly informative loss to follow-up comparable in magnitude to the loss rates found in previous i-CBT trials (Torous et al., 2020). Based on this result, we designed our trial to randomize 500 participants per intervention arm. Because the current report focuses on the aggregate treatment effects rather than heterogeneity of treatment effects, we were amply powered.

Analysis Methods

We began by evaluating treatment effects on both joint remission of GAD-7 and PHQ-9 and PHQ-ADS mean scores from an intent-to-treat perspective (Gupta, 2011) in the total sample, adjusting for loss to follow-up. We then conducted a per-protocol sensitivity analysis of these same primary outcomes in the subset of participants randomized to i-CBT who completed 3+ hours online compared to those randomized to TAU. We chose 3+ hours for the per-protocol analyses because 3 hours was the average amount of time Latin American college students spent in a prior positive trial of guided i-CBT for depression (Salamanca-Sanabria et al., 2020). It is also noteworthy that 25 minutes per module has been determined to be a minimally adequate dose in prior research on SilverCloud (Enrique et al., 2019b), which would transate into about 3 hours if 25 minutes were spent on each of the 7 core modules. Figure 1 shows the breakdown of the students invited to participate who did versus did not complete the SAQs, the breakdown of exclusion criteria for those who completed the SAQ but were not randomized, the distribution of follow-up SAQ completion among those randomized by arm, and the distribution of starting and completing 3+ hours online among those who completed the SAQ and were assigned to the i-CBT arms.

We also carried out exploratory subgroup analyses at the request of reviewers to examine variation in the preregistered primary analyses across universities, by country, and by whether the participants were selected from the general student body or clinic waiting lists. Finally, we carried out preregistered disaggregated analyses of intervention effects: (i) on GAD-7 (both remission and mean response) in the subsample of participants with baseline GAD-7= 10+; (ii) on PHQ-9 (both remission and mean response) in the subsample of participants with baseline PHQ-10=10+; and (iii) on joint remission and PHQ-ADS mean response in the subsample of participants with baseline scores of 10+ on both the GAD-7 and the PHQ-9.

Adjustment for loss to follow-up was implemented in all analyses with a doubly robust estimation method that used both outcome modeling (i.e., modeling of the sort used to impute missing values with multiple imputation) and propensity score modeling via the Targeted Minimum Loss-based Estimation (TMLE) method (Rose & van der Laan, 2018) in the tmle3 R package (Coyle, 2021). Our implementation of this approach used the random forests classifier in the ranger R package (Wright et al., 2021) to generate adjustment weights. See details in Supplement B.

In the analyses of remission, we report estimates of the proportion of participants in each arm who remitted after adjusting for nonrandom loss to follow-up along with adjusted risk differences (ARD) in these remission rates across arms. In the analyses of changes in mean symptoms, we report estimates of baseline and 3-month means after adjusting for loss to follow-up and within-arm changes over time, along with comparisons across arms in adjusted 3-month mean differences (AMD).

Participants in the trial could obtain other types of treatment. As noted above, the 3-month follow-up SAQ asked all participants regardless of intervention assignment if they had received any medication, psychotherapy, or other treatment for their anxiety or depression since the time of baseline assessment other than for the i-CBT provided by the study. In addition to using simple cross-tabulations to compare responses to these questions, we estimated a stacked generalization ensemble machine learning model to determine how well we could predict which participants in TAU sought other treatment using information obtained in the baseline SAQ. The goal in doing this was to determine if subgroups of the total sample could be defined based on such a model to distinguish between the subset of participants who would have had a high probability of obtaining treatment if assigned to TAU and the subset of participants who would have had a low probability of obtaining treatment if assigned to TAU. Such a distinction, if reliable, could be used to carry out subgroup analyses comparing the effects of i-CBT to TAU separately among participants who would and would not have been likely to receive treatment if randomized to TAU. See Supplement B for a discussion of the machine learning method used.

Note that all of the hypotheses, power analyses, data manipulations, and exclusions detailed throughout this report other than the exploratory subgroup analyses carried out at the request of reviewers were included in our preregistered study protocol.

Results

Sample Distribution

Table 1 presents the baseline descriptive statistics of participants. 55.0% came from Mexico (Table 1). 78.7% were female. 55.9% were first-generation university students. 70.3% met baseline criteria for clinically significant anxiety (GAD-7=10+), including 37.4% severe (GAD-7=15+) and 32.9% moderate (GAD-7=10–14). 90.2% met baseline criteria for clinically significant depression (PHQ-9=10+), including 37.0% either severe or moderately severe (PHQ-9=15+) and 53.2% moderate (PHQ-9=10–14). 60.6% met baseline criteria for both clinically significant anxiety and clinically significant depression. 21.4% of participants had both baseline severe anxiety and baseline moderately severe or severe depression. Distributions were quite similar across arms.

Patterns and Selected Predictors of Completing the 3-Month Follow-Up SAQ

Three-month (after randomization) SAQs were completed by 65.4% (n=291/445) of participants in the guided i-CBT arm, 62.2% (n=273/439) in the self-guided i-CBT arm, and 77.0% (n=335/435) in TAU. By far the two strongest predictors of completing the 3-month follow-up SAQ were intervention arm and country (Table 2), with the 3-month SAQ completion rate substantially higher in TAU than either the guided or self-guided i-CBT arms (77.0% versus 62.2–65.4%; χ²₂=26.7, p<.001) and higher in Mexico than in Colombia (77.4% versus 56.9%; χ²₁=63.9, p<.001). The 3-month SAQ completion rate also differed significantly depending on whether the university had a mental health clinic and, if so, the student was recruited from the clinic waiting list or the general student body (χ²₂=48.0, p<.001), from a high of 81.7% among students recruited from waiting lists to a low of 58.5% among students from universities that did not have clinics. The 3-month SAQ completion rate did not differ between first-generation students and students whose parents had university educations, by student gender, or by disorder severity. None of these associations between baseline predictors and 3-month completion differed across intervention arms.

Table 2.

Patterns and selected baseline predictors of completing the 3-month follow-up self-administered questionnaire (SAQ) among the 1,319 participants who completed the baseline SAQ

	3-month completion rate				Heterogeneity
	n	(%)	p a	df	p b	df
Total	899	(68.2)
I. Intervention arm			<.001	2
Guided	291	(65.4)
Self-guided	273	(62.2)
Treatment as usual (TAU)	335	(77.0)
II. Country			<.001	1	1.00	2
Colombia	338	(56.9)
Mexico	561	(77.4)
III. Sex			1.00	1	1.00	2
Male	184	(65.5)
Female	715	(68.9)
IV. Sexual orientation			.47	3	.31	6
Heterosexual	610	(66.6)
Gay or lesbian	39	(63.9)
Bisexual	141	(71.9)
Other (asexual, unsure, "other")	109	(74.7)
V. Age			.44	3	1.00	6
18–19	24	(69.8)
20	177	(70.5)
21–22	278	(70.0)
23 or older	197	(62.1)
VI. First generation university student			.26	1	1.00	2
Yes	485	(65.7)
No	414	(71.3)
VII. Did the university have a mental health clinic?			<.001	2	1.00	4
Yes and student recruited from waiting list	237	(81.7)
Yes but student recruited from student body	415	(68.4)
No (all students recruited from student body)	247	(58.5)
VIII. Severity of anxiety (GAD-7)c			1.00	2	1.00	4
Severe	335	(67.8)
Moderate	300	(69.1)
Mild or none	264	(67.5)
IX. Severity of depression (PHQ-9)d			.44	2	1.00	4
Severe	335	(68.6)
Moderate	466	(66.4)
Mild or none	98	(76.0)
X. Comorbidity			1.00	4	1.00	8
Severee on both GAD-7 and PHQ-9	190	(67.4)
Severee on one and moderate on the other	189	(66.5)
Severee on one and mild-none on the other	101	(75.4)
Moderate on both	158	(67.8)
Moderate on one and mild-none on the other	261	(67.6)

Note. Abbreviations: n=number of baseline participants who completed the 3-month follow-up self-administered questionnaire (SAQ); %=the proportion of baseline respondents represented by n; df=degrees of freedom; Guided=guided internet-delivered cognitive behavioral therapy (i-CBT); Self-guided=self-guided i-CBT; GAD-7=Generalized Anxiety Disorder-7; PHQ-9=Patient Health Questionnaire-9.

^aTests of the significance of the associations between baseline predictors and completing the 3-month follow-up SAQ in the total sample.

^bTests of the significance of variation in the association of the baseline predictor with completing the 3-month follow-up SAQ across intervention arms. All such tests were nonsignificant.

^cSevere refers to GAD-7=15+, moderate to GAD-7=10–14, mild or none to GAD-7=0–9.

^dSevere refers to PHQ-9=20+, moderately severe to PHQ-9=15–19, moderate to PHQ-9=10–14, mild or none to PHQ-9=0–9.

^eIncluding either severe or moderately severe PHQ-9.

Models to adjust for nonrandom loss to follow-up

We noted above that a doubly robust estimation method was used to adjust for loss to follow-up. This required the estimation of three different flexible machine learning models for participants in each of the three arms for the primary outcomes, each using all 282 baseline covariates in Supplement C as potential predictors. The first model estimated whether the participant completed the follow-up assessment. A propensity score weight of 1/p_i based on that model, where p_i was the predicted probability that participant completed the follow-up assessment, was then used to weight the data for each participant who completed the assessment. The outcome prevalence (for the dichotomous outcomes) or mean score (for the continuous outcomes) for that weighted sample was then compared across arms to obtain an estimate of intervention effects. The second and third models estimated the predicted outcome score in the subsample of respondents who completed the follow-up assessment. The coefficients from that second model were then used to estimate a predicted outcome prevalence (for the dichotomous outcomes) or mean score (for the continuous outcomes) for the total sample (i.e., including the respondents who were lost to follow-up) using baseline information that was compared across arms to obtain a second estimate of intervention effects. The two estimates were then combined using an optimization weight. See Supplement B for details.

The strength of these loss to follow-up models can be evaluated with the cross-validated area under the receiver operating characteristic curve (AU-ROC) for the dichotomous outcomes and the cross-validated coefficient of multiple determination (R²) for the continuous outcomes. We used 5-fold cross-validation (CV) in carrying out these calculations. AU-ROC was in the range 0.58–0.66 across subgroups to predict completion of follow-up SAQ and in the range 0.58–0.62 to predict joint remission of both GAD-7 and PHQ-9 (Table S1). R² was in the range 0.06–0.15 to predict PHQ-ADS scores. A total of 40 baseline covariates entered one or more of these models. The 10 most important of these predictors for each model are presented in Figures S1-S9.

Aggregate Comparative Treatment Effects

The estimated joint 3-month remission rate for both GAD-7 and PHQ-9 based on the doubly robust adjustment for loss to follow-up was 42.2% in the total sample (Table 3). This estimated remission rate varied significantly across arms (37.2–50.3%; χ²₂=2.4, p=.002) due to participants in the guided i-CBT arm having a significantly higher remission rate than those in the other arms: ARD=13.1% versus self-guided i-CBT, number needed to treat (NNT)=8 (χ²₁=10.4, p=.001); and ARD=11.2% versus TAU, NNT=9 (χ²₁=8.4, p=.004). The difference between remission rates in the self-guided i-CBT arm and TAU, in comparison, was nonsignificant (χ²₁=0.2, p=.63).

Table 3.

3-month joint remission rates of both GAD-7 and PHQ-9 across arms along with adjusted risk differences and number needed to treat^a

	Remission Rateb		Versus self-guided i-CBT				Versus TAU				Combined pc
	n	(%)	ARD	(SE)	p	NNT	ARD	(SE)	p	NNT
											.002
Guided	291	(50.3)	13.1%*	(4.1)	.001	8	11.2%*	(3.9)	.004	9
Self-guided	273	(37.2)	-	-	-	-	−1.9	(3.9)	.63	--
TAU	335	(39.0)					-	-	-		-
Total	899	(42.2)

Note. Abbreviations: GAD-7=Generalized Anxiety Disorder-7; PHQ-9=Patient Health Questionnaire-9; i-CBT=internet-delivered cognitive behavioral therapy; TAU=treatment as usual; n=number of participants who remitted; %=the remission rate associated with n; ARD=adjusted risk difference; SE=standard error of ARD; NNT=number needed to treat; Guided=guided i-CBT; Self-guided=self-guided i-CBT.

^aAdjusted for differences across universities and for informative loss to follow-up using targeted minimum loss-based estimation. See Supplement B for details.

^b3-month remission was defined as scores of 0–4 on both the GAD-7 and the PHQ-9.

^cThis is a 2-degree of freedom test for the significance of differences across all three arms.

^*Significant difference in ARD between the arm in the row heading and the arm in the column heading at the .05-level, two-sided test.

Similar comparative differences across arms were found for mean PHQ-ADS scores (Table 4). Although these means decreased substantially in all arms between baseline (28.4–28.7) and the 3-month follow-up assessment (13.1–15.6; Cohen’s d=0.93–1.07), the adjusted 3-month means varied significantly across arms (χ²₂=8.0, p=.019) due to participants in the guided i-CBT arm having lower means at follow-up than participants in either the self-guided i-CBT arm (AMD=−1.9, Cohen’s d=0.13; χ²₁=4.2, p=.039) or TAU (AMD=−2.4, Cohen’s d=0.18; χ²₁=7.3, p=.007) after adjusting for baseline scores. The difference in adjusted 3-month means between the self-guided i-CBT arm and TAU, in comparison, was nonsignificant (χ²₁=0.3, p=.55).

Table 4.

Baseline and 3-month mean PHQ-ADS scores across arms along with adjusted mean differences^a

	M						M 3-month differences
	Baseline			3-month			Versus self-guided i-CBT				Versus TAU				Combined pb
	N	M	(SD)	n	M	(SD)	AMD	(SE)	p	d	AMD	(SE)	p	d
Total sample															.02
Guided	445	28.7	(20.3)	291	13.1	(14.1)	−1.9	(0.9)	.04	0.13		−2.4*	(0.9)	.01	0.18
Self-guided	439	28.4	(20.2)	273	15.0	(14.2)	-	-	-	-		0.5	(0.9)	.55	0.04
TAU	435	28.7	(19.0)	335	15.6	(13.4)					-	-	-	-
Total	1,319	28.6	(19.9)	899	14.6	(13.9)

Note. Abbreviations: PHQ-ADS=the 0–48 Patient Health Questionnaire Anxiety and Depression Scale; M=observed mean; i-CBT=internet-delivered cognitive behavioral therapy; TAU=treatment as usual; N=number of participants randomized to the different intervention arms at baseline; n=number of participants randomized to the different intervention arms that completed the 3-month self-administered questionnaire; SD=standard deviation of the observed mean; AMD=adjusted mean difference; SE=standard error of AMD; d=Cohen’s d effect size; Guided=guided i-CBT; Self-guided=self-guided i-CBT.

^aAdjusted for differences across universities and for informative loss to follow-up using targeted minimum loss-based estimation. See Supplement B for details.

^bThis is a 2-degree of freedom test for the significance of differences across all three arms.

^*Significant difference between the arms in the row and column headings at the .05-level, two-sided test.

Exploratory subgroup analyses failed to find significant differences in these aggregate effects either on remission rates (Table S2) or on mean differences in PHQ-ADS scores (Table S3) across universities, by country, by whether the university had a mental health clinic, or whether participants were recruited from the general study body or from clinic waiting lists.

Per-Protocol Sensitivity Analysis

Comparative differences in remission rates across arms in the per-protocol sensitivity analysis were consistent with but more pronounced than those in the intent-to-treat analysis (Table S4). Specifically, the overall remission rate (44.0%) varied significantly across arms (34.6%−58.9%; χ²₂=15.4, p<.001) due to a significantly higher rate in the guided i-CBT arm than either the self-guided i-CBT arm (ARD=24.3%; χ²₁=7.8, p=.005) or TAU (ARD=20.3%; χ²₁=14.0, p<.001). As in the intent-to-treat analysis, the difference between the self-guided i-CBT arm and TAU was nonsignificant (ARD=−4.0%; χ²₁=0.3, p=.60). Three-month mean PHQ-ADS scores, in comparison, did not vary significantly across arms in the per-protocol analysis (χ²₂=5.3, p=.07).

Preregistered Disaggregated Analyses

Disaggregated analyses found that comparative treatment effects on GAD-7 remission among participants with baseline GAD-7=10+, on PHQ-9 remission among participants with baseline PHQ-9=10+, and on joint remission among participants with baseline scores of 10+ or both GAD-7 and PHQ-9 were all very similar to effects on the primary outcome in the total sample. Specifically, all these disaggregated remission rates were significantly higher for guided i-CBT than either self-guided i-CBT or TAU, but none was significantly different between self-guided i-CBT and TAU (Table S5).

Disaggregated analyses of mean differences were also comparable to those in the total sample in that all means decreased significantly more in the guided i-CBT arm than TAU (Table S6). However, two of the three disaggregated means also decreased significantly more in the guided i-CBT arm than the self-guided i-CBT arm -- for GAD-7 and PHQ-ADS -- whereas the mean difference was not significant in the total sample.

Probability of Obtaining Other Treatment

There were no significant differences across arms in probability of receiving any other type of treatment besides our i-CBT among respondents recruited from clinic waiting lists (66.4–68.7%; χ²₂=0.1, p=.95) (Table S7). However, among respondents recruited from the general student body, probability of receiving any other type of treatment was significantly higher in TAU (41.7%) than either the self-guided or guided i-CBT arms (23.5–24.4%, χ²₁=18.0–15.8, p<.001), with psychotherapy being more prevalent than psychotropic medication both in TAU (37.3% versus 12.1%, χ²₁=6.7, p=.010) and in the i-CBT arms (22.0–22.3% versus 5.1–5.2%, χ²₁=5.2, p=.022). Probability of receiving any other type of treatment was consistently higher among participants recruited from clinic waiting lists than from the general student body within each arm (χ²₁=18.0–53.2, p<.001).

As noted above in the section on analysis methods, we attempted to develop a machine learning model to predict which participants would seek other treatment if they were randomized to TAU. However, nested cross-validated AU-ROC was low (AU-ROC=0.53, SE=0.04), making it impossible to use this predicted value to support meaningful subgroup analysis. It is noteworthy, though, that the existence of significantly better treatment outcomes in the guided i-CBT arm than TAU despite the higher use of other treatments in TAU suggests that our results regarding the value of guided i-CBT are, if anything, conservative to the extent that other treatments had any positive effects on symptoms.

Discussion

We are aware of no other Latin American treatment trial that ever carried out a comparison of guided i-CBT, self-guided i-CBT, and TAU with a large a sample of anxious-depressed university students as in the current study. Our effect sizes were small based on the standard definition (Cohen, 1988). However, these results were robust given their consistency across per-protocol sensitivity analyses, preregistered disaggregated analyses, and post hoc exploratory subgroup analyses. The 37–50% of students who experienced joint remission of both GAD-7 and PHQ-9 were comparable to the 38–56% pooled response-remission rates for guided i-interventions and higher than the 21–35% pooled response-remission rates for self-guided i-interventions found in a meta-analysis of prior controlled trials for depression (Karyotaki et al., 2018) and comparable to the 35–70% pooled response-remission rates in found a separate meta-analysis of i-CBT for anxiety, depression, and other common behavioral health problems (Andersson et al., 2019; Davey et al., 2023; Simon et al., 2023).

Consistent with prior research (Eilert et al., 2020; Forand et al., 2019; Karyotaki et al., 2018; Karyotaki et al., 2021; Wright et al., 2019), we found significantly higher GAD-7 and PHQ-9 remission rates and mean differences in PHQ-ADS from baseline to 3-month for guided i-CBT than the other two arms. However, we found that self-guided i-CBT was not different from TAU, whereas most prior studies found better outcomes with self-guided i-CBT than controls (Karyotaki et al., 2017; White et al., 2022), although these earlier studies were for the most part based on waitlist controls or psychological placebo control groups rather than TAU (Andersson et al., 2019; Harrer et al., 2019). The equivalence of self-guided i-CBT with TAU in the current study might be important, given that self-guided i-CBT has a meager cost relative to TAU (Rohrbach et al., 2023; Romero-Sanchiz et al., 2017).

However, as somewhat more than half of participants randomized to TAU from the general student body subsample and one-third of those randomized to TAU from the clinic waiting list subsample failed to receive any treatment, the equivalence of self-guided i-CBT and TAU is difficult to interpret. This is true because we cannot distinguish between two plausible scenarios with the current design. Scenario 1 is that self-guided i-CBT might have been associated with significantly worse outcomes than TAU in the subsample of respondents who would have sought other treatment if randomized to TAU, but with significantly better outcomes than TAU in the subsample of respondents who would not have sought other treatment if randomized to TAU, resulting in nonsignificant differences between self-guided i-CBT and TAU in the total sample. Scenario 2 is that the effects of self-guided i-CBT and TAU might have been equivalent both in the subsample of participants who would have sought other treatment if randomized to TAU and in the subsample of respondents who would not have sought out other treatment if randomized to TAU.

It is easy to imagine how Scenario 1 could occur, as self-guided i-CBT might be “better than nothing” among students who would not seek treatment if randomized to TAU, but worse than the treatment obtained in TAU among students who would seek treatment if randomized to TAU. Scenario 2, in comparison, could occur either if the intensity of engagement both with self-guided i-CBT and with other treatments was low in the subsample of participants who would not have sought treatment if randomized to TAU and if intensity of engagement was high in the subsample of participants who would have sought treatment if randomized to TAU. We have no way to investigate these alternatives, as our machine learning model failed to predict whether participants randomized to TAU would seek other treatment. This means that we did not have a good proxy measure of this subgroup distinction in our sample. Because of this, we cannot rule out the possibility that self-guided i-CBT is superior to TAU for some subset of participants even though the aggregate difference is nonsignificant. Further analyses of average treatment effects among the treated (Hesser, 2020) would not shed any light on this uncertainty. Instead, analyses of heterogeneous treatment effects would be needed to investigate whether a subsample of participants could be found using baseline information to document interactions in the comparative effects of self-guided i-CBT and TAU (Maj et al., 2020; Schneider et al., 2015).

Limitations

The study has several noteworthy limitations. First, participants were diagnosed by SAQ rather than by a clinician, although GAD-7 and PHQ-9 are well-validated and commonly used outcomes in clinical trials (McMillan et al., 2010; Toussaint et al., 2020). Second, TAU was not uniform across universities, although recent research suggests that this variation might not have a large influence on effect size estimates (Cuijpers et al., 2021). Third, enrollment began during the COVID-19 pandemic, when university mental health services moved online, which means that results might not generalize to in-person TAU. Fourth, as with other i-intervention trials, we could not blind participants to treatment conditions. Fifth, again consistent with other i-intervention trials, engagement and completion of the i-CBT were low, especially self-guided i-CBT (Musiat et al., 2022). Sixth, although the 68% 3-month follow-up SAQ completion rate was high relative to most prior i-intervention trials (Donker et al., 2013; Harrer et al., 2019; McMillan et al., 2010), it was significantly higher in TAU, in Mexico, among treatment-seeking students, and in universities that had a mental health clinic than in the remainder of the sample. These differences might have introduced bias into some comparisons that violated the missing at-random assumptions made by the approach we used to adjust for loss to follow-up.

Strengths and Implications

Study strengths include the relatively large sample, the use of culturally-adapted i-interventions, and the focus on underserved Colombian and Mexican college students with anxiety and/or depression. The COVID-19 pandemic has accelerated the use of remote treatment for anxiety and depression, making it important to know if scalable, lower-cost i-interventions are feasible and as effective as TAU in resource-constrained environments such as these. Our results show clearly that treatment outcomes were meaningfully better with guided i-CBT than TAU. This was true, importantly, even among respondents recruited from clinic waiting lists, where roughly two-thirds of participants across arms received other treatments. Furthermore, effects were roughly equivalent in the subsample where recruitment was carried out in the general student body and the subsample selected from clinic waiting lists. This is a key result, as the general student body subsample was not seeking treatment but rather responded to our invitation. The email outreach method we used for recruitment had a meager cost. These results suggest that the outreach and guided i-CBT approaches used here could reduce barriers to treatment and improve the mental health of many students with clinically significant anxiety and depression who do not seek treatment on their own.

Future directions

As noted above, although we focused here on 3-month average treatment effects we will also investigate 12-month average treatment effects once data become available. In addition, we are investigating the existence of heterogeneity in comparative treatment effects across arms. If the latter investigation suggests that heterogeneity exists, it might be possible to develop an individualized treatment rule to optimize treatment outcomes by using self-guided i-CBT with some proportion of students. It is noteworthy that 80–84% as many participants remitted in the other arms as in the guided i-CBT arm, which means that substantial cost-savings could be achieved if we knew before intervention assignment which participants had an equally high probability of remission with self-guided i-CBT as guided i-CBT. It would also be valuable to know which participants had a high probability of remitting only with guided i-CBT, in which case special efforts could be made to give this intervention to those students. Finally, it would be valuable to know which participants had a low probability of remitting with any of the interventions considered here, in which case more intensive alternatives might be provided (Duffy et al., 2019). We will explore the possibility of developing such individualized treatment rules in future analyses using state-of-the-art precision treatment methods (Qiu et al., 2021).

Conclusion

We found evidence that guided i-CBT outperformed self-guided i-CBT and TAU over a 3-month follow-up period in treating university students with clinically significant anxiety and/or depression. Further research is needed to examine longer-term effects and whether heterogeneity of treatment effects can be documented. If so, it might be possible to develop individualized treatment rules to increase the cost-effectiveness of intervention assignments through the targeted expansion of access to i-CBT.

What is the public health significance of this article?

Anxiety and depression are significant public health problems in LMIC universities. A culturally adapted transdiagnostic guided i-CBT could help alleviate these problems as a low-threshold intervention component of a stepped-care treatment delivery model.

Appendix Table 1. Narrative Description

Thus far, in addition to the current report, we have another paper based on this project: Benjet, C., Zainal, N. H., Albor, Y., Alvis-Barranco, L., Carrasco-Tapias, N., Contreras-Ibáñez, C. C., Cudris-Torres, L., de la Peña, F. R., González, N., Guerrero-López, J. B., Gutierrez-Garcia, R. A., Jiménez-Peréz, A. L., Medina-Mora, M. E., Patiño, P., Cuijpers, P., Gildea, S. M., Kazdin, A. E., Kennedy, C. J., Luedtke, A., Sampson, N. A., … Kessler, R. C. (2023). A Precision Treatment Model for Internet-Delivered Cognitive Behavioral Therapy for Anxiety and Depression Among University Students: A Secondary Analysis of a Randomized Clinical Trial. JAMA psychiatry, e231675. Advance online publication. https://doi.org/10.1001/jamapsychiatry.2023.1675

The current report focuses on depression and anxiety outcomes. The other manuscript uses precision medicine approaches to determine if an individualized treatment rule (ITR) can identify a subgroup of patients who benefit equally from self-guided and guided internet-delivered cognitive-behavioral therapy (i-CBT). It focuses on an extensive array of baseline variables (socio-demographics, university-related factors, stressors related to COVID-19, other recent and lifetime stressors, anxiety and depression characteristics, comorbid mental disorders, mental health treatment, physical health, social networks and supports, personality/temperament and psychological resilience, and internet and literacy preferences) as potential prescriptive predictors of guided and self-guided i-CBT.