Enhancing Accessibility and Scalability of School-Based Programs to Improve Youth Attention and Behavior: Open Feasibility Trial of the Remote CLS-R-FUERTE Program in Mexico

Abstract

Neurodevelopmental disorders of inattention and disruptive behavior, such as attention-deficit/hyperactivity disorder and oppositional defiant disorder, are among the most common youth mental health conditions across cultures. There is a need to develop more accessible school-based intervention and training programs, as well as create a system with clinical research capacity for scalable school clinician training and evaluation, to support students with attention and behavior concerns worldwide. We adapted the collaborative life skills program for Mexico (i.e., CLS-FUERTE) for remote delivery (i.e., CLS-R-FUERTE) and conducted a three-school open trial with N = 67 participants (n = 7–8 students per school [ages 6–12] and their parents, teachers, and school clinicians). We examined fidelity to program content, attendance and adherence records, in vivo observations of program delivery, and postmeeting feedback informing iterative program changes between each school cohort. We also examined improvements in youth attention and behavior rated by parents and teachers to evaluate the remote program effectiveness. CLS-R-FUERTE feasibility, acceptability, and usability findings were promising. Iterative program changes between each school cohort were minor and included adapted curriculum order, enhanced engagement strategies, and technology adjustments. Many students demonstrated reliable change, and the pre–post program improvements were comparable to outcomes from the in-person CLS-FUERTE trial, indicating preliminary effectiveness. Our pilot CLS-R-FUERTE effort supports the process of iteratively adapting, implementing, and evaluating remote school-based intervention and training programs to enhance potential flexibility, accessibility, and scalability. Challenges emerging from technological problems and in context of the COVID-19 pandemic, as well as solutions, are discussed.

Neurodevelopmental disorders of inattention and disruptive behavior, such as attention-deficit/hyperactivity disorder (ADHD) and oppositional defiant disorder (ODD), are among the most common youth mental health conditions across cultures (American Psychiatric Association, 2022; Canino et al., 2010; Willcutt, 2012). Left untreated, the disorders persist and lead to other problems (American Psychiatric Association, 2022; Faraone et al., 2021; Pfiffner & Haack, 2014). Behavioral programs incorporating parent training, child skills training, and/or classroom management are evidence-based treatments (EBTs) shown to reduce ADHD/ODD symptoms and associated impairment compared to community practices (Pfiffner & Haack, 2014).

Despite promising effects of ADHD/ODD EBTs, most youth lack access (Danielson et al., 2018). Schools are a promising context for accessible delivery, given the potential to reallocate existing resources from nonempirically supported practices to those with effectiveness (Pfiffner et al., 2016). Currently, schools tend to offer non-EBTs for students with ADHD/ODD (Hustus et al., 2020; Lovett & Nelson, 2021; Murray et al., 2014; Spiel et al., 2014).

ADHD/ODD EBTs have been developed for delivery by school clinicians. The collaborative life skills (CLS) program (Pfiffner et al., 2016, 2018; a comprehensive school-home EBT featuring behavioral classroom management, child groups, and parent groups) was adapted for U.S. Spanish-speaking families (i.e., CLS-Spanish; Haack et al., 2019) and Mexican families (i.e., CLS-FUERTE; Haack et al., 2021). CLS-Spanish and CLS-FUERTE evaluations demonstrate comparable feasibility, acceptability, and treatment outcomes as the original trial (Haack et al., 2019, 2021; Pfiffner et al., 2016). Given that CLS and its adaptions are designed for school clinician delivery, once trained, clinicians may continue the EBTs in their schools at no cost to families thereby creating a sustainable and equitable method for students to receive treatments that work. Further, given that CLS and its adaptations target students with elevated ADHD/ODD symptoms and impairment (rather than students with documented diagnoses), the reach may be even greater than traditional ADHD/ODD interventions designed for clinical populations.

Supporting school-based EBT and training is valuable worldwide but particularly in settings with high unmet need, such as Mexico, where only 14% of youth with mental health disorders receive treatment and less than half of those treated receive more than minimally adequate care (Borges et al., 2006, 2008). Youth mental disorders also are prevalent in Mexico, with estimates at double those in the United States (Benjet et al., 2009). High unmet need likely is perpetuated by limited EBT training for school clinicians (Sanchez-Sosa, 2007; Stark et al., 2010) and for clinical researchers at partnering universities to gain capacity for scalable school clinician training, monitoring, and ongoing support.

One solution for enhancing the acceptability and scalability of school-based EBT and training programs is harnessing digital technology. CLS recently was adapted for web-based training, coaching, and fidelity monitoring in the United States, called CLS-Remote (CLS-R; Pfiffner et al., 2023). Preliminary results are consistent with existing studies suggesting comparable or even favorable results for training programs delivered remotely versus in-person (e.g., Khanna & Kendall, 2015). In the wake of COVID-19 preventing in-person groups during shelter-in-place, we recognized a need to adapt the CLS-FUERTE program for fully remote delivery in Mexico (i.e., CLS-R-FUERTE), such that school clinician training and implementation of EBT groups/meeting occurs via videoconferencing. A fully remote program would be valuable in engaging parents and students unable to attend in-person activities for various reasons, such as shelter-in-place situations due to COVID-19 surges or other illness outbreaks, safety lockdowns, or natural disasters.

The Present Study

As a first step in translating the in-person CLS-FUERTE program for fully remote delivery, we conducted an open feasibility trial in three public elementary schools in Sinaloa, Mexico, grounded in an iterative behavioral intervention design (Czajkowski et al., 2015; see Supplemental Figure 1). The primary objectives were to evaluate the CLS-R-FUERTE program feasibility, acceptability, usability, and preliminary effectiveness as a precursor to efficacy testing. Of note, the present study activities occurred in context of clinical research capacity building efforts aimed at advancing mental health research and research-informed practice in Mexico, as well as scalable EBT delivery and training. Specifically, emerging clinical researchers served as CLS-R-FUERTE trainers-in-training during the initial school cohort in preparation for serving as supervised program trainers in subsequent cohorts. We predicted:

Hypothesis 1: The CLS-R-FUERTE program would be feasible for school clinicians to implement with fidelity and for families and teachers of youth with ADHD/ODD symptoms to attend and adhere to;

Hypothesis 2: The CLS-R-FUERTE program would be acceptable to participating school clinicians, families, and teachers;

Hypothesis 3: The CLS-R-FUERTE program would be usable for participating school clinicians and families; and

Hypothesis 4: The CLS-R-FUERTE program would appear to be effective at improving student attention and behavior.

Method

Participants

A total of N = 67 (n = 7–8 students in grades 1–5 at each of three schools, as well as their parents, teachers, and school clinicians) participated during the 2021–2022 school year. For each student, one parent and one teacher were designated as “primary” and completed all measures; one or two school clinicians per school participated in the training and program implementation. See Table 1 for participant demographic information (which was collected on screening interviews) and student ADHD/ODD symptom presentation (based on parent and teacher baseline ratings).

Table 1.

Participant Demographics

School clinicians (n = 4)	School 1 (n = 1)	School 2 (n = 1)	School 3 (n = 2)	Teachers (n = 17)	School 1 (n = 5)	School 2 (n = 6)	School 3 (n = 6)
Gender	Female	Female	Males	Female (n, %)	3, 60%	6, 100%	6, 100%
Students (n = 23)	School 1 (n = 7)	School 2 (n = 8)	School 3 (n = 8)	Parents (n = 23)	School 1 (n = 7)	School 2 (n = 8)	School 3 (n = 8)
Female (n, %)	2, 29%	4, 50%	4, 50%	Female (n, %)	7, 100%	8, 100%	7, 87%
Age (n, %)				Marital status (%)
6–7	3, 43%	3, 38%	4, 50%	Married or cohabitating	6, 86%	7, 87%	8, 100%
8–10	3, 43%	5, 62%	2, 25%	Widowed	0, 0%	1, 13%	0, 0%
11–12	1, 14%	0, 0%	2, 25%	Divorced or separated	1, 14%	0, 0%	0, 0%
Grade (n, %)				Total annual household
1–2	3, 43%	3, 38%	4, 50%	Income (n, %)
3–4	3, 43%	5, 62%	2, 25%	Less than $10,000 (United States)	4, 57%	7, 87%	6, 75%
5	1, 14%	0, 0%	2, 25%	$10,000 (United States) or more	3, 43%	1, 13%	2, 25%
On medication (%)	1, 14%	0, 0%	2, 25%	Employment status (n, %)
ADHD presentation (n, %)a				Working full-time	2, 29%	0, 0%	2, 25%
Inattentive profile	1, 14%	4, 50%	4, 50%	Working part-time	3, 43%	1, 13%	1, 12.5%
Combined profile	6, 86%	4, 50%	4, 50%	Stay-at-home parent	0, 0%	1, 13%	4, 50%
ODD presentation (n, %)a				Unemployed	1, 14%	2, 25%	0, 0%
ODD profile	4, 57%	4, 50%	2, 25%	Other or prefer not to report	1, 14%	4, 50%	1, 12.5%

Note. N = 67. All participants represented Latine ethinicity. Teachers could have up to two students in the program; thus, the teacher (n) is lower than the student (n). ADHD = attention-deficit/hyperactisvity disorder; ODD = oppositional defiant disorder. CSI-4 = Child Symptom Inventory, 4th Edition.

^aADHD and ODD symptom presentation profiles based on number of symptoms endorsed by parents OR teachers on the CSI-4 (Sprafkin et al., 2002) at baseline.

Procedure

School and participant recruitment procedures matched that of the in-person CLS-FUERTE trial (Haack et al., 2021). Each participating school received the CLS-R-FUERTE program: a remote school clinician training and comprehensive behavioral intervention designed to improve attention and behavior in Mexican school-aged youth (grades 1–5). Please see our Supplemental Materials for a detailed CLS-R-FUERTE intervention protocol, including information about the program components, the school and student recruitment activities, and the iterative program changes from the present study,

Measures

Present study construct measurement and benchmarks were informed by previous school-based CLS program trials (Haack et al., 2019, 2021; Pfiffner et al., 2018, 2023). We measured feasibility through various sources of data, including fidelity ratings of program implementation, as well as rates of participant program attendance and strategy adherence. Study leads rated fidelity to program implementation and tracked participant attendance during in vivo program observation with benchmarks of at least 80% of program material delivered, quality of program delivery rating averaging at least four on a 5-point Likert scale, and participants averaging at least 80% attendance at program sessions. We measured parent strategy adherence through weekly self-report questionnaires with benchmarks of at least 80% reporting using the home routine strategy more days than not; we measured teacher strategy adherence through the electronic daily report card (DRC) system with the benchmark of DRC use more days than not. We measured acceptability through satisfaction ratings by school clinicians, parents, and teachers on postprogram questionnaires with benchmarks of at least 80% reporting the two most favorable options on items rated with a 5-point Likert scale; we measured student satisfaction on weekly verbal questionnaires with benchmarks of at least 80% reporting the most favorable option on items rated with a 4-point Likert scale. We measured usability through System Usability Scale (SUS; Brooke, 1996) ratings from school clinicians and parents on weekly and postprogram questionnaires with benchmarks of ratings reaching 68 out of 100 throughout the course of the program. We measured effectiveness through parent and teacher ratings of student ADHD and ODD symptoms on the Child Symptom Inventory, 4th Edition (CSI-4) and related impairment on the Impairment Rating Scale (IRS) at pre- and postprogram; we examined reliable change and compared pre–post effect sizes to our prior trial. See below for a detailed description of each measure.

Fidelity to Program Implementation

Study leads rated fidelity to the CLS-R-FUERTE program intervention based on amount of session content covered (0 = not at all to 2 = fully; allowing for a calculation of overall percentage of session content covered) and quality (1 = low to 5 = high, allowing for a calculation of average session quality) during in vivo observation for each program session. This method has been successfully used and iteratively updated in previous CLS program trials (Haack et al., 2019, 2021; Pfiffner et al., 2011, 2016, 2023).

Participant Attendance

Study leads also tracked parent and student group attendance during observation of each program session, allowing for a calculation of overall percentage of session attendance. For hybrid student groups, study leads qualified if students attended in-person or remotely via videoconferencing.

Participant Satisfaction

We measured school clinician, parent, and teacher satisfaction based on the percentage of participants who responded to the postprogram questionnaire item about overall program quality with the two most favorable options on a 5-point Likert scale (i.e., 4 = satisfied and 5 = very satisfied). We measured student satisfaction based on the percentage of responses on the weekly two-item verbal questionnaire (i.e., “how much did you like group today?” and “how much did you learn in group today?”) rated as the most favorable option on a 4-point Likert scale (i.e., 0 = not at all to 3 = a lot). This method has been successfully used and iteratively updated in previous CLS program trials (Haack et al., 2019, 2021; Pfiffner et al., 2016, 2023).

Participant Strategy Adherence

We measured parent strategy adherence based on the percentage of responses on the weekly questionnaire item regarding use of the home routine strategy rated as “often/more than half of the days” or “very often/most days.” We measured teacher strategy adherence using the electronic DRC system facilitating automated, daily emails with links for teachers to submit their student’s DRC points online, which are automatically sent to parents, school clinicians, and clinical research team members. This system allows for a calculation of percentage of days the DRC is used out of the number of school days the student attended during the program. We supplemented electronic DRC reports via discussion with teachers about how many days they used the DRC but did not lock-in points electronically when data were missing. This system was used and iteratively updated in the remote CLS training program open trial in the United States (Pfiffner et al., 2023).

The SUS (Brooke, 1996) was used to collect weekly and postprogram usability ratings from school clinicians and parents. The SUS is a normed 10-item measure with a 5-point Likert scale widely used and adapted for technology usability testing, including evaluations of web-based intervention programs (Lyon et al., 2021) and the remote CLS training program open trial in the United States (Pfiffner et al., 2023). SUS total scores range from 0 to 100, with higher scores indicating better usability. A recent meta-analysis demonstrates that the widely accepted SUS benchmark of good usability at >68 is psychometrically sound for evaluating digital health programs (Hyzy et al., 2022). The SUS has high internal consistency (α = .91) and high convergent validity with a separate rating of usability and user satisfaction (r = 0.81; Bangor et al., 2008).

The CSI-4 (Gadow & Sprafkin, 1997) facilitates parent- and teacher ratings of ADHD and ODD symptom severity at pre- and postprogram. Each symptom is rated on a 4-point scale (0 = never to 3 = very often) allowing for a calculation of average severity for each symptom subscale of interest (i.e., inattention, hyperactivity/impulsivity, and oppositionality), as well as a symptom count for each subscale of interest. Specifically, each symptom is deemed “present” if the parent or teacher endorses a rating of 2 = often or 3 = very often. The symptom count can be used to classify profiles of ADHD and ODD presentation, as well as examine reliable change following treatment. If a student has at least six of nine inattention symptoms present and less than six of nine hyperactivity/impulsive symptoms present, they are classified with a symptom profile reflective of ADHD, inattentive presentation; if they have at least six of each subscale, they are classified with a symptom profile reflective of ADHD, combined presentation; if they have at least four ODD symptoms present, they are classified with a symptom profile reflective of ODD Presentation.

The Spanish CSI-4 version has normative data, acceptable test–retest reliability, predictive validity for ADHD and ODD diagnosis (Gadow & Sprafkin, 1997). The CSI-4 was used in the in-person CLS-FUERTE trial (Haack et al., 2021) and demonstrated high internal consistency in the Mexican sample (αs = .87–.95). We found comparable consistency in our present study sample via coefficient omega, a reliability estimate appropriate for small samples sizes (Dunn et al., 2014), (range = .86–.91) with one exception (parent-rated inattention = .60).

The IRS (Fabiano et al., 2006) facilitates parent- and teacher ratings of ADHD/ODD-related impairment (i.e., academics and peer relations) at pre- and postprogram. Each item is rated on a 7-point scale (1 = no problem to 7 = extreme impairment) allowing for a calculation of average impairment across items. The IRS has excellent psychometric properties including reliability, convergent validity, discriminant validity, and predictive validity for an ADHD diagnosis (Fabiano et al., 2006). The IRS was used in the in-person CLS-FUERTE trial (Haack et al., 2021) and demonstrated high internal consistency in the Mexican sample (αs = .72–.85). We found comparable consistency in our present study sample via coefficient omega (range = .75–.76).

Data Analytic Plan

All data analysis was conducted using Statistical Package for the Social Sciences Version 28. We analyzed descriptive statistics (i.e., frequency percentages and mean averages, as described above) of program feasibility (i.e., fidelity to program implementation, as well as participant attendance and strategy adherence), acceptability (i.e., participant satisfaction), and usability (i.e., SUS ratings). To examine preliminary effectiveness, we calculated the percent of students who demonstrated reliable change in ADHD and ODD symptom counts rated by parents and teachers, using the Reliable Change Index formula proposed by Jacobson and Truax (1992):

$$\begin{gathered} \text{RCI}=X_{\text{post}}-X_{\text{pre}}/\text{Sdiff};\text{Sdiff}=\sqrt{2}{\left(S_E\right)}^2; \\ {S}_E=S{D}_1\sqrt{1}-r_{xx}. \end{gathered}$$ (1)

In this formula, r_xx = test–retest reliability of the measure; Sdiff = standard error of the difference between the two test scores; SD₁ = standard deviation of the current sample at Time 1; S_E = standard error of measurement. If the Reliable Change Index was less than or equal to −1.96, students were categorized as demonstrating reliable change in symptom count improvement.

We also used repeated-measures t tests and effect sizes to examine pre–post changes on parent- and teacher ratings of student ADHD/ODD symptom and impairment severity. We set alpha at .05 and used the Benjamini and Hochberg (1995) correction procedure to control family-wise Type 1 error rate given the number of statistical comparisons. We calculated Hedge’s g pre–post effect sizes, which includes an adjustment factor and is recommended for small sample sizes (Fritz et al., 2012). Given COVID-19 surges fluctuating between the three schools, which appeared to impact participant attendance and strategy adherence, we conducted all analyses by school.

Results

Student Characteristics: ADHD/ODD Symptom Presentation Profiles at Baseline

As seen in Table 1, the distribution of ADHD and ODD presentation profiles that emerged for students in the present study based on symptom count at baseline varied across the three schools. The majority of students in School 1 showed profiles reflective of ADHD, combined presentation (i.e., 86%); in Schools 2 and 3, there was an even split of students who showed profiles reflective of ADHD, inattentive presentation (i.e., 50%) and ADHD, combined presentation (i.e., 50%). Regarding how many students showed profiles reflective of a comorbid ODD presentation, there was a slight majority of students in School 1 (i.e., 57%), an even split in School 2 (i.e., 50%), and a minority of students in School 3 (i.e., 25%). The average baseline severity as rated by parents and teachers ranged between 1.42 and 2.25 (rated on a scale of 0 = never to 3 = very often) for ADHD symptoms and between 0.43 and 1.36 (also rated on a scale of 0 = never to 3 = very often) for ODD symptoms.

Feasibility, Acceptability, and Usability Findings

Our first hypothesis was that the CLS-R-FUERTE program would be feasible for school clinicians to implement with treatment fidelity and for families and teachers of youth with ADHD/ODD symptoms to engage in and adhere to. As seen in Table 2, trainers and school clinicians covered over 95% of program content at quality levels above four out of five for all components (with one exception: the percentage of classroom component content covered at School 3 was 70%). Participant attendance and strategy adherence varied by school.

Table 2.

CLS-R-FUERTE Open Trial Feasibility, Acceptability, and Usability

School clinician training	School 1	School 2	School 3	Classroom component	School 1	School 2	School 3
% of content delivered in training/consultation meetings by trainers, quality ratingsa	99%	99%, 4.69 out of 5	99%, 4.93 out of 5	% of teacher orientation content delivered by school clinicians, quality ratings	100%, 4.22 out of 5	95%, 4.56 out of 5	70%, 2.56 out of 5
School clinician attendance at trainings	100%	100%	100%	% DRCs in traditional three-goal format for in-person classes	74%	100%	100%
% school clinicians reporting being “satisfied” or “very satisfied”	100%	100%	100%	% days DRC used	25%	98%	59%
% school clinicians with system usability ratings reaching ≥68	100%	100%	100%	% teachers reporting being “satisfied” or “very satisfied”	86%	100%	71%
Parent group component	School 1	School 2	School 3	Child group component	School 1	School 2	School 3
% of content delivered in groups by school clinicians, quality ratings	100%, 4.79 out of 5	100%, 4.89 out of 5	100%, 4.39 out of 5	% of content delivered in groups by school clinicians, quality ratings	97%, 4.45 out of 5	100%, 4.71 out of 5	98%, 4.68 out of 5
Parent group attendance	51%	92%	69%	Child group attendance	57%	90%	73%
% parents using home routine strategy most or every day	100%	100%	67%	% students attending child group who joined remotely	100%	100%	15%
% parents reporting being “satisfied” or “very satisfied”	100%	100%	100%	% students reporting that they liked group “a lot”	100%	93%	100%
% parents with system usability ratings reaching ≥68	86%	75%	100%	% students reporting that they learned “a lot” in group	87%	93%	91%

Note. N = 67. CLS-R-FUERTE = The Collaborative Life Skills Remote Program: Familias Unidas Emprediendo Retos y Tareas para el Éxito/Families United in Undertaking Challenges for Success; DRC = daily report card.

^aUnavailable for School 1 when study leads were primary trainers. Parent/teacher satisfiaction were self-reported on 5-point Likert scale questions at posttreatment; student satisfaction and parent adherence was averaged from reports by those attending groups each week. We calculated DRC format (i.e., traditional three-goal for in-person instruction vs. modified for distance learning) and use from the automatic email reports supplemented by discussion with teachers when email report data were missing.

Our second hypothesis was that the CLS-R-FUERTE program would be acceptable to participating school clinicians, families, and teachers with satisfaction ratings comparable to findings from the in-person CLS-FUERTE program. Over 86% of participants rated high satisfaction (with one exception: the percentage of teachers rating high satisfaction at School 3 was 71%; see Table 2). As seen in the Supplemental Materials, we uncovered meaningful themes regarding program satisfaction in postsession focus groups.

Our third hypothesis was that the CLS-R-FUERTE program would be usable for participating school clinicians and families. As seen in Table 2, 75% reported good program usability with SUS ratings reaching ≥68. We uncovered meaningful themes regarding the program usability satisfaction in postsession focus groups (see Supplemental Materials).

Preliminary Effectiveness

Our final hypothesis was that the CLS-R-FUERTE program would appear to be effective at improving student attention and behavior, as evidenced by reliable changes in ADHD/ODD symptom counts rated by parents and teachers, as well as significant pre–post improvement in parent and teacher symptom and impairment severity outcomes with effect sizes comparable to the in-person CLS-FUERTE trial. As seen in Table 3, the percentage of students demonstrating reliable change varied by outcome and across schools. Regarding statistically significant changes from pre- to postprogram, a similar pattern emerged across outcomes (see Table 3), such that students’ ADHD symptoms and impairment rated by parents and teachers improved significantly after CLS-R-FUERTE. Changes in ODD symptoms rated by parents and teachers were only significant in School 3. Pre–post effect sizes ranged from medium to large. Due to the nature of the open trial design without a control group, we lack the ability to conclude with certainty that the CLS-R-FUERTE program caused the pre–post improvements.

Table 3.

CLS-R-FUERTE Open Trial Effectiveness

Proportion of students demonstrating reliable change
		School 1 (n = 7)		School 2 (n = 8)		School 3 (n = 8)
Outcome	M (SD)	n (%) showing reliable change	M (SD)	n (%) showing reliable change	M (SD)	n (%) showing reliable change
Parent-rated inattentive symptom count improvement	3.14 (3.44)	5 (71)	4.50 (2.73)	7 (88)	5.75 (2.31)	8 (100)
Parent-rated hyperactive-impulsive symptom count improvement	2.57 (3.55)	5 (71)	2.13 (2.75)	6 (75)	3.38 (2.50)	7 (88)
Parent-rated ODD symptom count improvement	0.14 (3.02)	3 (43)	1.75 (2.87)	5 (63)	2.25 (2.38)	5 (63)
Teacher-rated inattentive symptom count improvement	4.00 (2.65)	6 (86)	5.71 (2.43)	7 (100)a	5.00 (3.06)	6 (86)a
Teacher-rated hyperactive-impulsive symptom count improvement	1.71 (3.30)	4 (57)	3.43 (3.10)	6 (86)a	1.71 (2.21)	3 (43)a
Teacher-rated ODD symptom count improvement	0.80 (0.84)	3 (43)	2.14 (2.79)	5 (71)a	0.43 (0.79)	2 (29)a

Statistically significant pre-post improvement outcomes
	School 1 (n = 7)			School 2 (n = 8)			School 3 (n = 8)
Outcome	M (SD)	t(df)	Effect size [95% CI]	M (SD)	t(df)	Effect size [95% CI]	M (SD)	t(df)	Effect size [95% CI]
Parent-rated ADHD†
Baseline	2.25 (0.21)	t(6) = 4.05*	g = 1.33 [0.332, 2.28] (large)	1.73 (0.34)	t(7) = 3.20*	g = 1.00 [0.18, 1.79] (large)	1.97 (0.56)	t(7) = 6.05*	g = 1.90 [0.73, 3.03] (large)
Posttreatment	1.44 (0.65)			1.07 (0.62)			0.97 (0.45)
Teacher-rated ADHD†
Baseline	1.68 (0.37)	t(6) = 2.65*	g = 0.87 [0.05, 1.65] (large)	1.56 (0.82)	t(6) = 5.09*	g = 1.67 [0.53, 2.78] (large)	1.42 (0.58)	t(6) = 4.77*	g = 1.57 [0.47, 2.62] (large)
Posttreatment	1.21 (0.75)			0.60 (0.37)			0.80 (0.38)
Parent-rated ODD
Baseline	1.36 (0.46)	t(6) = 1.47	g = 0.48 [-0.23, 1.16] (medium)	1.16 (0.48)	t(7) = 2.25	g = 0.71 [-0.03, 1.40] (medium)	1.33 (0.72)	t(7) = 3.58*	g = 1.12 [0.26, 1.95] (large)
Posttreatment	1.00 (0.92)			0.64 (0.38)			0.59 (0.31)
Teacher-rated ODD†
Baseline	0.65 (0.66)	t(4) = 2.42	g = 0.86 [-0.07, 1.74] (large)	1.00 (0.88)	t(6) = 2.25	g = 0.74 [-0.05, 1.48] (large)	0.43 (0.39)	t(6) = 3.03*	g = 1.00 [0.13, 1.82] (large)
Posttreatment	0.30 (0.39)			0.30 (0.34)			0.21 (0.29)
Parent-rated impairment†
Baseline	5.25 (1.26)	t(6) = 9.83*	g = 3.23 [1.33, 5.11] (large)	4.46 (1.12)	t(7) = 3.32*	g = 1.04 [0.21, 1.84] (large)	4.54 (1.28)	t(7) = 4.51*	g = 1.42 [0.45, 2.35] (large)
Posttreatment	3.23 (1.20)			2.47 (1.23)			3.08 (0.81)
Teacher-rated impairment†
Baseline	4.69 (1.26)	t(6) = 3.33*	g = 1.09 [0.19, 1.95] (large)	4.50 (1.48)	t(6) = 5.06*	d = 1.66 [0.52, 2.76] (large)	4.98 (1.10)	t(6) = 7.09*	g = 2.33 [0.88, 3.75] (large)
Posttreatment	3.08 (1.78)			2.04 (1.22)			2.29 (1.00)

Note. N = 23 students; two had missing teacher ODD baseline data and two had missing postdata due to illnesses during the collection period. ADHD = attention-deficit/hyperactivity disorder; ODD = oppositional defiant disorder; CSI-4 = Child Symptom Inventory, 4th Edition; IRS = Impairment Rating Scale; CI = confidence interval; CLS-R-FUERTE = The Collaborative Life Skills Remote Program: Familias Unidas Emprediendo Retos y Tareas para el Éxito/Families United in Undertaking Challenges for Success.

^aPercentages are calculated out of n = 7 due to missing teacher data from one student due to illnesses during the collection period; improvement in ADHD and ODD symptom count reflects the number of items endorsed as occurring “often” or “very often” the CSI-4 (Gadow & Spafkin, 2002) at posttreatment compared to baseline; reliable change in symptom count improvement was calculated using the Reliable Change Index; Jacobson and Truax (1992) Higher ratings of ADHD and ODD symptom severity (on a 0–3 scale from the CSI-4; Gadow & Spafkin, 2002) and impairment (on a 1–7 scale from the IRS; Fabiano et al., 2006) indicate more severe difficulties.

^*Significant after Benjamini and Hochberg correction (q* = .04).

^†Significant in the CLS-FUERTE in-person trial; g = Hedge’s G.

Discussion

Remote delivery of school clinician training and intervention to improve youth attention and behavior in Mexico appears promising. Our three-school CLS-R-FUERTE open trial demonstrated high ratings of program fidelity, satisfaction, and usability. Participant attendance and strategy adherence varied by school with high rates for those least impacted by COVID-19 surges (i.e., School 2). Importantly, many students demonstrated reliable change in ADHD/ODD symptom counts and the pre–post effect sizes for ADHD symptom and impairment severity improvement were similar to the in-person CLS-FUERTE trial (Haack et al., 2021), providing support for the potential effectiveness of CLS-R-FUERTE. That said, subsequent efficacy testing employing a control group is needed before causal inferences can be made about the CLS-R-FUERTE program’s role in reducing student ADHD/ODD symptoms and related impairment. It is compelling that present study results occurred in context of a capacity building effort in which novel trainers shadowed and then trained new school clinicians. We hope that this process (by which emerging clinical researchers observe and subsequently lead training) may provide a model for promoting school program implementation and evaluation in global settings with high unmet need.

Results support the process of iteratively adapting, implementing, and evaluating school-based programs to enhance potential flexibility, accessibility, and scalability (aligned with recommendations by Cappella et al., 2011). Informed by participant suggestions and program observation, we made minor program changes between each school, including adapting the curriculum order, enhancing engagement strategies, and adjusting technology (e.g., video volume). Despite promising findings, challenges did arise. Some emerged from technological issues (e.g., occasional internet connectivity problems) and others in context of COVID-19 (e.g., absences due to illness and fluctuating in-person vs. distance learning). Regarding nonpandemic challenges, including technology specialists on our team appeared beneficial in helping participants trouble-shoot difficulties, such as connecting to videoconferencing for initial sessions. Regarding pandemic-related challenges, flexibility in adapting methodology was crucial. We encouraged school clinicians to hold makeup sessions for those unable to attend groups. Thus, attendance ratings are likely an underestimate of participant program engagement given their lack of accounting for makeup sessions (which were not monitored by our team). We responded to pandemic surges and improvements by modifying the DRC structure to accommodate teacher ratings of remote class attendance (yes/no) and work completion versus classroom behavior. We also allowed hybrid in-person/remote meetings as schools returned to in-person learning. Interestingly, pre–post ADHD and impairment severity improvement outcomes were relatively similar across schools despite variability in attendance and DRC completion as COVID-19 surged during School 1, improved for School 2, and surged again during School 3. However, the percentage of students demonstrating reliable change in ADHD/ODD symptom counts appeared lower in School 1 (which also demonstrated lower ratings of attendance and DRC usage) compared to Schools 2 and 3; further, ODD symptom severity improvement was only significant in School 3. Given that program adherence is a mechanism of treatment change (Dvorsky et al., 2021; Meza et al., 2020), future investigation should explore how much adherence is necessary for program success.

Constraints on Generality

Notably, we examined a relatively small sample size in one urban school district and thus are unable to determine if our study methods and results would generalize to other settings, such as rural school districts with limited internet access. Adqueate internet access is a necessary setting characteristic which should be present for the remote CLS-R-FUERTE procedures; however, schools with limited internet connectivity could follow in-person CLS-FUERTE procedures previously published (Haack et al., 2021). In addition, we acknowledge that our decision to target students with attention and behavior difficulties rather than diagnosed ADHD/ODD conditions limits generalizability of our methods and findings to students with elevated ADHD/ODD symptoms and results may not generalize to clinical populations. Indeed, baseline symptom severity ratings appear lower than what may be expected in a clinical study of diagnosed youth. However, given that the CLS-R-FUERTE program is designed to be implemented in schools, the decision to forgo a requirement of formal ADHD and/or ODD diagnosis increases the accessibility of the program, as well as likelihood of replicable study procedures and findings, in real-world settings where formal diagnosis may not be possible.

Our study occurred in context of an ongoing collaboration between the clinical research team and partnering school district, which has included in-person CLS-FUERTE program implementation/evaluation efforts since 2016. Thus, it is unclear if findings would differ in novel settings without an existing partnership; future research should explore the need and methods for expanded research capacity building efforts in partner universities supporting program implementation in novel school districts. Finally, some students started the program in distance learning due to COVID-19; it is unclear if this impacted accuracy of teacher ratings.

Other Limitations and Future Directions

In addition to constraints on generality described above, other present study limitations should be noted and addressed in future research. As previously discussed, because the nature of an open feasibility trial does not feature efficacy testing with a control group, we are unable to know if outcomes are in part due to factors other than the CLS-R-FUERTE program, such as typical school supports or the passage of time. Moreover, the statistically significant changes observed were in the context of small sample size; the preliminary nature of these outcomes needs to be considered. Taking into consideration these limitations, our next steps include implementing and evaluating the iteratively adapted CLS-R-FUERTE program in an eight-school clustered Randomized Controlled Trial. Evaluation of CLS-R-FUERTE mechanisms of change, such as improvements in parenting and teaching effectiveness, also are warranted. Subsequent future directions include adapting, implementing, and evaluating CLS-R-FUERTE in novel school districts across Mexico.

Conclusions and Implications

Given the widespread prevalence and negative consequences of ADHD and ODD worldwide, efforts are needed to develop feasible and accessible intervention and training programs for school clinicians to support students with attention and behavior concerns. Our study results support harnessing technology and creating a system with clinical research capacity for more accessible and scalable school clinician training and evaluation. These efforts could not only address a significant public health concern in Mexico but also could be used to expand accessible and scalable school clinician training for EBTs delivered remotely in other settings and/or for other mental health presentations, ultimately reducing disparities for our most underserved global populations.

Impact and Implications.

The widespread prevalence and impact of ADHD and ODD worldwide warrant global efforts to develop feasible and accessible school clinician training and intervention programs aimed at improving student attention and behavior. Our findings support harnessing technology and creating a system with clinical research capacity to tackle this need. These efforts could not only address a significant public health concern in Mexico but also could be used to inform scalable school clinician training for treatments that work more broadly.