Effectiveness of the Challenging Horizons After-School Program for Young Adolescents With ADHD

Abstract

There are no empirically supported psychosocial treatments for adolescents with attention-deficit hyperactivity disorder (ADHD). This study examined the treatment benefits of the Challenging Horizons Program (CHP), a psychosocial treatment program designed to address the impairment and symptoms associated with this disorder in young adolescents. In addition to evaluating social and academic functioning outcomes, two critical questions from previous studies pertaining to the timing, duration, and family involvement in treatment were addressed. Forty-nine students recruited in two cohorts were randomly assigned to receive either the CHP or a community care condition. Outcomes suggested that students who received the CHP improved compared to students in the control condition on measures of symptoms and impairment. Implications related to timing, duration, and family involvement are reported, as well as recommendations for future studies.

Attention-deficit hyperactivity disorder (ADHD) is a high-incidence, chronic disorder associated with adverse outcomes throughout the life span. Although ADHD symptoms appear less conspicuous with increased chronological age, the related impairments can actually prove more costly during adolescence than childhood (e.g., motor vehicle accidents, teenage pregnancy, and occupational failure), and adolescents with ADHD are at significantly higher risk than their peers for school suspensions, academic failure, social impairment, classroom behavior problems, and school dropout (Barkley, Fischer, Edelbrock, & Smallish, 1990).

Empirically supported treatments for children with ADHD consist of pharmacotherapy, behavioral interventions, and their combination (Pelham & Fabiano, 2008); in comparison, the limited research focusing on adolescents suggests that medication treatment may be beneficial (Evans et al., 2001) but its use is limited by problems such as side effects, lack of long-term benefits, and poor adherence (Lerner & Wigal, 2008). Such findings point to a critical need to develop effective psychosocial alternatives for adolescents; but to date, no psychosocial treatments meet criteria to be considered “evidence based,” according to the guidelines offered by most professional organizations.

Two treatment studies have evaluated family therapy to treat adolescents with ADHD. Barkley and colleagues compared behavioral parenting training, family therapy, and problem-solving communication training (Barkley, Edwards, Laneri, Fletcher, & Metevia, 2001; Barkley, Guevremont, Anastopoulos, & Fletcher, 1992). Self-report ratings of family conflict significantly improved with all treatments, but only a minority of participants demonstrated reliable change. Another family intervention developed for middle school youth with behavior problems is the Family Check-Up (FCU; Dishion & Kavanagh, 2003). Although the FCU is not specifically targeted to families of adolescents with ADHD, it includes many aspects appropriate to families of these adolescents. The FCU intervention incorporates motivational interviewing techniques to help parents pursue services. Evaluations of this intervention have shown long-term benefits related to self-regulation, decreases in depressive symptoms, and improved school engagement (Stormshak, Fosco, & Dishion, 2010).

Intervention development and evaluation research for adolescents with ADHD has been conducted primarily with school-based interventions (see DuPaul & Evans, 2008, for a comprehensive review). Some of these interventions are components of a comprehensive school-based psychosocial treatment program for adolescents with ADHD called the Challenging Horizons Program (CHP). Various aspects of the CHP, such as note taking (Evans, Pelham, & Grudberg, 1995), academic organization (Evans et al., 2009; Langberg, Epstein, Urbanowicz, Simon, & Graham, 2008), and homework completion (Raggi, Chronis-Tuscano, Fishbein, & Groomes, 2009), suggested beneficial stand-alone effects on academic functioning. Although there is evidence that these interventions can be effective stand-alone interventions, the intent when developed was that they be provided in the context of the total CHP.

The Challenging Horizons Program

The after-school model of the CHP involves students staying after school to participate in group and individual interventions targeting academic and social functioning. The program is held 2–4 days per week, for about 2 hours each, for a portion or all of the academic year. Family and parenting groups are part of the CHP and all interventions are intended to help students learn to master the skills necessary to independently succeed socially and academically. The studies of the CHP after-school program reported to date provide considerable evidence for improvements in social and academic functioning, although the effect sizes vary considerably and the samples have been relatively small (range = 7 to 48). In addition, the timing and duration of the treatment has varied across studies. Two studies evaluated the benefit of the CHP offered only during the fall term (16 weeks and 10 weeks, respectively; see Langberg et al., 2006; Molina et al., 2008); although the program was offered 4 days per week in the Langberg and colleagues study and only 2 days per week in the Molina and colleagues study. In the two studies conducted by Evans and colleagues the program was provided 3 days per week for 22 weeks (November–May; Evans, Axlerod, & Langberg, 2004) or for the entire academic year (Evans, Langberg, Raggi, Allen, & Buvinger, 2005). Rationale for these variations in timing, length, and frequency were not provided; although dosage of psychosocial interventions can affect outcomes, and there is evidence that time of the academic year interacts with academic functioning (Evans, Langberg, et al., 2005; Schultz, Evans, & Serpell, 2009). To date, studies indicate that the academic performance of students with ADHD tends to decline over the course of the academic year and one potential benefit of the CHP may be to prevent that decline.

The efficiency of the CHP may be optimized if the program is offered during the second half of the academic year when this decline is greatest. In addition, the effect sizes were notably smaller in the trial that lasted only 10 weeks and provided the program 2 days per week (Molina et al., 2008) compared to the other studies suggesting that a longer duration may be necessary. The benefits of family interventions reported in the two Barkley and colleagues studies (1992, 2001), as well as the success of the FCU (Stormshak et al., 2010), suggest that enhancing the family interventions provided as part of the CHP may increase the benefits for the participants. In the study by Langberg and colleagues (2006) there were no parent interventions, the study by Molina and colleagues (2008) included three 2-hour parent meetings, and the Evans, Langberg, and colleagues studies (2005) included 90-minute monthly parent meetings. Engaging parents and providing family-focused interventions ahead of the period of academic decline in the second half of the academic year may optimally take advantage of both the school-based and family interventions that have been supported for adolescents with ADHD.

The Present Study

In the present study we examined the efficacy of providing the FCU in the fall and the CHP from January through May during the spring semester. Improvements in social and academic functioning as well as symptoms were contrasted between this study and others in order to gain an improved understanding of the timing, frequency, and duration of providing the program under these conditions. These questions are particularly important for a treatment program such as the CHP as it is labor expensive to provide the interventions. In order to maximize our opportunity to compare the response to treatment of participants in this study to those reported in other studies, the participants in this study were disproportionately assigned to the treatment group. In addition to enhancing our ability to contrast effect sizes with previous studies, maximizing the size of the treatment group allowed us to investigate patterns of responding to some of the interventions in the program (Evans et al., 2009). Finally, we examined questions of teacher satisfaction and perceived effectiveness in order to continue to learn about modifications we can make to the CHP to improve acceptance in the schools and overall feasibility.

Method

Participants

Participants were 49 students in sixth through eighth grades in two Virginia middle schools recruited in two cohorts over 2 successive academic years. Thirty-one participants were randomly assigned to the treatment group over both years (n = 15 in year 1 and n = 16 in year 2) and 18 were randomly assigned to the control group (n = 9 in year 1 and n = 9 in year 2. Overall, there were 24 total participants in year 1 and 25 in year 2. Participants ranged in age from 10 to 13 at the time of entry into the study (Mdn = 11). Most participants (71%) were boys, and the majority of the sample was Caucasian (70%), followed by African American (14%), Latino (12%), and Asian (4%). Slightly more than half (55%) of caregivers were married at the time of intake, and the remainder reported to be divorced (19%), single (14%), or legally separated (6%). Additionally, one participant (2%) resided in a foster home at the time of intake, and the remaining two participant families (4%) did not report caregiver marital status. Parent income and education are reported in Table 1.

Table 1.

Participant Demographic, IQ, and Achievement Data at Intake

	Treatment Group (n = 31)				Control Group (n = 18)
	M	SD	Min	Max	M	SD	Min	Max
Demographics
Estimated family income	43,870	36,577	0	130,000	46,670	29,306	10,000	110,000
Mother level of education	13.4	1.9	10	18	13.6	1.5	12	16
Father level of education	13.5	2.8	10	18	12.9	2.6	8	18
IQ/Achievement
WISC-IV Full-Scale IQ	94.2	9.2	80	113	98.3	9.2	84	118
WIAT-II Reading	96.7	10.1	75	117	96.6	13.5	63	122
WIAT-II Num Oper	95.5	14.5	60	121	96.1	15.2	76	128
WIAT-II Spelling	93.8	7.8	75	104	97.7	15.7	72	129
WIAT-II Writ Exp	90.5	13.8	67	126	94.5	15.4	68	130

Note. Demographic data was provided by primary caregiver during intake evaluation. All variables appear to be normally distributed and no significant differences were found between the treatment and control group at intake. WISC-IV = Wechsler Intelligence Scale for Children (4th ed.); WIAT-II = Wechsler Individual Achievement Test (2nd ed.); Reading = reading subtest; Num Oper = numerical operations subtest; Spelling = spelling subtest; Writ Exp = written expression subtest.

Procedures

Recruitment

Two cohorts of participants were recruited during 2 consecutive school years, commencing with study announcement letters that were mailed to all families with children attending the two participating middle schools. Each cohort was involved for 1 academic year. Primary caregivers (hereafter “parents”) who responded to these mailings were screened by telephone interview for the likelihood of eligibility (i.e., child exhibited impulsivity, hyperactivity, or inattention) and then scheduled for an initial evaluation at a university-based clinic. The evaluation included psychological testing (child intelligence and academic achievement testing), diagnostic interviews with the parent and child, and rating scale completion by the parent and child. Criteria for inclusion in the study required that children (a) attended one of the two participating middle schools; (b) met diagnostic criteria for at least one subtype of ADHD based on the Kiddie Schedule for Affective Disorders and Schizophrenia (K-SADS; Kaufman et al., 1997), parent and teacher ratings on the Behavioral Assessment System for Children (BASC-2; Reynolds & Kamphaus, 2004), and ADHD Rating Scale-IV (DuPaul, Power, Anastopoulos, & Reid, 1998) were consistent with the diagnosis; (c) demonstrated academic or social impairment based on parent or teacher report on the Impairment Rating Scale (IRS; Fabiano et al., 2006); (d) demonstrated an IQ of 80 or above as measured by the Wechsler Intelligence Scale for Children–Fourth Edition (WISC-IV; Wechsler, 2003); and (e) did not meet diagnostic criteria for pervasive developmental disorder or any of the following: bipolar disorder, psychosis, substance dependence other than tobacco, or obsessive-compulsive disorder.

Data from the K-SADS were used to establish the primary ADHD diagnosis and all participants met criteria according to this interview. Data from the BASC-2 and ADHD Rating Scale–IV were used to supplement the parent interview data by confirming or discomfirming the presence or absence of ADHD symptoms. Sixty-two evaluations were completed and 11 were not eligible. Ten of them did not meet criteria for an ADHD diagnosis based on data from the K-SADS. One met criteria for ADHD, but the IQ was below 80. The mean scores on the ADHD Rating Scale–IV for those not accepted due to not meeting the diagnostic criteria were 7.9 (SD = 5.3) for the inattention factor and 4.7 (SD = 2.9) for the hyperactivity/impulsivity factor. These are more than two standard deviations below the mean for the participants accepted into the study and below the 75th percentile (DuPaul et al., 1998). The two senior investigators (first two authors) reviewed the results of each evaluation and unanimous agreement on ADHD diagnoses was required before determining eligibility (Leckman, Sholomskas, Thompson, Belanger, & Weissman, 1982). Two were accepted but withdrew very shortly after the initiation of the study. Participants accepted into the study were randomly assigned to participate in either the treatment or a community care control group at a ratio of 2:1, respectively, within each school.

Challenging Horizons Program

Both cohorts of participants randomly assigned to the CHP received the FCU in the fall semester, followed by the after-school program beginning in January of each school year. The FCU in the fall consisted of three 90-minute sessions. The three family meetings included an assessment that focused on family coping, family interactions, and how the family solves problems together. These meetings were scheduled individually with each family and conducted by doctoral-level clinicians in a university-based clinic. The techniques utilized within the FCU are informed by motivational interviewing techniques, and the goal of the intervention is to highlight potential ambivalences in each family’s motivation for making changes. At the end of the last session, all families discussed additional services that could help them reach their family goals. One of these options was to meet with the doctoral-level clinician on the research team to complete individual family interventions using the procedures reported by Barkley and colleagues (1992, 2001).

Beginning in January in both years of the study, students in the treatment condition participated in the after-school program, which was held at each school site twice per week for 2 hours and 15 minutes per meeting (excluding holidays and school breaks), and operating until the end of the school year. The schedule of activities in the after-school program included an education group, an interpersonal skills group, recreation, and individual meeting times between student participants and “primary counselors.” In the present study, primary counselors were undergraduate students who were responsible for establishing a relationship with each participant and then implementing specific group and individual behavioral interventions designed to target student disorganization and study skills. In addition, counselors communicated with their students’ teachers on a biweekly basis. The purpose of this communication was to share what was being done in CHP and gather information about participants’ progress and problems in the classroom. The information gathered from teachers was used to inform the interventions provided in CHP (e.g., missing assignments, behavior problems).

In the education group, students received instruction in four “phases” of academic skills, including study skills, note taking, and summarizing notes. Over time, these skills were increasingly applied to actual schoolwork. The interpersonal skills group is designed to help students identify personal social goals and then demonstrate progress toward those goals in interactions with peers, adults, and family members. Activities in the interpersonal skills group included social problem solving, individual goal setting, and frequent staff feedback (i.e., ratings) on actual performances in structured and unstructured social activities. The after-school program also included 30 minutes of recreation time during which the primary counselors and the students played competitive and cooperative games for the purpose of practicing social skills in real-life scenarios. As part of these activities, participants were taught sports skills relevant to basketball, soccer, baseball, and football in preparation for use in informal social settings outside of the program. During the entire after-school program, participants’ behaviors were monitored and tracked using a “behavior call” procedure, where program staff provided immediate verbal feedback on a predefined set of prosocial (e.g., complimenting) and antisocial (e.g., teasing) behaviors. The behavior calls in the CHP were a modification of a similar system used in the Summer Treatment Program (Sibley et al., in press). In the present study, a “levels” system of rewards and privileges was added to help shape behaviors that are typically expected in middle school classrooms. Additional details about the specific interventions provided in the CHP are available elsewhere (Evans et al., 2009; Sadler & Evans, in press).

To ensure adequate treatment adherence in the after-school program, procedures were outlined in a treatment manual provided to the graduate and undergraduate student staff. All staff members were required to score 90% or greater on a test of the manual content before working in the program. In addition, primary counselors were supervised by graduate students and the first two authors at the school sites and in weekly individual and group meetings. Observation-based treatment adherence measures were developed in the first year of the program and tested to ensure adequate interrater reliability, reaching an average absolute agreement rate of 88.5%. Observers recorded whether procedures were followed (yes/no) and assessed a variety of aspects of implementation including the amount of time spent in activities, spacing of the counselors during activities, proper implementation of intervention steps, and behavior calls. In the second year, these checklists were formally implemented, with independent raters observing the counselors implementation of the procedures on 18 separate days. Results of these observations suggested that treatment adherence was adequate across 90.5% of the 81 specific checklist items.

In addition to the FCU and the after-school program, treatment participants were free to pursue treatment in the local community during their year of participation. At the end of the study, roughly half of treatment group families (n = 16; 52%) reported use of medications for their child during the school year; however, 15 of the youth were taking medication at the beginning of the study as well. In all but one of these 16 instances, participants received a medication typically prescribed for ADHD (e.g., psychostimulant). In two of these 16 instances, an additional psychiatric medication was prescribed (in addition to the stimulant) that is not typically used to treat ADHD (e.g., antidepressant).

Community Control Condition

The families of participants randomly assigned to the community control condition received a packet of contact information for local community resources. Additionally, a summary of the intake evaluation was sent to the school psychologists at the respective schools (all parents consented to this release of information). Beyond these efforts, participants in the control condition did not receive any services other than what the families independently pursued in their schools and communities. Interviews conducted with parents at the end of each school year indicated that 12 of the 18 participants (67%) received medications at some point during the study (nine were taking medication at the beginning of the study). In almost all instances (n = 11; 92%), control participants who took medications during the study received medication to treat ADHD and, in some cases, an additional psychiatric medication (n = 3; 27%), but in one case (8%), the participant received only an antidepressant.

Measures

Assessments to determine outcomes and diagnosis included several parent- and teacher-rating instruments, a diagnostic interview and an assessment of cognitive abilities, as well as school measures (grades) and a satisfaction survey. These measures are listed below in alphabetical order and briefly described.

ADHD Rating Scale–IV

The ADHD Rating Scale–IV (DuPaul et al., 1998) is a parent- or teacher-rated symptom checklist for the symptoms of ADHD. The authors provide normative data and report adequate validity and reliability data in the manual. Only the parent version was used in this study to assist in the diagnostic process.

Behavior Assessment Scale for Children, Second Edition (BASC-2)

The BASC-2 (Reynolds & Kamphaus, 2004) is a comprehensive behavior rating scale with separate versions that can be completed by parents and adolescents in about 20 minutes each. Each version of the scale has multiple item scales and yields several subscales that are useful for diagnosis and assessing outcomes. There are also several adaptive scales, such as social skills and study skills. The BASC-2 scales have good psychometric properties and an excellent normative database (Reynolds & Kamphaus).

The Disruptive Behavior Disorders Questionnaire (DBD)

The DBD (Pelham, Gnagy, Greenslade, & Milich, 1992) is a narrow-band rating scale that requires respondents to rate target children on behavioral symptoms related to ADHD, oppositional defiant disorder (ODD), and conduct fisorder (CD). Research on previous iterations of the DBD suggests strong internal reliability as well as strong negative predictive power (NPP) and moderate positive predictive power (PPP; Pelham et al.).

Impairment Rating Scale (IRS)

The IRS (Fabiano et al., 2006) is a brief rating scale that assesses several broad areas of impairment, including academic impairment, social impairment, and impairment in adult–child relationships. Most items on the IRS are scored along a 7-point scale, anchored on one end by no problem, definitely does not need treatment and on the other end by extreme problem, definitely needs treatment. There are separate parent and teacher versions of the IRS, and each focuses respectively on home and school impairment. The IRS has been found to be sensitive to treatment-related changes, with good test–retest reliability (r = .74, −.96) and good convergent and discriminant validity (PPP = .90, NPP = .74). Further, the IRS was found to have moderate to high correlations when compared to other teacher instruments that measure impairment (Fabiano et al.). In the present study, we focused on the subscales designed to measure social and academic impairment because these areas were targeted with the CHP interventions.

Classroom Performance Survey (CPS)

The CPS (Children and Adults With Attention Deficit Disorder, 1996) is a 20-item teacher survey designed to assess the unique performance demands of secondary schools. Each item on the CPS inquires about classroom behavior, such as participation in activities, paying attention, and completing classwork. Each item is rated using a 5-point Likert response format ranging from 1 (always) to 5 (never). As such, higher scores represent poorer classroom performance. Although there are no psychometric data published on this measure, it is recommended by experts in the field and targets key areas of performance that middle school students with functional impairment frequently fail (Robin, 1998, p. 106). In the present study, we computed a single raw score by adding together all of the items. Analyses were conducted for teacher ratings from language arts, math, social studies, and science classes (hereafter “core courses”) separately, as well as averaged together.

Kiddie Schedule for Affective Disorders and Schizophrenia (K-SADS)

The K-SADS (Kaufman et al., 1997) is a widely used semistructured clinical interview for assessing psychopathology in youth according to DSM-IV diagnostic criteria. Psychometric properties are well established with test–retest reliability coefficients for ADHD in the good range (.63) and interrater scoring agreement over 90% (Kaufman et al.). A doctoral-level clinician conducted the K-SADS interviews with the parents during the initial assessment.

School Grades

Grades for each participant in the study were collected directly from the school offices at both sites. Grades were converted into grade point averages (GPAs) for each of the core courses, where A = 4.0, B = 3.0, C = 2.0, D = 1.0, and F = 0.0.

Teacher Evaluation of Services

Teacher satisfaction with the CHP was assessed using a brief, eight-item satisfaction questionnaire developed by the authors and distributed at the end of the school year to core subject teachers with treatment students in their classes. The first seven items use 4-point Likert response formats, with higher scores indicating greater satisfaction. The higher scores are anchored with statements such as very helpful or very satisfied, whereas the lower scores are anchored with statements such as no, it seemed to make things worse, or quite dissatisfied. The final item is an open-ended question that invites additional comments.

Wechsler Intelligence Scale for Children–Fourth Edition (WISC-IV)

The WISC-IV (Wechsler, 2003) is an individually administered test of cognitive abilities that is considered the standard in the field for children ages 6 to 16. Psychometric properties are solid and are reported in the manual. The standard battery was administered by a qualified school psychologist and the full-scale IQ was used to contribute to determining eligibility for this study.

Results

Analyses were conducted to compare symptom and functioning outcomes between the treatment and control conditions using parent and teacher ratings on the DBD and IRS, as well as teacher ratings on the CPS. For all participants, there were six measurement occasions, beginning with the intake assessment and then once each month from January through May. Descriptive statistics for each outcome measure over each measurement occasion are provided in Table 2. All outcome data were analyzed using hierarchical linear modeling (HLM) in SAS 9.1 (proc mixed procedure). Time points were nested within students, and the student factor was modeled as random. Time was modeled as a linear trend; the quadratic trend of time was not statistically significant at the .05 level for any outcome. In each case, we investigated Condition × Time interactions, because a significant interaction would indicate the treatment and control groups changed differently over time. Additionally, a potential Condition × Time × Site interaction was investigated in all analyses to assess the possibility of differential impact of the CHP at the two participating schools, but only significant findings are reported. Although site would conceptually be a random factor, it was modeled as a fixed factor because there were only two sites. Similarly, cohort effects were also evaluated and revealed no significant differences; accordingly, cohort was not included in the outcome analyses.

Table 2.

Descriptive Statistics for Outcome Variables Over Six Measurement Occasions

Measure		Time 0		Time 1		Time 2		Time 3		Time 4		Time 5
		M	SD	M	SD	M	SD	M	SD	M	SD	M	SD
DBD-IA (Parent)	T	20.4	4.0	20.2	5.1	18.1	7.1	17.5	5.9	16.1	6.7	16.4	5.6
	C	19.1	4.9	18.4	4.0	17.1	5.3	17.2	5.0	15.9	7.5	16.9	6.5
DBD-HI (Parent)	T	14.5	4.4	10.6	4.1	11.1	4.1	11.0	4.6	10.3	4.9	9.9	4.0
	C	12.8	5.4	11.9	3.4	12.8	6.1	11.6	5.6	11.6	6.0	12.5	6.2
DBD-IA (Teacher)	T	14.7	7.7	14.9	7.1	13.8	7.4	12.4	6.9	13.9	7.4	13.9	6.9
	C	13.1	7.7	14.5	7.3	13.3	6.9	12.4	6.0	13.2	7.3	13.6	6.2
DBD-HI (Teacher)	T	9.0	7.0	8.4	6.5	8.8	7.0	7.8	6.5	8.2	6.5	8.4	6.5
	C	6.7	6.7	7.8	7.2	7.0	7.4	6.5	6.9	6.9	6.4	7.5	6.4
IRS Social (Parent)	T	3.6	1.8	3.3	1.9	3.0	1.6	3.4	1.5	2.4	1.6	2.4	1.9
	C	2.9	2.1	2.0	2.0	2.2	2.3	2.5	1.9	2.7	2.4	2.2	2.1
IRS Social (Teacher)	T	2.3	2.0	2.6	1.8	2.3	1.7	2.2	1.7	2.2	1.8	2.2	1.8
	C	1.8	1.9	2.7	2.0	2.4	1.9	2.2	1.8	2.3	1.9	2.4	1.9
IRS Academic (Teacher)	T	4.0	2.0	4.1	1.8	3.8	1.8	3.4	1.8	3.6	1.9	3.8	2.0
	C	3.1	2.1	3.8	2.1	3.7	2.1	3.2	2.2	3.5	2.1	3.4	2.1
CPS (Teacher)	T	51.3	14.3	52.1	15.2	50.6	15.0	46.7	14.9	50.7	15.9	51.1	15.8
	C	48.7	16.5	53.9	14.8	52.7	15.8	49.1	13.8	50.2	13.9	52.4	11.7

Note. Statistics are based on observed measurements and may differ from estimates in the HLM analyses.

DBD = Disruptive Behavior Disorders Rating Scale; IA = inattention subscale; HI = hyperactivity/impulsivity subscale; IRS = Impairment Rating Scale; CPS = Classroom Performance Survey; T = treatment; C = control.

We examined the potential for a Condition × Time × Medication Status interaction; however, in no cases did medication status significantly moderate outcomes. There was a significant main effect of medication use on several of the variables including parent ratings of inattention on the DBD and parent ratings of peer relationships, parent–child relationships, family impairment, and overall severity on the IRS. In each case, the severity of the impairment and symptoms was greater for participants taking medication. If medication status was not balanced across conditions, it could potentially bias the estimation of the treatment effects; however, rates of medication use were equivalent at baseline (50% control and 48% treatment) and similar throughout the study (67% control and 52% treatment).

We analyzed teacher satisfaction data, which were collected at the end of both school years, using analysis of variance (ANOVA) techniques in SPSS 15.0. We also analyzed end-of-year student report card data using a life table analysis because previous research suggests that psychosocial interventions such as those used in the CHP may be most helpful in preventing serious academic decline, rather than improving student performance relative to baseline performance (Schultz et al., 2009).

Functional Impairment

In order to evaluate the effect of treatment on ADHD-related impairment, similar analyses were conducted using parent and teacher ratings on the IRS. There were no significant Condition × Time interactions for the parent ratings of social and academic impairment on the IRS, and no significant changes were noted in the teacher ratings of participants’ social impairment. However, analyses of teacher ratings of participants’ academic progress revealed a significant Condition × Time interaction in the predicted direction, t(213) = −2.04, p = .04. Specifically, participants in the treatment condition were rated as appreciably more impaired at intake, but these ratings declined (d = −0.45) to a level below that of participants in the control condition (d = 0.10) by the end of the school year, suggesting that teachers observed less academic impairment in the treatment group as compared to the control group.

Classroom Performance

Teacher ratings on the CPS were also analyzed using HLM (proc mixed procedure) in SAS 9.1. When all core course teacher ratings were averaged together, there were no significant Condition × Time interactions. Since previous studies have documented high rates of inconsistency between middle school teacher ratings of student symptoms and impairments (e.g., Evans, Allen, Moore, & Strauss, 2005), the teacher ratings were also analyzed separately by class. The Condition × Time interaction was statistically significant for language arts, t(188) = −3.09, p < 0.01, and social studies, t(179) = −2.26, p = 0.02. In both cases, impairment in classroom performance decreased over time for the treatment group (d = −0.37 and −0.23, respectively), but increased for the control group (d = 0.40 and 0.29, respectively). A similar pattern emerged for math teachers, although the effect was not statistically significant. Although there was no statistically significant Condition × Time interaction in science teacher ratings, there was a three-way Condition × Time × Site interaction, t(166) = 2.02, p = 0.04. Specifically, impairment increased for the control group and decreased for the treatment group at one site, while the opposite was true at the other site, washing out the overall Condition x Time interaction. Additionally, in social studies there was a three-way Condition x Time x Site interaction, t(179) = 2.40, p = 0.02, suggesting that the significant Condition × Time interaction was mostly due to an interaction at one site (treatment d = −0.35, control d = .76) and not the other (treatment d = −0.12, control d = −.13).

School Grades

Average core course GPAs were used as an additional outcome measure of academic functioning. In previous research, we have noted a curvilinear trend in GPA, characterized by declining grades in late fall and spring followed by partial recovery late in the year (Evans, Langberg, et al., 2005). Additionally, the second semester grading periods corresponded to the after-school intervention program, so the trend was predicted to change for the treatment group during this time. Thus, we analyzed a two-part trend over time by modeling grade as a function of an intercept (predicted grade halfway through the year), experimental condition, linear trend of time in the first semester, linear trend of time in the second semester, condition by first semester, and condition by second semester. The results of this analysis for core GPA suggested that there were no significant differences between the trends of the treatment and control groups during the first semester, t(220) = 1.04, p = .30, or the second semester, t(220) = 0.93, p = .35. Follow-up analysis of the constituent classes revealed that differences between treatment and control over time only reached significance in one instance; specifically, treatment participants appeared to outperform control participants in the second semester in their math classes, t(225) = 2.51, p = 0.01. The magnitude of improvement over first semester performance for the treatment group was small (d = .14), as most of the apparent benefit of treatment was relative to a performance decline in the control group (d = −.39).

Similarly, previous research has indicated that even in cases when statistically significant benefits are not found using continuous measures of GPA, differences may exist in the proportion of participants who experience failure events over time (Schultz et al., 2009). In the present study, the proportion of students who experienced failing grading periods—defined as overall GPA dropping below 1.0—at some point in the school year was 32% in the treatment group and 39% in the control group. Although these differences are in the predicted direction, life table analysis revealed that the trend did not reach statistical significance (Z = 0.18, p = .67).

ADHD Symptoms

Changes in ADHD symptoms were assessed over time using parent and teacher ratings on the DBD. The estimated intercepts and slopes of the treatment and control groups for these and all rating scale outcomes are reported in Table 3. The results suggest that there were no significant Condition × Time interactions for teacher ratings of either inattention or hyperactivity–impulsivity. On parent ratings, a significant Condition × Time interaction was found on ratings of hyperactivity–impulsivity, t(163)= −3.37, p < .01, but not for ratings of inattention. Specifically, parent ratings of hyperactivity–impulsivity remained stable for participants in the control condition (d = −0.16), while parent ratings of participants in the treatment condition indicated a decline in symptoms over time (d = −1.03). In follow-up analyses of the parent ratings, a significant three-way interaction was noted when school site was entered into the model (Condition × Time × Site),t (163) = 2.32, p = .02. Specifically, improvement in ratings for treatment participants at one of the schools (d = −1.76) far exceeded improvement for treatment participants at the other (d = −0.40), suggesting differential impact of the CHP as a function of unspecified contextual factors.

Table 3.

Estimated Intercepts and Slopes When Modeling Change in Parent and Teacher Ratings of Participant Performance Over Time

Outcome	Dependent Measure	Intercepts		Slopes
		Control	Treatment	Control	Treatment
ADHD Symptoms	DBD-IA (Parent)	18.6	20.4	−0.509	−0.998
	DBD-HI (Parent)	12.2	13.8	−0.148	−0.976*
	DBD-IA (Teacher)	13.1	14.7	0.124	−0.376
	DBD-HI (Teacher)	7.2	8.5	0.043	−0.018
Functional Impairment	IRS-Social (Parent)	2.7	3.6	−0.069	−0.183
	IRS-Social (Teacher)	2.2	2.3	0.049	−0.010
	IRS-Academic (Teacher)	3.4	4.1	0.030	−0.144*
Classroom Performance	CPS (Language Arts)	48.6	53.0	1.091	−1.015*
	CPS (Social Studies)	46.1	50.4	0.853	−0.671*
	CPS (Math)	48.7	47.6	1.077	−0.088
	CPS (Science)	58.2	54.5	−0.892	−0.041
	CPS (All Teachers)	50.8	51.1	0.460	−0.389

Note. The intercept represents the average outcome measure at the initial measurement occasion, whereas the slope represents the average change in the outcome measure for each 1-month increment of time. Slopes for outcome measures on different scales are not comparable. ADHD = attention-deficit hyperactivity disorder; DBD = Disruptive Behavior Disorders Rating Scale; IRS = Impairment Rating Scale; CPS = Classroom Performance Survey.

^*p < .05.

Teacher Satisfaction

The third question being addressed in this study has to do with the degree of teacher satisfaction with the program. All core course teachers who taught the treatment participants (n = 89) received the Teacher Evaluation of Services at the end of school year. The return rate was 76% (n = 68). The mean response to all items was close to three (range = 2.8 to 3.1), which was typically anchored by mostly satisfied. Based on ANOVA analysis, there were no significant site differences on any of the items; however, there were significant differences between the 2 years of the study, with teachers’ satisfaction with the program significantly greater during the second year than the first on four of the seven items. Specifically, a main effect of time was noted for the items related to whether teachers would recommend the CHP to friends (Item 3; F = 8.098, p = .006, η² = 0.11), satisfaction with the amount of help students received (Item 4; F = 10.528, p = .002, η² = 0.14), satisfaction with interactions with the CHP staff (Item 6; F = 4.667, p = .004, η² = 0.12), and on the item related to the responsiveness of the CHP staff (Item 7; F = 2.259, p = .011, η² = 0.10). In all instances, the responses suggested that there were significant improvements in teacher perception of the program from year 1 to year 2. The mean scores for the other three items improved from the first to the second year, but not to a statistically significant degree.

Parent Involvement

Finally, we examined the number of parents who chose to participate in family treatment sessions provided by our research clinician or obtain assistance entering other services from outside agencies. Out of the 31 participants in the treatment condition who completed the FCU, only two began individual sessions with the clinician to target specific behavioral concerns, and only one family requested assistance obtaining services in the community based on the results of their FCU. The first two families requested extra assistance addressing their child’s behaviors at home, and the latter family requested more intensive family therapy to address concerns in the marital relationship.

Discussion

The present study examined the benefits of a school-based psychosocial treatment program designed to improve academic and social functioning of young adolescents with ADHD. The results indicate that after 5 months of the CHP following the FCU, there were benefits for the treatment group, but improvement appears to be less than previously reported when the program was provided for the entire academic year (Evans, Langberg, et al., 2005) or 4 days per week during the first semester (Langberg et al., 2008). In addition, as in previous studies, there appears to have been a decline in academic functioning beginning in the fall semester. These problems began earlier in the year in this study than in the Evans, Lanberg, et al. (2005) study, but the fall decline was similar to the one reported in the Langberg and colleagues (2008) study. Consistent with other studies of the CHP, teachers reported being mostly satisfied with the program, and satisfaction increased from year 1 to year 2. Finally, in spite of providing the FCU for all families in the treatment group, very few pursued additional services.

Academic and Social Functioning

We analyzed data from five measures of school functioning including parent and teacher ratings of social impairment on the IRS, teacher ratings of school impairment on the IRS, teacher ratings on the CPS, and grades. Parent and teacher ratings of social impairment on the IRS revealed no statistically significant treatment benefit, although a review of between-group effect sizes indicates that the treatment group improved more than the control group according to both teachers and parents. At baseline the treatment group was rated as more severely socially impaired than the control group (teacher ratings d = .29; parent ratings d = .37) and at the end of treatment teachers rated the treatment group as less impaired than the control group (d = −.11) and parent ratings indicate a reduced difference between the groups with the control group still less impaired (d = .10). The within-group effect size for the treatment group (d = .66) is equivalent to the one reported in the study of the year-long CHP (d = .70) as are the improvements in parent rating of social functioning for the control group (this study d = .33; Evans, Langberg, et al., 2005, d = .21). Teacher ratings of social functioning indicated no improvement for either group and teacher scores suggest less perceived social impairment for all participants than parent reports. Langberg and colleagues did not analyze the parent ratings of social impairment since the participants were not in the impaired range at baseline (score ≥ 3 on IRS). These findings indicate that offering the CHP for 5 months of the academic year instead of the entire year may make little difference in terms of impact on social functioning. Although these treatment effects are not statistically significant, the effect sizes of parent reports indicate some benefit of treatment on social functioning. Given the lack of effective treatments for the social impairment typically associated with children and adolescents with ADHD, this positive finding indicates that the interventions targeting social impairment in the CHP warrant additional development and evaluation.

The three measures of academic functioning indicated a trend toward improvement for the treatment group. Teacher ratings of academic impairment on the IRS indicated a moderate within-subject effect size for the treatment group (d = 0.45) and a statistically significant treatment effect. Although the effect of the intervention was modest, the slopes defining change in teacher ratings of academic impairment were headed in opposite directions for the two groups (see Table 3). The results of the analyses with the CPS indicate that teacher ratings of classroom performance improved for the treatment group according to a majority of the teachers accounting for the statistically significant Group × Time interactions for language arts and social studies teachers. Teacher ratings of participants in the control group generally became worse, but there were exceptions in the ratings of some of the science and social studies teachers. There were a couple of statistically significant Group × Time × Site interactions indicating some inconsistency in these findings, but in all of these situations except one, the treatment group improved. In spite of the inconsistencies, the findings with the CPS are consistent with the trend of the classroom measures indicating benefit for the treatment group.

Our experience indicates that grades matter more to parents, students, and teachers than most other academic outcomes, but there are difficulties when using grades as a measure of treatment response. For example, an “A” in an academically challenging honor’s class certainly means something different than an “A” in a special education class that has no expectation for work outside of the classroom. Nevertheless, there have been reports of the CHP preventing a decline in grades (e.g., Evans, Langberg, et al., 2005; Molina et al., 2008; Schultz et al., 2009). In the present study, there were no such benefits for the treatment group, with the exception of math grades in the second semester. Overall, most initial failure events (12 of 17) occurred in the first half of the school year.

Although teacher ratings of academic functioning significantly improved for the treatment group compared to the controls, the benefits for the treatment group were largely preventative as the decline in the control group was substantial. Previous studies of the CHP did demonstrate a decline in academic functioning for the control groups; however, they also reported improvements in academic functioning for the treatment groups. The study conducted during the fall semester (Langberg et al., 2008) reported moderate within-group effect sizes (d = .74) and the year-long CHP study (Evans, Langberg, et al., 2005) reported similar findings (d = .76). As noted in the previous paragraph, many of the participants in this study were experiencing failure in the fall semester before the after-school program started. These findings suggest that in order to optimally benefit academic functioning, the CHP should begin at the start of the school year.

Symptom Improvement

Parent ratings of both hyperactivity/impulsivity and inattention on the DBD improved for treatment participants as indicated by negative slopes of nearly 1.0 or an improvement of nearly 1 point per month on a scale with a range of 27. However, due to improvements in parent ratings of inattention with participants in the control condition (slope = −.509), only analyses of the parent ratings of hyperactivity–impulsivity resulted in statistically significant treatment effects. The improvement in parent ratings of inattention in the control sample has not been reported in other studies of the CHP (Evans, Langberg, et al., 2005; Evans, Serpell, Schultz, & Pastor, 2007; Langberg et al., 2006; Molina et al., 2008) and certainly confounds interpretation. Symptom improvement in this study was as large or larger than that reported in other CHP studies indicating that the time of year and dosage of the CHP may be less of a factor for impacting symptom change than it is for impacting functioning.

Family Treatment

This study included the FCU as part of the CHP interventions for all participants in the treatment condition for each cohort in the fall. In spite of these efforts to increase the focus on families as compared to previous CHP studies, there was very little interest by parents in family treatment following the FCU. In order to identify why these services were rejected by participants we reported this finding to members of our community development team (Evans, Green, & Serpell, 2005) consisting of parents, educators, and mental health care providers. They suggested that parents may be best engaged by beginning with a group treatment that may not be as intimidating to them as the individual FCU sessions. The team hypothesized that participation in the group may adequately “hook” parents and increase the likelihood that they engage in family or individual interventions following the group.

Summary of Outcomes

The results of this study indicate that providing the CHP for 5 months during the second half of the academic year may result in benefits in social functioning and symptom reduction similar to offering it during the entire academic year; however, the academic benefits of providing it at this time appear to be less than other times of the year. In addition to the smaller effects on academic functioning in this study than previous work, the outcome scores at posttreatment for academic impairment indicated that the participants were still performing poorly in spite of the improvement in the treatment group. Scores on the IRS were still above the impairment threshold on the instrument and data from the CPS indicated that problems persisted. It is important to note that it is not possible to determine which aspects of the CHP (e.g., FCU, academic interventions, interpersonal skills group) were most salient in producing the improvements.

Medication use did not meaningfully differ between the groups and there were no significant interactions on any of the outcome variables that included medication. There was a significant main effect of medication on many parent ratings of symptoms and impairment indicating that those taking medication were rated as more severe than participants not taking medication. This may reflect increased motivation to pursue medication treatment for parents who perceive their children to be most impaired. Given the lack of interactions with treatment effects, the medication did not appear to impact outcomes.

Satisfaction

Teachers reported that they were generally satisfied with the CHP and believed the program to be effective. For example, 67% of all teachers reported that the CHP was somewhat or mostly helpful for their students, but perhaps the most interesting finding is the increase in teacher satisfaction from one year to the next. Conceivably, the quality of relationships between the CHP staff and the school staff improves over time. Nevertheless, the frequency of communication with teachers was the most frequently noted area for improvement. For example, one teacher reported, “I really do not know what all is being addressed with the kids. It would be nice to get an update now and then about what help they are getting.” Typically, staff asked teachers about students’ upcoming assignments or performances on tests and quizzes once every other week. Another common concern voiced by teachers was the perceived burden created by the paperwork (i.e., monthly rating scales). One representative teacher comment was “Apparently some of the [CHP staff] don’t realize the amount of paperwork we do on a regular basis!” Such comments suggest that future iterations of the CHP may benefit from pragmatic and streamlined approaches to outcomes measurement and increased staff–teacher communication.

Limitations

One of the limitations of this study is the inability to specifically analyze moderators and mediators that may have affected outcomes. For example, although medication use was included in the analyses, the sample size was inadequate to consider variations in types of medication, dosage, and adherence to prescribed schedules. In addition, there may have been differences in the practices of educators between the two schools that may have impacted the responsiveness of the participants to the interventions. As the CHP is comprised of a variety of interventions, the outcomes may have been primarily attributable to one or a few of the interventions. The careful measurement of variables that may moderate or mediate outcomes in studies with larger samples than this one will be important in order to understand the impact of the CHP.

Another limitation involves the lack of outcome data provided by sources blind to treatment condition. This problem is common when interventions include parents and teachers since these individuals tend to be the most relied upon sources for assessments. The problem is even more challenging with adolescents than young children as parents are frequently less involved with secondary education than elementary education resulting is less direct knowledge of performance. In addition, adolescents tend to conduct more of their social interactions unobserved by parents and teachers than elementary school-age children making the assessment of social functioning difficult. The identification of sources of assessment data for school and social functioning that can be kept blind to the intervention condition will be a valuable advance in our field and improve our ability to evaluate interventions free from this potential confound.

There are also questions about the feasibility of the after-school model of the CHP. Schools may not be adequately staffed to operate such a program and community organizations that do provide after-school programming usually do not have staff with the necessary mental health expertise. It may be more cost-effective to provide some of the individual CHP interventions during the school day than to offer an after-school program. Comparing the benefit of providing individual CHP interventions to providing the after-school model using cost–benefit analyses may reveal answers about the efficiency of these interventions as a function of the child’s characteristics.

Conclusion

These findings suggest that further development of the after-school model of the CHP should include starting the program at the beginning of the school year and continuing it for most or all of the academic year. Additional data collection may suggest other modifications to timing and duration that may warrant testing. Improved systems for communication and coordination of efforts with teachers are likely to enhance teacher satisfaction with the program and may improve the positive impact on academic functioning. Continued development of models for involving parents in treatment is needed if we are going to be able to take advantage of their potential to improve their children’s behavior. Although parent involvement is considered critical for treatment focused on young children, their potential to meaningfully augment treatment of adolescents with ADHD may be limited.