Evaluation of the Homework, Organization, and Planning Skills (HOPS) Intervention for Middle School Students with ADHD as Implemented by School Mental Health Providers

Abstract

The purpose of the study was to evaluate the Homework, Organization, and Planning Skills (HOPS) intervention for middle school students with Attention-Deficit/Hyperactivity Disorder (ADHD) as implemented by school mental health (SMH) providers using a randomized trial design. Seventeen SMH providers from five school districts implemented the HOPS intervention. Forty-seven middle school students with ADHD (grades 6–8) were randomly assigned to receive the HOPS intervention or to a waitlist comparison group. Parent and teacher ratings of organizational skills and homework problems were collected pre- and post-intervention and at a 3-monoth follow-up, and school grades were also collected. Intervention participants demonstrated significant improvements relative to the waitlist comparison across parent-rated organized action (d = .88), materials management (d = .63), planning (d = 1.05), and homework completion behaviors (d = .85). Intervention participants did not make significant improvements relative to the comparison group according to teacher ratings. SMH providers were able to implement the HOPS intervention with fidelity despite the fact that no formal ongoing consultation was provided.

Children with Attention-Deficit/Hyperactivity Disorder (ADHD) typically experience clinically significant impairment in the school setting as evidenced by lower school grades and achievement scores and higher rates of school dropout in comparison to their peers (DuPaul & Stoner, 2003; Frazier et al., 2007). Emerging evidence suggests that organizational skills problems characteristic of children with ADHD are strongly associated with academic impairment. Organizational skills is a broad term that encompasses both the ability to manage materials and belongings (e.g., transfer of homework assignments to and from school) and time (e.g., planning ahead to ensure adequate time is spent studying). Parent and teacher ratings of materials management and planning behaviors have been shown to predict school grades, with materials management behaviors predicting grades above and beyond the impact of intelligence (Langberg, Epstein et al., 2011). Further, parent ratings of homework materials management in elementary school have been shown to predict grade point average (GPA) in high school (Langberg, Molina et al., 2011). The association between homework materials management and academic performance is present even after controlling for stimulant medication use and receipt of school services (Langberg, Molina et al., 2011).

Problems with organization tend to increase in severity as children progress through school (Booster, DuPaul, Eiraldi, & Power, 2010; Langberg et al., 2010). In particular, problems with organization often escalate following the transition to middle school (Evans, Serpell, & White, 2005). The transition to middle school is marked with numerous environmental changes and represents a significant challenge for children with externalizing behavior problems (Langberg, Epstein, Altaye et al., 2008; Moilanen, Shaw, & Maxwell, 2010). A greater number of teachers, increased demands for independence, and larger workloads make the transition to middle school difficult (Evans, Langberg, Raggi, Allen & Buvinger, 2005; Evans, Serpell et al., 2005). Middle school children with ADHD frequently lose homework assignments or fail to turn them in on-time, misplace school materials such as books, pencils, and classwork, and procrastinate and fail to adequately prepare for tests (Evans et al., 2009; Langberg, Epstein et al., 2011).

Given the association between organizational skills and academic performance (Langberg, Vaughn et al., 2011), and the fact that medication does not normalize these problems (Abikoff et al., 2009), psychosocial interventions have been developed. Organizational skills interventions have typically focused on academic aspects of organization, such as classroom preparation, homework management, and managing time during and after school, in addition to the physical organization of school materials. Strategy and skills training are typically the core features of organizational interventions for children with ADHD. Behavioral therapeutic techniques such as rehearsal, prompting, shaping and contingency management are used to teach and promote skills use and their generalization.

Most organizational skills interventions include point systems or token economies to monitor and reward adherence to a structured organizational skills system (see Langberg, Epstein, & Graham, 2008 for a review). On a periodic basis, children are awarded points for meeting operationalized goals. Points are typically applied towards purchasing rewards. The ultimate goal of all programs is to reduce the frequency of monitoring and overt reward and/or to transfer monitoring and reward responsibilities from the clinician to school staff or to a parent/guardian. To this end, many organizational skills programs for children include intervention with parents/guardians or school mental health (SMH) providers (e.g., Gureasko-Moore, DuPaul & White, 2006; 2007; Pfiffner et al., 2007). Parents or school staff are trained to take over the monitoring of organization and application of rewards in an effort to promote skills generalization.

Organizational skills training has been included as part of a number of multicomponent interventions for children with ADHD (e.g., Evans, Langberg et al., 2005; Evans, Serpell, Schultz, & Pastor, 2007; Evans et al., 2009; Hechtman et al., 2004; Pfiffner et al., 2007; Power et al., under review). These interventions are multicomponent because in addition to targeting organization and time management, they often target behavior problems, social skills, and other educational skills (e.g., study skills). These multicomponent interventions have been shown to lead to significant improvements in interpersonal functioning and organizational skills (Evans et al., 2009; Pfiffner et al., 2007; Pfiffner et al., 2011) as well as decreases in parent and teacher ratings of overall academic impairment (Evans, Langberg et al., 2005). Given that it is difficult to disentangle the specific impact of organization skills training versus other interventions in multicomponent studies, the literature review below focuses on interventions designed specifically to target organizational skills.

Gureasko-Moore, DuPaul, and White (2006, 2007) used a multiple baseline design to examine the efficacy of self-management training for improving the organizational skills of young adolescents with ADHD. Participants were taught to monitor and record their own classroom preparation and homework behaviors daily on checklists. Participants reviewed the checklists with an SMH provider and operationalized goals for improvement. The efficacy of this intervention was evaluated across two studies using three and six middle school students (M_age = 12) respectively. Post intervention, all participants were completing classroom preparation behaviors nearly 100% of the time. Similarly, participants exhibited low percentages of homework behaviors at baseline (range = 18–66%) and improved to nearly 100% by completion of the 6-week intervention.

Abikoff and Gallagher (2008) pilot-tested a 10-week, 20-session clinic-based individual intervention designed to improve physical organization of materials, time-management, assignment tracking and planning skills. Twenty children in grades 3–5 diagnosed with ADHD received the intervention delivered by clinical psychologists. This pilot study focused on evaluating feasibility, acceptability and effectiveness, and so no comparison group was included. Participants made significant improvements on parent and teacher ratings of organizational skills and on parent ratings of homework performance. Parents and teachers were highly satisfied with the intervention and attendance was high with no children dropping out of the intervention. In addition, the investigators recently completed a large randomized trial of the organizational skills intervention. Participants (N = 158) in grades 3–5 were randomly assigned to three conditions, including one of two different organizational skills interventions or a waitlist comparison. Preliminary results show that children with ADHD in both of the organizational skills intervention groups made significant gains according to parent and teacher rated organizational skills, homework problems and academic proficiency (Abikoff et al., 2011).

Langberg, Epstein, Urbanowicz, Simon, and Graham (2008) evaluated the efficacy of an 8-week intervention called the Homework, Organization, and Planning Skills (HOPS) intervention for middle school age students with ADHD. Thirty-seven students (M_age = 11) were randomly assigned to receive the HOPS intervention (n = 24) or to a waitlist comparison (n = 13). The intervention focused on improving participants’ physical organization (i.e., bookbag, binder, and locker) and homework management (i.e., accurate homework and test recording and planning) and was delivered by undergraduate college students as a school-based after-school program. The intervention included two parent training sessions that focused on transferring behavior monitoring responsibilities and contingency management to the home setting. According to parent ratings, intervention participants in this study made large gains in materials organization and homework management relative to the comparison and these improvements were largely maintained at an 8-week follow-up. Further, participants in the intervention group made small to moderate improvements in overall GPA. Teachers rated minimal improvements in academic performance that were not statistically significant.

In summary, organizational skills interventions appear to be highly effective at improving organization and time management skills and homework problems in children and young adolescents with ADHD. Youth with both ADHD-Inattentive Type and ADHD-Combined Type have been included in prior studies and, to date, there is no evidence for differential intervention effectiveness. There is also some evidence that these improvements translate into gains in overall academic performance as measured by teacher ratings and school grades (e.g., Abikoff et al., 2011; Langberg et al., 2008). However, the primary limitation of the organizational skills intervention work completed to date is that the interventions have been implemented by trained research staff under controlled conditions. For example, in the Langberg et al. (2008) study, research staff received in-depth training and daily observation and supervision to promote high levels of treatment fidelity. Failure to evaluate interventions as implemented in their intended settings by community providers has been identified as one of the primary barriers to successfully disseminating evidence-based treatments (Chorpita, 2003; Weisz, Jensen, & McLeod, 2004). If organizational skills interventions are to be widely disseminated, they must be feasible for clinicians/schools to implement using existing infrastructure (e.g., staff and time; Kataoka, Rowan, & Hoagwood, 2009). Weisz and colleagues (Weisz, 2000; Weisz et al., 2004) proposed the Deployment Focused Model (DFM) as a method of developing treatments that can overcome the research to practice gap. This model suggests that effectiveness research should take place early in the intervention development process with intervention protocols piloted in their intended settings. As part of this process, feedback should be gathered from stakeholders regarding feasibility of implementation and modifications made to the protocol to increase the potential for widespread dissemination. The intervention is then tested, typically using randomized trial methodology, to determine if the modified protocol can be implemented in the intended setting with fidelity and produce clinically significant improvements in participant functioning. Assessment of treatment fidelity is a critical component of effectiveness research in order to gauge the amount of training and supervision that will be necessary when the intervention is disseminated.

With this goal in mind, Langberg, Vaughn et al. (2011) modified and refined the HOPS intervention for young adolescents with ADHD so that it could be feasibly implemented by SMH providers during the school day. Using an open trial design, SMH providers (N=10) from three separate school districts implemented the HOPS intervention, each with one middle school student with ADHD. SMH providers and teachers participated in focus groups and provided feedback on ways to improve the feasibility and usability of the HOPS intervention. These qualitative data, along with a review of audio-recorded HOPS sessions, were used to systematically refine the HOPS intervention protocol. A number of substantial changes were made, including adding scripts for SMH providers to use to engage students in session, devoting additional sessions to troubleshooting, increasing the frequency of rewards provided for skills implementation, and moving parent sessions earlier in the intervention (see Langberg, Vaughn et al., 2011 for further detail).

The purpose of the present study is to complete an evaluation of the refined HOPS intervention using a randomized controlled design. As in the two previous studies of HOPS, the primary dependent measures were ratings of homework problems and organizational skills. It was hypothesized that participants in the HOPS intervention group would demonstrate significantly greater improvements in homework problems and organizational skills in comparison to participants in a waitlist comparison group. It is also critical that studies of organizational skills interventions also evaluate change in more distal outcomes, in order to demonstrate that improvements in organizational skills impact academic performance. Accordingly, school grades were also examined in this study. In keeping with a focus on feasibility and potential for dissemination, SMH providers working in local school districts were recruited to implement the refined HOPS intervention. SMH providers were provided with the HOPS treatment manual but did not receive formal consultation from research staff during intervention implementation. Accordingly, another important aspect of this study is to preliminarily evaluate SMH providers’ ability to implement the HOPS intervention with fidelity.

Methods

Participants

Schools and SMH Providers

Seventeen SMH providers (seven school psychologists and ten school counselors) from five school districts and twelve distinct schools were recruited to participate in this study. The school districts involved in the study were diverse, with urban, suburban and rural school districts represented. The three urban schools in this study each had a >90% minority student body with >85% of students receiving free or reduced lunch. SMH providers were recruited through a series of face to face meetings with the first author. At these meetings, SMH providers were told that they would receive a copy of the HOPS treatment manual (Langberg, 2011) and a $100 honorarium for their participation. In addition, SMH providers were told that they would receive new school materials for each participant they provided intervention to (e.g., school binder, folders, and paper) and that incentives earned by participants would be provided by the study. The SMH provider participation rate was 100% at three of the five districts where presentations were made. Specifically, at those three districts, all middle school counselors and school psychologists in the district participated. In the fourth district there were two middle schools and the SMH providers at one of the two schools agreed to participate. The fifth district was a large urban district and a single, 10-minute presentation was made to all 36 school psychologists who served middle school students. Interested school psychologists were asked to follow-up by calling the first author, and 4 of 36 called and signed consent to participate (11%). As a condition of participation, SMH providers each had to agree to work with a minimum of two students at their school. This was to allow random assignment of participants to occur at the SMH provider level. For example, if an SMH provider worked with two study participants, one was randomly assigned to intervention and the other to waitlist comparison. All of the SMH providers who participated were female and Caucasian. The SMH providers were diverse in terms of age (M = 39; SD = 12.7; Range = 27 – 66), educational background (N = 7 Ed.S; N = 7 M.A; N = 3 M.Ed.), and years of service (M = 10.1; SD = 7.8; Range = 1 – 26).

Student Participants

All student participants (N = 47) were in grades 6–8 with an age range of 11–14 (see Table 1 for additional student demographics). Students were referred to the study by the SMH providers. Specifically, SMH providers were provided with recruitment flyers which described the study and stated that students in grades 6–8 with attention problems and academic difficulties and/or students with a diagnosis of ADHD were eligible to participate. SMH providers then contacted the parents/guardians of students that they thought would be a good fit for the study. Parents who called study staff to express interest in participation were scheduled for an inclusion/exclusion evaluation if their child met the phone screen criteria (≥4 of 9 symptoms of inattention endorsed over phone or a previous diagnosis of ADHD). Sixty-three families completed an inclusion/exclusion evaluation and 47 met full inclusion/exclusion criteria and were enrolled. To be included in the study, students had to meet DSM-IV criteria for a diagnosis of ADHD -Inattentive Type or Combined Type and have an estimated full scale IQ > 75. Diagnosis was determined using a combination of a structured interview administered to the parent, the Diagnostic Interview Schedule for Children – IV (DISC-IV; Shaffer, Fischer, Lucas, Dulcan, & Schwab-Stone, 2000), and teacher ratings on a DSM-based scale, the Vanderbilt ADHD Teacher Rating Scale (VATRS; Wolraich, Feurer, Hannah, Baumgaertel, & Pinnock, 1998). To be eligible for participation, students had to meet criteria for ADHD on the DISC-IV and have at least four symptoms in one domain endorsed as often or very often on the VATRS. Children with comorbid conditions were included in the study (see Table 1) unless they met criteria for Bipolar Disorder, Psychotic Disorder, or Substance Dependence. Full scale IQ was estimated using four subtests from the Wechsler Intelligence Scale for Children – 4^th Edition (WISC IV; Wechsler, 2003).

Table 1.

Participant Demographics

Demographic Variable	Intervention Group (N = 23)	Waitlist Control Group (N = 24)	T	p
	M (SD)	M (SD)
WISC Estimated IQ	98.5 (14.7)	98.1 (11.9)	.10	.92
WIAT-II
Reading	95.3 (11.5)	92.0 (11.1)	.99	.33
Math	96.1 (18.3)	88.2 (14.5)	1.65	.11
Spelling	97.1 (14.9)	93.8 (15.0)	.77	.45
	% (n)	% (n)	χ2	p
Male	73.9 (17)	79.2 (19)	.18	.67
Minority	21.7 (5)	33.3 (8)	.79	.37
Comorbid Diagnosesa
ODD	39.1 (9)	50 (12)	.56	.45
Anxiety	4.3 (1)	8.3 (2)	.001c	.97
Mood	0.0 (0)	0.0 (0)	--d	--
Highest Level of Parent Educationb			.90	.64
High School	19.0 (4)	31.6 (6)
Some College/Associate’s Degree	28.6 (6)	21.1 (4)
Completed College Degree	52.4 (11)	47.4 (9)
Family Income			2.34	.31
< $25,000	8.7 (2)	20.8 (5)
$25,000 – 75,000	47.8 (11)	29.2 (7)
> $75,000	43.5 (10)	50.0 (12)
ADHD Medication
Medicated pre	69.6 (16)	62.5 (15)	.26	.61
Medicated post	65.2 (15)	67.0 (16)	.01	.92
Started medication	0.0 (0)	4.2 (1)	.002c	.96
Change medication/dose	30.4 (7)	25.0 (6)	.17	.68
School Services
IEP	34.8 (8)	20.8 (5)	1.14	.29
504 Plan	0.0 (0)	8.3 (2)	.48c	.49
Resource room	8.7 (2)	4.2 (1)	.001c	.97
Homework support	21.7 (5)	12.5 (3)	.21c	.65

Note. ADHD = attention-deficit/hyperactivity disorder; IEP = Individualized Education Plan; ODD = oppositional defiant disorder; WIAT-II = Wechsler Individual Achievement Test, Second Edition; WISC = Wechsler Intelligence Scale for Children; Started medication = started taking ADHD medication during the intervention period; Change medication/dose = a change was made in either the type of medication or the dose during the intervention period.

^aComorbid diagnoses established based on parent-report on the Diagnostic Interview Schedule for Children (DISC); anxiety counted as present if social phobia, separation anxiety, or generalized anxiety criteria were met on the DISC.

^bSeven parents did not indicate their level of education.

^cA Yates’ correction for continuity was employed given the frequency counts below 5.

^dChi-square was not calculated given the observed frequencies of zero.

Participants were randomized at the SMH provider level to receive the intervention immediately (at the beginning of the school year) or to a waitlist comparison condition that would receive intervention as soon as the SMH provider finished working with intervention participants. The interventions that participants on the waitlist received were determined by the SMH provider in collaboration with the family. Specifically, the SMH provider and family could decide to implement all of HOPS, parts of HOPS, or to try a different intervention or accommodation. To ensure that equivalent numbers of students in the intervention and comparison groups were on ADHD medication, random assignment was completed blocking on ADHD medication status (see Table 1). For example, if an SMH provider was working with four students and two of them were taking ADHD medications, random assignment was blocked to ensure that only one of the two students assigned to the intervention condition was taking ADHD medication. The median number of participants assigned to each SMH provider was three (M = 2.76; Range = 2–5). The study was approved by the IRB and SMH providers, parents, and children either consented or assented to participate in the study.

Measures

Outcome Measures

Parents and teachers completed ratings for both the intervention and comparison groups pre-and post-intervention. Parents and teachers also completed a 3-month follow-up for students in the intervention group. Two teachers, Math and Language Arts, completed ratings for each participant.

Homework Problems Checklist (HPC; Anesko, Schoiock, Ramirez, & Levine, 1987)

Homework completion and homework materials management behaviors were assessed using the 20-item parent-completed HPC. For each item, parents rate the frequency of a specific homework problem on a 4-point Likert scale (0 = never, 1 = at times, 2 = often, 3 = very often). Higher scores on the measure indicate more severe problems. The measure has excellent internal consistency, with alpha coefficients ranging from .90 to .92 and corrected item-total correlations ranging from .31 to .72 (Anesko et al., 1987). Factor analyses indicate that the HPC has two distinct factors (Langberg et al., 2010; Power et al., 2006) measuring homework completion behaviors (HPC Factor I) and homework materials management behaviors (HPC Factor II). These factors are consistent across general education and clinical samples. Example items from Factor I (Homework Completion) include: a) Must be reminded to sit down and start homework; b) Daydreams during homework; c) Doesn’t complete work unless someone does it with him/her; and d) Takes an unusually long time to complete homework. Example items from Factor II (Homework Materials Management) include: a) Fails to bring home assignments and materials; b) Forgets to bring assignments back to class; and c) Doesn’t know exactly what has been assigned. In the present study, internal consistencies were high (Factor I α = .87, Factor II α = .88).

Children’s Organizational Skills Scale (COSS; Abikoff & Gallagher, 2008)

The COSS is a measure of organization, planning and time-management skills that has parent, teacher, and child versions. The COSS yields three subscale scores that have been validated through factor analysis: Task Planning, Organized Actions, and Memory and Materials Management. Items on the Task Planning subscale relate to children’s proficiency with planning out the steps needed to complete tasks in order to meet deadlines. Items on the Organized Actions subscale relate to children’s use of tools (e.g., planners and calendars) and strategies (e.g., lists) to accomplish tasks. Items in the Memory and Materials Management subscale relate to whether children lose items and how well they manage their materials (e.g., bookbags, binders, and supplies). The items from these subscales can be combined to generate a COSS Total Score. There are also two additional subscales, Life Interference and Family Conflict, which assess for the presence of functional impairment due to organizational skills problems. Scoring the COSS generates raw scores for each subscale which were used in the analyses. The raw scores can be turned into T-scores with scores > 60 indicating a clinically significant problem. T-scores between 60 and 69 are considered elevated (more problems than typical) and scores > 70 are considered to be very elevated (many more concerns than typical). Internal consistency for the items included in the COSS total score as reported in the COSS Technical Manual (Abikoff & Gallagher, 2008) is high for the parent version (.98) and teacher version, (.97). Test-retest reliability with the three COSS subscales is also high for the parent (.94 –.99) and teacher (.88 – .93) versions. In the present study, each participant’s parent/guardian and Math and Language Arts teacher completed the COSS. The COSS subscales had adequate internal consistencies in the present study (parent αs = .74–93; Language Arts teacher αs = .89-.96; Math teacher αs = .82-.94).

Vanderbilt ADHD Diagnostic Parent Rating Scale (VADPRS)

The VADPRS is a DSM-IV-based scale that includes all 18 DSM-IV symptoms of ADHD. Parents rate how frequently each of symptoms occur on a 4-point Likert scale (0 = never, 1 = occasionally, 2 = often, 3 = very often). The VARS produces an Inattention score (sum of the nine inattention items) a Hyperactivity/Impulsivity score (sum of the nine hyperactive/impulsive items) and a Total score. The VADPRS has excellent psychometric properties (Wolraich et al., 2003) and internal consistencies were high in the present study (Inattention α = .92, Hyperactivity/Impulsivity α = .96, Total ADHD α = .94).

School Grades

At the end of the school year, report cards containing school grades were collected for all study participants. All of the districts involved in the study used the same scale for grades where A = 4.0, A− = 3.7, B+ = 3.3, B = 3.0, B− = 2.7, etc. Grade point average (GPA) was calculated as the average of participants’ core class grades (math, science, history, language arts). Participants’ overall GPA served as the criterion variable in the analyses.

Parent Skills Implementation Questionnaire

At the 3-month follow-up, parents of intervention participants completed a brief questionnaire asking them to indicate if they continued to monitor and reward their child’s use of the HOPS skills. Specifically, parents were asked if they had been: (1) monitoring their child’s homework assignment completion; (2) checking their child’s planner for homework recording accuracy; and (3) monitoring their child’s materials organization using the HOPS organization checklist. If parents answered yes to any of the above questions, they were asked to indicate how often per week they were monitoring and if rewards and/or consequences were being provided.

Satisfaction Measures

Parent Satisfaction

A nine item satisfaction questionnaire was modified and used in this study (Langberg, Vaughn et al., 2011). The majority of items assessed parent satisfaction related to specific components of the HOPS intervention. For example, parents were asked to rate the level of communication between the SMH provider and the parent and how well the binder organization system worked for their child. In addition, parents responded to more general questions about overall satisfaction with the intervention. Parents indicated their agreement with each statement on a 5-point Likert scale (0 = strongly disagree, 1 = disagree, 2 = neutral, 3 = agree, 4 = strongly agree). Statements were phrased so that higher scores represented greater satisfaction (e.g., “I found the two parent meeting with my child’s school counselor/psychologist to be helpful”).

SMH Provider Satisfaction

The SMH providers completed a satisfaction questionnaire after implementing the intervention. All items were Likert-type items and SMH providers were asked to indicate if the strongly disagreed, disagreed, were neutral, agreed, or strongly agreed with ten statements. Example items included, “The HOPS interventions were feasible to implement in the school setting”, and “The HOPS treatment manual was user friendly and easy to follow”, and “I am likely to use this intervention again for students in the future”. All items were scored from 0 (strongly disagree) to 4 (strongly agree), with higher scores representing greater satisfaction.

Fidelity Measures

Organizational Skills Checklist

The Organizational Skills Checklist has been utilized in a number of treatment outcome studies with adolescents with ADHD (e.g., Evans et al., 2009; Langberg et al., 2008). This checklist consists of 14 operationalized criteria for binder (7 criteria), bookbag (4 criteria), and locker (3 criteria) organization. Example items include: 1) There are no loose papers in the bookbag; and 2) All papers in the binder are filed in the appropriate class section. SMH providers completed the organizational skills checklist at the beginning of every HOPS session and recorded either “Yes” or “No” to indicate whether participants met each criterion. The organizational skills checklist was used to assess treatment fidelity in this study. Specifically, research staff completed the checklist independently from the SMH provider and agreement was calculated. In addition, SMH provider checklists from all HOPS sessions were collected at the end of the intervention period and reviewed for accuracy.

HOPS Components Checklist

The HOPS Components Checklist was developed for this study. Each of the 16 sessions in the HOPS treatment manual was reviewed by research staff and the first author. A separate checklist was created for each HOPS session as some sessions included more steps than others (e.g., time-management is introduced in later sessions). The number of criteria on each checklist ranged from 8 – 11, depending on the session. Example items include: 1) SMH provider completed the time-management checklist; 2) SMH provider reviewed the evening schedule completed last session with the student; 3) SMH provider spent time helping the student troubleshoot difficulties with the organization system; and 4) SMH provider introduced and explained the self-management checklist. The checklist also asked research staff to record how long the session took. SMH providers did not have access to the HOPS components checklists as they were not included in the treatment manual. When research staff observed HOPS sessions, the HOPS Components Checklist was completed as a measure of fidelity.

Procedure

SMH providers received the HOPS intervention manual (Langberg, 2011) to review at the beginning of the school year and began implementing the HOPS intervention with children assigned to the intervention group in September, 2010. The first author met individually with each of the SMH providers for 1 hour prior to intervention implementation. Half of this meeting was spent reviewing study procedures. Example issues discussed included: when ratings would be administered, when SMH providers could start working with students in the waitlist group, and how treatment fidelity observations would be scheduled. During the second half of this meeting, the first author provided an overview of the HOPS intervention treatment manual and procedures. Specifically, the first author outlined when each particular skill would be introduced (e.g., organization versus time management) and demonstrated how to complete the progress monitoring checklists provided in the HOPS manual. These are the checklists SMH providers use to monitor participant progress with organization, teacher initials, and time management skills and also to track the number of points earned. SMH providers were informed during the consent process that in order to test the feasibility and usability of the HOPS manual, the first author would not provide any ongoing consultation while they were implementing the interventions.

HOPS Intervention

The HOPS intervention delivered in this study was an individual (i.e., 1:1), 16-session intervention, delivered during the school day, with each session designed to last no longer than 20 minutes. Initial sessions occurred twice weekly and then moved to once-a-week for the last six sessions. As a result, the 16 sessions can be completed over an 11-week period. The specific skills areas targeted with intervention did not change from the Langberg, Vaughn et al. (2011) study to the current study. Three main skills areas were covered: school materials organization, homework recording and management, and planning/time-management. Materials organization and homework recording and management skills were introduced first and time-management/planning was introduced second.

For materials organization, the SMH provider taught the student a specific system of bookbag, school binder, and locker organization. The student also was taught to implement an organization system for transferring homework materials to and from school. For homework recording and management, the SMH provider taught the student how to accurately and consistently record homework assignments, projects and tests in a planner. In the planning/time-management portion of the program, SMH providers taught students how to break projects and studying for tests down into small, manageable pieces, and how to plan for the timely completion of each piece. Participants were also taught how to plan out after school activities using an evening schedule to balance extracurricular activities and school responsibilities. Skills instruction was completed by session 10, after which the SMH providers met with students once per week and focused on problem-solving difficulties and self-monitoring and maintaining skills (for further details about the HOPS intervention see Langberg, 2011).

The HOPS intervention included a point system. SMH providers completed skills tracking checklists at every intervention session that included operationalized definitions of materials organization and homework management. At each HOPS session, students’ materials (e.g., binder, bookbag, and planner) were visually inspected by the SMH provider. Students received points for each criterion they met on the skills tracking checklists (e.g., no loose papers in bookbag = 1 point). In later sessions, the SMH providers also completed a checklist containing operationalized definitions of time-management, and the student earned points for effectively planning and studying for tests and projects (e.g., recorded a test in the planner = 1 point; designated a time to study for the test = 1 point). These points accumulated and students traded in the points for gift card rewards.

The HOPS intervention included two 1-hr parent meetings. These meetings were held at the school and included the SMH provider, the student, and one or both parents. The first meeting took place early in the intervention and was designed to orient the parent/guardian to the program. The second meeting took place near the completion of the intervention. The goal of the second parent meeting was to teach the parent how to manage the HOPS checklist completion and reward responsibilities once the intervention period ended. Parents learned about the point system and worked with the SMH provider to establish a plan for providing home-based rewards.

Treatment Fidelity

All SMH providers consented to having one randomly selected HOPS sessions observed and audio-taped. SMH providers were not told which sessions would be observed until the week the session was held. Study staff spread out the fidelity observations to ensure that HOPS sessions 2 – 15 were each observed at least once. There were three separate processes for evaluating fidelity to the intervention procedures outlined in the HOPS manual. First, HOPS intervention component checklists were developed that listed the specific topics to be covered by the SMH provider in each intervention session. Study staff completed these checklists during the observed sessions to evaluate SMH providers’ fidelity to the intervention procedures. Second, during session observations, study staff completed the relevant skills checklists (e.g., organizational skills checklist) independent of the SMH provider. Each checklist contains a number of operationalized criteria (e.g., organization checklist contains 14 criteria) and the SMH provider indicates yes/no whether the student met each criterion. Agreement between the study staff checklists and the SMH provider checklists was examined. Third, all SMH provider-completed checklists were photocopied at the end of the intervention. This allowed study staff to evaluate SMH providers’ fidelity to completing the checklists to monitor and reward progress with organizational skills at all intervention sessions as specified in the HOPS manual.

Statistical Analyses

Prior to intervention effect analyses, baseline equivalence between groups was examined using independent sample t-tests and chi-square analyses. Next, repeated measures multivariate analyses of variance (MANOVAs) were conducted to examine main effects of group (intervention and comparison), time (pre- and post-intervention), and group × time interactions for the primary dependent measures (i.e., COSS and HPC). Four MANOVAs were conducted for parent ratings and one MANOVA was conducted for teacher ratings. The four MANOVAs for parent ratings included: (1) two HPC factors (Homework Completion and Materials Management); (2) three COSS factors that make up the COSS Total Score (Task planning, Organized actions, Memory and Materials Management); (3) the two COSS impairment factors (Life Interference and Family Conflict); and (4) the two subscales from the Vanderbilt ADHD Rating Scale (Inattention Total Score and Hyperactivity/Impulsivity Total Score). The MANOVA for teacher ratings included the three COSS factors (Task Planning, Organized Actions, Memory and Materials Management). The Math and Language Arts teachers’ ratings were entered simultaneously into the teacher MANOVA. For all MANOVAs, when group × time interactions were significant, effects at the subscale level were examined using repeated measures ANOVAs with Bonferonni corrections. When two follow-up tests were conducted (e.g., for the two subscales on the HPC), statistical significance was set to .025 and when three follow-up tests were conducted (e.g., three subscales on the COSS), statistical significance was set to .017. Eta-squared (η²) effect sizes were calculated to represent the magnitude of the group × time interactions and Cohen’s d effect sizes were calculated using standardized mean difference scores to examine the magnitude of between group differences (Kline, 2004). For Cohen’s d effect sizes, .20 considered small but likely meaningful, .50 considered a medium effect and .80 is considered large (Cohen, 1988). For η² effect sizes, .01 is considered small, .06 medium, and .14 large (Cohen, 1988). We also conducted paired sample t-tests in order to examine whether intervention participants’ gains evident at post-intervention were maintained at the 3-month follow-up.

In addition to the primary analyses, we wanted to explore the impact of the intervention on more distal measures of functioning. Therefore, in secondary analyses we also examined the impact of the intervention on school grades. Independent sample t-tests were used to compare overall GPA between the intervention and comparison groups for the 1^st and 2^nd quarters of the school year (i.e., the intervention period) and Cohen’s d effect sizes were calculated. GPA during the 3^rd and 4^th quarters was also examined as part of the follow-up analyses. Independent sample t-tests were used to examine GPA because no baseline grade data were available. Specifically, participants had received more than a full month of intervention prior to the end of Quarter 1.

Results

Treatment Fidelity

The HOPS intervention can be completed in 11 weeks if no sessions are missed or canceled. In this study, SMH providers took between 11 and 19 weeks to implement the HOPS intervention (M = 13.8 weeks; Mdn = 14 weeks). Seven participants completed all 16 sessions in 11 weeks. Reasons sessions were missed included child or SMH provider illness/absence and snow days. Seventeen fidelity observations were completed, one for each SMH provider in the study. The observed sessions ranged in length from 10–35 min (M = 22.5; SD = 7.1). At each session observation, study staff completed the HOPS components checklist and the organizational skills checklist independent of the SMH providers. Fidelity to the intervention procedures as assessed by the HOPS components checklist was high (M = .91 of session criteria implemented correctly; SD = .13). Agreement between SMH provider and study staff on the organizational skills checklist criteria was calculated using kappa, and was high for the binder (M = .98; SD = .11) and bookbag (M = .89; SD = .20) and moderate for the locker (M = .82; SD = .34). Finally, review of the SMH providers’ records following completion of the intervention showed that the HOPS skills checklists were completed at 96% of all sessions. This is important because it indicates that SMH providers were consistently monitoring and rewarding students’ progress with organizational skills, a critical component of any intervention rooted in behavioral theory.

Outcomes

Baseline Equivalence

Independent sample t-tests and chi-square analyses demonstrated pre-treatment equivalence between the intervention and comparison groups on all demographic variables (see Table 1) and on all parent-rated measures and subscales (i.e., COSS, HPC, and ADHD symptoms on the VADPRS). There were also no significant differences at baseline according to Math teachers’ ratings on the COSS. Language Arts teachers rated participants in the comparison group higher (worse) on the Organized Actions subscale at baseline t(45) = −2.49, p = .016, but there were no statistically significant differences on the Task Planning and Memory and Materials Management subscales of the COSS (see Table 2).

Table 2.

Means, Standard Deviations, and Repeated Measures Multivariate Analyses of Variance (MANOVAs) Group × Time Interaction Results

Variable	Intervention Participants		Comparison Participants		df	Group × Time Interaction			da
	Pre M (SD)	Post M (SD)	Pre M (SD)	Post M (SD)		F	p	η2
Parent Measures
COSS Organization Domains
Omnibus	---	---	---	---	3, 42	5.81	.002	.29	---
TP	14.91 (4.23)	11.64 (3.17)	15.75 (3.10)	15.75 (4.52)	1, 44	8.19	.006	.16	1.05
OA	33.55 (4.14)	28.18 (4.96)	32.75 (4.87)	32.54 (5.00)	1, 44	16.00	.000	.27	0.88
MMM	22.32 (6.81)	19.27 (5.99)	23.54 (7.68)	23.67 (7.90)	1, 44	4.52	.04	.09	0.63
COSS Impairment Domains
Omnibus	---	---	---	---	2, 42	8.10	.001	.28	---
Life Interference	12.19 (2.98)	9.29 (3.27)	11.58 (2.84)	11.58 (3.37)	1, 43	16.40	.000	.28	0.69
Family Conflict	10.48 (2.93)	8.43 (2.69)	11.17 (3.19)	10.96 (3.62)	1, 43	4.70	.04	.10	0.79
HPC
Omnibus	---	---	---	---	2, 43	6.18	.004	.22	---
Homework Completion	30.77 (6.64)	25.05 (7.36)	32.54 (8.13)	31.75 (8.30)	1, 44	12.42	.001	.22	0.85
Materials Management	18.14 (5.73)	15.55 (5.10)	20.42 (6.21)	20.25 (6.28)	1, 44	3.60	.06	.08	0.82
Vanderbilt
Omnibus	---	---	---	---	2, 41	3.03	.059	.13	---
ADHD-I Mean	2.02 (0.55)	1.62 (0.64)	2.00 (0.54)	1.97 (0.70)	1, 42	5.88	.02	.12	0.52
ADHD-H/I Mean	1.33 (0.71)	1.22 (0.71)	1.14 (0.66)	1.18 (0.69)	1, 42	1.28	.26	.03	0.06
Teacher Measure
COSS
Omnibus	---	---	---	---	6, 35	0.88	.52b	.13
Math TP	11.82 (3.65)	12.32 (3.55)	14.55 (5.72)	13.50 (5.49)	---	---	---	---	---
Math OA	33.01 (6.38)	31.14 (7.63)	35.35 (8.65)	33.60 (10.28)	---	---	---	---	---
Math MMM	19.36 (6.92)	20.45 (6.73)	26.10 (10.32)	24.90 (11.46)	---	---	---	---	---
Language Arts TP	12.59 (4.09)	11.64 (4.24)	13.95 (5.06)	14.85 (6.08)	---	---	---	---	---
Language Arts OA	30.23 (7.00)	29.91 (8.38)	36.25 (6.58)	35.15 (8.96)	---	---	---	---	---
Language Arts MMM	17.77 (5.76)	18.18 (5.48)	24.00 (9.83)	25.30 (10.26)	---	---	---	---	---

Note. ADHD-H/I = attention-deficit/hyperactivity disorder hyperactive/impulsive symptoms; ADHD-I = attention-deficit/hyperactivity disorder inattentive symptoms; COSS = Children’s Organizational Skills Scales; HPC = Homework Problems Checklist; MMM = Memory and Materials Management; OA = Organized Actions; TP = Task Planning.

^aBetween group effect size, calculated as standardized mean difference score

^bFollow-up ANOVAs were not conducted due to the omnibus MANOVA not approaching or reaching statistical significance.

Pre-Post Analyses

On the parent-rated COSS, the omnibus MANOVA revealed a significant main effect of time, F(3,42) = 6.67, p = .001, η² = .32, as well as a significant group × time interaction, F(3,42) = 5.81, p =.002, η² = .29. Follow-up ANOVAs demonstrated that intervention participants made large and significant improvements relative to the comparison group on the Task Planning and Organized Actions scales (see Table 2). There was also a moderate intervention effect (d = .63) on the Memory and Materials Management subscale, but this difference did not reach significance at the Bonferonni-corrected p-value. No significant effects of group, time, or group × time interactions were found for the teacher COSS MANOVA (ps > .10; see Table 2).

On the parent-rated COSS impairment scales, the omnibus MANOVA revealed a significant main effect of time, F(2,42) = 8.03, p = .001, η² = .28, as well as a significant group × time interaction, F(2,42) = 8.10, p =.001, η² = .28. Follow-up comparisons showed that participants in the intervention group made significant improvements on the Life Interference scale in comparison to the waitlist group (see Table 2). Improvements on the Family Conflict scale (p=.04) were not significant at the Bonferonni-corrected p-value (p<.025).

On the parent-rated HPC, the omnibus MANOVA revealed a significant main effect of time, F(2,43) = 11.15, p < .001, η² = .34, as well as a significant group × time interaction, F(2,43) = 6.18, p =.004, η² = .22. Parents indicated that participants in the intervention group significantly improved relative to the comparison on both the Homework Completion factor but not on the Materials Management factor (see Table 2).

The parent-reported ADHD symptom MANOVA revealed a significant main effect of time, F(2,41) = 4.05, p = .03, η² = .17, as well as a marginally non-significant group × time interaction, F(2,41) = 3.03, p =.059, η² = .13. Follow-up tests revealed that participants in the intervention group made significant improvements on parent-rated inattentive symptoms relative to the comparison group after applying the Bonferonni correction.

Independent sample t-tests revealed that participants in the intervention group had higher overall GPAs compared to participants in the comparison group during the first and second quarters of the school year (i.e., the intervention period; ps = .01-.03) with medium-to-large effect sizes (d = .82 for Q1; d =.69 for Q2). Mean GPAs for participants in the intervention group were in the high C range (Q1 = 2.99; Q2 = 2.84), whereas participants in the comparison group had mean GPAs in the low C range (Q1 = 2.14; Q2 = 2.12).

Follow-up Analyses

Paired sample t-tests using post-intervention and 3-month follow-up scores were used to evaluate whether gains evidenced by the intervention group from pre- to post-intervention were maintained at the three-month follow-up. Results demonstrated that participants showed some decline in parent-reported Task Planning from post-intervention to the 3-month follow-up, t(20) = −2.14, p = .045, d = −.34, but all other treatment gains from the parent-rated COSS and HPC were maintained (ps > .10; see Figures 1 and 2). In addition, intervention participants’ mean GPAs did not decline during the third (M = 2.84) or fourth (M = 2.89) academic quarters despite the fact that they were no longer receiving the HOPS intervention.

Figure 1.

Parent-reported COSS Total Score for Intervention and Comparison Groups.

Note. COSS = Children’s Organizational Skills Scales; COSS Total Score includes all items from the three main subscales, Memory and Materials Management, Task Planning, and Organized Actions; Values in figure are T-scores where ≥ 60 = a clinically significant problem.

Figure 2.

Parent-reported HPC Total Score for Intervention and Comparison Groups.

Note. HPC = Homework Problems Checklist; HPC Total Score includes all factor I and factor II items.

Parent Skills Implementation

When asked at the 3-month follow-up if they were monitoring their child’s assignment completion, 79% of intervention group parents indicated that they were. Of those parents who continued to monitor assignment completion, they indicated that they monitored it frequently (M days per week = 3.8; SD = 1.8) and 60% indicated that rewards/consequences were tied to assignment completion. Seventy-nine percent of parents also indicated that they were monitoring their child’s assignment recording accuracy in their planner. Parents indicated that they monitored the accuracy of assignment recording frequently (M days per week = 4.4; SD = 1.2), and 40% were using rewards/consequences. Finally, 55% of parents indicated that they were monitoring their child’s materials organization using the HOPS organizational skills checklist. Those parents who were monitoring organization using the checklist were doing so frequently (M days per week = 4.3; SD = 1.6) and were using rewards/consequences (80% using rewards/consequences for organization).

Satisfaction

Parent Satisfaction

Examination of item mean scores revealed that parents were satisfied with the HOPS binder organization system (M = 3.0; SD = 1.14), rewards system (M = 3.48; SD = .60), and with the content of the two parent meetings (M = 3.0; SD = 1.1). Parents also indicated that they would strongly recommend the intervention to other families at their child’s school (M = 3.4; SD = .68). Parents were neutral to moderately satisfied with the amount of parental involvement (M = 2.3; SD = 1.0), the level of communication between parents and the SMH provider (M = 2.6; SD = .92) and in their ability to continue the HOPS interventions (M = 2.7; SD = .97).

SMH Provider Satisfaction

Overall, SMH providers indicated that they agreed (item score = 3) or strongly agreed (item score = 4) with all statements, indicating a high level of satisfaction (overall scale item M = 3.2; SD = .12). SMH providers indicated that the intervention was feasible to implement during the school day (M = 3.0; SD = .65), that the manual was user friendly and easy to follow (M = 3.3; SD = .59), that they were likely to use the intervention again in the future (M = 3.4; SD = .74), and that they prefer this type of intervention over what they were previously using for students with ADHD (M = 3.1; SD = .74). The SMH providers strongly agreed that the HOPS intervention would benefit other types of students (i.e., not just students with ADHD; M = 3.6; SD = .51).

Discussion

This study evaluated the effectiveness of the HOPS intervention for young adolescents with ADHD as implemented by SMH providers during the school day. Forty-seven middle school students with ADHD were randomly assigned to receive the HOPS intervention or to a waitlist comparison group. Intervention participants demonstrated significant improvements relative to comparison participants across parent-rated materials management and planning skills, life interference due to organizational skills problems, and homework problems outcomes. These effects were largely maintained at a 3-month follow-up assessment relative to the post-intervention timepoint. Intervention participants also had significantly higher GPAs than the comparison group during the intervention period and their GPAs did not decline during the post-intervention period. In contrast to parent ratings, significant effects on organizational skills were not observed on teacher ratings. Preliminary evidence also suggests that SMH providers were able to implement the intervention with fidelity despite the fact that no formal ongoing consultation was provided.

These findings further support the effectiveness of the HOPS intervention as implemented by SMH providers (Langberg, Vaughn et al., 2011). Similar to previous work, effects in this study were found on parent ratings but not on teacher ratings (Langberg et al., 2008, 2011). Further, the magnitude of between group effects on homework problems in this study as implemented by SMH providers (HPC Total Score; d = .83) was similar to the effects found in the previous randomized trial implemented by trained and supervised research staff (d = .71; Langberg et al., 2008). This study adds to previous work evaluating the efficacy of the HOPS intervention for young adolescents with ADHD by using a randomized controlled design along with SMH provider implementation.

The findings that SMH providers were able to implement the HOPS intervention without formal ongoing supervision or consultation, and that SMH providers found the intervention feasible to implement during the school day, are perhaps the two most important findings from this study. Typical randomized controlled trials use research staff to implement the intervention. Interventionists often receive weekly supervision to ensure that protocols are followed closely. Further, psychosocial interventions tested in randomized trials are often time and resource intensive (Chorpita, 2003; Weisz et al., 2004). As a result, evidence-based psychosocial interventions are rarely disseminated into community settings (Kataoka et al., 2009). When they are disseminated, fidelity is often an issue, either because interventions need to be modified so that they are feasible to implement, or because the community providers do not have the training, supervision, and/or infrastructure necessary to implement the procedures (Frazier, Formoso, Birman, & Atkins, 2008; Schoenwald & Hoagwood, 2001; Weisz, Donenberg, Han, & Kauneckis, 1995).

The HOPS intervention was specifically designed and refined with these dissemination concerns in mind. For example, during the development of the HOPS intervention SMH providers indicated that it would not be feasible to have parents attend more than two sessions. Therefore, while it might be ideal to include more parent sessions, only two sessions were included. It is important to note that attendance at the parent meetings in this study was 100%, with at least one parent/guardian attending two sessions for all intervention participants. This finding lends credence to SMH provider input regarding parent involvement in school-based interventions.

During intervention development, SMH providers also indicated that sessions needed to be fewer than 20 minutes in length if the intervention was to be implemented during the school day. While longer sessions would allow the intervention to be delivered over a shorter period of time, based on this input, the manual was written so that each session should take no longer than 20 minutes to implement. In this study, the mean session length was 22.5 min with some sessions taking as few as 10 min. The fact that SMH providers were able to implement the intervention during the school day is promising from a dissemination perspective. These findings also demonstrate the value of involving community-based providers in intervention development, and in conducting effectiveness work under real world conditions, prior to completing large scale efficacy trials. Such an intervention development model is counter to current intervention development theory but may result in more evidence-based interventions reaching the community.

In terms of resources, implementation of HOPS requires SMH provider time, space to implement the intervention, and a source for providing students with rewards. In the current study, students were provided with gift cards as rewards for consistently implementing materials organization and planning skills. Outside of the context of a research study, SMH providers may not have access to funds for gift cards and may need to use other types of rewards. The HOPS manual suggests that the SMH provider create a rewards menu, listing multiple reward options that do not cost money, such as playing a game with the SMH provider, a get out of homework free pass, or time on a computer or video game system (Langberg, 2011). SMH providers also received a 1 hr meeting with the first author prior to implementing the intervention, and approximately 30 minutes of that time was spent orienting the SMH provider to the treatment manual and checklists. It is currently unclear if the 1 hr meeting or provision of gift cards are critical components of the HOPS intervention, and future research will need to examine these questions.

The finding that intervention participants had significantly higher school grades than comparison participants strengthens the evidence supporting the efficacy of the HOPS intervention because school grades are less subject to rater biases. Further, the fact that no significant effects were found on teacher ratings, yet intervention participants had higher school grades and parent-rated improvements in functional impairment, supports the assertion that middle school teachers may not be able to accurately rate the constructs of organization and time management (Evans, Allen, Moore, & Strauss, 2005; Langberg, Vaughn et al., 2011). Specifically, middle school teachers may not have sufficient opportunity to observe what students record in their planners or how they organize their backpacks and lockers given the brief amount of time students spend in each class and the large number of students in each class. Alternatively, it may be that the effects generated by the HOPS intervention are not large enough to be noticed by teachers or did not meet teacher expectations. Additional research is needed to determine what types of behaviors middle school teachers are able to accurately rate, perhaps by comparing teacher ratings to objective skills observations, or by providing “don’t know” options on rating scales.

Intervention-related improvements in parent-rated materials management, organized actions, and homework completion during the intervention were largely maintained at the 3 month follow-up (see Figures 1 & 2) and school grades did not decline during the follow-up period. It is possible that this maintenance of gains was due to the fact that many parents continued to monitor and reward the HOPS skills. At the 3 month follow-up assessment, 80% of intervention group parents indicated that they continued to monitor their child’s assignment completion and homework assignment recording accuracy on a frequent basis. Fifty-five percent of parents also indicated that they were monitoring their child’s use of organizational skills by completing the HOPS organizational skills checklist multiple times each week. Many of the parents in the sample also reported that they were providing rewards and consequences for their child’s use of the homework and/or organizational skills. Another possible explanation for the generalization of effects across time is that the HOPS manual encourages SMH providers to add frequent monitoring of organization and time management skills using the checklists to students’ IEP and 504 plans. However, this hypothesis cannot be tested with the data collected in this study.

Limitations

In this study, randomization was completed blocking on ADHD medication status to ensure that an equal number of students on and off medication were in the HOPS and comparison groups. Medication changes made during the intervention period were also tracked (see Table 1). A stronger design would be to control for the impact of ADHD medication through the analyses or to evaluate whether ADHD medication status moderated outcome. The sample size in this study is not sufficient for these types of analyses. Similarly, it would be important to control for other types of school or therapeutic services that students may have received (see Table 2).

Parents and teachers were involved in the intervention and therefore could not be blind to condition. Accordingly, rater-biases may be present. Further, the comparison condition was a waitlist comparison condition and as such, the potential impact of nonspecific therapeutic effects (e.g., the SMH provider/student relationship) cannot be accounted for. It will be important for future studies to compare the HOPS intervention to an active comparison group where students in the comparison receive the same amount of therapist attention. An active comparison group may also reduce rater-basis as both groups of parents would be expecting to see improvements. Further, group differences on school grades must be interpreted with caution because baseline equivalence could not be established, although it also worth noting that the intervention and waitlist control groups did not differ on standardized measures of IQ and academic achievement.

Another important limitation is that the SMH providers volunteered to participate in this study and therefore, may represent a unique group of motivated school practitioners. The results may not generalize to SMH providers as a group. Further, SMH providers took part in the process of selecting students to participate in the intervention. As such, the findings may not generalize to all middle school students with ADHD. It could be that the middle school students in this study were selected because they had particular difficulties with organizational skills or fewer difficulties in other areas (e.g., learning problems). However, it should be noted that the participants in this study were recruited from a diverse group of schools, and were relatively diverse in terms of race, comorbid mental health disorders, and parent education level and income (see Table 1).

Finally, treatment fidelity was assessed through live observation of randomly selected sessions for each SMH provider. Although SMH providers were given short notice that they were going to be observed, having an observer present may have changed their behavior (i.e., the Hawthorne effect). A stronger method of assessing fidelity would have been to audio-record all sessions and to complete components checklists based upon those recordings. However, that would not have permitted assessment of checklist completion accuracy which requires that an observer complete checklists independently. Future research with the HOPS intervention needs to assess fidelity as a multi-dimensional construct (Sanetti & Kratochwill, 2009), including examining session length as a potential predictor of outcomes (Nock & Ferriter, 2005).

Future Research

The HOPS intervention appears to have considerable promise as an effective school-based intervention for improving the organizational skills of adolescents with ADHD. Larger studies of the HOPS intervention are necessary to answer questions about moderators and mediators of treatment response. It may be that the HOPS intervention works well for some students but less well for others. For example, it may be that students with severe oppositional defiant behaviors or with comorbid learning disorders respond less well to the intervention or need a higher intervention dose to achieve a clinically meaningful response. In this study, while participants made large improvements in homework problems according to parent ratings, there was still additional room for improvement, and a longer intervention may be necessary in some cases. It is also possible that the HOPS intervention could be applied to a broader group of students than students with ADHD, and could potentially have a larger impact. In terms of mediation, it will be important to evaluate mechanisms of change within the HOPS intervention. For example, it may be that students’ use of certain skills (e.g., time management) drives improvements in overall school performance. It is also possible that student perception of the SMH provider or satisfaction with the intervention plays an important role in predicting outcomes. A limitation of this study is that we only assessed SMH provider and parent satisfaction and did evaluate satisfaction from the students’ perspective. It will also be critically important for further research to compare the HOPS intervention to an active comparison group to account for potential nonspecific therapeutic effects. It would be useful to compare the HOPS intervention to the types of services typically provided in school settings to address problems with homework and organization (e.g., a homework tutoring condition).

Conclusion

The HOPS intervention has now undergone a systematic process of evaluation and refinement during which stakeholder input was gathered at multiple points. The hope is that by focusing on feasibility of intervention delivery with treatment fidelity up front, the HOPS intervention will be able to overcome the oft-cited research to practice gap following proof of efficacy. The HOPS intervention appears promising for improving the organizational skills and academic performance of students with ADHD. Additional research comparing HOPS to an active control group, and with a stronger evaluation of fidelity, is needed before efficacy can be firmly established.