Refinement of an Organizational Skills Intervention for Adolescents with ADHD for Implementation by School Mental Health Providers

Abstract

The purpose of the study was to modify, test, and refine the Homework, Organization, and Planning Skills (HOPS) intervention for adolescents with ADHD for use by school mental health (SMH) providers. Ten SMH providers from three school districts implemented the HOPS intervention with 11 middle school students with ADHD. Parent and teacher ratings of materials organization and homework management were collected pre- and post-intervention and treatment fidelity was assessed. SMH providers and teachers participated in focus groups and provided feedback on ways to improve the feasibility and usability of the HOPS intervention. Students made large improvements in organization skills (d = 1.8) and homework problems (d = 1.6) according to parent ratings however, no improvements were observed on teacher ratings. Qualitative data generated from coding the focus groups and audio-recorded HOPS sessions were combined with the quantitative results to systematically refine the HOPS intervention for further evaluation of intervention effectiveness and disseminability.

Attention-Deficit/Hyperactivity Disorder (ADHD) is one of the most commonly diagnosed childhood disorders, with prevalence rates estimated at 8% (Froehlich et al., 2007). Several of the primary diagnostic features of ADHD relate to problems with temporal and materials organization (i.e., often has difficulty organizing tasks and activities, often loses things, is often forgetful, and often fails to finish school-work, chores, or duties; American Psychiatric Association, 2000). In the school setting, problems with temporal and materials organization manifest as forgetting to complete or losing homework assignments, difficulties planning for the completion of long-term projects and studying for tests, procrastination, and problems keeping class materials organized (Langberg, Epstein, & Graham, 2008; Langberg et al., 2010; Power et al. 2006). These problems contribute to the poor school grades and high rates of grade retention experienced by children and adolescents with ADHD (Frazier et al., 2007; Langberg et al., in press; Molina et al., 2009).

Difficulties with temporal and materials organization become particularly problematic in middle school and can result in considerable academic impairment (Evans, Serpell, & White, 2005). The transition to middle school is marked with numerous environmental changes and represents a significant challenge for children with ADHD (Langberg, Epstein, Altaye et al., 2008; Wolraich et al., 2005). Changes in class size, number of teachers, increased demands for independence and greater workloads make the transition to middle school difficult (Evans, Langberg, Raggi, Allen & Buvinger, 2005; Evans, Serpell et al., 2005). Further, there is a significant reduction in adult supervision and monitoring following the transition to middle school as teachers work with close to 150 different students each week. As a result, teachers are not afforded the ability to closely monitor and support individual student’s organization of materials and homework management (Evans et al., 2005). Not surprisingly, middle school is often when difficulties with organization and planning/time management become particularly problematic.

Given the relationship between temporal and materials organization and poor school performance, there is a clear need for interventions that address these difficulties. A few interventions have been developed that target the organization and time-management skills of children with ADHD (e.g., Abikoff & Gallagher, 2008; Evans et al., 2007; Evans et al., 2009; Pfiffner et al., 2007; see Langberg, Epstein, & Graham, 2008 for a review). However, these are all either clinic-based interventions or include organizational skills interventions as part of a larger package of psychosocial interventions (e.g. including classroom behavior management and/or and interpersonal skills intervention). Given the importance of organization and planning skills for academic performance, there is a need for focused intervention targeting these behaviors. Further, organizational skills interventions that can be directly implemented in the school setting are valuable because the school is where most of these difficulties occur. Delivering the intervention in the school setting is likely to improve, access to care, intervention outcomes and generalization of improvements (Evans, Langberg, & Williams, 2003).

Langberg, Epstein, Urbanowicz et al. (2008) developed the Homework, Organization, and Planning Skills (HOPS) intervention to address the organization and planning difficulties of children with ADHD. The intervention was evaluated by completing a small randomized trial with 37 students (31 boys and 6 girls) with ADHD (age range 9 – 14; grades 4–7). Students were randomly assigned to receive the intervention in the fall school semester or put on a waitlist to receive the intervention later in the school year (i.e. a waitlist control group).

The intervention was delivered as part of a 2 day per week after-school program. The after-school program was held on-site at the school for 8 weeks and students received 75 minutes of intervention each program day. The after-school program was staffed by university undergraduate psychology students with a ratio of 3:1, students to counselors. Each day, all students received 20 minutes of individual intervention time (1:1 student to counselor) during which time the organization and homework/time-management interventions were implemented. Participants were taught to develop and maintain systems of school materials organization for school binders, bookbags and lockers. Participants also were taught how to accurately record homework assignments and tests in a planner and to plan for long-term projects and tests by breaking tasks into smaller, more manageable, components. The intervention also included two 1 hr parent meetings that took place at the school. The goal of these meetings was to have parents establish a plan for monitoring and rewarding students’ organization and planning behaviors once the after-school program ended.

Intervention participants made significant improvements in homework recording and were accurately recording assignments and tests for all core classes 72% of the time at post-intervention. In comparison, waitlist control participants were recording homework assignments accurately in their planners 21% of the time. Parents indicated that participants in the intervention group significantly improved their homework problems during the 8-week intervention, whereas the waitlist control group made no significant change. Analyses of students’ GPA revealed a significant improvement in core class grades for those students who participated in the intervention (Langberg et al., 2008).

While the intervention model focused primarily on organizational and planning skills and was school-based; a primary limitation was that the interventions were implemented in an after-school program operated by research staff. If organization and planning interventions are to be widely disseminated, they must be feasible for schools to implement using existing infrastructure (i.e., staff and time).

The primary goal of the present study was to adapt and refine the HOPS intervention protocol (Langberg et al., 2008) to create a product that is highly acceptable to parents and schools and is feasible for schools to implement during the school day. We have chosen to use School Mental Health (SMH) providers (i.e.., school counselors and school psychologists) for intervention delivery since they are on the frontlines when it comes to delivering school-based interventions targeting academic problems. They also have daily access to students and their teachers to facilitate implementation of academic interventions. Further, SMH providers have the ability to identify problems early and to intervene before difficulties compound and lead to more severe negative academic outcomes, such as failing grades.

In this study, we engaged in a collaborative intervention development process. Ten SMH providers from three separate school districts each implemented the intervention with at least one middle school student with ADHD. Parent, teacher and self-ratings of organization and time-management skills and homework problems were collected pre- and post-intervention. SMH providers and teachers participated in focus groups and provided feedback on ways to refine the intervention. Data generated from the outcome measures, focus groups, and session observations were used to systematically modify and revise the intervention protocol.

Methods

Participants

SMH Providers

Ten SMH providers from three separate school districts and eight distinct schools were recruited to participate in this study. The SMH providers were diverse in terms of age, education background, and years of service (see Table 1). Each SMH provider worked with one student with ADHD with the exception of one SMH provider who chose to work with two students. The SMH provider who worked with two students met with each student individually.

Table 1.

School Mental Health Provider Demographics (N = 10)

Demographics	M (SD) or Percentage
Age	35.4 (9.6); Range = 27–58
Gender	100% Female
Ethnicity	100% Caucasian
Grades Served
K-8	50%
6–8	50%
Years Experience	8.1 (5.1); Range = 1–15
Degrees
M.A.	40%
Ed.S.	40%
M.Ed.	10%
Psy.S	10%

Student Participants

All student participants (N = 11) were in grades 6–8 (see Table 2 for student demographics). Students were referred to the study by teachers. Teachers were provided with recruitment flyers which described the study and stated that students in grades 6–8 with attention problems and academic difficulties and/or students with a diagnosis of ADHD were eligible to participate. Teachers provided student names to SMH providers who contacted parents. Parents who called study staff to express interest in participation were scheduled for an inclusion/exclusion evaluation. To be included in the study, students had to meet DSM-IV criteria for a diagnosis of ADHD - Inattentive Type or Combined Type and have an estimated full scale IQ > 75. Diagnosis was determined using a combination of a semi-structured interview administered to the parent, the Diagnostic Interview Schedule for Children – IV (DISC-IV; Shaffer, Fischer, Lucas, Dulcan, & Schwab-Stone, 2000), and teacher ratings on a DSM-based scale, the Vanderbilt ADHD Rating Scale (Wolraich, Feurer, Hannah, Baumgaertel, & Pinnock, 1998). To be eligible for participation, students had to meet criteria for ADHD on the DISC-IV and have at least four symptoms in one domain endorsed as often or very often on the teacher rating. Full scale IQ was estimated using the Wechsler Abbreviated Scale of Intelligence (WASI; Psychological Corporation, 1999). The study was approved by the IRB and SMH providers, parents, and children either consented or assented to participate in the study.

Table 2.

Student Participant Demographics (N = 11)

Demographics	M (SD) or Percentage
Age	12.0 (1.18); Range = 11 – 14
Grade	6.5 (.82); Range = 6 – 8
Gender	90% Male
Race	40% Minority
WASI IQ	98.7 (13.7)
WIAT-II
Reading	102 (9.1)
Math	84.7 (16.2)
Spelling	96.3 (12.9)
ADHD
Inattentive Type	73%
Combined Type	27%
Comorbid Diagnoses
ODD	36%
Anxiety	36%
Mood	09%
Parent Education (Highest Level)
High School	27%
Some College	9%
College Degree	45%
Masters Degree	18%
Family Income
< 25,000	36%
25,000 – 75,000	45%
> 75,000	18%
Parent Marital Status
Married	36%
Divorced/Separated	64%
ADHD Medication
Medicated	64%

Note – Comorbid diagnoses as reported by parents on the DISC; Anxiety counted as positive if Social Phobia, Separation Anxiety, or Generalized Anxiety criteria were met on the DISC.

Procedures

SMH providers received the HOPS intervention manual (Langberg, in press) to review in November. SMH providers began implementing the HOPS intervention in January after students returned from winter break. The study principal investigator (PI) met individually with each of the SMH providers for 1 hour prior to intervention implementation. During this meeting, the PI provided an overview of the HOPS intervention treatment manual and procedures. SMH providers were informed during the consent process that in order to test the feasibility and usability of the HOPS manual, there would be no formal consultation with the PI during the study.

HOPS Intervention

In the Langberg et al. (2008) study, the organization and planning interventions were delivered through two, 20-min sessions per week over a period of 8 weeks (total of 16 sessions). In this study, the HOPS intervention was also delivered through 16, 20-min sessions. However, in this study, the HOPS manual specified that SMH providers meet with students twice per week for the first ten sessions and once per week for the last six sessions. As a result, the 16 sessions were completed over an 11 week period instead of an 8 week period. This change was made to facilitate generalization of improvements over time. Specifically, when students meet less frequently with SMH providers, they have greater opportunity to implement and problem-solve self-management skills. SMH providers in this study either implemented sessions before school (N=3) or during the school day (N=7). Three main skills areas were covered: school materials organization, homework management and planning/time-management. Materials organization and homework management skills were introduced first (sessions 1–5) and time-management/planning were introduced second (sessions 6–10). For materials organization, the SMH provider taught the student a specific system of bookbag, school binder, and locker organization. The student also learned to implement an organization system for transferring homework materials to and from school. For homework management, the SMH provider taught the student how to accurately and consistently record homework assignments, projects and tests in a planner. In the planning/time-management portion of the program, SMH providers taught students how to break projects and studying for tests down into small, manageable pieces, and how to plan for the timely completion of each piece. Participants were also taught how to plan out after school activities using an evening schedule to balance extracurricular activities and school responsibilities. Skills instruction was completed by session 10, after which the SMH providers met with students once per week and focused on problem-solving difficulties and self-monitoring and maintaining skills.

The HOPS intervention included a point system. SMH providers completed checklists at every intervention session that included operationalized definitions of materials organization and homework management. Students received points for each criterion they met on these checklists (e.g. no loose papers in bookbag = 1 point). In later sessions, the SMH providers also completed a checklist containing operationalized definitions of time-management, and the student earned points for effectively planning and studying for tests and projects (e.g. recorded a test in the planner = 1 point; designated a time to study for the test = 1 point). These points accumulated and students traded in the points for gift card rewards.

As with Langberg et al. (2008), the HOPS intervention also included two, 1 hr parent meetings. These meetings were held at the school and included the SMH provider, the student, and one or both parents. The first meeting took place halfway through the intervention and was designed to orient the parent to the program. The second meeting took place near the completion of the intervention. The goal of the second parent meeting was to teach the parent how to take over the HOPS checklist completion and reward responsibilities from the SMH provider. Parents learned about the point system and worked with the SMH provider to establish a plan for providing home-based rewards.

Focus Groups

Following completion of the HOPS intervention, SMH providers participated in focus groups. Three focus groups were run, one for each of the school districts involved in the study. Focus groups were 2.5 hrs long. The first 1.5 hrs was attended only by the SMH providers. During the second hour, teachers invited by the SMH providers joined the focus group in order to examine how the students’ teachers perceived the intervention. SMH providers each invited one teacher to the group (total teacher N = 8). Prior to attending the focus group, SMH providers and teachers were provided with a list of topic areas and questions that would be asked so that they could prepare. For the SMH providers, 45 min were spent on these predetermined questions and 45 min were spent using an open forum format. Likewise, when teachers joined the focus groups, 30 min were spent on the predetermined questions and 30 min were devoted to an open forum.

Treatment Fidelity and Integrity

All SMH providers consented to having two randomly selected HOPS sessions observed and audio-taped. SMH providers were not told which sessions would be observed until the week the session was held. Study staff spread out the fidelity observations to ensure that all HOPS sessions were observed at least once. There were three separate processes for evaluating integrity to the intervention procedures outlined in the HOPS manual. First, HOPS intervention component checklists were developed that listed the specific topics to be covered by the SMH provider in each intervention session. Study staff completed these checklists during the observed sessions to evaluate SMH providers’ fidelity to the intervention procedures. Second, during session observations, study staff completed the relevant skills checklists (e.g. organizational skills checklist) independent of the SMH provider. Each checklist contains a number of operationalized criteria (e.g. organization checklist contains 14 criteria) and the SMH provider indicates yes/no whether the student met each criterion. Agreement between the study staff checklists and the SMH provider checklists was examined. Third, all SMH provider-completed checklists were photocopied at the end of the intervention. This allowed study staff to evaluate SMH providers’ fidelity to completing the checklists at all intervention sessions as specified in the HOPS manual.

In addition, process integrity was evaluated by coding the transcribed audio-taped sessions. A qualitative methodologist coded each of the transcribed sessions for themes related to participant and SMH provider engagement. Specifically, sessions were coded to evaluate student engagement in the session as well as the language that SMH providers used to motivate and engage students and to promote student ownership of the intervention process.

Measures

Homework Completion/Materials Management

Homework performance was assessed at baseline and post-intervention using the parent-completed Homework Problems Checklist (HPC; Anesko, Schoiock, Ramirez, & Levine, 1987). The HPC is a 20 item parent-report instrument. For each of 20 items, parents rate the frequency of a specific homework problem on a 4-point Likert scale (0 = never, 1 = at times, 2 = often, 3 = very often). Higher scores on the measure indicate more severe problems. The measure has excellent internal consistency, with alpha coefficients ranging from .90 to .92 and corrected item-total correlations ranging from .31 to .72 (Anesko et al., 1987). Factor analyses indicate that the HPC has two distinct factors (Langberg et al., 2010; Power et al., 2006) measuring homework completion behaviors (HPC Factor I) and materials management behaviors (HPC Factor II).

Children’s Organizational Scale (COSS; Abikoff & Gallagher, 2008)

The COSS is a comprehensive measure organization and time-management skills that was completed by parents, teachers, and adolescents pre- and post-intervention. The COSS yields three subscale scores that have been validated through factor analysis; task planning, materials organization, and materials management. The COSS also produces a total score. Scoring the COSS generates T-scores for each of the subscales with scores >60 indicating a clinically significant problem. T-scores between 60 – 69 are considered elevated (more problems than typical) and scores > 70 are considered to be very elevated (many more concerns than typical). Internal consistency for the items included in the COSS total score is high for the parent version (.98), teacher version, (.97), and self-version (.94). Test-retest reliability with the three COSS subscales is also high for the parent (.94 – .99), teacher (.88 to .93), and self versions (.94 – .96). Children with ADHD have been shown to score significantly higher (worse) on the COSS as compared to their peers (Abikoff & Gallagher, 2008) and the COSS has been used in other ADHD treatment development studies (Pfiffner et al., 2007).

Vanderbilt ADHD Rating Scale (VARS)

The VARS are DSM-IV-based scales with teacher-report and parent-report forms. The VARS include the 18 DSM-IV ADHD symptoms which are rated on a 4-point Likert scale (0–3) with 0 indicating whether each ADHD symptoms occurs Never (0), Occasionally (1), Often (2), or Very Often (3). The VARS produces an Inattention score (sum of the nine inattention items) a Hyperactivity/Impulsivity score (sum of the nine hyperactive/impulsive items) and a Total score. The VARS has excellent psychometric properties (Wolraich et al. 2003).

Organizational Skills Checklist

The Organizational Skills Checklist has been utilized as an outcome measure in a number of treatment outcome studies with adolescents with ADHD (e.g., Evans et al., 2009; Langberg et al., 2008). This checklist consists of 14 operationalized criteria for binder (7 criteria), bookbag (4 criteria), and locker (3 criteria) organization. Example items include: 1) There are no loose papers in the bookbag; and 2) All papers in the binder are filed in the appropriate class section. SMH providers completed the organizational skills checklist at the beginning of every HOPS session and record either “Yes” or “No” to indicate whether participants met each of the criteria. The checklist was completed during the first two HOPS sessions prior to the introduction of materials organization skills to establish a baseline.

Parent Satisfaction

A nine item satisfaction questionnaire was developed for this study. The majority of items assessed parent satisfaction related to specific components of the HOPS intervention. For example, parents were asked to rate the level of communication between the SMH provider and the parent and the how well the binder organization system worked for their child. In addition, parents responded to more general questions about overall satisfaction with the intervention. All items were Likert-type items on a scale from 1 – 6. For the majority of items (N = 7), parents were asked to indicate their level of agreement with a statement where 1 = not at all, 3 = somewhat, and 6 = very much. On the other two items, parents were asked to rate the level of parent involvement in HOPS and level of communication with the SMH provider. For these items the anchors were 1 = not enough, 3 = just right, and 6 = too much.

Statistical Analyses

The primary dependent measures, the HPC and COSS, were completed pre- and post-intervention. The HPC was completed by parents and the COSS was completed by parents, adolescents (i.e. participants), and teachers. Paired sample t-tests were used to examine change from pre to post-intervention. Separate Paired sample t-tests were run for the two HPC factors and for the HPC total score. Separate t-tests were also run for the three COSS subscales and for the COSS total score. Cohen’s d effect sizes, measuring the magnitude of pre- to post-intervention change were calculated.

The VARS was also completed by parents and teachers pre- and post-intervention and was examined as a secondary dependent measure. Paired sample t-tests were run separately for the inattentive and hyperactive/impulsive subscales (i.e. the 9 DSM items from each domain).

Qualitative Data Analysis

The audio-recorded focus groups and HOPS sessions were transcribed verbatim. Transcripts were reviewed before open coding began. Open coding is the process of identifying meaning units within the data and establishing a representative code for each meaning unit. Next, frames of analysis were created to reduce data to only those data relevant to addressing research questions (Hatch, 2002). In this case, the research questions were focused on improving the feasibility, usability and functionality of the HOPS intervention.

Quantitative Results

Fidelity

Eighteen fidelity observations were completed. Eight SMH providers were observed twice and two SMH providers were each only observed once due to scheduling conflicts. The observed sessions ranged in length from 5 – 35 min (M = 21.7; SD = 7.9). At each session observation, study staff completed a HOPS components checklist and the organizational skills checklist independent of the SMH providers. Fidelity to the intervention procedures as assessed by the HOPS components checklist was high (M = .90 criteria implemented correctly; SD = .12). Agreement between SMH provider and study staff on the organizational skills checklist criteria was moderate for the binder (M = .78) and high for the bookbag (M = .85) and locker (M = .89). Finally, review of the SMH providers records showed that the HOPS checklists (e.g. organizational skills checklist) were completed at 96% of all sessions.

Program Measures

Organizational Skills Checklist

Percentage of organization criteria met on the organizational skills checklist was recorded at baseline (sessions 1 & 2) and at each intervention session for the duration of the intervention. At baseline (average of sessions 1 & 2), participants met 25% of binder criteria, 49% of bookbag criteria, and 45% of locker criteria. Over the last four HOPS sessions, participants averaged 85%, 82%, and 89% of binder, bookbag, and locker criteria met, respectively (see Figure 1).

Figure 1.

Percentage of Criteria Met on the Organizational Skills Checklist

Rating Scales

Results of analyses with the primary and secondary dependent measures, HPC, COSS, and VARS are presented in Table 3. In summary, parents rated participants as making large and significant pre- to post-intervention improvements on the HPC and COSS on all subscales/factors (all ps <.05). Post-intervention, mean scores on all parent-rated COSS subscales were no longer in the clinically significant problem range (i.e. M = < 60; see Table 3). Participants did not make any significant improvements on the COSS subscales according to teachers. Participants rated themselves as making moderate improvements on each of the COSS subscales and improvement on the Materials Management subscale of the COSS was significant (p < .05). Analyses of the parent and teacher completed VARS revealed that participants made significant improvements on the inattention subscale according to parents. Improvements were not statistically significant for the hyperactive/impulsive subscale on the parent completed VARS or for either subscale on the teacher completed VARS.

Table 3.

Means, Standard Deviations, and Effect Sizes for Parent and Teacher Ratings

	Pre	Post
	M (SD)	M (SD)	t	p	d
Parent Ratings
HPC – Homework Completion	37.7 (7.6)	25.9 (5.0)	3.9	.003	1.8
HPC - Materials Management	24.3 (5.3)	16.9 (5.7)	3.2	.010	1.3
HPC - Total Score	64.4 (13.5)	44.4 (10.1)	3.5	.005	1.6
COSS - Task Planning	68.7 (11.0)	59.4 (9.7)	2.7	.021	0.9
COSS – Organized Actions	62.1 (3.9)	54.7 (6.0)	3.4	.007	1.6
COSS – Materials Management	70.2 (8.0)	55.0 (15.9)	2.9	.014	1.6
COSS – Total Score	71.2 (4.9)	58.1 (9.0)	4.3	.002	1.8
VARS – Inattention	23.4 (3.1)	14.8 (7.4)	4.1	.002	1.6
VARS – Hyper/Impulsive	14.7 (9.0)	9.9 (8.0)	2.2	.056	0.9
Teacher Ratings
COSS - Task Planning	61.9 (12.5)	59.4 (12.3)	0.8	.440	.20
COSS – Organized Actions	58.6 (7.1)	60.5 (8.1)	0.8	.450	−.25
COSS – Materials Management	63.6 (10.2)	64.5 (13.2)	0.2	.847	−.08
COSS – Total Score	63.3 (8.7)	63.9 (11.3)	0.2	.842	−.06
VARS – Inattention	20.7 (4.6)	17.6 (6.9)	1.4	.202	.54
VARS – Hyper/Impulsive	8.3 (8.8)	6.0 (7.2)	0.9	.368	.29
Self Ratings
COSS - Task Planning	55.1 (11.4)	49.1 (6.6)	1.5	.163	.67
COSS – Organized Actions	50.6 (7.5)	46.6 (5.7)	1.8	.109	.61
COSS – Materials Management	57.0 (11.9)	46.3 (12.7)	2.4	.036	.87
COSS – Total Score	52.8 (8.7)	47.3 (6.4)	1.8	.096	.72

Note – HPC = Homework Problems Checklist; COSS = Children’s Organizational Skills Scales; VARS = Vanderbilt ADHD Rating Scale

Parent Satisfaction

The first two questions asked parents about their satisfaction with the level of parent involvement in the HOPS intervention and the level of communication between the parent and SMH provider. Parents responded on a 1–6 scale where 1 = not enough, 3 = just the right amount; 6 = too much. Parents indicated that the level of parent involvement was adequate (M = 2.5; SD = .93) and the level of communication was just right (M = 2.9; SD = .94). The next seven items were also on a 6-point Likert scale and the anchors were 1 = not at all; 3 = somewhat; 6 = very much. Overall, parents were highly satisfied with the HOPS intervention and indicated that they would recommend the program to other students/families (M = 5.6; SD = .92). Parents were also highly satisfied with the HOPS binder materials organization system (M = 5.0; SD = 1.3) and points system (M = 5.3; SD = 1.0). The lowest scores were on the items related to parents’ confidence in maintaining the HOPS interventions (M = 4.4; SD = .92) and continuing to manage their child’s organization and homework management (M = 3.8; SD = 1.3) post-intervention.

Qualitative Results

Focus Group Analyses

Qualitative analysis of focus group data was conducted to evaluate SMH provider and teacher perceptions of the feasibility, usability, and functionality of the HOPS intervention. Across the three focus groups, SMH provider and teacher feedback clustered around five main topic areas or themes: 1) Intervention structure and pace; 2) Incentive system; 3) Teacher involvement; 4) Parent involvement; and 5) Student engagement and Agency. The results of the qualitative analyses are summarized in Table 4.

Table 4.

Results of the Qualitative Analyses of Focus Group and HOPS Session Transcripts

Theme	Specific Feedback	Modifications and Revisions
Intervention Structure/Pace	Pace of skills introduction is too fast. More time needed to problem-solve materials organization and homework management skills.	Two sessions added that are devoted entirely to problem-solving the materials organization and homework management systems. Specific suggestions for troubleshooting commonly noted problems are also added.
Incentives System	Takes too long for students to earn the first reward and SMH providers need flexibility in delivering points.	Point values for demonstrating skills in the first few sessions increased. Manual changed to specify that SMH providers can be flexible with points. Examples provided for using bonus points.
Teacher Involvement	Not enough guidance provided on the role of teachers in the intervention and ways to involve teachers. Intervention should include tracking and incentives for improving missing assignments.	Section added that specifies the role of teachers in the HOPS intervention. Template letters included in manual that can be used to inform teachers about the HOPS intervention and student progress. Missing Assignment tracking and rewarding added to the manual. Parents track and provide rewards for assignment completion.
Parent Involvement	Meetings with parents need to occur earlier in the intervention process.	Parent meetings 1 and 2 moved earlier in the intervention process. The first parent meeting now takes place between HOPS sessions 5 and 6.
Student Engagement/Agency	Scripts for SMH providers should provide examples of techniques for engaging students in session and for promoting student agency and ownership of the intervention.	Scripts added to the manual that incorporate language SMH providers used during pilot study to effectively engage students and to promote student agency.

Intervention Structure and Pace

In the HOPS intervention, materials organization and homework management skills are introduced first (sessions 1 – 4), time-management skills are introduced next, (sessions 5 – 10) and self-management skills are introduced last (sessions 11 – 16). SMH providers’ liked the order that the skills were presented. One SMH provider put it this way:

“I feel like the sequence of skills introduced was very appropriate. It started off very basic, just recording assignments and keeping up with it, and then planning for things. The progression was very good.”

SMH providers suggested that the pace new skills were introduced was too quick. Specifically, SMH providers felt that more time needed to be devoted to problem-solving materials organization and homework management skills before time-management skills were introduced. SMH providers noted that students were still learning to implement the materials organization and teacher initials systems when time-management was introduced in session five. SMH providers also indicated that the session where the student’s new materials organization system was established took longer than 20 min which in turn meant that they got behind in delivering the next session’s content. One commonly noted difficulty was that it took students a long time to determine which papers to keep and which to throw away when cleaning out binders and bookbags. In addition, it took students time to “acclimate” to the new binder system. One SMH provider explained it this way, “at the beginning [he was] still putting things in the wrong place and would just stick everything in front of the binder, so we would have to go through all those papers, so it was pretty time consuming.” In the words of one SMH provider, “I felt like since nothing was firm, we just kept sort of exposing them and not really mastering any skills.” In terms of homework management, SMH providers indicated that students often had a difficult time obtaining teacher initials. Problems reported included students “being afraid to ask for initials,” “forgetting to ask,” “being rushed at the end of class,” and “avoiding negative attention”. SMH providers felt the intervention could benefit from having more session content and time be devoted to strategies for overcoming these problems.

Incentive System

In the HOPS intervention, student’s earned points for implementing the HOPS skills and the points could be turned in for gift cards. SMH providers reported that the HOPS points system was “highly motivating”. However, SMH providers gave feedback suggesting that: 1) it took students too long to earn their first reward (i.e. the first gift card); and 2) that more flexibility was needed in distributing points. For example, one SMH provider stated, “I know my student was extremely excited about the potential to earn points, and then when it didn’t happen after the first couple of sessions, it became a distant concept.” With regard to flexibility, SMH providers wanted the flexibility to distribute bonus points when students’ met interim goals. For example, the first time a student came prepared to session and met all of the criteria on the organizational skills checklist the student could earn 25 bonus points.

Teacher Involvement

Guidelines for how SMH providers should involve teachers were not included in the HOPS manual. Despite the fact that the specifics for teacher involvement were not outlined in the HOPS manual, many of the SMH providers actively sought teacher involvement. Further, many of the teachers in the focus groups expressed interest in learning more about the intervention. SMH providers took the initiative to get teachers involved in a number of different ways. For example, one SMH provider reported that she “had the teacher come to that [clean out] session so we didn’t get rid of anything important”, whereas another SMH provider requested that a teacher assist the student with the task of deciding what papers to throw away after class. Obtaining teacher initials was another area where SMH providers felt teacher involvement would be helpful. Although SMH providers acknowledged the importance of students taking the initiative to request teacher initials, teachers might be in the position to offer unobtrusive prompts to help students remember to ask. SMH providers also felt that sharing pieces of the HOPS intervention with teachers, such as planning for long-term assignments and studying for tests might be beneficial as teachers could use the interventions with other students.

When teachers were asked to describe changes in student behaviors that occurred as a result of the intervention, teachers said that students were better able to locate needed materials and that some students made academic improvements (e.g., F to a C in one class). However, a number of teachers noted that better organizational skills did not necessarily translate into “getting more work done” and turned into the teacher. Missing or late assignments was consistently noted as a problem by teachers. One 7^th grade teacher noted the distinction between organization and academic performance:

“It wasn’t so much an issue of this particular individual forgetting the math worksheet or losing the reading worksheet; it was there. It was whether or not it was completely completed that was a whole other issue, but it was there.”

Taken together, qualitative evidence suggests that formalizing procedures for teacher involvement in the HOPS intervention and adding a missing assignments intervention that incorporates monitoring and rewarding of assignment completion would enhance the intervention.

Parent Involvement

The HOPS intervention included two parent meetings. These meetings took place towards the end of the intervention; between sessions 9 and 10 and between sessions 14 and 15. The purpose of these meetings was to transfer the monitoring (i.e. checklist) and incentive systems from the SMH provider to the parent. SMH providers were to facilitate the completion of a written transition plan that contained detailed responsibilities of families and students and a home-based incentive plan. SMH providers indicated that the two parent meetings should occur earlier in the HOPS intervention. In particular, SMH providers felt that the first parent meeting should be earlier in the intervention so that they could provide parents with an overview of the intervention and get parents involved in providing incentives more quickly.

Student Engagement and Agency

During focus groups, SMH providers and teachers noted that some students seemed to get more out of the intervention than others. SMH providers suggested that this might be a function of certain students understanding the purpose of the HOPS intervention and “wanting to improve.” A couple of the SMH providers noted that the scripts for SMH providers in the manual did little to emphasize the greater purpose of the intervention. They suggested that if students made the connection between implementation of the HOPS skills and improved academic performance this might facilitate student engagement and motivation. For example, one SMH noted that for her student, “Never was it that there was a clear connection that, oh, this is helping me track my time at home and utilizing my time better is helping me with my academics or getting my grades up.” Another SMH provider commented that her student didn’t understand that the purpose of the “whole thing [using agenda and receiving teacher initials] is to see the benefit of writing down assignments.” The idea that the scripts written for SMH providers did not do enough to emphasize the connection between the HOPS skills and larger academic outcomes and to establish student engagement in the intervention was also evident from coding the sessions transcripts.

HOPS Session Content Analyses

The primary theme revealed from coding the session transcripts was that the strategies SMH providers used to engage students in the HOPS intervention varied considerably. Inspection of the language used by SMH providers to facilitate student agency and engagement revealed that SMH providers primarily used four strategies for engaging students in the session: 1) pairing student behaviors with academic performance; 2) thinking out-loud to teach decision-making process skills; 3) scaffolding responsibility of the intervention for the student; and 4) promoting student ownership of the organization and homework systems. Below we provide a few examples of how each of these strategies was used.

One SMH praised the student’s behavior noting, “You did a really good job of writing those [assignments] down.” She followed the praise statement asking, “Is it helpful then when you go home and you know what you have to do? What ways does it help?” This type of language afforded the student the opportunity to link his assignment recording behavior with additional positive outcomes (i.e. knows what to work to complete, and/or parents don’t get upset). Thinking out-loud was another technique that worked well for some SMH providers. For example, while examining a paper, one SMH provider stated, “Is this something that needs to be turned in? Oh, it looks like it has already been graded, this is a B, and so it looks like this is a paper that can go home. So, I’ll put it in front.” Another example occurred during completion of the organization checklist: “I’m giving you a yes for this, because your keys are the only thing extra in your bag, and those are a necessity.”

SMH providers also used language to scaffold student decision making. For example, one SMH provider began with, “So, do you want to put this in your language arts folder, or do you think it would be better in your homework folder?” After the student answered, she followed with, “Right, the language arts folder because it does not have to be turned in tomorrow.” Another SMH provider used the strategy of taking turns with the student evaluating the individual criterion on the organization checklists (e.g., I do one, you do one).

Analysis of the session content revealed that some SMH providers afforded students choices when possible for how they set-up the organization system. For example, one SMH provider allowed a student to choose where in his organization system to keep his assignment notebook and pencil bag. This helped the student take ownership of the system. Other SMH providers used language to convey that students were in charge of making changes. One SMH provider noted, “This is your binder, so you have to take care of it.” Another took care to notice the efforts made by her student and challenged him to do more: “You are making it a priority so that is great. The next thing I want to talk to you about is how we can step it up a bit, and I can give you a little more challenge, right?” These types of statements encouraged the students to take more responsibility and ownership of the changes they were making with the intervention.

Discussion

The purpose of this study was to modify, test, and refine the Homework, Organization, and Planning Skills (HOPS) intervention for adolescents with ADHD for use by SMH providers in school settings. This study used a combination of qualitative and quantitative methodologies to systematically test and refine the HOPS intervention. Outcome measures collected pre- and post-intervention revealed that participants made large improvements in organization, homework and time-management skills according to parents. Teacher ratings indicated that participants did not improve in these areas (see Table 3). Coding of SMH provider and teacher focus group data revealed five consistent themes that led to a number of important revisions to the HOPS intervention manual (see Table 4).

The results of this study (i.e. improvements on an objective skills checklist, significant parent ratings and non-significant teacher ratings) are nearly identical to the outcomes produced with the initial evaluation of the HOPS intervention as implemented by research staff (Langberg et al., 2008). The organization skills checklist completed by SMH providers at all HOPS intervention sessions showed that participants improved their binder, bookbag, and locker materials organization during the intervention period (see Figure 1). In addition, parents reported that participants made large and significant improvements in homework management, homework completion, organization, and planning as measured by the HPC and COSS and participants reported making moderate gains with organization on the COSS self-ratings (see Table 3). It is noteworthy that the mean total score on the parent COSS was above 70 pre-intervention (severe problems with organization and time-management), and below 60 post-intervention (within the average range). The fact that outcomes generated by research staff who received intensive supervision were replicated in this study by SMH providers who received no formal supervision or consultation provides encouraging preliminary evidence supporting the potential for dissemination of the HOPS intervention.

In contrast to the objective skills measures and parent ratings, teacher ratings on the COSS indicated that student’s made no improvements with organization or planning. This discrepancy is not particularly surprising for two reasons. First, many of the skills taught by the HOPS intervention are not directly observable by teachers. For example, it unlikely that a middle school teacher would have reason to look through a student’s bookbag or locker or to review the student’s planner in great detail. Further, many of the time-management skills participants’ learned were implemented outside of the school setting (e.g. developing evening schedules to plan out and manage after-school activities). As noted by teachers during the focus groups, the aspect of organization and homework management that is most salient to teachers is assignment completion, which contributes directly to students’ class grades. One of the main revisions made to the HOPS intervention based on focus group data was the addition of a missing assignment intervention component. Specifically, in the next iteration of the HOPS manual, parents will be taught during the first parent meeting to provide daily contingencies for the number of assignments students fail to turn in each class day. This addition to the HOPS intervention may result in noticeable teacher-rated intervention effects in future studies. Second, the validity of teacher ratings in middle school has been questioned in multiple studies (e.g., Evans, Allen, Moore, & Strauss, 2005). Middle school teachers typically provide instruction for over 100 students every school day and observe students less than 1 hour per day. As a result, middle school teachers may have difficulty accurately rating less observable, covert behaviors such as materials organization and homework recording.

One of the primary aims of this study was to determine if SMH providers could implement the HOPS intervention with fidelity. This is a critical step in the intervention development process and a step where many empirically-based interventions fall short (Weisz, Donenberg, Han, & Weiss, 1995). That is, when interventions are disseminated to real-world settings they are often not implemented with fidelity and therefore, fail to produce the types of improvements that were generated in research studies (Weisz, Donenberg, Han, & Kauneckis, 1995). SMH providers in this study were able to implement the HOPS intervention with fidelity. Specifically, direct observation data showed that SMH providers implemented the HOPS sessions as specified in the manual and completed the skills checklists accurately. High levels of fidelity were achieved despite the fact that no formal consultation was provided on intervention implementation. This is an important finding that improves the potential for the HOPS intervention to be widely disseminated. Specifically, consultation services are not always available to SMH providers for a variety of reasons (e.g. lack of specialists and resources) and accordingly, interventions that do not require outside supports to implement have greater potential for dissemination.

Qualitative analysis of focus group data and HOPS session content revealed a number of ways to further refine the HOPS intervention (see Table 4). These changes would not have been identified if a collaborative intervention development approach had not been used. Interventions are often developed with little or no input from community and SMH providers. This lack of collaboration during the intervention development has been identified as an important factor contributing to the research to practice gap (Weisz, 2000). In this study, data collected from SMH providers during the focus groups revealed that the pace of skills introduction was too fast. Specifically, SMH providers indicated that additional sessions needed to be devoted to problem-solving materials organization and homework management skills. Teacher focus group data indicated that improvements in organization and planning do not necessarily translate into improvements with assignment completion. Teachers suggested that a missing assignments intervention might be a useful addition to the HOPS intervention. Finally, coding of session transcripts revealed considerable variability in the strategies SMH providers used to engage students in the intervention and to promote student ownership of the intervention systems. Based upon the results of the qualitative analyses, two additional sessions were added to the intervention that focus entirely on problem-solving materials organization and homework management skills, a missing assignments component was added, and scripts that provide examples of ways to promote student engagement were included. The HOPS intervention with these modifications is now being evaluated.

Limitations

This was a pilot study and did not include a control group. Accordingly, firm conclusions about the efficacy of the intervention cannot be drawn from the outcome data reported. Research with the HOPS intervention using randomized controlled methodology is needed before the intervention can be considered efficacious. In addition, selection bias related to the SMH providers who implemented the intervention may limit generalizability of the findings to other SMH providers. Specifically, the SMH providers in this study may be a unique group given that they volunteered to participate in the study. It should be noted however that the SMH providers in this study were diverse in terms of age, years of experience, and education training background, which may improve the generalizability. There are also limitations with our treatment fidelity data. SMH providers were aware that their sessions were being observed and this may have influenced their behavior. However, we were able to evaluate SMH providers’ completion of the HOPS skills checklists for all intervention sessions which provides a measure of fidelity not subject to this bias. Finally, this study did not include a follow-up assessment to determine if gains with organization and homework management skills were maintained after participants stopped working with SMH providers.

Future Directions

An important next step is to examine predictors and moderators of response to the HOPS intervention. It may be that certain types of children (e.g. children with a Learning Disorder) benefit to a lesser or greater extent from the HOPS intervention than do other children. This would be important information for SMH providers to have when they are choosing students to target with the HOPS intervention. Currently, the HOPS intervention is designed to be implemented on an individual 1:1 basis. This type of intervention model may limit the potential for generalization as schools may not have the resources to deliver individualized intervention to large numbers of students. Accordingly, a group-based delivery model of the HOPS intervention should be evaluated. Finally, it is likely that students other than young adolescents with ADHD can benefit from these types of interventions. The efficacy of the HOPS intervention with adolescents generally considered at-risk but not meeting criteria for ADHD and with elementary age children with ADHD should be evaluated.