Parent-Reported Homework Problems in the MTA Study: Evidence for Sustained Improvement with Behavioral Treatment

Abstract

Parent-report of child homework problems was examined as a treatment outcome variable in the MTA - Multimodal Treatment Study of Children with Attention-Deficit/Hyperactivity Disorder (ADHD). Five hundred seventy-nine children ages 7.0–9.9 were randomly assigned to either medication management, behavioral treatment, combination treatment, or routine community care. Results showed that only participants who received behavioral treatment (behavioral and combined treatment) demonstrated sustained improvements in homework problems in comparison to routine community care. The magnitude of the sustained effect at the 24-month assessment was small to moderate for combined and behavioral treatment over routine community care (d = .37; .40, respectively). Parent ratings of initial ADHD symptom severity was the only variable found to moderate these effects.

The Multimodal Treatment Study of ADHD (MTA) was a multisite, randomized clinical trial of well-established treatments for children with Attention-Deficit/Hyperactivity Disorder (ADHD). Children were randomly assigned to receive 1 of 4 treatments: systematic medication management (MedMgt), intensive behavioral treatment (Beh), the combination of the two (Comb), or routine community care comparison (CC). The active treatment portion of the MTA study was 14 months in length, with follow-up assessments collected thereafter (e.g., at 24 and 36 months post-randomization). The main outcome analyses examined the impact of the MTA treatments on six outcome domains (comprising 19 separate variables), including 1) ADHD symptoms; 2) Aggression-ODD; 3) Internalizing Symptoms; 4) Social Skills; 5) Parent-Child Relations; and 6) Academic Achievement (MTA Cooperative Group, 1999). These variables were also examined in secondary analyses as a global composite outcome measure including all 19 variables (Conners et al., 2001) and as a symptom-based narrow composite including parent and teacher ADHD- and ODD-related symptoms (Swanson et al., 2001).

Across the six outcome domains, all four groups showed significant improvement between baseline and 14 months. Analyses using a broad composite variable showed a statistically significant effect for Comb over MegMgt (d = .28), MedMgt over Beh (d = .26), and for Comb and MedMgt over CC (d = .70; d = .35, respectively), but not Beh over CC (d = .09; Conners et al., 2001). Similarly, analyses using a symptom-based narrow composite variable showed a statistically significant effect of Comb over MedMgt (d = .26), Comb over CC (d = .83) and MedMgt over CC (d = .45) but not Beh over CC (d = .09; Swanson et al., 2001). Analyses completed at 24 months (i.e., 10 month follow-up) revealed sustained treatment effects for two of the original six outcome domains, ADHD and ODD symptoms (MTA Cooperative Group, 2004). Specifically, there was a significant effect for participants who received the MTA medication treatment (Comb and MedMgt) versus those who did not (Beh and CC) at 24 months. Treatment group differences were no longer evident at 36 months (Jensen et al., 2007).

Many of the 19 variables included in the global composite outcome have also been examined separately and in more detail (e.g., Hinshaw et al., 2000; Wells, Epstein et al., 2000). For the most part, similar patterns of results have emerged across outcome variables. One noteworthy finding is that the Beh group demonstrated statistically significant superiority over CC on only one outcome variable, parent-rated negative parenting, a difference that did not maintain at the 24-month assessment (Wells, Epstein et al., 2000). The lack of a statistically significant difference between Beh and CC across the vast majority of outcome variables may be attributed to the fact that many participants in the CC group (67.4%) received medication through community prescribers (MTA Cooperative Group, 1999).

Compared to other MTA outcome domains, especially symptomatology, the academic functioning domain has received relatively little attention and analysis. This is surprising given that the focus of many of the MTA behavioral interventions was on school functioning. As part of the behavioral intervention parents learned techniques for improving their children’s academic performance, participants’ teachers received bi-weekly consultation, and a paraprofessional aide was assigned to work directly with each child in the classroom for a 3 month period (MTA Cooperative Group, 1999; Wells, Pelham et al., 2000). Further analysis of academic outcomes in the MTA is warranted because academic impairment is one of the most prevalent and problematic impairments associated with ADHD (DuPaul & Stoner, 2003). Children with ADHD consistently underachieve and are more likely than their peers to be retained, to be placed in special education, and to drop out of school (Barkley, Fischer, Edelbrock, & Smallish, 1990; Faraone et al., 1993; Hinshaw, 1992).

The only academic outcome variables that have been examined in the MTA are standardized achievement test scores: the Wechsler Individual Achievement Test (WIAT) Reading, Math and Spelling subtest scores. From baseline to 14 months, children in all four groups made significant gains on WIAT Reading subtest scores, whereas no significant gains were made on the WIAT Math or Spelling subtests. Pairwise contrasts for WIAT Reading scores revealed that at 14 months, Comb had significantly higher scores than Beh and CC, with no significant difference between Comb and MedMgt (MTA Cooperative Group, 1999). Within-group effect size calculations for the Comb group show that participants made small gains in reading from baseline to 14 months, d =.20. The Beh group did not fare better than CC on any WIAT subtest. At 24 and 36 months, there were no longer significant treatment effects on any WIAT subtest (Jensen et al., 2007; MTA Cooperative Group, 2004).

In addition to assessing MTA participants’ performance on standardized achievement tests, parent ratings of participants’ homework problems were collected through the 36-month assessment. These ratings have not previously been examined. Homework completion is a major component of children’s academic functioning and accounts for approximately 20% of the total time students invest in academics (West Chester Institute for Human Services Research, 2002). Furthermore, the amount of time spent on homework and the amount of homework completed are both positively correlated with class grades and achievement test scores (Cooper, Lindsay, Nye, & Greathouse, 1998; Cooper, 1989). The relationship between homework and academic achievement is moderated by grade in school and is strongest in secondary school (Cooper, Robinson, & Patall, 2006). Homework also serves as a means of promoting family involvement in education and plays and important role in both the family-school relationship and parent-child relationship (Rogers, Wiener, Marton, & Tannock, 2009).

Children with ADHD have significantly more problems with homework than their peers (Power, Werba, Watkins, Angelucci, & Eiraldi, 2006). Specifically, children with ADHD often forget to record their assignments or record them inaccurately, fail to complete assignments, complete assignments but forget to turn them in, and make careless mistakes in their work (DuPaul & Stoner, 2003; Langberg, Epstein, Urbanowicz, Simon, & Graham, 2008; Raggi & Chronis, 2006). Children with ADHD also have significant difficulties with homework management, such as planning for the completion of long-term projects and studying for tests (Power, Karustis & Habboushe, 2001).

The primary goal of the present study is to examine the impact of the MTA treatments on parent-rated homework problems. Portions of the MTA behavioral parent training curriculum, teacher consultation protocol, and child-focused interventions were specifically focused on improving problems with homework completion and management. Stimulant medication produces marked improvements in symptoms of inattention and distractibility which contribute to homework problems in children with ADHD. Accordingly, we predicted that participants in all of the MTA treatment groups (MedMgt, Comb & Beh) would have significantly fewer homework problems at 14 months than children in the CC group. Previous publications have documented significant decreases in medication use for the Comb and MedMgt groups following active treatment (i.e. after 14 months; Jensen et al., 2007). However, parents in the Comb and Beh groups might be expected to continue implementing behavioral strategies related to homework management post-treatment. Accordingly, we predicted that only participants in the Comb and Beh groups would exhibit a sustained effect of treatment over CC at the 24-month assessment. Given that no treatment group differences have been found on any variable at 36 months, we predicted that there would be no significant treatment group differences on homework problems at that point.

A secondary aim of this study was to explore possible moderation effects. Previous research has shown that children with a Learning Disability (LD) and/or ADHD have significantly more homework problems than their peers (Epstein, Polloway, Foley, & Patton, 1993; Lahey et al., 1994; Power et al., 2006). Further, boys typically have more severe homework problems in comparison to girls (Power et al., 2006). Finally, African American children have historically experienced lower academic achievement when compared to Caucasian children (Tucker & Herman, 2002). Accordingly, we examined the potential moderating effects of LD status (reading, math, and spelling), Full Scale IQ, receipt of school services, severity of parent and teacher rated ADHD symptoms, gender, and race. We also included medication use as a variable in the moderator analyses to test our hypothesis that the decrease in ADHD medication use for the Comb and MedMgt groups following active treatment would be associated with an increase in parent-rated homework problems.

Method

Participants

Participants were children (n = 579) between 7.0–9.9 years of age (grades 1 – 4) who had a diagnosis of ADHD Combined Type at the time of recruitment (American Psychiatric Association, 2000). This diagnosis was determined using the Diagnostic Interview Schedule for Children, Parent Report (DISC-P 4.0; Shaffer, Fischer, Lucas, Dulcan, & Schwab-Stone, 2000), supplemented with up to two symptoms identified by children’s teachers using the SNAP-IV (Swanson, 1992) for cases falling just below the DISC diagnostic threshold by parent report. Co-occurring oppositional defiant or conduct disorders (54%), anxiety disorders (33.5%), and affective disorders (3.8%) were diagnosed with the DISC-P. Sixty-one percent were Caucasian, 20% African-American, 8% Hispanic, and 11% other (Asian, Pacific Islander, mixed, etc.). Eighty percent of the sample were boys.

Procedures

In a 4-group randomized clinical trial design, children were randomly assigned to MedMgt, Beh, Comb, or CC for 14 months of treatment. In order to assess treatment response, assessments were performed at baseline, 3 months, and 9 months into treatment, and at the end of treatment (14 months). Multi-domain and multi-source follow-up assessments were completed at 24 months and 36 months, and the sample continues to be followed. The measure of interest for this study, the Homework Problems Checklist (HPC), was completed at all assessment points through 36 months. Participant retention rate was 97% at 14 months, 93% at 24 months and 84% at 36 months. At the 36-month follow-up participants ranged in age from 10 to 14 years (M = 11.8). There were no significant differences in baseline characteristics between subjects participating in the 36-month assessment and those that did not complete the assessment and follow-up rates did not vary significantly across the four treatment groups (see Jensen et al., 2007). Further, at baseline, the four treatment groups did not differ on key demographic variables, including Wechsler Intelligence Scale for Children – Third Edition (WISC-III) Full Scale IQ (M across groups = 100.9; SD = 14.8), race, gender, comorbidities, and severity of ADHD symptoms (see MTA Cooperative Group, 1999).

Children and their parents provided informed assent and consent, respectively, during the baseline assessments, using each site’s IRB-approved procedures and documents. These included consent for the collection of rating scales reported in this study. A more complete description of the design, assessment battery, interventions, and follow-up procedures and assessment battery is described elsewhere (Arnold, 1997; Hinshaw et al., 1997; MTA Cooperative Group, 1999; Wells, Pelham et al., 2000). The components of the behavioral intervention directly related to child homework performance are briefly described below.

The MTA behavioral intervention curriculum consisted of parent training, a school intervention component, and a summer treatment program (Wells, Pelham et al., 2000). The initial parent training sessions focused on setting up a Daily Report Card (DRC) to facilitate communication between parents and teachers regarding child behavior and work completion. In later sessions, parents were also taught how to set up a token economy system and how to use the system to support homework completion. Parents were given a script for establishing a DRC with future teachers and a script for monitoring the implementation of the DRC for fidelity. Parents were also taught strategies for structuring the setting where homework was completed to reduce potential distractions. As part of the school intervention component, teachers received consultation on how to use contingency management to improve child behavior and work completion. Each child was also assigned a half-time classroom aide for 12 weeks, who used behavior modification techniques to encourage positive behavior and work completion. Finally, in the Summer Treatment Program (Pelham, Fabiano, Gnagy, Greiner, & Hoza, 2005), children spent 3 hrs daily (out of 9 hrs overall) in classroom settings and earned rewards for assignment completion and accuracy.

Outcome Measure

Homework Problem Checklist

(HPC; Anesko, Schoiock, Ramirez, & Levine, 1987): The HPC is a parent report instrument consisting of 20 items that is commonly used as a screening tool for and outcome measure of homework problems. For each item, parents rate the frequency of the problem on a 4-point Likert scale (0 = never, 1 = at times, 2 = often, 3 = very often). The measure has excellent internal consistency, with alpha coefficients ranging from .90 to .92 and corrected item-total correlations ranging from .31 to .72 (Anesko et al., 1987). Factor analyses conducted by Power et al., (2006) in a sample of general education students (N = 675) and in a clinic-based sample (N = 356) indicate that the HPC has two distinct factors, Inattention/Avoidance of Homework (IA) and Poor Productivity/Nonadherence to Homework Rules (NA). Twelve items load on Factor I and eight items load on Factor II (Power et al., 2006). Example items from Factor I include: a) Must be reminded to sit down and start homework; and b) Puts off doing homework, waits until last minute. Example items from Factor II include: a) Fails to bring home necessary materials (textbooks); and b) Doesn’t know exactly what homework has been assigned (see Anesko et al., 1987 for a list of all HPC items). These factors were consistently extracted both in a large general education sample and in a clinic sample containing children with ADHD. Both factors have moderate to high correlations (Factor I = .67; Factor II = .61) with the Inattention factor on the Behavior Assessment Scale for Children (BASC; Reynolds & Kamphaus, 1992), parent version (Power et al., 2006). The IA and NA factors can be combined to produce an HPC Total Score.

Statistical Analyses

To ensure accurate comparisons between these analyses and those from previous MTA analyses, we replicated the statistical techniques utilized in the original 14-month treatment outcome paper (MTA Cooperative Group, 1999). Specifically, we used intention-to-treat (ITT) mixed-effects regression procedures using SAS Proc Mixed. For the outcome variables of interest (HPC Factor I, Factor II, and Total Score), we completed tests for site, time, time X treatment group (treatment group effects over time), site X time, and site X treatment group X time. These analyses were conducted separately for the HPC Factors and Total Score at the 14-, 24-, and 36-month assessments; all available assessment points were included each time (i.e., 3- and 9-month assessments). As with the original 14-month analyses, time was expressed as the log of the number of days since randomization for each assessment point (MTA Cooperative Group, 1999). The intercept and time on a log scale are treated as random effects while treatment group and site are treated as fixed. For all analyses, unstructured variance covariance structure was used since it produces the smallest Bayesian Informational Criterion values when compared to other structures considered. When omnibus regression analyses comparing all four groups were significant, six pairwise comparisons were performed: (1) MedMgt v. Beh; (2) MedMgt v. Comb; (3) Comb v. Beh; (4) CC v. MedMgt; (5) CC v. Comb; and (6) CC v. Beh. To remain consistent with the MTA Cooperative Group (1999) paper, we applied Bonferonni corrections to the six pairwise contrasts to control for Type I error (p<.05 / 6 = p<.008).

Moderator Analyses

For the moderator analyses, we completed the above described mixed-effects regression analyses including each moderator variable as a main effect and interaction. When a significant three-way interaction was found (i.e. time x treatment group x moderator variable) we examined the data further by separating the moderator variable into levels (e.g. male vs. female) and representing the data graphically. We then performed the pairwise comparisons of treatment group and examined interactions of treatment condition with time at each level of the moderator.

Currently, there is disagreement about the best way to diagnose a LD. IQ/Achievement discrepancy approaches vary from 1 SD difference to 2 SD difference (see Dombrowski, Kamphaus, & Reynolds, 2004 for a discussion of this topic). A score of < 85 on a subtest of the WIAT indicates a basic skills deficit (i.e., in reading, math or spelling) that would likely necessitate specific, direct instruction intervention. Accordingly, the < 85 definition was used for the moderator analyses as representative of a students with a potential LD. Using this definition, N = 108 students met criteria for potential Reading Disability, N = 95 students for Math, and N = 128 students for Spelling. The percentage of children with each type of potential LD did not vary significantly at baseline as a function of treatment group. Each type of LD was examined separately in the moderator analyses.

ADHD symptom severity was measured using the SNAP-IV Rating Scale (Swanson, 1992). The SNAP includes the 18 ADHD items from DSM-IV (9 DSM inattention and 9 DSM hyperactive/impulsive symptoms). Parents and teachers respond on a 4-point Likert scale rating the severity of symptoms (i.e., 0 = not at all, 1 = just a little, 2 = pretty much, and 3 = very much). For the moderator analyses, the SNAP ADHD Total Score was examined (sum of 18 DSM ADHD items) separately for parent and teacher ratings. Receipt of school services was examined as the total hours per week of special education, counseling or therapy in school, and/or other school services (e.g. tutoring) as reported by parents on the Services Use in Children and Adolescents-Parent Interview (SCA-PI; Jensen et al., 1994). This structured interview was administered every 6 months, either by phone or during the face-to-face assessments. Parents also reported about children’s ADHD medication use on the SCAPI. The percent of days in the interval between the last assessment and the current assessment that any stimulant medication was taken was used as an indicator of ADHD medication usage. Test-retest reliability using an 18 day between test interval for reporting medication use on the SCAPI is excellent, kappa = .97 (Hoagwood et al., 2004).

Missing Data

We analyzed the impact of missing data on the findings by completing the analyses in two ways: once with inclusion of all subjects, and then with only those subjects who provided data over multiple time points. No differences emerged between these two sets of analyses. To examine the magnitude of treatment gains in homework problems we calculated between-groups Cohen’s d effect sizes, using standardized mean difference scores.

Results

Table 1 presents the effects of site, time, treatment group, and their interaction at the 14-, 24-, and 36-month assessments. The results of the pairwise comparisons and between-group Cohen’s d values are presented in Table 2. The mean values for the HPC Factor scores and Total Score at each assessment are presented in Table 3 and represented graphically in Figures 1 – 3.

Table 1.

Random Effects Regression Analyses

Effect	df	F	p	df	F	p	df	F	p
	HPC Factor I			HPC Factor II			HPC Total Score
Model to 14 months
Site	5, 742	2.65	0.0221	5, 742	1.98	0.0802	5, 742	2.36	0.0386
Time	1, 507	476.36	<.0001	1, 507	145.10	<.0001	1, 507	403.05	<.0001
Treatment Group	3, 742	1.44	0.2312	3, 742	2.74	0.0426	3, 742	1.94	0.1222
Treatment Group x Time	3, 742	7.22	<.0001	3, 742	2.18	0.0894	3, 742	6.01	0.0005
Site x Time	5, 742	0.62	0.6856	5, 742	1.12	0.3456	5, 742	0.66	0.6524
Site x Treatment Group x Time	15, 742	1.64	0.0595	15, 742	0.80	0.6817	15, 742	1.3	0.1979
Model to 24 months
Site	5, 1203	2.76	0.0173	5, 1203	2.25	0.0474	5, 1203	2.59	0.0244
Time	1, 528	471.35	<.0001	1, 528	116.67	<.0001	1, 528	382.88	<.0001
Treatment Group	3, 1203	2.31	0.0742	3, 1203	2.93	0.0326	3, 1203	2.60	0.0505
Treatment Group x Time	3, 1203	3.24	0.0213	3, 1203	0.78	0.5077	3, 1203	2.54	0.0548
Site x Time	5, 1203	0.78	0.5647	5, 1203	0.98	0.4282	5, 1203	0.63	0.6774
Site x Treatment Group x Time	15, 1203	1.31	0.1852	15, 1203	0.38	0.9836	15, 1203	0.84	0.6289
Model to 36 months
Site	5, 1655	2.72	0.0187	5, 1655	2.24	0.0479	5, 1655	2.56	0.0259
Time	1, 553	427.02	<.0001	1, 553	59.14	<.0001	1, 533	307.33	<.0001
Treatment Group	3, 1655	3.67	0.0119	3, 1655	3.72	0.0111	3, 1655	3.87	0.0090
Treatment Group x Time	3, 1655	1.17	0.3183	3, 1655	0.42	0.7420	3, 1655	0.80	0.4933
Site x Time	5, 1655	1.20	0.3076	5, 1655	0.72	0.6085	5, 1655	0.77	0.5680
Site x Treatment Group x Time	15, 1655	1.45	0.1180	15, 1655	0.73	0.7574	15, 1655	1.17	0.2849

Table 2.

Pairwise Comparisons at the 14- and 24-month Assessment Points

	t	p	Cohen’s d	t	p	Cohen’s d	t	p	Cohen’s d
	HPC Factor I			HPC Factor II			HPC Total Score
14-month Assessment
MedMgt vs. Beh	−0.11	0.9102	−.02	1.19	0.2282	.16	0.38	0.7048	.05
MedMgt vs. Comb	1.87	0.0623	.25	1.58	0.1146	.21	1.89	0.0596	.25
Comb vs. Beh	−2.24	0.0257	−.29	−0.52	0.6054	−.07	1.80	0.0733	−.23
CC vs. MedMgt	2.75	0.0064*	.37	0.87	0.3878	.12	2.13	0.0340	.29
CC vs. Comb	4.93	<.0001*	.63	2.55	0.0112	.33	4.36	<.0001*	.57
CC vs. Beh	2.99	0.0031*	.39	2.22	0.0276	.29	2.88	0.0044*	.39
24-month Assessment
MedMgt vs. Beh	0.72	0.4702	.09	1.74	0.0827	.23	1.15	0.2518	.15
MedMgt vs. Comb	0.96	0.3358	.12	0.87	0.3847	.11	1.00	0.3190	.13
Comb vs. Beh	−0.31	0.7545	−.04	0.86	0.3929	.11	0.08	0.9398	−.01
CC vs. MedMgt	1.92	0.0555	.26	1.37	0.1724	.18	1.85	0.0653	.24
CC vs. Comb	2.92	0.0038*	.37	2.26	0.0246	.29	2.86	0.0045*	.37
CC vs. Beh	2.83	0.0051*	.36	3.09	0.0023*	.40	3.09	0.0022*	.40

Note. MedMgt = Medication Management Group; Beh = Behavioral Treatment Group; Comb = Combined (Medication + Behavioral) Group; CC = Routine Community Care Group; Pairwise comparisons were not significant at 36 months for either HPC factor or for the HPC Total Score. Cohen’s d = between treatment group effect calculated using standardized mean difference scores; treatment group 1 mean at assessment point - treatment group 2 mean at assessment point / pooled standard deviation. Negative d indicates that direction of difference favors the first treatment group of the pair;

^*represents a statistically significant effect (p<.008) after the Bonferonni correction.

Table 3.

Means, Standard Deviations, N, and Cohen’s d for the HPC at all Assessment Points

Treatment Group	Assessment Point	Mean	SD	N	Mean	SD	N	Mean	SD	N
		HPC Factor I			HPC Factor II			HPC Total Score
CC	0	25.33	7.52	141	9.28	5.62	141	34.67	11.95	141
	3	18.82	8.50	55	6.73	4.99	55	25.55	12.53	55
	9	20.57	9.29	130	7.78	5.76	130	28.35	14.24	130
	14	18.66	8.46	118	6.97	5.43	118	25.63	13.02	118
	24	18.77	8.84	115	7.70	5.34	115	26.48	13.46	115
	36	18.45	9.58	111	8.14	6.10	111	26.61	15.13	111
MedMgt	0	23.91	8.03	140	8.76	5.27	140	32.73	12.03	140
	3	13.80	8.86	60	4.42	3.98	60	18.33	12.34	60
	9	14.93	8.44	122	5.66	4.73	122	20.6	12.44	122
	14	15.49	8.85	107	6.35	5.30	107	21.85	13.55	107
	24	16.51	8.74	111	6.78	4.63	111	23.3	12.34	111
	36	17.82	8.83	108	7.97	5.48	108	25.89	13.64	108
Comb	0	23.59	8.49	139	8.18	4.99	139	31.87	12.51	139
	3	12.59	7.99	110	3.75	3.28	110	16.38	10.59	110
	9	13.73	8.36	125	5.06	4.42	125	18.79	11.93	125
	14	13.38	8.12	121	5.31	4.54	121	18.71	11.46	121
	24	15.40	9.17	131	6.27	4.57	131	21.67	12.86	131
	36	16.47	8.63	118	7.29	5.43	118	23.76	13.37	118
Beh	0	24.01	8.21	139	7.71	4.79	139	31.72	11.95	139
	3	17.98	8.74	105	5.53	4.39	105	23.51	12.16	105
	9	17.96	7.80	120	6.35	4.69	120	24.32	11.6	120
	14	15.61	7.13	117	5.60	3.91	117	21.24	10.18	117
	24	15.73	7.81	126	5.81	3.97	126	21.56	10.97	126
	36	17.37	8.42	120	7.48	5.25	120	24.84	12.87	120

Note. MedMgt = Medication Management Group; Beh = Behavioral Treatment Group; Comb = Combined (Medication + Behavioral) Group; CC = Routine Community Care Group; Ns at the 3-month assessment are smaller because this assessment point fell during the summer for many of the participants; those in Comb or Beh were in the all-day summer treatment program and more accessible to data collection.

Figure 1.

Homework Problems Checklist Factor I score through 36 months. Higher scores on the Y axis indicate more homework problems. Comb = combination of medication management and behavioral treatments; Med = medication management; Beh = Behavioral Treatment; CC = Routine Community Care Comparison.

Figure 3.

Homework Problems Checklist Total Score through 36 months. Higher scores on the Y axis indicate more homework problems. Comb = combination of medication management and behavioral treatments; Med = medication management; Beh = Behavioral Treatment; CC = Routine Community Care Comparison.

14-month analyses

For HPC Factor I, the omnibus tests revealed a significant effect for site, time, and a significant time X treatment group interaction. Pairwise contrasts revealed three statistically significant effects: MedMgt over CC (p=.0064); Beh over CC (p=.0031); and Comb over CC (p<.0001). No other comparisons reached statistical significance with the adjusted p value of .008. Between-group effect size calculations revealed that the Beh and MedMgt groups made small to moderate gains compared to CC (d = .39; d = .37, respectively) and Comb made a moderate to large gain compared to CC (d = .63). For HPC Factor II, the omnibus tests revealed a significant effect for time and treatment group. Pairwise contrasts did not reveal any significant effects with the adjusted significance cutoff.

For the HPC Total Score, the omnibus tests revealed a significant effect for site, time, and a treatment group x time interaction. Pairwise contrasts revealed two statistically significant effects: Comb over CC (p<.0001) and Beh over CC (p=.0044). Between-group effect size calculations revealed that, relative to CC, Comb made moderate to large gains (d = .57) and Beh made moderate gains (d = .39).

24-month analyses

For HPC Factor I, omnibus tests revealed a significant effect of site, time, and a significant time X treatment group interaction. Pairwise contrasts revealed two statistically significant effects: Beh over CC (p=.0051) and Comb over CC (p=.0038). No other comparisons reached statistical significance. Between-group effect size analyses revealed that the difference between the Beh and CC group was sustained from the 14-month assessment (d = .36). The difference between Comb and CC was also in the small-to-moderate range (d = .37), not the medium-to-large range it had been at 14 months. Between-group effect size calculations revealed small or negligible differences for all other comparisons (see Table 2). For HPC Factor II, omnibus tests revealed significant effects of site, time and treatment group. Pairwise contrasts revealed only one statistically significant effect: Beh over CC (p = .0023). Between-group effect size calculations revealed a moderate difference between Beh and CC at 24 months (d = .40).

For the HPC Total Score, omnibus tests revealed significant effects of site, time and treatment group. The treatment group x time interaction narrowly missed significance at the .05 level (p = .0548). Pairwise contrasts revealed two statistically significant effects: Comb over CC (p = .0045; d = .37) and Beh over CC (p = .0022; d = .40).

36-month analyses

For HPC Factor I, II and for the Total Score, there were significant effects of site, time and treatment group. None of the pairwise comparisons reached significance. Representation of the HPC Factor II data graphically at 36 months (see Figure 2) revealed that the groups that received behavioral treatment (Comb and Beh) were performing somewhat better than groups that did not (MedMgt and CC). However, exploratory analyses revealed this difference was not statistically significant: Comb + Beh over MedMgt + CC (p = .20).

Figure 2.

Homework Problems Checklist Factor II score through 36 months. Higher scores on the Y axis indicate more homework problems. Comb = combination of medication management and behavioral treatments; Med = medication management; Beh = Behavioral Treatment; CC = Routine Community Care Comparison.

Moderator Analyses

Only one of the moderator variables examined resulted in a significant (p<.05) three-way interaction. Parent ratings of ADHD symptoms at baseline moderated the 14 month treatment effects for HPC Factor I, F (3, 728) = 3.14, p < .05, and for the HPC Total Score F (3, 728) = 2.65, p < .05. Given the exploratory nature of the moderator analyses and consistent with the MTA Cooperative Group (1999) paper, pairwise comparisons separated by level of moderator were examined without Bonferonni corrections. Consistent with previous MTA moderator analyses examining initial ADHD symptom severity (Owens et al., 2003), a SNAP item mean score of 2.33 was used as a cut-point. Twenty-five percent of the sample had a baseline SNAP item mean score greater than 2.33 (N = 159; SNAP Item M = 2.61; SD .21). None of the pairwise comparisons were significant for children in the highest 25% of the sample in terms of parent-rated baseline ADHD symptom severity. Specifically, participants in all treatment groups made large improvements (average within group d = 1.28) however, the groups did not differ from each other at 14 months. For children with low to moderate ADHD symptom severity at baseline (bottom 75%; N = 414; SNAP Item M = 1.71; SD .43) Comb was relatively more effective. Specifically, as with the pairwise contrasts for the entire sample, there were significant effects for Comb and Beh over CC at 14 months (ps < .05) but not for MegMgt (p = .11). Unique to the analyses with the moderate baseline severity group was a significant effect for Comb over MedMgt (p = .009; d = .42) and Comb over Beh (p = .006; d = .43) at 14 months.

Discussion

This article extends prior analyses of the MTA study’s effects on academic functioning. In the present study, the three MTA treatment groups, Comb, MedMgt and Beh, did not differ significantly from each other on homework problems ratings at any of the assessment points. All three MTA treatment groups had significantly greater decreases in problems related to inattention and avoidance during homework completion (HPC Factor I) than did routine community care (CC). However, only participants who received the behavioral therapy component (Beh and Comb) sustained this effect over time (i.e. 10 month follow-up). In contrast, none of the MTA treatment groups had significantly greater decreases than CC immediately post-treatment (i.e. 14 month assessment) in problems related to poor homework productivity and non-adherence to homework rules (HPC Factor II). Only the Beh group demonstrated an effect over CC on HPC Factor II at the 10 month follow-up. In terms of overall parent ratings of homework problems (HPC Total Score), only children that received behavioral treatment (Beh and Comb) demonstrated significantly greater decreases relative to CC immediately post-treatment and this effect was sustained at the 10 month follow-up.

These findings are unique and noteworthy for several reasons. First, the HPC is only the second MTA outcome variable examined to show a significant difference between Beh and CC post-treatment (i.e., 14 months). Second, this is the first outcome variable examined for which the advantage of Beh over CC was sustained out to 24 months (i.e., the 14-month effect on negative parenting was not sustained in the 24-month analyses). Third, this is the first time sustained treatment effects (i.e., present at both 14 and 24 months) have been demonstrated on any MTA measure of functioning (i.e. a non-symptom measure).

Moderator analyses examining gender, race, LD status, IQ, medication use, school service use, and teacher-rated ADHD symptoms did not reveal any significant effects. Parent-rated ADHD symptom severity at baseline was the only significant moderator. None of the pairwise comparisons were significant for children in the highest quartile of initial ADHD symptom severity. It is noteworthy that despite the lack of treatment group differences, children with high ADHD severity made large and significant improvements in homework problems from baseline to 14 months. This effect is counter to previous MTA moderator analyses which found that high symptom severity was associated with reduced treatment efficacy for the Comb and MedMgt groups (see Hinshaw, 2007 for a review of MTA moderator effects). Pairwise comparisons with the 75% of the sample with moderate initial ADHD symptom severity produced similar results as pairwise comparisons with the entire sample. One noteworthy finding for this group of children was that participants in the Comb group had significantly fewer homework problems than the MedMgt group and the Beh group at 14 months and this difference was moderate, d > .40. This suggests that Comb treatment was the most effective treatment option for children with moderate parent-rated ADHD symptom severity.

The results of the main analyses varied based upon the aspect of homework problems assessed. Factor I on the HPC (Inattention/Avoidance of Homework) relates to problems that occur during homework completion (Power et al., 2006). For example, parents rate their child’s efficiency of work completion, distractibility, inattention, and the parent-child interactions that occur during homework completion. The behavioral parent training curriculum in the MTA included training in techniques directly related to these problems. For example, parents learned strategies for structuring the homework environment (e.g. selecting a quiet location to minimize distractions), providing effective instructions, and setting up reward systems to encourage on-task behavior. It is evident from numerous studies that medication produces marked reductions in symptoms of inattention and distractibility. The MedMgt protocol took this into consideration and deliberately gave a third dose in the late afternoon to cover homework time. Accordingly, it follows that participants in all three MTA treatment groups (MegMgt, Beh, and Comb) would have significantly fewer homework problems post-treatment than children in the CC group.

Factor II on the HPC (Poor Productivity/Non-Adherence with Homework Rules) relates predominately to behaviors that take place outside of actual homework completion time. Most of the items relate to organization of homework and homework materials (e.g. does not know what homework has been assigned, fails to bring home assignments, and forgets to bring assignments back to class). The behavioral parent training and teacher consultation portions of the MTA treatment protocol included implementation of a daily report card (DRC) with the purpose of increasing communication between parents and teachers surrounding these issues. For example behavioral targets on the DRC often related to assignment completion (e.g. child turned in assigned work today). Stimulant medication may serve to improve some aspects measured by HPC Factor II, but likely not all aspects. For example, medication may improve forgetfulness, but does not teach children skills related to organizing their school materials, planning for tests/projects, accurately recording homework assignments and does not increase parent/teacher communication. This assertion is supported by the data which shows immediately post-treatment, participants in MedMgt made negligible improvements on Factor II relative to CC (d = .12) in comparison to Comb and Beh which made small improvements (d = .33; .29, respectively). Further, only participants in the Beh group were performing significantly better than CC at 24 months (d = .40). A recent study, which examined the effects of stimulant medication on children’s organization, time management and planning behaviors, provides additional support for the specificity of medication effects on these areas of academic functioning (Abikoff et al., 2009).

One possible explanation for the effect of behavioral therapy over routine community care on the HPC relates to the measurement of outcomes in behavioral intervention research. The HPC assesses an aspect of child behavior that was directly targeted for improvement by the MTA behavioral interventions. It is noteworthy that the only other variable to show an effect of behavioral therapy over community care at 14 months, negative parenting, was also directly targeted by the MTA behavioral interventions (Wells, Epstein et al., 2000). Indeed, there may be an association between change in parent behavior and change in child homework behavior. The specific possibility that changes in parenting style mediate improvements in child homework performance should be examined in future research.

The only MTA academic outcome measure previously examined is the WIAT, a standardized achievement test. For achievement deficits, (e.g. reading or math difficulties) best-practice treatment includes explicit direct instruction involving intensive, 1:1 or small group intervention (Lyon et al., 2006). As the MTA behavioral treatment protocol did not include this type of intensive direct instruction, it is not surprising that sustained effects of treatment were not found for standardized achievement test scores. Similarly, behavioral and academic interventions do not explicitly target all DSM ADHD symptoms (e.g., short attention span), a domain for which medication is known to produce substantial improvements (Swanson et al., 2001). Thus, it follows that Beh would outperform CC on a measure of homework problems but not on measures of ADHD symptoms. In fact, current recommendations for the evaluation of behavioral interventions state that the focus of assessment should be on functional impairment rather than on ADHD symptoms (e.g. DuPaul et al., 2004; Pelham, Fabiano & Massetti, 2005; Pelham & Fabiano, 2008).

The sustained treatment effect out to 24 months on the HPC for children who received behavioral treatment (Beh and Comb) is encouraging. The sustainability of behavioral treatment effects is especially important given what is known about long-term patterns of medication use and adherence for children and adolescents with ADHD. For example, analyses of medication use patterns in the MTA sample revealed a significant decrease in medication use for the Comb and MedMgt groups between 14 and 36 months (Jensen et al., 2007). This decrease became even more prominent as the MTA sample moved through adolescence (62% decrease; Molina et al., 2009). Our analyses revealed the decrease in medication use between 14 and 36 months was not associated with the increase in homework problems for Comb and MedMgt participants during this same period (see Table 3). The medication use variable examined in this study represents the percentage of days MTA participants were taking ADHD medications between each assessment interval. An alternate hypothesis is that a decrease in medication efficacy, rather than in medication use, might be associated with the increase in homework problems. Specifically, following active treatment, MTA participants received medication through community providers and medication was not monitored and titrated as frequently or consistently (Jensen et al., 2001). Regardless of the explanation, the combination of decreased medication use and the potential for increased academic difficulties during adolescence (Hinshaw, Owens, Sami, & Fargeon, 2006; Wolraich et al., 2005) magnifies the importance of having behavioral treatment alternatives that produce sustainable academic improvements.

At 36 months, homework problems remained significantly improved over baseline for all groups despite some deterioration between 24 and 36 months (see Table 3). These findings are not surprising given that previous MTA analyses revealed that all group differences present at 24 months (ADHD and ODD symptoms) were no longer evident at 36-months (Jensen et al., 2007). Interestingly, approximately 50% of the MTA sample (N = 258) made the transition to middle school between 24 and 36 months. Recent analyses with the MTA sample found that participants experienced a significant increase in ADHD symptoms during the transition to middle school (Langberg, Epstein, Altaye, et al., 2008). The transition to middle school is associated with numerous environmental changes and increased academic demands. Accordingly, it is possible that the increase in homework problems witnessed between 24 and 36 months is associated with this transition. The deterioration in homework problems following active treatment highlights the fact that ADHD is a chronic disorder that necessitates ongoing treatment.

Limitations

All participants in the MTA sample met DSM-IV criteria for ADHD Combined Type and the results may not generalize to children with Inattentive Type. Some subtype differences related to academic impairment have been identified (Langberg & Epstein, 2009; Pfiffner et al., 2007). For example, the persistence of symptoms of inattention predicts academic performance deficits over time whereas symptoms of hyperactivity and impulsivity do not (Massetti et al., 2008). Thus it may be necessary to separately evaluate the impact of behavioral interventions on the homework problems of children with ADHD Inattentive Type.

The HPC is a parent completed measure. As parents were directly involved in many aspects of the MTA behavioral treatment and were not blind to group assignment, their ratings were subject to rater bias and expectancy effects. However, any such expectancy effects did not prevent the same parents from rating MedMgt and Comb significantly better than Beh on many other measures (MTA Cooperative Group, 1999). Unfortunately, no objective observations of homework problems were completed as was done for parenting behaviors (see Wells et al., 2006). Recently, a teacher-report measure of homework problems was developed (Power, Dombrowski, Watkins, Mautone, & Eagle, 2007). Future research on homework problem interventions should include teacher-report as part of a multi-informant assessment strategy. Another limitation is that some of the HPC items overlap with symptoms of ADHD making it hard to measure the constructs independently. Future research with children with ADHD might use instruments that exclude items directly rated to the core symptoms of ADHD (e.g., Power et al., 2007), or perhaps exclude such items from analysis.

Implications for Research, Policy, and Practice

The Beh vs. CC comparison remains important and highly relevant 10 years after the MTA treatment phase ended. The behavioral treatments that were delivered in the MTA (e.g. parent training and classroom contingency management) continue to be the only behavioral treatments that qualify as well-established (Chronis, Jones & Raggi, 2006; Pelham & Fabiano, 2008). Given our findings, increased efforts are needed to increase the availability of intensive behavioral therapies by training of therapists in these methods and by ensuring necessary health care benefits. Given the temporary sustainability of these efforts (i.e. to the 10-month follow-up), the findings also suggest that studies are needed to determine whether developmentally tailored booster treatments would enhance the longevity of the effects. Relevant to this consideration is the observation that during the time when the behavioral interventions were being faded out (9 – 14 months; Wells, Pelham et al., 2000) the Beh group made continued gains and the Comb group trend was stable (see Figure 3). Arnold et al., (2004) noted that this trend confirmed the effectiveness of the MTA behavioral generalization procedures (i.e. prevention of deterioration) and could form a basis for designing a booster/maintenance session plan.

Children in the Beh group made larger gains in reducing homework problems than children treated in the community. These findings may have implications for children with ADHD whose main area of difficulty is academic functioning. Many parents of children with ADHD would prefer to try a behavioral treatment before pursuing medication (Jensen et al., 1999; Pelham, 2008). This study suggests that for children whose primary area of concern is academic functioning, a behavioral first approach may well be successful. It is noteworthy that academic functioning was not the primary focus of many of the MTA behavioral interventions. For example, only a few of the behavioral parent training sessions were directly related to academic functioning and many academic achievement improvement strategies were not presented (e.g., organization of materials, note-taking or study skills). Further, not all participants had academic functioning targets on the DRC, a core component of the behavioral intervention. It is likely that behavioral interventions specifically designed to target academic functioning would produce even more impressive effects. This assertion is supported by the fact that the HPC Total score means at the end of treatment (see Table 3) were still five to ten points higher (worse functioning) than means from normative samples of children without ADHD (Power et al., 2006). This suggestion is further strengthened by recent studies demonstrating that children with ADHD make large improvements on standardized achievement scores, homework problems, and report card grades with targeted academic intervention (i.e. strategy and skills training) that incorporates behavioral therapeutic techniques (Evans, Serpell, Schultz, & Pastor, 2007; Jitendra et al., 2007; Kern et al., 2007; Langberg, Epstein, Urbanowicz et al., 2008; Raggi et al., 2009).

Intervention protocols are needed that are assessment-based rather than one-size-fits-all. Problems in a multitude of areas can lead to academic impairment. Children can experience difficulties with behavioral functioning in the classroom, achievement deficits (e.g. reading and math skills), homework management problems, and/or materials organization problems, all of which can cause academic impairment (e.g. failing grades). Not all children with ADHD experience problems in all of these areas. Further, as documented in this study, there can be significant within-domain differences in impairment (i.e. problems with certain aspects of homework problems but not with others). Therefore interventions targeting all of these areas at once may be inefficient. Further, the areas of impairment that a child with ADHD exhibits change over time. A child might struggle academically in elementary school due to an achievement deficit (e.g. reading skills) but have difficulty in middle school primarily due to problems with homework management. Therefore interventions are needed that use assessment to guide recommendations for targeted intervention (Pelham & Fabiano, 2008; Pelham, Fabiano & Massetti, 2005).