An Analysis of Teacher Investment in the Context of a Family-School Intervention for Children with ADHD

Abstract

Intervention researchers have often failed to assess treatment integrity; when integrity is examined, the focus is typically on whether the steps of intervention have been applied and not on quality of implementation. In the few studies that have investigated intervention quality, the emphasis has been on how intervention is delivered and not how it is received or the degree of participant engagement. This study was designed to examine participant engagement, specifically teacher investment, in the context of family interventions for children with attention-deficit/hyperactivity disorder (ADHD) that were linked with the school. The interventions included Family-School Success (FSS), a program that engages families and schools in a problem solving partnership, and Coping with ADHD through Relationships and Education (CARE), a program designed to provide education and support to families. Participants included the families and teachers of children in grades 2 through 6; 45 sets of families and teachers participated in FSS and 48 were in CARE. The Teacher Investment Questionnaire (TIQ) was designed to assess teacher engagement in intervention. The TIQ is a rating scale for clinicians to be completed during and after treatment. The findings provided support for the reliability and validity of the TIQ. Level of teacher involvement was demonstrated to decline with advancing grade level for FSS, but not CARE. Parent ratings of the quality of the family-school relationship as well as the level of teacher support for homework, assessed at baseline, were shown to be significantly related to clinician ratings of teacher investment post intervention. The findings highlight the importance of assessing participant engagement in intervention. Strategies to refine the assessment of teacher investment were discussed.

Numerous research investigations have documented the effectiveness of psychosocial approaches for the treatment of attention-deficit/hyperactivity disorder (ADHD). Three types of psychosocial interventions have been demonstrated to meet the criteria for well-established treatments for children with this disorder: behavioral parent training, behavioral classroom management, and behavioral peer interventions (see Pelham & Fabiano, 2008 for a review). In addition, evidence is accumulating to support the effectiveness of behavioral interventions involving the collaborative efforts of the family and school, including daily report cards and strategies targeting homework (see Power & Mautone, 2008).

Intervention Implementation

Although behavioral approaches to managing ADHD generally have been demonstrated to be effective, virtually every intervention study has documented substantial variability in response to intervention. Variability in the degree of intervention implementation is one source of differential outcomes. Problems such as poor attendance at sessions and low adherence to program recommendations have been well documented in family treatment studies of children with disruptive behavior problems (Armbruster & Kazdin, 1994). Further, teacher implementation of school-based interventions for learning and behavior problems generally is poor, and tends to decline after the initial phases of intervention (Noell et al., 2005). Research studies have demonstrated repeatedly that level of intervention integrity is positively correlated with degree of effectiveness (e.g., Henggeler, Melton, Brondino, Scherer, & Hanley, 1997; Hirschstein, Edstrom, Frey, Snell, & MacKenzie, 2007; Noell, Gresham, & Gansle, 2002), with the magnitude of the relationship in the low to moderate range.

Treatment implementation research generally has focused on how much of an intervention is provided (i.e., dosage received and clinician adherence); there has been much less emphasis on how well the intervention has been applied (Dane & Schneider, 1998; Waltz, 1993). In the few studies that have examined quality of implementation, the focus generally has been on how the intervention has been delivered (e.g., Hirschstein et al., 2007) and not how the intervention has been received or the extent of participant engagement (Dane & Schneider, 1998). Participant engagement is a critical aspect of intervention quality; indeed, interventions delivered by practitioners with high quality have little value if they do not contribute to strong engagement on the part of participants (Power, Blom-Hoffman, Clarke, Riley-Tillman, Kelleher, & Manz, 2005).

Implementation in the Context of Family-School Interventions

Family involvement in education can enhance children’s academic and social competence in educational settings (see Christenson & Sheridan, 2001). Researchers have identified three fundamental ways in which families can be involved in education: (a) involvement in the home, such as supporting children with homework; (b) collaboration between family and school, such as conferencing with teachers to resolve issues interfering with the child’s education; and (c) involvement in the school, such as volunteering to be a class aide (Epstein, 1995; Fantuzzo, Tighe & Childs, 2000). Promoting home-based family involvement and collaboration between family and school appear to be especially promising in improving student performance in school (Power & Mautone, 2008).

Teachers have a critical role in promoting family involvement by supporting family involvement in education in the home setting and fostering collaborative partnerships between family and school (Christenson & Sheridan, 2001). However, teachers vary substantially with regard to their commitment and efforts to promote family involvement. Intervention programs have been developed to foster family involvement in education, such as Conjoint Behavioral Consultation (CBC; Sheridan & Kratochwill, 2008) and Family-School Success (FSS; Power, Soffer, Clarke, & Mautone, 2006). These programs involve parents and teachers in a partnership to promote the development of academic and social competence and resolve problems related to educational performance (Power & Mautone, 2008). The success of these programs depends to a large extent upon the willingness of teachers to participate and their level of engagement in the intervention process.

This study examines participant responsiveness, more specifically teacher engagement, in the context of the implementation of two family-based interventions designed specifically to address the concerns of children diagnosed with ADHD and demonstrating significant homework problems. The first intervention program, known as FSS, is a family-school behavioral intervention designed to foster collaborative partnerships between parents and teachers in order to provide evidence-based interventions to children with ADHD in the home and school settings. This intervention incorporates the methods of CBC and includes the use of daily report cards or school-home notes (Kelley & McCain, 1995) and homework strategies, including goal setting with contingency management (Kahle & Kelley, 1994; Power, Karustis, & Habboushe, 2001). This study also explores teacher involvement in the context of a second intervention, known as the Coping with ADHD through Relationships and Education (CARE) program, which involves less investment on the part of teachers relative to FSS. CARE is designed to provide parents with education about ADHD and related impairments and to provide parents with support in coping with this condition. The study builds upon the limited research base related to participant involvement in intervention, which to date has focused primarily on family engagement.

Purpose of Study

This study reports on the development of a practitioner-rated measure of teacher engagement, the Teacher Investment Questionnaire (TIQ), which was developed for both FSS and CARE. This investigation examines variations in teacher investment or engagement in the intervention process as a function of child grade level, degree of school adversity, and family socioeconomic status (SES). Further, the study investigates how teacher engagement is related to the quality of the parent-teacher relationship and the level of teacher support with homework. In addition, the study explores whether teacher investment varies as a function of severity of attention and behavior problems within a high-risk sample of children diagnosed with ADHD and demonstrating significant homework problems. The following research questions and hypotheses were addressed in this study:

1. To what extent do clinician ratings of teacher investment vary as a function of student grade level? Research generally supports the hypothesis that higher grade level is associated with an approach to education that is less family-centered (McWilliam, Maxwell, & Sloper, 1999; Rimm-Kaufman & Pianta, 1999), therefore teacher investment in interventions involving the family was expected to decrease with advancing grade.

2. To what extent is teacher investment related to school adversity, as assessed by the percentage of children in the school who are eligible for free or reduced-price lunch, and family adversity, as assessed by family SES? It was hypothesized that higher levels of student poverty would be associated with lower levels of teacher investment. Teachers in less advantaged schools typically have fewer resources available and cope with greater levels of student aggression (Hughes, Cavell, Meehan, Zhang, & Collie, 2005; Thomas, Bierman, & The Conduct Problems Prevention Research Group, 2006). Also, based upon research demonstrating that lower family SES is associated with lower levels of family-school communication (Manz, Fantuzzo, & Power, 2004), it was expected that level of family SES would be positively correlated with teacher investment.

3. To what extent are clinician ratings of teacher investment in intervention related to the quality of the parent-teacher relationship, as rated by parents and teachers, and teacher support of students’ homework, as rated by parents? We expected that teacher investment in interventions involving the family would be positively correlated with indicators of family and school collaboration at pretreatment, including ratings of the quality of the parent-teacher relationship and the level of teacher support of student homework.

4. To what extent is teacher investment related to severity of child problems, as assessed by severity of ADHD and oppositional behavior? Based upon expectations that the family-school relationship would be more conflictual for cases in which there were higher levels of student problems, it was expected that teacher investment in intervention would be inversely related to severity of child problems.

This study was designed to evaluate the usefulness of a tool for assessing teacher level of engagement in inventions involving the family and school. In addition, the purpose was to identify factors associated with variable levels of teacher engagement that might be useful in developing strategies to improve teacher investment.

Method

Participants

This study was conducted through an ADHD center situated within a tertiary-care pediatric hospital located in the Northeast section of the U.S. Data were collected in the context of a federally-funded study evaluating the effectiveness of a family-school intervention for children with ADHD. Referrals to the study were obtained through the ADHD clinic within the center, which receives referrals for services at the initiative of parents, as well as from school professionals and primary care providers who referred parents directly to the study. Inclusion criteria were the following: (a) Children enrolled in grades 2 through 6; (b) Children meeting criteria for ADHD, Combined Type (ADHD/COM) or ADHD, Inattentive Type (ADHD/I) on the Schedule for Affective Disorders and Schizophrenia for School Age Children – DSM IV (K-SADS-P IVR; Ambrosini, 2000), based upon parent-report; (c) Children rated at or above the 85^th percentile on the Inattention or Hyperactivity-Impulsivity factor of the school version of the ADHD Rating Scale-IV (DuPaul, Power, Anastopoulos, & Reid, 1998), or the Attention Problems or Hyperactivity subscales of the Behavior Assessment System for Children, Second Edition (BASC-2; Reynolds & Kamphaus, 2004); (d) Children scoring at or above 0.75 of a standard deviation above the mean on the Homework Problem Checklist (Anesko, Schoiock, Ramirez, & Levine, 1987); and (e) Children scoring at or above an estimated IQ of 75 on the Wechsler Abbreviated Scale of Intelligence (WASI; Psychological Corporation, 1999).

Children meeting DSM-IV criteria for a psychotic disorder, bipolar disorder, chronic tic disorder or Tourette’s disorder, obsessive-compulsive disorder serious enough to warrant separate treatment, history of major neurological illness, and history of suicidal or homicidal behavior or ideation were excluded. Furthermore, children were excluded if they were currently receiving psychotropic medications, and their parents chose not to undergo a new medication trial as part of the study. Children with learning disabilities (as assessed using tests administered for this study or as reported by multidisciplinary school evaluations), disruptive disorders (oppositional defiant disorder and conduct disorder), and internalizing disorders (anxiety and mood disorders with the exception of bipolar disorder) were included.

Referrals were received for 215 children; 107 cases (50%) met eligibility criteria and consented to the study. The primary reasons that families did not meet eligibility criteria were failure to meet teacher rating criteria for inattention or hyperactivity, failure to meet parent rating criteria for homework problems, and parental decision that they could not fulfill study responsibilities. In 99 cases, teacher consent for study participation was also provided (6 parents withdrew shortly after consenting, and in 2 cases, the school did not consent). Of the 99 eligible families with teacher consent, 93 (94%) provided sufficient data to be evaluable cases. At the time of consent, the family was given the choice of receiving psychosocial intervention with or without medication for treatment of the child’s ADHD symptoms. Families electing medication, including those whose children were receiving medication prior to enrollment, were referred to the medication team of the study for titration and monitoring to determine the optimal medication and dose prior to randomization to an intervention group. Participants were randomly assigned to FSS and CARE by stratifying according to child gender, medication status, ADHD subtype (ADHD/COM versus ADHD/I), learning disability status (present or absent), externalizing disorder status (present or absent), and internalizing disorder status (present or absent). Randomization was limited by accommodating schedule restrictions reported by parents and by efforts to assign students attending the same school to the same intervention group.

Of the 93 cases in the study, 45 participated in FSS and 48 were in CARE. Table 1 reports the distributions for gender, grade level, single-parent family status, ethnic groupings, family socioeconomic status, subsidized lunch status of school, ADHD subtype, learning disability status, medication status, and externalizing and internalizing disorder status (evaluated using the K-SADS-P IVR). The groups differed only in terms of grade level. However, the difference was relatively small; the mean grade level for both groups was in the third grade.

Table 1.

Background Information About Participants in the FSS (n=45) and CARE (n=48) Groups

	FSS	CARE	Difference
Gender (% female)	33.3%	31.3%	n.s.
Grade level (M and SD)	3.9 (1.2)	3.3 (1.1)	p < .05
Single parent status (%)	24.4%	25.0%	n.s.
African American (AA), AA/White (%)	17.8%	22.9%	n.s.
White (%)	75.6%	68.8%	n.s.
Hispanic (%)	6.7%	0.0%	n.s.
Asian (%)	4.4%	0.0%	n.s.
SES (% Levels III, IV, V on Hollingshead)	100%	98%	n.s.
Subsidized lunch status of schools (average %)	17.5%	22.0%	n.s.
ADHD, Combined (%)	44.4%	52.1%	n.s.
ADHD, Inattentive (%)	55.6%	47.9%	n.s.
Learning disability status (% with LD)	13.6%	21.3%	n.s.
Externalizing disorder (% with disorder)	28.9%	22.9%	n.s.
Internalizing disorder (% with disorder)	35.6%	31.3%	n.s.
Medication status (% on medication)	36.8%	48.0%	n.s.

Note: SES refers to socioeconomic status, as assessed by the Hollinghead (1975) index of social status. Levels III, IV, and V reflect the middle to high levels of the scale. Subsidized lunch status refers to the average percentage of students who are eligible for free or reduced-price lunch across the schools represented in the FSS and CARE groups.

Intervention Programs

Family-School Success (FSS)

FSS was a family-school intervention designed to improve family involvement in education as well as child functioning at home (i.e., homework performance, child behavior) and school (i.e., academic performance, school behavior). This 12-session intervention was designed to build a family-school partnership and guide parents and teachers through the steps of CBC (i.e., problem identification, problem analysis, plan implementation, and plan evaluation). Parents and teachers worked collaboratively to design a daily report card and targeted homework interventions.

Children were actively involved in the intervention and attended all sessions, except two meetings that were held in the school. While the parents were attending group sessions, their children attended a group that was designed to introduce them to the strategies being taught to their parents. Although child sessions were educational, they were provided in a manner that was fun for children (see Power et al., 2001). One psychosocial clinician (doctoral-level psychologist or post-doctoral fellow) was assigned to work with each cohort of parents. The clinician conducted group sessions and had responsibility for working with families in individualized family sessions and school-based sessions. In addition, three clinical assistants (graduate students in applied psychology) were assigned to work with the child groups.

FSS included three formats: (a) parent group meetings (6 sessions) held simultaneously with separate child group sessions; (b) individualized family therapy (4 sessions) including the parents and child; and (c) family-school consultations (2 sessions) held at the school, including parents and teachers. Sessions were held on a weekly basis. The initial session, which was held on a Saturday, lasted 3 hours. Subsequent group sessions were 90 minutes in length. Individualized family sessions lasted 60 minutes. Each school session was 45 minutes in duration. Two phone conferences between the clinician and the teacher (10 minutes each) were conducted to monitor the child’s progress and to modify interventions, if needed.

Several procedures were used to elicit teacher investment at the outset (for both FSS and CARE): (a) a letter briefly explaining the study was sent to teachers and their principals after the family had consented to the study; (b) a phone call was placed from the assigned clinician to the child’s teacher to introduce the study to the teacher and to schedule a time for a face-to-face meeting; (c) the assigned clinician visited the school to obtain authorization from the principal for the teacher to participate in the study, and to obtain teacher consent; and (d) teachers participating in the study were provided an explanation of the treatment being offered during the meeting held at school to obtain their consent. During this meeting, the clinician established rapport with the teacher and identified the primary concerns the teacher had with the child.

In this study, seven FSS cohorts were conducted. The number of families per group ranged from 3 to 9, with a mean of 6.4 families. Four clinicians conducted FSS groups: one conducted three groups with a total of 17 families; a second conducted two groups with a total of 16 families; a third had one group of 4 families; and the fourth had one group of 8 families. Forty-four teachers participated in the intervention; one teacher was involved in FSS for two children who were included in the same cohort.

Coping with ADHD through Relationships and Education (CARE)

CARE was a 12-session program designed to provide support and education to parents. There were three components to this program, which are similar to the components of most support groups for parents coping with ADHD: (a) discussing children’s progress at home and school, (b) establishing a context within which parents can support each other in coping with their children’s difficulties, and (c) providing generic education to parents about ADHD. Education provided to parents focused on ADHD, its associated features, and the challenges these children often encounter at home, in school, and with peers. Although parents were informed about potentially useful intervention strategies, they were not provided training in the use of empirically supported psychosocial interventions.

The purpose of CARE was to control for the non-specific effects of intervention. The content of CARE sessions did not address the primary components of FSS. CARE did not involve parents and teachers in the process of problem solving, nor did it involve training parents in the use of contingency management strategies. During CARE, children met in groups while parents attended group sessions. Children in CARE received education on topics covered in the parent groups and engaged in fun, recreational activities. One psychosocial clinician (doctoral-level psychologist or post-doctoral fellow) was assigned to work with each cohort of parents, and three clinical assistants (graduate students in applied psychology) worked with each child group.

CARE included 11 group sessions and 1 school meeting. The initial session was conducted on a Saturday for 3 hours and subsequent meetings were 75 minutes (approximately the average amount of time spent with families in FSS). Sessions were held on consecutive weeks. The purpose of the school meeting was to acquire information about school functioning and not to engage in problem solving or behavioral consultation. The same procedures were used to obtain teacher consent and investment as were described for FSS.

In this study, seven CARE cohorts were conducted. The number of families per group ranged from 5 to 10, with a mean of 6.9 families. Four clinicians conducted CARE groups: one conducted three groups with a total of 16 families; a second conducted two groups with a total of 17 families; a third had one group of 8 families; and the fourth had one group of 7 families. Forty-seven teachers participated in the intervention; one teacher was involved in CARE for two children who were included in separate cohorts.

Measures

Teacher Investment Questionnaire (TIQ)

The TIQ is a clinician-report measure designed to assess teacher engagement in intervention. Items on the scale measure teacher support, cooperation, and involvement in the intervention process. Clinicians rate each item on a four-point scale (1 = not at all true, 2 = a little true, 3 = somewhat true, 4 = very true). Separate versions of the TIQ were developed for FSS and CARE. The TIQ-FSS version consisted of 13 items assessing engagement with various components of the FSS program. The TIQ-CARE version consisted of 8 items reflecting engagement in this program. All of the TIQ-CARE items were included on the TIQ-FSS; in addition, the FSS version included some program-specific items. Table 2 provides a brief description of each item.

Table 2.

Corrected Item-Total Correlations, Means (Standard Deviations), and Range of Ratings for the TIQ-FSS and TIQ-CARE

Item	Item-Total Correlation	Mean (SD)	Range of Ratings
TIQ-FSS
1. Teacher was willing to modify homework expectation	.51	3.38 (0.71)	1 – 4
2. Teacher was cooperative in making arrangements for school meetings	.67	3.77 (0.52)	2 – 4
3. Teacher was interested in working with parent and clinician	.86	3.55 (0.72)	1 – 4
4. Teacher was supportive of family involvement in program	.67	3.79 (0.51)	2 – 4
5. Teacher provided enough time during meetings	.58	3.67 (0.67)	1 – 4
6. Teacher was engaged in discussion about child during meetings	.28	3.81 (0.45)	2 – 4
7. Teacher, parent, and clinician worked collaboratively to identify target behaviors	.47	3.81 (0.40)	3 – 4
8. Teacher actively participated in development of school-home note	.69	3.75 (0.49)	2 – 4
9. Teacher showed interest in child’s progress	.67	3.77 (0.52)	2 – 4
10. Teacher recognized the family’s efforts to support the child’s progress	.64	3.55 (0.65)	2 – 4
11. Teacher welcomed contact with family outside of school meetings	.58	3.48 (0.72)	1 – 4
12. Teacher followed the use of a daily school-home note	.14	3.38 (0.79)	1 – 4
13. The teacher-parent-clinician team jointly evaluated intervention implementation	.61	3.72 (0.50)	2 – 4
TIQ-CARE
1. Teacher was cooperative in making arrangements for school meetings	.79	3.54 (0.86)	1 – 4
2. Teacher was interested in working with parent and clinician	.87	3.30 (0.89)	1 – 4
3. Teacher was supportive of family involvement in program	.85	3.54 (0.91)	1 – 4
4. Teacher provided enough time during meetings	.76	3.56 (0.84)	1 – 4
5. Teacher was engaged in discussion about child during meetings	.68	3.70 (0.65)	1 – 4
6. Teacher showed interest in child’s progress	.84	3.36 (0.83)	1 – 4
7. Teacher recognized the family’s efforts to support the child’s progress	.81	3.20 (1.01)	1 – 4
8. Teacher welcomed contact with family outside of school meetings	.83	3.26 (0.97)	1 – 4

Note: TIQ refers to the Teacher Investment Questionnaire. FSS refers to the Family-School Success Program, and CARE refers to the Coping with ADHD through Relationships and Education program. Item wording was shortened for brevity.

These measures were designed specifically for the purpose of assessing teacher investment for the interventions used in this study. A pilot study of this measure examining teacher engagement in FSS (n = 25), which was conducted prior to the current study, demonstrated that the scale had high internal consistency (alpha > .90). Further, clinician ratings on this scale were negatively correlated (r = −.47, p < .05) with change scores on the ADHD Rating Scale - School Version (DuPaul et al., 1998) from pre-intervention to post-intervention.

Parent-Teacher Involvement Questionnaire (PTIQ)

The PTIQ (Kohl, Lengua, McMahon, & Conduct Problems Prevention Research Group, 2000) was developed to assist in evaluating outcomes for the FAST Track study, an efficacy trial of a comprehensive prevention program designed to reduce conduct problems among young children at risk for developing antisocial behavior. This measure, which is rated by parents and teachers, is firmly rooted in theory and empirical research related to family-school relationships. A factor analysis of the PTIQ uncovered an 11-item Quality of Parent-Teacher Relationship factor consisting of 6 parent-reported items and 5 teacher-reported items (Kohl et al., 2000). Only the parent-rated items were analyzed in this study. Parents were requested to rate each item with regard to how well it described the target child on a five-point scale from 0 = not all to 4 = a great deal. An example of an item on this scale is: “You feel your child’s teacher cares about your child.” The reliability of this factor was found to be high (alpha coefficient = .89).

Homework Performance Questionnaire – Parent Scale (HPQ-PS)

The HPQ-PS was developed to assess parent perceptions of children’s performance on homework (Power, Dombrowski, Watkins, Mautone, & Eagle, 2007). Each of the 32 items on this questionnaire is rated on a four-point scale (0 = rarely/never, 3 = always/almost always) with regard to how often the behavior has occurred during the previous 4 weeks. A factor analysis of this scale uncovered a factor related to parent perceptions of teacher support regarding homework. Eight salient items loaded on this factor (alpha = .77). Examples of items on this factor include: “The teachers communicate effectively with me about homework;” and “The teachers seem willing to help if we have homework problems.”

School Adversity

The level of adversity experienced by a school has been assessed by combining reports of economic disadvantage among students (i.e., percent of students qualifying for free or reduced-price lunch) with rates of student mobility (i.e., percent of students who enrolled and left during the school year; Hughes et al., 2005). For this study, subsidized lunch status and mobility rates were obtained for each school by viewing web-based electronic reports or obtaining reports from authorized school officials. School adversity, which was calculated as the sum of the subsidized lunch rate plus the mobility rate, was correlated with subsidized lunch rate alone. Because the correlation was very high (r = .88) using Spearman’s rho and because information about lunch status was available for more cases than data about mobility rates, school adversity in this study was determined using lunch subsidy rate alone.

Family adversity

Family adversity was operationalized as socioeconomic status (SES), which was assessed using Hollingshead’s four-factor index (Hollingshead, 1975). Levels of SES were grouped into five categories, ranging from a high of V to a low of I. The Hollingshead scale also provided information about marital status.

Student Problem Severity

The Swanson, Nolan, and Pelham Questionnaire - Teacher Rating Scale (SNAP-IV-TRS; see Swanson et al., 2001) was used to assess child level of Inattention, Hyperactivity, and Oppositional Defiant Disorder. The SNAP-IV-TRS is commonly used in research and clinical practice, and these scales have been shown to have stronger than adequate reliability and validity (Bussing et al., 2008).

Assessment Procedures

The PTIQ (Quality of Parent-Teacher Relations factor), HPQ-PS, and BASC-TRS were administered at baseline, prior to the application of the interventions. Also, information about school and family adversity was obtained at baseline. Clinicians were requested to complete the TIQ to assess level of teacher engagement for each case after the intervention was completed. In addition, mid-treatment ratings on the TIQ were obtained for 12 cases in FSS and 16 cases in CARE.

Data Analyses

As a first step, an item analysis was conducted of the TIQ-FSS and TIQ-CARE. Corrected item-total correlations were computed for each version separately. In addition, the mean, standard deviation, and range of ratings for each item were determined. A priori, it was determined that items would be retained on the scale if: (a) the corrected item-total correlation was ≥ .40, indicating a relatively high loading on the aggregate factor; and (b) the range of ratings on the items was at least 2 points (e.g., range = 2 – 4, indicating some variability in responding).

The stability of the TIQ was determined by correlating ratings at mid-treatment with those at post-treatment, which occurred 6 weeks apart, for the subset of teachers for whom there were both sets of ratings (n = 12 for FSS; n = 16 for CARE). Spearman rho correlations were calculated because the distribution of ratings on the TIQ deviated markedly from normality.

Seasonal effects were evaluated using the Kruskal-Wallis test, given the non-normality of the distributions for the TIQ. Groups were conducted three times per year (fall, winter, spring). Clinician effects were evaluated by examining means and standard deviations across clinicians. Tests of significance were not used because of the relatively low number of cases treated by some providers. FSS groups were conducted by four clinicians (n = 17 for clinician (C)-1; n = 16 for C-2; n = 4 for C-3; n = 8 for C-4). CARE groups also were facilitated by four clinicians (n = 8 for C-1; n = 16 for C-2; n = 17 for C-3; n = 7 for C-4).

Differences in TIQ ratings as a function of grade level were evaluated using the Kruskal-Wallis test. Associations between TIQ ratings and levels of school adversity, family adversity, and child behavior problem severity were evaluated using Spearman rho correlations. With regard to grade level, ratings for grades 5 and 6 were collapsed because of the small cell sizes for each of these grades.

Results

Psychometric Properties of the TIQ

Corrected item-total correlations for each version of the TIQ are presented in Table 2. For the TIQ-FSS, all of the item-total correlations exceeded .40 except those for item 6 (r = .28) and item 12 (r = .14). Also, the range of ratings per item was at least 2 points, except for item 7 (1 point). Therefore, items 6, 7, and 12 were deleted from the scale in subsequent analyses. Coefficient alpha for the revised 10-item version of the TIQ-FSS was .90. For the TIQ-CARE, all of the item-total correlations exceeded .40 and had a range of ratings of 3 points. Therefore, none of the TIQ-CARE items were deleted in subsequent analyses. Coefficient alpha for the 8-item TIQ-CARE was .95. Correlations reflecting the stability of the TIQ over a 6-week period were .73 for FSS and .98 for CARE.

Seasonal effects were relatively small; differences in TIQ ratings across seasons were non-significant for both FSS and CARE. With regard to clinician effects, FSS providers generally were consistent with regard to their use of the scale. The variation in mean ratings across clinicians was relatively small (3.43 to 3.83). Standard deviations generally were consistent across clinicians (in the 0.42 to 0.58 range for three clinicians), although one clinician had a relatively low standard deviation (0.14). Clinicians for CARE were less consistent in how they assigned ratings using the TIQ than the FSS clinicians. The variation in mean ratings was relatively large (2.68 to 3.96). Standard deviations also fluctuated markedly across providers (ranging from 0.12 to 1.21).

Differences by Grade Level

As expected, level of teacher investment varied as a function of grade level, although this relationship was significant for FSS (Χ² = 9.40, p < .05) only, not CARE. Figure 1 illustrates the marked decline in level of teacher investment with increasing grade level for FSS. The findings suggest a decline in teacher investment between grades 2 and 3 as well as between grades 4 and 5/6, although sample sizes within grade levels are relatively low (range from 7 to 16).

Figure 1.

Variations in level of teacher investment in FSS and CARE as a function of student grade level (levels of teacher investment are in rank order ranging from 1 to 45 for FSS and 1 to 48 for CARE)

Association with Level of Adversity

Contrary to expectation, level of teacher investment did not vary as a function of school adversity or family adversity. The correlation (Spearman rho) between teacher investment and school adversity was −.08 for FSS and −.09 for CARE. The correlation between teacher investment and family adversity was −.28 (n.s.) for FSS and .21 (n.s.) for CARE. The effect of marital status, as determined by the Mann-Whitney test, was also non-significant for FSS and CARE.

Association with Family-School Relationship

Parent ratings of family-school collaboration at baseline, as assessed by scores on the PTIQ Quality of the Parent-Teacher Relationship factor as well as the HPQ Teacher Support factor, demonstrated a significant correlation with clinician reports on the TIQ post-intervention. The correlations were significant for both the FSS and CARE groups (range from .38 to .46 across groups). Table 3 presents intercorrelations among these variables. It was noted that the correlations between the PTIQ and HPQ factors were also significant (.37 and .58).

Table 3.

Intercorrelations Among the TIQ Teacher Investment, PTIQ Quality of Parent-Teacher Relationships, and HPQ Teacher Support of Homework Factors

	FSS		CARE
	PTIQ	HPQ	PTIQ	HPQ
PTIQ Quality of Parent-Teacher Relationships	--	--	--	--
HPQ Teacher Support of Homework	.37*	--	.58**	--
TIQ Teacher Investment	.46**	.38**	35*	.42**

Note: FSS refers to Family School Success, and CARE refers to Coping with ADHD through Relationships and Education. TIQ refers to the Teacher Investment Questionnaire, PTIQ refers to the Parent-Teacher Involvement Questionnaire, and HPQ refers to the Homework Performance Questionnaire.

^*p < .05

^**p < .01

Association with Student Attention and Behavior Problems

Teacher ratings of Inattention, Hyperactivity, and Oppositional Defiant behavior, as assessed using the SNAP-IV-TRS administered at baseline, were not found to be correlated with ratings on the TIQ assessed at post treatment. Correlations generally were in the expected direction (negative), but were not significant for either the FSS or CARE cohorts (range from .02 to −.20).

Discussion

Research and clinical practice clearly indicate that teachers vary widely in their investment and implementation of evidence-based interventions for children with attention, learning, and behavior problems (Biggs, Vernberg, Twemlow, Fonagy, & Dill, 2008). Further, variations in teacher investment and level of implementation have been demonstrated repeatedly to be linked with student outcomes (Hirschstein et al. 2007; Noell et al., 2002). The findings of this study generally indicated that the clinician-rated TIQ is a valuable measure for assessing the construct of teacher investment. There were some important differences in how the TIQ operated when applied in the context of FSS versus CARE. The overall pattern of findings indicated that the TIQ was a more accurate and sensitive indicator when used in the context of FSS; CARE clinicians demonstrated a higher level of variability in their ratings across groups than did FSS clinicians. Clinicians in FSS are much more involved with teachers than CARE clinicians (i.e., FSS involves an additional school visit and more telephone contact with teachers), and therefore, FSS clinicians have more information about teachers upon which to base their ratings.

Effect of Grade Level

The results confirmed our expectations that level of teacher involvement is inversely related to student grade level, although this pattern was evident for FSS and not CARE. The findings suggest that involving teachers in interventions that entail a substantial amount of family-school collaboration, such as FSS, can become increasingly challenging as students advance in their schooling. Practitioners may need to invest more effort in engaging teachers in the upper elementary and middle school grade levels than in the lower grades.

Effect of Adversity

The findings did not confirm our hypothesis that teacher involvement is related to level of school and family adversity. The range of school adversity scores (rates of students eligible for subsidized lunch) and family adversity levels (Hollingshead categories) was substantially restricted. Schools with relatively high rates of students eligible for subsidized lunch and families from the lowest two (out of five) Hollingshead categories were significantly underrepresented in this study. Investigations demonstrating the effect of school adversity have included schools with much higher rates of subsidized lunch (Hughes et al., 2005; Thomas et al., 2006). Also, it is possible that the methods used to assess school adversity (rates of subsidized lunch) and family adversity (Hollingshead scores) were too global to capture the association between these constructs and level of teacher engagement.

Effect of Family-School Relationship

The results confirmed that level of teacher involvement in intervention is associated with the degree of family-school collaboration, as assessed by parent ratings of the Quality of the Parent-Teacher Relationship (PTIQ) as well as Teacher Support of Student Homework (HPQ-PS). The correlations generally were in the moderate range for both FSS and CARE. The longitudinal relationship of the variables (PTIQ and HPQ were administered at baseline and TIQ was administered post intervention) suggests that level of family-school collaboration prior to intervention may be predictive of the degree of teacher investment in intervention. For practitioners, obtaining information about the degree of family-school collaboration prior to intervention may be highly useful in determining whether it will be challenging to engage teachers in intervention, suggesting the need for additional consultation and support throughout the intervention period.

Effect of Student Attention and Behavior Problems

The findings failed to demonstrate that severity of child attention and behavior problems assessed prior to intervention was associated with teacher level of investment as assessed after intervention. Although findings generally were in the expected direction (higher levels of attention and behavior problems inversely related to level of teacher investment), the correlations were low and non-significant. The restricted range of ratings in this sample of children who had relatively high ratings of attention and behavior problems may have served to suppress the magnitude of the correlations to some extent.

Reliability and Validity of the TIQ

The TIQ demonstrated strong internal consistency when applied in the context of both FSS and CARE. However, there were some important differences in how the TIQ operated across interventions. For FSS, the 6-week stability of the TIQ was .73, reflecting some shifting in ratings from mid- to post-intervention. The change in ratings was understandable, because in FSS there is considerable contact between provider and teacher throughout the entire course of the 12-week intervention. For CARE, the 6-week stability of the TIQ was extremely high (.98). The limited contact between CARE clinicians and teachers in the second half of the intervention may explain the unusually high level of stability in TIQ ratings. Also, the consistency of ratings across providers was relatively high for the FSS and considerably lower for CARE, suggesting that rater bias (i.e., differences due to variations in how raters use the scale) is less problematic for FSS than CARE. Greater involvement with teachers among FSS providers (as compared to CARE clinicians) affords these clinicians more information upon which to base their ratings, which may serve to minimize potential rater bias.

Limitations

Several limitations should be noted in considering the findings of this study. First, as noted above, schools and families experiencing relatively high levels of adversity were underrepresented in this study. It is possible that families and schools facing greater levels of adversity might show a different pattern of findings. Second, the sample sizes for FSS and CARE were relatively small, which placed limits on the power in the analyses and the types of analyses that could be employed (e.g., a factor analysis of the TIQ could not be conducted). Third, when using the TIQ, clinicians typically rated teachers at the higher end of the scale (e.g. 3 = somewhat true, 4 = very true). In the future, it might be helpful to extend the upper end of the scale and compress the lower end (e.g., 1 = not at all, 2 = somewhat true, 3 = mostly true, 4 = very true). In this way, the scale may be more sensitive to differences among teachers and ratings are likely to be more normally distributed. Fourth, a limited number of clinicians (four for each program) completed ratings for all the teachers involved in FSS and CARE. This practice may have placed limits on the degree to which ratings of teachers were independent from one another. For example, bias in the ratings of a particular clinician may have had an effect on multiple teacher ratings. To address this issue methodologically, it is recommended that more than one clinician (e.g., a co-therapist or supervisor) complete ratings. Also, it would be useful to obtain ratings from clinician pairs on two occasions approximately 2 weeks apart. With these data a generalizability analysis could be conducted to sort out the unique contribution of rater bias and time effects (see Bergeron, Floyd, McCormack, & Farmer, 2008). Fifth, in this study teacher investment was assessed using clinician ratings alone. The use of parent ratings along with clinician ratings would provide a multi-informant assessment of the construct that would likely prove more useful in research and practice. Finally, the process of randomization was limited by schedule restrictions on the part of parents and efforts to assign children from the same school to the same intervention group. However, it is not likely these factors had a differential effect across groups because family availability to come to sessions on a Tuesday versus a Thursday night should not have had an effect on the results, and there were few children who attended the same school.

Implications for Practice

This study indicates that the TIQ may be a useful tool for assessing teacher engagement in interventions involving the family and school. As such, the study suggests that the TIQ may be able to capture some of the variability in teacher investment that is related to variable intervention outcomes, although additional research is needed to evaluate the relationship between teacher engagement and outcomes. Assessing teacher engagement relatively early in the intervention process may be useful in promoting favorable outcomes, in that relatively low ratings of engagement may suggest the need for strategies to provide additional support for teachers to elicit greater investment. Although the version of the TIQ that was administered to FSS clinicians in this study included several items specific to FSS, both versions of the TIQ (FSS and CARE forms) included the same set of eight items, and this set may be useful in assessing teacher engagement in other intervention programs involving the family and school.

Conclusions

Assessing level of teacher investment in intervention is important because of variations in their level of investment and its relationship to student outcomes. This study expands the science base by introducing a measure for assessing teacher engagement in intervention. In addition, this study demonstrates that degree of teacher engagement may vary as a function of the quality of parent-teacher collaboration. The TIQ appears to be a promising measure for assessing teacher investment. In this study, level of clinician-rated teacher investment was demonstrated to vary significantly as a function of student grade level (i.e., lower investment with higher grade levels) in the context of an intervention for students with ADHD designed to build family-school partnerships and resolve learning and behavioral problems. Also, the findings suggested that the quality of the family-school relationship, as reported by parents, and level of teacher support with homework issues at baseline may be indicative of the degree of teacher investment in family-school interventions. Additional research is needed to further refine the TIQ, explore the potential contribution of rater bias, provide a multi-informant method for assessing teacher investment (using both clinician and parent reports), and assess this construct across a broader range of school and family adversity levels.