Teacher Network of Relationships Inventory: Measurement Invariance of Academically At-risk Students Across Ages 6 to 15

Abstract

We tested the longitudinal measurement invariance of the Teacher Network of Relationships Inventory (TNRI), a teacher-report measure of teacher-student relationship quality (TSRQ), on a sample of 784 academically at-risk students across ages 6 to 15 years by comparing the model for each subsequent year with that of the previous year(s). The TNRI was constructed with 22 items to form three correlated factors: Warmth, Conflict, and Intimacy. Cronbach’s alphas (α) ranged from .87 to .96 across 9 years. Both metric and scalar measurement invariance held for 9 years, indicating that scores on the TNRI have similar meaning across these ages. The TNRI also demonstrated measurement invariance across gender and race/ethnicity. Findings support that the TNRI is an appropriate measure for investigating substantive issues related to developmental changes in TSRQ from early childhood through adolescence, including gender and ethnic/racial differences in TSRQ across these ages. Based on RM-ANOVAs, each scale decreased across the 9 years, although the growth patterns for scales differed somewhat: Conflict had a linearly decreasing pattern, Warmth declined most notably as students make the transition to adolescence, whereas Intimacy scores dropped off noticeably at the transition from early to late childhood. Research limitation and implication for practice are discussed.

An extensive body of research documents the importance of a supportive relationship with one’s teacher for a student’s school success (for narrative review see Hamre & Pianta, 2006; for meta-analytic review see Roorda, Koomen, Spilt, & Oort, 2011). Students at risk for academic failure also benefit from a supportive teacher-student relationship, and in fact are more affected by relational supports at school compared with the normative sample (Hamre & Pianta, 2005 and Hughes et al, 2012). Therefore, research on teacher-student relationship among the at-risk students is a worthwhile pursuit. In particular, teacher reports of support (i.e., closeness or warmth) and conflict in the teacher-student relationship predict students’ social, behavioral, and academic outcomes, above prior levels of outcomes, from preschool through the elementary grades (Hughes, Luo, Kwok, & Loyd, 2008; Ladd, Birch, & Buhs, 1999; Pianta, Steinberg, & Rollins, 1995). After the elementary grades, teacher-student relationship quality (TSRQ) is commonly assessed with student questionnaires, which are also associated with positive academic and behavioral adjustment (Wentzel, 1999). In developing teacher report measures of TSRQ, researchers have drawn from attachment theory (Pianta, 2001), social support theory (Hughes & Kwok, 2007), and self system models of motivation (Furrer & Skinner, 2003). Despite the diversity of theoretical frameworks, teacher report measures of TSRQ consistently identify a supportive dimension (i.e., close, warm, accepting) and a conflict dimension (Hughes & Kwok, 2007; Murray, Murray, & Waas, 2008; Pianta, 2001), with some measures identifying a third dimension of dependency (Pianta & Stuhlman, 2004) or intimacy (Hughes & Kwok, 2007). Both teacher support scores and teacher conflict scores are associated with children’s social, emotional, and academic adjustment (Hamre & Pianta, 2001; Meehan, Hughes, & Cavell, 2003).

Recently, researchers have investigated developmental changes in teacher-reported TSRQ across the elementary grades and their implications for student performance. These studies have found that average levels of overall relationship quality tend to decrease across the elementary grades (O’Connor & McCartney, 2007; O’Connor, 2010). When closeness and conflict in relationships were examined separately from kindergarten to grade 6, a decrease in closeness across this period and an increase in conflict in the early grades followed by a decrease in grade 5 were found (Jerome, Hamre, & Pianta, 2009). Surmising that average growth trajectories may obscure subtypes of children who follow atypical trajectories, Spilt, Hughes, Wu, and Kwok (2012) identified subgroups of trajectory classes for warmth and conflict for boys and girls across grades 1 to 5 on a sample of students at risk for school failure. The typical child had high but somewhat declining warmth and low, stable conflict across these years. However, not all children followed these normative trends. Children whose trajectories were characterized by variably high conflict showed lower growth in reading and math across the elementary grades than did students with stable and low conflict. Because these researchers did not test the assumption that scores on their measures have the same meaning across ages (i.e., may not demonstrate longitudinal measurement invariance), conclusions regarding developmental changes in relationship constructs may not be justified.

Measurement invariance of tests of TSRQ

Definition of measurement invariance

In its most general sense, measurement invariance refers to the equivalence of measurement across groups. Groups can be based on any number of factors, such as ethnicity, gender, or age. Testing measurement invariance involves a series of increasingly stringent criteria (Millsap, 2011). Configural invariance is the first criterion, and examines if the number of factors and the pattern of zero factor loadings are the same among groups. Next, metric invariance (also referred to as weak invariance) refers to a finding that factor loadings are equivalent across groups (i.e., that the relationship between the items and the underlying latent construct is the same across groups). Metric invariance is sufficient for examining the issue of factorial validity of a measure across groups. The next level of invariance is scalar invariance, which refers to a finding that item intercepts are also equivalent across groups. A finding of scalar invariance means that the construct is measured on a similar scale across groups. When both metric and scalar invariance are found, the measure is considered to demonstrate strong invariance. In order to reach valid conclusions regarding observed differences between group means or group differences in the patterns of correlations of the measure with external criteria, a measure must demonstrate strong invariance (Meredith, 1993).

Longitudinal measurement invariance

Longitudinal measurement invariance means that a measure assesses similar constructs in the same way at different ages or assessment waves. Substantive interpretations of observed score differences between different ages are not valid unless the measure demonstrates invariance across ages (Meredith, 1993). Measurement invariance across age is often assumed but rarely tested in studies assessing age-related changes in TSRQ. An exception is a study by Koomen, Verschueren, van Schooten, Jak, and Pianta (2012), which investigated measurement invariance of a Dutch-adaptation of the Student Teacher Relationship Scale (STRS; Pianta, 2001) across ages 3 to 12 and across gender. The STRS, a teacher-report measure derived from literature on parent-child attachment, was originally intended for use with children ages 3 to 8 and yields three scores: Closeness, Conflict, and Dependency. Although the three-factor structure of the STRS was supported between younger and older age levels and gender, neither metric invariance (i.e., equivalence of factor loadings) nor scalar invariance (i.e., equivalence of item intercepts) across age levels was supported. Gender non-invariance problems were found to be minor, indicating scores on the STRS have similar if not identical meanings across gender. The authors concluded that the STRS measures different constructs at different age levels; consequently, scores do not mean the same thing for younger and older students. The authors also concluded that the STRS, which is based on attachment theory, may be more appropriate for children under the age of 8, or even under the age of 6, than for older children.

The primary purpose of this study was to test longitudinal measurement invariance of the Teacher Network of Relationships Inventory (TNRI; Hughes et al., 2008) from age 6 to 15. The TNRI was developed from the Network of Relationships Inventory (NRI; Furman & Buhrmester, 1985), a measure designed to assess children’s and adolescents’ perceptions of relationship quality across different types of relationships or across different ages or groups. The NRI is based on social support theory (Weiss, 1974) and has been used extensively in research to assess how youths’ perceptions of overall support and negative interactions in a given relationships (e.g., best friend or parent) are associated with other individual outcomes (e.g., loneliness or self-concept) or relationship outcomes (e.g., continuation or termination of relationships) (Furman & Buhrmester, 2009). The TNRI was adapted from the child-report NRI and assesses teachers’ reports of the provision of various types of social support to students and yields three scales: Warmth, Intimacy, and Conflict. The TNRI has demonstrated good internal consistency and construct and predictive validity in grades 1–5, with Warmth and Support scores being positively associated with peer acceptance, behavioral engagement, and reading and math achievement and negatively associated with aggression (Meehan, Hughes & Cavell, 2003; Hughes & Kwok, 2007; Hughes et al., 2008;Liew, Chen & Hughes, 2010; Hughes, 2011; Hughes et al., 2012). For example, in a 3-year longitudinal study, TNRI Support scores (which includes Warmth and Intimacy scales) predicted changes in elementary students’ academic achievement, an effect that was mediated by the effect of Support scores on children’s behavioral engagement in the classroom (Hughes et al., 2008). The TNRI Warmth scale has demonstrated good convergent and discriminant validity with both child and peer reports of teacher warmth and child conduct problems (Li, Hughes, Kwok & Hsu, 2012).

The hypothesis of longitudinal measurement invariance of the TNRI across elementary and middle school is reasonable because the TNRI assesses the provision of sources of social support (e.g., affection, intimacy, enhancement of worth, low conflict) that have been found to be relevant to individuals from early childhood to adulthood (Furman, 1996; Furman & Buhrmester, 2009). The availability of a measure of TSRQ that is invariant across elementary and middle school would facilitate efforts to understand the development of TSRQ across childhood and adolescence, age differences in TSRQ, and differences in correlates of TSRQ across developmental levels. Given the saliency of gender and race/ethnicity both to teacher-student interactions and to educational outcomes, it is also important to investigate invariance of the TNRI across gender and race/ethnicity. The limited research on the role of gender and race/ethnicity in the measurement of TSRQ is summarized next.

Gender and Measurement of TSRQ

Consistently, studies report that teachers report greater closeness and lower conflict in their relationships with girls than with boys (Spilt et al., 2012; Wu, Hughes, Kwok, 2010). However, without a finding of metric and scalar invariance across gender, a conclusion that gender differences in observed scores on a measure of TSRQ reflect differences on the same construct is not warranted. The current study tested measurement invariance of the TNRI across gender in a sample of youth from age 6 to 15.

Race/Ethnicity and Measurement of TSRQ

Researchers have found that teachers report lower levels of support and higher levels of conflict in their relationships with African American students than with Hispanic or White students, who experience similar levels of teacher support and conflict (Hamre & Pianta, 2001; Meehan, Hughes & Cavell, 2003; Hughes & Kwok, 2007). These race and ethnic differences are minimized, however, when teachers and students are matched on race or ethnicity (Saft and Pianta, 2001). To the authors’ knowledge, only one study has investigated ethnic or racial invariance of measures of TSRQ. Using CFA procedures, Webb and Neuharth-Pritchett (2011) examined the factorial validity of the STRS and its measurement equivalence across African American and White children in a sample of pre-kindergarten children (age 5). The hypothesized three-factor model of the STRS did not obtain an adequate fit to the whole sample, and the factor structure was not equivalent across racial groups. Relatively large racial discrepancies were found in patterns of factor loadings for items on the Closeness and Dependency scales. The authors concluded that studies reporting ethnic differences on observed scores of the STRS may not reflect differences in the latent constructs. In the current study, we tested measurement invariance of the TNRI across Hispanic, White, and African American students.

Study Purpose

The primary purpose of the current study was to investigate measurement invariance of the TNRI across ages 6 to 15 (generally corresponding to grades 1 to 9) as well as across gender and ethnicity. We pursue these aims in an ethnically and linguistically diverse, academically at-risk sample of students (see participants). The TNRI was administered to students’ teachers annually for 9 years, beginning when the children were in first grade. A finding of strong measurement invariance for the TNRI across age, gender, and race would support its use in substantive investigations of developmental patterns in TSRQ and of gender and race/ethnicity differences in students’ relationships with teachers. Upon a finding of longitudinal, ethnic, and gender measurement invariance of the TNRI, the secondary purpose of the study was to describe developmental changes in each construct measured by the TNRI (i.e., Warmth, Intimacy, and Conflict) from age 6 to 15.

Method

Participants

Participants were recruited from three school districts in Texas (1 urban and 2 small cities) across two sequential cohorts in first grade during the fall of 2001 and 2002. Children were eligible to participate in the longitudinal study if they scored below the median score on a state-approved district-administered measure of literacy, spoke either English or Spanish, were not receiving special education services, and had not previously been retained in first grade. The study was conducted under the approval of the Institutional Review Board of the second author’s university. School records identified 1,374 children as eligible to participate. Because teachers distributed consent forms to parents via children’s weekly folders, the exact number of parents who received the consent forms could not be determined. Small gifts to children and the opportunity to win a larger prize in a random drawing were instrumental in obtaining 1,200 returned consent forms, of which 784 parents (65%) provided consent.

Analyses of a broad array of archival variables, including performance on the district-administered test of literacy, age, gender, ethnicity, eligibility for free or reduced-price lunch, bilingual class placement, cohort, and school context variables (i.e., ethnic composition and percentage of economically disadvantaged students), did not indicate any differences between children with and without consent. The resulting sample of 784 participants (52.6% male) closely resemble the population from which they were drawn on demographic and literacy variables relevant to students’ educational performance. The ethnic composition of the study sample was 37% Hispanic (of whom 39% were Spanish language dominant), 34% White, 23% African American, and 6% other; 62% of the children qualified for free or reduced cost lunch. In Year 1, the mean scores on the Woodcock Johnson III Tests of Achievement (WJ-III; Woodcock McGrew, & Mather, 2001) Broad Reading and Broad Math were 98.25 (SD = 17.26) and 101.70 (SD = 13.80), respectively.

In Year 1, the 784 students were in 207 classrooms in 36 schools. The sample was followed for 9 years. The effective sample differed by year, based on student attrition and teacher completion of the TNRI. By Year 9, 488 students (55.1% male) were still active in the study. The largest factor in subject attrition was parental non-response to a request for continuation in the study after the first 5 years, at which time parental written consent for continued participation was received for 569 participants. Attrition analysis of Year 9 sample found that attritted participants scored lower on math achievement¹ at Year 1 (ΔM_{WJ–III Math} = −2.09, t(754) = 2.08, p = .038), but did not differ on a wide range of other variables (e.g., gender, ethnicity, bilingual education status, economically disadvantaged status, retaintion status, literacy scores, reading achievement, school engagement, parent education level and parent literacy status)². Across 9 years, on average 243.3 (MIN-MAX, 148–335) teachers from 72 (36–108) schools completed the TRNI and provided demographic information each year. The number of teachers and schools involved each year increased over time due to the increasing dispersion of students outside the original three participating school districts. Most teachers were female (54.3%–87%) and White (58.8%–73.9%), while 1.4%–11.7% of teachers were Hispanic and 1.9%–8.7% were African American. From Year 1 to Year 9, there were 2.5%–6.4% teachers with less than one year of experience, 11%–19.9% with 1–3 years, 11%–15.5% with 4–6 years, 4.5%–11.2% with 7–9 years, 3.5%–8.2% with 10–12 years and 20.4%–37.7% with more than 12 years. As expected for an educationally at-risk sample, some students were retained in grade. By Year 9, 154 (31.5%) students had been retained once and 10 (2.0%) students had been retained twice.

Teacher Network of Relationships Inventory (TNRI)

In the spring of each year, from Year 1 to Year 9, teachers were mailed questionnaires for each study child in their classroom that included the TNRI. Teachers received $25 for each questionnaire returned. In the elementary grades, students’ primary classroom teacher completed the questionnaire. Beginning at the transition to middle school, questionnaires were completed by the student’s language arts teacher (93%) or a teacher named by the language arts teacher as having greater knowledge of the student (7%).

As described above, the 22-item TNRI was developed from the child version of the Network of Relationships Inventory (Furman & Buhrmester, 1985). Specifically, NRI items were rephrased so that teachers reported on a 5-point Likert scale (not at all true to very true) their provision of six types of social provisions (reliable alliance, dependable bond, enhancement of worth, affection, intimacy, and nurturance) as well as negativity and conflict in the relationship. Some items were reworded so that the focus of the item was on the child rather than on the teacher’s behavior in order to reduce the threat of the question and minimize a teacher’s tendency to respond in a socially desirable manner. For example, an item on the child version asked “How much does your teacher treat you like you’re admired and respected?” The comparable item on the teacher version is “This child gives me many opportunities to praise him/her.” Other items were changed minimally (e.g., from “How satisfied are you with your relationship with your teacher?” to “I am satisfied with my relationship with this child”). Earlier exploratory and confirmatory factor analysis (Hughes, Gleason & Zhang, 2005) with the same longitudinal sample in Year 3 identified three factors: Warmth (13 items; α= .96), Intimacy (3 items; α= .86) and Conflict (6 items;α= .91). Based on their high correlation, the Warmth and Intimacy items are sometimes combined to form a Support score (Hughes & Kwok, 2006). As stated previously, the TNRI has demonstrated good convergent and discriminant validity.

Data Analysis Overview

In order to examine the longitudinal stability of the measurement structure of the TNRI across 9 years, a series of measurement invariance (MI) analyses were conducted to sequentially test the configural, metric and scalar invariance assumptions within CFA framework. The normality of TNRI item responses was checked across 9 years. All the skewness and kurtosis values were within the commonly used criteria (e.g. skewness ranging within ±3 and kurtosis within ±8, Kline, 2010)², and the normality assumption held for our responses across 9 years. We followed the practices of prior researchers assessing teacher-student relationships in treating items as continuous responses on a 5-point scale (e.g. Hughes, 2012; Koomen, Verschueren, van Schooten, Jak, & Pianta, 2012; Pianta, 2001; Solheim, Berg-Nielsen, & Wichstrøm, 2012 to name a few), thus permitting comparisons between our results and those of other researchers and allowing us to investigate the complex factorial structures with invariance assumptions (Rhemtulla, Brosseau-Liard, & Savalei, 2012; Stark, Chernyshenko, & Drasgow, 2006). The nested nature of our dataset was taken into consideration by using TYPE = COMPLEX routine in Mplus 6.12 with the rescaled model-fit test statistic and robust standard error estimator (L. K. Muthén & Muthén, 2010; Wu & Kwok, 2012). Participants with scores on the TNRI for at least one assessment wave were included in the analysis, and the missingness of the dataset was handled with full information maximum likelihood (FIML) parameter estimator (Enders, 2010).

We tested the three-factor CFA model from previous research with the TNRI (Hughes, Gleason & Zhang, 2005) with the 9 annual datasets. Besides model fit chi-square test statistics, the commonly used criteria of model fit indices suggested by Hu and Bentler (1999) were used to evaluate the model goodness of fit. Modification indices were used to provide statistical evidence for the unknown underlying relationship among items. An examination of modification indices found that 4 pairs of correlated items had consistently large modification indices across 9 years (i.e., modification index >= 40). Item 14 and item 7 both involved praising child; item 20 and item 19 both involved affection/nurturance; item 17 and item 9 both involve general satisfaction with relationship; item 21 and item 6 both involved need to discipline. Similarity in wording or phrasing on these pairs may suggest that correlated residuals are necessary (Brown, 2006). Therefore, we revised the original three-factor CFA with these four pairs of correlated items. This revised three-factor CFA demonstrated an adequate model goodness of fit across the 9 datasets; the average of model-fit indices were as follow:

$$\overline{\mathit{CFI}}=.928,\overline{\mathit{TLI}}=.918,\overline{\mathit{RMSEA}}=.080\text{and}\overline{\mathit{SRMR}}=.047.$$

This revised CFA structure was then used as the factor structure in testing the longitudinal measurement invariance of the TNRI measures across 9 years. First, we examined the configural invariance of measurement structures; second, we tested the metric invariance of factor loadings; last, we tested the scalar invariance of intercepts (Millsap, 2011). For instance, in Table 1, we first confirmed the configural invariance assumption with Model 1.1 and proceeded to test metric invariance. The longitudinal MI analysis was conducted by comparing the model for each subsequent year with that of the previous year(s). This approach has two major advantages. First, we can know if the parameters are the same across 9 years. Second, we can detect when items become non-invariant. To do so, metric invariance between Year 1 and Year 2 was tested (Λ₁ = Λ₂) with the factor loadings of the rest of years being freely estimated in Model 2.1. In Model 2.2, we fixed Year 1 and Year 2 factor loadings to be the same and tested if the loadings are invariant through Year 3 (Λ₁ = Λ₂ = Λ₃) with the factor loadings of the following years being freely estimated. The same procedure was applied through Model 2.8, where we fixed all the factor loadings of CFA models for 9 years to be the same. After we confirmed the metric invariance from Model 2.1 to Model 2.8, the same yearly comparison was applied to test scalar invariance from Model 3.1 to Model 3.8.

Table 1.

Model fit test statistics and fit indices and their changes for the measurement models of Year 1 through Year 9

Model fit test statistics and fit indices								Change of model fit test statistics & fit indices
		χ2	df	CFI	TLI	RMSEA	SRMR	Δχ2	Δdf	ΔCFI	ΔTLI	ΔEVCI	RDR
Step 1: Configural Invariance
1.1		6220.03	1818	.926	.915	.068	.048	–	–	–	–	–	–
Step 2: Metric Invariance
2.1	Λ1 = Λ2	6243.64	1837	.925	.916	.068	.049	23.61	19	.001	−.001	−.0002	.018
2.2	Λ1 =…= Λ3	6281.10	1856	.925	.916	.068	.049	19.65	19	0	0	−.0001	.035
2.3	Λ1 =…= Λ4	6319.09	1875	.925	.917	.068	.051	18.60	19	0	−.001	0	.036
2.4	Λ1 =…= Λ5	6357.22	1894	.925	.917	.067	.051	10.42	19	0	0	0	.036
2.5	Λ1 =…= Λ6	6357.22	1913	.925	.918	.067	.052	16.08	19	0	−.001	−.004	.002
2.6	Λ1 =…= Λ7	6396.41	1932	.924	.919	.067	.053	21.19	19	.001	−.001	.0002	.037
2.7	Λ1 =…= Λ8	6445.90	1951	.924	.919	.067	.055	34.69	19	0	0	.002	.045
2.8	Λ1 =…= Λ9	6475.03	1970	.924	.920	.066	.056	29.13	19	0	−.001	−.001	.026
Step 3: Scalar Invariance
3.1	τ1 = τ2	6512.04	1989	.923	.920	.066	.056	–	–	–	–	–	–
3.2	τ1 =…= τ3	6593.60	2008	.922	.920	.066	.056	87.78	19	.001	0	.006	.065
3.3	τ1 =…= τ4	6678.76	2027	.921	.919	.067	.056	92.60	19	.001	.001	.007	.067
3.4	τ1 =…= τ5	6762.61	2046	.920	.919	.067	.057	88.08	19	.001	0	.007	.066
3.5	τ1 =…= τ6	6842.98	2065	.919	.919	.067	.057	86.28	19	.001	0	.006	.064
3.6	τ1 =…= τ7	6929.38	2084	.917	.917	.068	.058	88.57	19	.002	.002	.009	.073
3.7	τ1 =…= τ8	7029.21	2103	.917	.918	.067	.059	103.26	19	0	–.001	.010	.078
3.8	τ1 =…= τ9	7125.00	2122	.915	.917	.068	.060	212.39	19	.002	.001	.008	.072

Note. Λ : factor loading matrix; τ: item intercept vector; the subscript indicates the year of the measures collected.

Bold font indicates the difference between two comparative models is statistically negligible.

To evaluate the longitudinal measurement invariance, the Satorra-Bentler scaled differential chi-square test (Δχ²) was conducted to take the data nesting structure into consideration (Muthén & Muthén, 2010). Given the sensitivity of the Δχ² to the slight difference between two comparative models for large sample size dataset, four other criteria of invariance analysis were used, including changes in CFIs (ΔCFI; Cheung & Rensvold, 2002), and TLIs (ΔTLI; Little, 1997), Root Deterioration per Restriction Index (RDR; Browne & Du Toit, 1992), and the Expected Cross-Validation Index difference (ΔEVCI, Browne & Cudeck, 1993; Oort, 2009). When ΔCFI ≤ 0.02 (Cheung & Rensvold, 2002), ΔTLI ≤ 0.05 (Little, 1997), RDR ≤ .08, and/or the 90% confidence interval of ΔEVCI includes zero (Dudgeon, 2004), it would suggest that the two comparative models were not substantially different from each other. To the author’s knowledge, there is not a single criterion that can claim measurement invariance; therefore a finding that the majority of criteria are within the suggested thresholds is interpreted as evidence of measurement invariance (Koomen et al., 2012; Vandenberg & Lance, 2000).

We conducted multi-group comparison analyses to test if the invariance assumptions hold for gender and ethnicity. Three ethnic groups were included in this analysis, African American (AA), Hispanic (HIS) and White (W). In the case of non-invariance, modification indices were used to locate the non-invariant items across different groups.

After we confirmed the invariance assumptions across years, genders, and ethnicities of the TNRI, we examined the developmental pattern of TNRI factors across 9 years. The shift and the growth trend of factor means of the TNRI scales were tested using RM-ANOVA with Proc GLM in SAS 9.3 (SAS Institute, 2012). RM-ANOVA allowed us to investigate if the TNRI construct means would maintain or shift across 9 years. If significant mean shift was found, we tested the linear, quadratic, and cubic average growth trend of the TNRI scales to see if the decreasing or increasing trends would change linearly or if the trends would level off using the polynomial contrast option in Proc GLM procedure.

Results

Invariance of Model Structure across Years

The model information indicated the revised CFA model with Warmth, Conflict and Intimacy factors and four pairs of corrected items adequately fit the longitudinal data, χ²(1818) = 6220.03 with p < .001, CFI = .926, TLI = .915, RMSEA = .068 and SRMR = .048. Therefore, the configural factor invariance with the same pattern of freely estimated parameters held for 9 year data.

Metric and Scalar Longitudinal Measurement Invariance

The results of the metric and scalar longitudinal measurement invariance tests are reported in Table 1. Though the model fit test statistics (χ²) indicated that the hypothesized models fit did not exactly fit to the data, all model fit indices were within the commonly used criteria (Hu & Bentler, 1999) and demonstrated adequate goodness of fit of the hypothesized models to the data.

Metric Invariance

For yearly MI analyses from Model 2.1 to Model 2.8, except for Model 2.7, the differential chi-square test statistics were smaller than the critical values, which indicated the hypothesized model equivalence was not rejected. The Δχ² between Model 2.6 and Model 2.7 were significant but the other four model comparison criteria indicated no substantial difference between the factor loadings of Year 8 construct and those of the seven previous years, Δχ²(19) = 34.69 with p = .02, ΔCFI=0, ΔTLI = 0, ΔEVCI= .002 with 90% CI=[−.002, .003], and RDR=.045. Therefore, metric invariance was established and we proceeded to examine the scalar/intercept invariance hypothesis.

Scalar Invariance

When we conducted the analyses of scalar invariance for yearly measurements from Model 3.1 to Model 3.8, the pattern of the model comparison information was similar. For all the yearly MI analyses, the majority of criteria (i.e., ΔCFI ≤ 0.02, ΔTLI ≤ 0.05, and RDR ≤ 0.08) suggested negligible differences between the constrained and baseline models (Koomen et al., 2012; Vandenberg & Lance, 2000). The MI result supported both metric and scalar invariance and suggested that the constructs of TNRI scales were longitudinally equivalent in the sense of strong invariance.

Measurement Invariance across Gender and Ethnicity

Table 2 presents the multigroup comparison result on gender and ethnicity. In Model G.1, we had freely estimated factor loadings and item intercepts for boys and girls. In Model G.2, we fixed factor loadings to be the same for boys and girls but allowed item intercepts to be freely estimated for the two groups. In Model G.3, we further fixed the item intercepts to be the same for boys and girls. For ethnic group comparison of AA, HIS, and W, we followed the same procedure as that for the gender group comparison and yielded Model E.1, Model E.2, and Model E.3. All models exhibited adequate goodness-of-fit to the data.

Table 2.

Model fit test statistics and fit indices and their changes for the measurement models for gender and ethnicity

Model fit test statistics and fit indices								Change of model fit test statistics & fit indices
		χ2	df	CFI	TLI	RMSEA	SRMR	Δχ2	Δdf	ΔCFI	ΔTLI	ΔEVCI	RDR
Gender
G.1	Λ♀ ≠ Λ♂ & τ♀ ≠ τ♂	4102.04	404	.931	.921	.063	.047	–	–	–	–	–	–
G.2	Λ♀ = Λ♂ & τ♀ ≠ τ♂	4138.13	423	.930	.924	.061	.048	26.09	19	.001	−.003	−.0004	.020
G.3	Λ♀ = Λ♂ & τ♀ = τ♂	4163.60	442	.930	.927	.060	.048	2.28	19	0	−.003	−.0028	.012
Ethnicity
E.1	ΛAA ≠ ΛHS ≠ ΛW& τAA ≠ τHS ≠ τW	4110.97	606	.934	.924	.062	.045	–	–	–	–	–	–
E.2	ΛAA = ΛHS = ΛW& τAA ≠ τHS ≠ τW	4206.83	644	.932	.927	.061	.048	53.21	38	.002	−.003	.0044	.032
E.3	ΛAA = ΛHS = ΛW& τAA = τHS = τW	4382.12	682	.930	.929	.060	.049	163.77	38	.002	−.002	.0222	.049

Note. Λ represents the factor loading matrix; τ is the item intercept vector; the subscript indicates the group of the measures collected; AA = African American; HIS = Hispanics; W=Whites.

G.1/E.1: freely estimated Λ and τ for gender and ethnicity group; G.2/E.2: fixed Λ to be the same and freely estimated τ for gender and ethnicity group; G.3/E.3: fixed Λ and τ to be the same for gender and ethnicity group;

Bold font value indicates the difference between two comparative models is statistically negligible.

Comparing Model G.2 to Model G.1 and Model E.2 to E.1, the Δχ² test results were not statistically significant, revealing that neither gender nor ethnicity moderated the factor loadings in the CFA models for both gender and ethnicity, $\mathrm{\Delta }{\chi }_{\mathit{Metric}\mathit{Inv}.,\mathit{Gender}}^2\left(19\right)=26.09$ with p =.13, and $\mathrm{\Delta }{\chi }_{\mathit{Metric}\mathit{Inv}.,\mathit{Ethnicity}}^2\left(38\right)=53.21$ with p =.05. The Δχ² tests of scalar invariance hypothesis was not significant for gender but was marginally significant for ethnicity. Nevertheless, except for ΔEVCI, the other three model comparison criteria provided evidence for intercept equivalence for ethnic groups, ΔCFI =.002, ΔTLI = −.002, ΔEVCI = .022 with 90% CI[.014, .032], and RDR = .049. Therefore, we concluded that the TNRI measure is largely invariant across gender and ethnic groups. Coefficient alphas were also adequate across the 9 years, as shown in Table 3, ranging from .869 to .962.

Table 3.

The factor means, correlations and Cronbach’s alpha of TNRI scales across 9 years.

	Factor correlations									Factor means (κ)									Reliability (α)
	Conf with Warf			Intf with Warf			Intf with Conf			Warf mean			Conf mean			Intf mean			Warf	Conf	Intf
	Est.		SE	Est.		SE	Est.		SE	Est.		SE	Est.		SE	Est.		SE	13 items	6 items	3 items
Year1	−.446	**	.038	.376	**	.035	−.112	**	.032	.000	—	.000	.000	—	.000	.000	—	.000	.949	.919	.884
Year2	−.470	**	.042	.384	**	.038	−.077	*	.034	−.070		.048	−.035		.045	−.062		.058	.951	.916	.880
Year3	−.378	**	.040	.407	**	.038	−.069	*	.030	−.058		.050	−.077		.045	−.220	**	.058	.956	.916	.863
Year4	−.385	**	.042	.393	**	.039	−.077	*	.030	−.093		.052	−.114	**	.044	−.318	**	.060	.950	.909	.869
Year5	−.379	**	.039	.412	**	.039	−.022		.034	−.145	**	.051	−.121	**	.044	−.402	**	.060	.944	.926	.901
Year6	−.434	**	.047	.323	**	.042	−.042		.031	−.262	**	.057	−.157	**	.046	−.725	**	.060	.960	.937	.900
Year7	−.459	**	.055	.324	**	.042	.014		.031	−.393	**	.059	−.158	**	.048	−.857	**	.059	.960	.937	.904
Year8	−.391	**	.045	.420	**	.046	−.016		.029	−.532	**	.062	−.201	**	.044	−.935	**	.059	.962	.923	.913
Year9	−.293	**	.037	.263	**	.036	−.016		.021	−.438	**	.058	−.253	**	.043	−.916	**	.057	.949	.919	.886

Note. Est. = parameter estimate; SE = Standard error; Conf = Conflict factor; Warf = Warmth factor; Intf = Intimacy factor. Factor means (κ) reported were computed from standardized factor scores with Year1 factor mean as the baseline, which can be interpreted as Cohen’s d (Cohen, 1992).

^*p ≤ .05,

^**p ≤ .01.

Developmental Changes in Mean Level of TNRI scales

The yearly factor means of TNRI constructs were reported in standardized scores with factor means at Year1 as the baseline, which is equivalent to Cohen’s d (Cohen, 1992), in Table 3. The growth trends of three scales were depicted in Figure 1. The Warmth factor means were consistent across Year 1 to Year 4 but those of Year 5 to Year 9 were significantly smaller than the Year 1 mean (partial η² = 10.1%, F_(8,6.676)=12.052, p<.001)³. The cubic growth trend fit the data better with the factor means decreasing after Year 4 and leveling off at Year 8 (R² =3.6%, F_(3,4652)=57.27, p<.001). Though Conflict factor means were consistent for the first three years, they were statistically different from Year 4 to Year 9 (partial η²=3.5%, F_(8,6.477)=12.052, p<.001) and showed a linearly decreasing pattern (R² =1.2%, F_(1,4654)=56.31, p<.001). Like the Warmth scores, the Intimacy factor means exhibited a cubic growth pattern (R² = 13.7%, F_(3,4652)=246.54, p<.001). Despite the consistent factor means for the first two years, the Intimacy factor means decreased form Year 3 and leveled off at Year 8 (partial η²=22.5%, F_(8,6.768)=31.08, p<.001).

Figure 1.

The development change in mean Level of TNRI scales. From Year 1 to Year 9, Warmth and Intimacy show cubic growth pattern, while Conflict demonstrates linearly decreasing trend. The solid regression line best fitted the means of factor scores shown as in circles.

Discussion

Measurement Invariance

The primary purpose of the current study was to investigate the invariance of the measurement model of the TNRI across grades 1 to 9 (ages 6 – 15) as well as across gender and ethnicity. In each test of invariance, we followed the sequence of steps recommended by Millsap (2011). First, we tested whether the same three-factor structure of the TNRI was a satisfactory representation of dimensions of teacher-student relationship quality across groups (i.e., configural invariance). Next, we applied multi-group invariance analysis across groups to test equivalence of factor loadings (metric invariance) and then equivalence of item intercepts (scalar invariance), using a variety of model comparison statistics. Statistical support for strong invariance of the TNRI across grades/ages, gender, and a racial/ethnic group was found.

These results mean that observed scores on the TNRI scales have the same or very nearly the same meaning across these groups, which provide evidence to our hypothesis of longitudinal measurement invariance of the TNRI across elementary and middle school based on social support theory (Furman, 1996). Therefore, the TNRI is an appropriate research instrument for answering substantive questions about developmental changes from age 6 to 15 and about gender or racial/ethnic differences in TSRQ among students within this broad age span.

Developmental Shifts in of TNRI Scales

This study is the first to investigate changes in mean levels of TSRQ from the early elementary grades to the end of middle school. Intimacy showed the steepest decline (partial η²=22.5%,), followed by Warmth (partial η²=10.1%,) and Conflict (partial η²=3.5%,). As shown in Figure 1, Intimacy and Warmth had a cubic average growth pattern and Conflict had a linearly decreasing pattern. Findings suggest that the normative decline in positive aspects of the teacher-student relationship reported across the elementary grades (Jerome et al., 2009; O’Connor, 2010) continues in the early years of middle school before leveling off in Year 8, and that the normative decline in conflict observed at the end of elementary school (Jerome et al., 2009) continues through middle school. Prior research has found that middle school students’ perceptions of teacher support is a strong predictor of their sense of school belonging and engagement in school (Barber & Olsen, 2004; Niehaus, Rudasill, & Rakes, 2012; Roeser, Eccles, & Sameroff, 1998). Furthermore, low school engagement in middle school is a strong predictor of low academic achievement in high school and dropping out of school (Casillas, Robbins, Allen, Kuo, & Hanson., 2012; Johnson, McGue, & Iacono, 2006). Thus the downward trend in perceived teacher support in middle school underscores the need for school psychologists and other school practitioners to implement school-wide interventions focused on building positive, supportive relationships during middle school. Findings from a recent randomized clinical trial suggests that providing middle school teachers with the opportunity to reflect on individualized feedback on classroom teaching interactions with a supportive consultant results in more positive teacher-student interactions and achievement (Allen, Pianta, Gregory, Mikami, & Lun, 2011).

Study Limitations Implications for Future

Although the study models acceptably fit our longitudinal dataset, findings need to be considered in light of certain limitations. The study sample was selected based on scoring below the median on a district-administered measure of literacy at the beginning of first grade; however, their scores at baseline on the nationally standardized Woodcock Johnson III Broad Reading and Broad Math were similar to those in the standardization sample in terms of the sample mean and standard deviation. Thus study results may not generalize to normally and higher achieving students, because this study did not establish measurement invariance with regard to more diverse sample. Analyses of whole school climate and student behavioral norms that involve a broader population and also use the TNRI must be approached with caution. Additionally, teacher support was assessed only from the perspective of the teacher. Student report of teacher social support might lead to different conclusions regarding developmental trends. Future research is needed on measurement properties of the TRNI with more diverse samples, including its correspondence with student report of similar relationship constructs.

Conclusion

The TNRI appears to meet the need for a measure of TSRQ with strong invariance across age, gender, and race/ethnicity. Thus, the TNRI is the only measure of TSRQ in the published literature that meets measurement invariance standards for investigations of developmental changes in teacher support from early childhood through early adolescence. Among questions that could be pursued with the TNRI are “Do boys and girls or racial/ethnic groups differ in their growth trajectories for TSRQ across elementary and middle school?” and “What are the implications of Warmth and Conflict trajectories on adolescents’ commitment to school and high school completion?” The TNRI is also appropriate for investigations of age-related changes in the associations between TSRQ and student social and academic outcomes as well as research on the mechanisms that account for these associations at different developmental periods. For example, in the early elementary grades a supportive relationship with one’s teacher may provide a secure basis for children, thereby promoting behavioral self-regulation (Liew, Chen & Hughes, 2010). In middle school, teacher social support may buffer students from the normative decline in identification with and bonding to school (Catalano, Haggerty, Osterle, Fleming, & Hawkins, 2004; Liljeberg, Eklund, Fritz, & Klinteberg, 2011).