A Confirmatory Factor Analysis of Home Environment and Home Social Behavior Data from the Elementary School Success Profile for Families

Abstract

The purpose of the current study was to test the factor structure and scale quality of data provided by caregivers about the home environment and child behavior at home using the Elementary School Success Profile (ESSP) for Families. The ESSP for Families is one component of the ESSP, an online social–environmental assessment that also collects information from students and teachers. Confirmatory factor analyses with Mplus and weighted least squares means and variances adjusted estimation took into account the hierarchical nature and ordinal level of the data. The sample comprised caregivers of 692 third- through fifth-grade students from 13 elementary schools in four districts. A primary model and an alternative model were tested. Models were tested on a random calibration sample and validated with another sample. A nine-factor first-order solution demonstrated superior fit to the data. Scores from the nine scales also demonstrated acceptable internal consistency reliability. Implications for practice and further research are presented.

The theoretical and empirical knowledge bases of developmental psychopathology (Costello & Angold, 1996; Sroufe, 1997), risk and resilience (Rutter, 2001; Sameroff, 2000), ecological theory (Bronfenbrenner, 1992), and developmental contextualism (Lerner, 1986) suggest that students’ social and academic behaviors in schools are influenced by their experiences in the social environment. Research supporting these theories demonstrates the impact that the social environment has on concurrent and future child outcomes (for example, Barnard, 2004; Chase-Lansdale & Gordon, 1996). In spite of the pivotal role of social–environmental experiences in child development, few well-validated social–environmental assessments exist for school-based practitioners. An exception is the School Success Profile (SSP), a self-report social–environmental assessment for middle and high school students, which was developed in the early 1990s (G. L. Bowen, Richman, & Bowen, 2002; G. L. Bowen, Rose, & Bowen, 2005). Based on an “eco-interactional-developmental” perspective (G. L. Bowen et al., 2002), the SSP assesses dimensions of the neighborhood, school, peer system, and family as experienced by youths. Youths’ perceptions of dimensions of their own psychological and physical well-being and school performance are also assessed. Individual- and group-level data are provided to schools to guide change efforts.

In response to practitioner requests for an elementary version of the SSP, the Elementary School Success Profile (ESSP) (N. K. Bowen, 2006; N. K. Bowen & Powers, 2005; Woolley, Bowen, & Bowen, 2004) was developed through an iterative process incorporating expert, practitioner, and respondent feedback. The ESSP makes it possible to assess the social environment of children in grades 3 through 5. The ESSP assesses the same major domains as the SSP but with differences in specific dimensions that reflect the developmental stage of middle childhood. Because the ESSP assesses younger children who might not be developmentally capable of providing valid self-report data about all aspects of their social environments, ESSP data are collected from caregivers and teachers in addition to children.

Together, the three components of the ESSP collect data about the following social domains: neighborhood (child and caregiver report), school (child and caregiver report), friends (child report), caregiver education involvement (caregiver and teacher report), and family (child and caregiver report). Data are also collected on social behavior at home and school (caregiver and teacher report) and school performance (teacher report). Sources of data within domains overlap when appropriate, providing multiple perspectives on the same social environmental factors.

Analysis of data collected from caregivers supported the reliability of scales, with alphas ranging from .69 to .94 (Bowen, 2008). Results of exploratory factor analyses indicated adequate convergent and discriminant validity of dimensions. Construct validity has been supported through significant correlations between scales on the ESSP for Families (previously called the “ESSP for Parents”) and similar scales on the ESSP for Teachers and ESSP for Children. More information on the three-part ESSP, including its theoretical approach, is available in previously published work (N. K. Bowen, Bowen, & Woolley, 2004; see also http://schoolsuccessonline.com).

Like the SSP, the ESSP provides individual- and group-level reports that are used by school staff to guide intervention efforts and promote school success. The theory of change derived from the eco-interactional-developmental perspective (G. L. Bowen et al., 2002) posits that social–environmental factors influence well-being, which in turn influences behavior and performance. According to this theory of change, intervening in malleable environmental dimensions is expected to promote school success. The purpose of the ESSP is consistent with recent developments in the school intervention literature, such as calls for evidence-based practices, school-level prevention approaches and data-driven decision making and a focus on social and emotional learning issues in addition to academic issues (Graczyk, Domitrovich, Small, & Zins, 2006).

The current study adds to the validation evidence for the caregiver component of the ESSP for Families. In addition to providing a solution to the potential problem of unreliable data from child informants, collection of information about student performance from multiple sources and across different settings is considered a hallmark of high-quality assessment (Merrell, 2003; Pepler & Craig, 1998; Silverman & Saavedra, 2004). Documented differences in perspective among sources regarding the same phenomena support the importance of obtaining multiple views (Achenbach, McConaughy, & Howell, 1987; Offord et al., 1996; Pepler & Craig, 1998). Pepler and Craig, for example, outlined the strengths and limitations of different sources, concluding that a combination of information from multiple sources provided the most complete understanding of phenomena. Data collected with the ESSP for Families complement the data collected from children and teachers. Because caregivers have the most knowledge of their own children over time and across settings (Pepler & Craig, 1998), they may be the most important source of information on children (Achenbach et al., 1987). Establishing the quality of data collected with the ESSP for Families, therefore, is a critical step in validating the ESSP as a whole.

The purpose of the current study was to test the factor structure and scale quality of family-report items assessing the home environment and child behavior at home. Findings of the study permitted progress toward an additional goal: reducing the length of the ESSP for Families. The caregiver report component of the ESSP is the longest of the three survey components, with 185 items and an average response time of 30 minutes. School staff have commonly reported that length has prevented some respondents from completing the ESSP for Families. Excessive length might also be expected to reduce the reliability of data collected from caregivers. The present analysis included only items belonging to scales assessing child social behavior at home and the home environment, focusing on 50 items from the original 185-item instrument.

Method

Sample

Caregivers of 692 third- through fifth-grade students attending 13 low-performing schools in four school districts in a southeastern U.S. state constituted the sample. Each school was participating in one of four ESSP projects during the 2008–09 school year. The schools by district, number, percentage of responses per school, and percentage of urbanicity/rurality of the sample are displayed in Table 1. Five elementary schools in two rural, economically depressed districts (districts C and D) (see Table 1) constituted 64% of the sample. In a project in more urban district (district B) (see Table 1), students in five elementary schools were randomly selected from among low-achieving third-, fourth-, and fifth-graders. Data were collected from about 63.7% of the targeted sample in these schools, primarily because one of the five schools failed to collect data. In another district (district A) (see Table 1), all fourth-graders in four schools were targeted. Students in three of the schools were in the second year of a three-year project that started when the students were third-graders. School number four (see Table 1) was a new school that was added to the project because many of its students had been reassigned from the three schools originally targeted. Data were collected on about 80% of the targeted population in this project.

Table 1.

School Responses by District and Location (N = 1,251)

District and School	Urban (U) or Rural (R)	Percentage of Full Sample	Percentage of Cases with Family Data
A
1	U	56	28 (4.6)
2	U	56	27 (4.4)
3	U		46 (7.5)
4	U		30 (4.9)
B
5	U		27 (4.4)
6	U		33 (5.4)
7	U		12 (2.0)
8	U		18 (3.0)
C
9	R		78 (12.8)
10	R		54 (8.9)
D
11	R		110 (18.0)
12	R		72 (11.8)
13	R		75 (12.3)

The ESSP is designed to be useful to schools wanting to assess only those students who are struggling with social or academic problems in school or schools wanting to understand the positive and negative influences and experiences of all their students. By combining samples from multiple districts using the ESSP in both ways, we were able to obtain a sample that was large enough for the planned psychometric tests and heterogeneous enough to ensure adequate variance on the items being tested. Substantively, the combination of samples allowed adequate representation of students most in need of services at a variety of schools and settings, including both urban and rural, as well as of various socioeconomic statuses.

Caregiver data were available for 54% (n = 692) of the 1,251 students represented in the data from the 13 schools. Schools typically are able to collect data from a higher percentage of targeted students and teachers than caregivers. Students, for example, are taken to the computer lab at school. Teachers as a group may be more comfortable with computers than caregivers are and hear more at school about the purpose and value of ESSP data than caregivers do. Low-performing schools, furthermore, are less likely to have strong, trusting relationships with caregivers that can facilitate data collection. Chi-square tests on variables from the teacher survey of the ESSP showed that caregivers who completed the survey were more likely to have a child that the teacher perceived as capable of performing at or above grade level in reading [χ²(2, N = XX) = 41.69, p < .001] and in math [χ²(2, N = XX)= 23.68, p < .001). Significant differences were also seen with regard to actual student achievement, as caregivers completing the survey were more likely to have children earning passing grades in reading [χ²(4, N = XX), p <.001) and in math [χ²(4, N = XX)=54.28, p < .001).

The majority of respondents to the ESSP for Families (82.9%) identified themselves as their child’s mother or stepmother. Almost 10% identified themselves as the father or stepfather, 4.6% as a grandparent, and 2.7% as another relative or “other.” More than half (56%) of the caregiver sample reported that their children received free or reduced-price lunches at school, indicating that the families had limited household incomes. Just over half (52%) of caregivers reported their child as female. Forty-five percent of caregivers reported their children’s race/ethnicity as white, whereas 40% indicated African American, 7% indicated Latino, 2% indicated Asian, and 1% indicated Native American race/ethnicity. The remainder (4%) classified their children’s race/ethnicity as “other.” According to teacher report, approximately 64% of the children were performing at or above grade level in math, and 63% were performing at or above grade level in reading.

Measures

The analysis focused on 50 items comprised by scales about the home environment and child behavior at home on the original ESSP for Families instrument. Due to length considerations, survey items are not listed; however, Table 2 presents a brief description of the items that were analyzed. The items were hypothesized to load on nine dimensions. Three dimensions were derived from scales on the ESSP’s “parent” instrument, the SSP (G. L. Bowen et al., 2005). These scales are Home Educational Environment, Warmth and Encouragement, and Family Who Care. Three other scales were newly defined for the ESSP on the basis of theoretically salient aspects of the home life of children in middle childhood (Home Learning Activities, Sibling Relationships, Positive Parenting). Items on these first six scales were assessed with four options: “always,” “often,” “sometimes,” and “never.” Three scales were derived from dimensions of social behavior defined in the Carolina Child Checklist (Macgowan, Nash, & Fraser, 2002). These scales are Is Sociable with Other Children, Uses Good Social Skills, and Interacts Peacefully. The Carolina Child Checklist is an adaptation of the Social Health Profile and the Teacher Observation of Classroom Adaptation–Revised, as described by Macgowan et al. Response options for items in the three behavior dimensions were “never,” “rarely,” “sometimes,” “often,” “very often,” and “always.”

Table 2.

Current Home Environment and Social Behavior Dimensions on the Elementary School Success Profile for Families

Dimension	Description
Home Educational Environment	Six items assessing how often the caregiver talks to the child about what he or she is learning in school, what happens in school, and homework assigned; check on homework, offering praise for hard work; provide supplies for homework
Home Learning Activities	Four items assessing the frequency with which the caregiver manages the child’s time on TV, play, reading, and learning activities
Is Sociable with Other Children	Six items assessing how often the child is liked by playmates, is friendly, initiates interactions and plays with others, and is accepted by other well-behaved children
Uses Good Social Skills	Nine items assessing how often the child thinks before acting, gives suggestions without being bossy, calms down when excited, helps others, controls temper, accepts authority, understands others’ feelings, breaks rules, and acts stubbornly
Interacts Peacefully	Four items assessing how often the child fights, harms others, plays aggressively, or hits on purpose
Warmth and Encouragement	Five items assessing how often the caregiver expresses warmth, encouragement, and affection to the child and spends free time with the child
Family Who Care	Six items assessing how often family members support each other, play together, give each other attention, have open discussions, work together to solve problems, and feel loved
Sibling Relationships	Three items assessing how often the child gets along with, shares with, or comforts other children in the home
Positive Parenting	Seven items assessing how often the caregiver praises the child, punishes for misbehavior, listens to the child before punishing, explains punishment, uses time out or takes away privileges, and remains calm when punishing

Data Collection

Caregivers completed the ESSP for Families using either the online version or a paper copy of the survey. Data collection windows for caregivers, teachers, and children were set by each school. Some schools collected data in the fall; others completed their data collection early in the spring of the 2008–09 school year. Caregivers were provided with instructions for accessing the online survey or completing and returning hard copies to the school. Hard copies were scannable teleforms that were scanned after completion and saved into text files. The text files were then added to the online survey database. All data are anonymous and reside on a secure server managed by the Internet technology company that has developed the online capabilities of the SSP and ESSP. The ESSP for Families is available in English and Spanish and has an online audio option. Many Latino caregivers opt to complete the survey in Spanish, but the current study does not include data on language choice or format (hard copy versus online).

Data Analysis

Confirmatory factor analysis (CFA) was selected as the analysis method because the scales on the ESSP for Families have a theoretical base established in prior work on the ESSP and its parent instrument, the SSP (G. L. Bowen et al., 2002, 2005; N. K. Bowen et al., 2004). Results of earlier exploratory factor analysis on the ESSP for Families scales (Bowen, 2008) support the hypothesized factor structure of the current analysis.

Mplus version 4.21 (Muthén & Muthén, 2007) was chosen to perform the CFA because of the program’s ability to handle characteristics of ESSP data appropriately, such as the clustering of caregiver respondents in 13 schools, missing data, and ordinal variables. The weighted least squares means and variances adjusted (WLSMV) estimator was used because ESSP variables are both ordinal and nonnormally distributed. WLSMV is a robust weighted least squares approach, and studies suggest that it is yields accurate estimates and standard errors under a variety of conditions (Flora & Curran, 2004). When a user selects the WLSMV estimator and specifies categorical variables, Mplus generates a polychoric correlation matrix and an associated weight matrix (Muthén & Muthén, 1998–2007). Polychoric correlation matrices are based on underlying theoretical normal distributions of categorical variables derived from bivariate distributions of the original variables (Bollen, 1989; Kline, 2005). Mplus’s full information (FIML) procedure for analyzing data sets with missing values was used, allowing all available information to be used in analyses without deleting cases with missing values. FIML has been shown to perform better than other common approaches to handling missing values (Enders & Bandalos, 2001).

Validation of structural equation modeling (SEM) models on independent samples is considered a component of rigorous factor analysis (de Vet, Adèr, Terwee, & Pouwer, 2005) and is especially important if modifications have been made (Byrne, Shavelson, & Muthén, 1989; Kline, 2005). The caregiver data were divided into two random subsamples. The first subsample (n = 329) was used as a calibration sample. The second subsample (n = 363) was used to validate the results of analyses on the calibration sample. T tests conducted using SPSS version 17.0 (SPSS, 2009) confirmed that there were no statistically significant differences between the calibration and validation samples on gender of respondent, race/ethnicity, and free/reduced-price lunch status. Rates of missing values were low across the 50 analysis variables, ranging from 0.6% to 2.7%, with the exception of three variables. Between 17.6% and 18.2% of values were missing on three variables assessing students’ relationships with other children in the home due to the fact that some students lived in homes with only one child.

Another recommended SEM analysis practice is the testing of alternative models (Kline, 2005). Testing multiple models permits a comparison of fit that provides more confidence in retained models. Our primary model hypothesized that the 50 indicators related to the home loaded onto nine first-order factors corresponding to the dimensions shown in Table 2, which were based on theory and previous empirical work (see Figure 1). The alternative model hypothesized that the nine first-order factors identified in the first model loaded onto three second-order factors related to ecological domains. Home Educational Environment and Home Learning Activities were indicators of a second-order Home Education factor; Sibling Relationships, Family Who Care, Warmth and Encouragement, and Positive Parenting were indicators of a second-order Family Relations factor; and Is Sociable with Other Children, Uses Good Social Skills, and Interacts Peacefully were indicators of a second-order Social Behavior factor. The names of these factors were chosen to reflect a consistent positive coding direction and because they make sense to school staff. Interacts Peacefully measures aggressive behaviors but is coded so that high scores indicate an absence of aggressive behaviors.

Figure 1. Initial First-Order Factor Model of 50 Items from the Elementary School Success Profile for Families.

Note: All latent factors were correlated in the confirmatory factor analysis.

All models were evaluated in terms of the magnitude of factor loadings, significance of loadings, and model fit indices. We sought high, statistically significant, and similar factor loadings among items loading on any one factor. Mplus provided the following fit indices when the WLSMV estimator was used: Satorra Bentler scaled chi-square(SB χ²), Tucker-Lewis index (TLI), comparative fit index (CFI), root mean square error of approximation (RMSEA), and weighted root mean square residual (WRMR).

The SB χ² is similar to the traditional chi-square statistic, adapted for WLSMV calculation by Mplus. As with the traditional chi-square test of model fit, nonsignificant SB χ² values indicate that the reproduced correlation matrix is statistically equivalent to the input matrix of observed indicator correlations (Muthén & Muthén, 2004). Browne and Cudeck (1993) suggested that RMSEA values less than or equal to .05 indicate close fit, whereas values between .05 and .08 indicate “reasonable” fit. Values exceeding .95 are desirable for the CFI and the TLI (Hu & Bentler, 1999). WRMR values should be below .90 (Muthén & Muthén, 1998–2007).

The change in SB χ² per change in degrees of freedom is used to compare the fit of two nested models. When there is no statistically significant difference in fit between a more and less constrained model, the more parsimonious model is chosen. If the deterioration in fit is statistically significant, the less restrictive model is retained (Kline, 2005). The chi-squarecomparison tests were calculated by the Mplus program because the usual manual test does not apply to chi-squareand degree of freedom statistics obtained with WLSMV (Muthén & Muthén, 2004).

Polychoric correlations tend to be higher than Spearman or Pearson correlations among ordinal items (Bollen, 1989). Although it is the preferred approach, analyzing the polychoric correlation matrix of our indicators may have complicated our analyses in two ways. First, the majority of factor loadings in the first-order model were unexpectedly high (for example, over .90), suggesting that many items were redundant. Second, high primary loadings of many items were accompanied by secondary loadings that were greater than about .20 and statistically significant. If double-loading items are retained and permitted to load on only one factor, model fit is reduced. If they are allowed to load on more than one factor, they typically fail to load at desirable levels (for example, over .30) on one or both factors. Little guidance for handling double-loading items is provided in the literature; however, a general preference for maintaining simple structure (each item loading on only one factor) is expressed. Because a goal of this analysis was to reduce the number of survey items, deletion of double-loading items was the logical strategy, provided that the deletion of any item did not adversely affect the reliability of the scale from which it came.

Although the double loadings indicated that the scales on the ESSP for Families should be shortened, multicollinearity among multiple indicators within factors and multiple high secondary loadings made it difficult to systematically select items for deletion or improve model fit. Therefore, we used the jigsaw piecewise technique described by Bollen (2000) to develop the first-order factors in the primary model. With this technique, each set of indicators is fitted to its hypothesized factor before multiple factors are analyzed together. Modification indices were used in a systematic manner to identify indicators with high double loadings. The expected parameter change (EPC) values provided in Mplus’s modification indices were examined to determine the number and magnitude of double loadings. Items with the largest sums of EPC values were considered for deletion, with theoretical arguments determining the order in which high-sum items were deleted. The validation of the final first-order model and the testing of an alternative model were especially important given that our analysis became more of a “model generation” (Jöreskog, 1993) exercise and less confirmatory than planned.

To ensure that the ESSP for Families scales met traditional standards for internal consistency reliability in spite of item deletion during the CFA process, reliability statistics were obtained for all scales in the final model, using SPSS for Windows version 17.0 (SPSS, 2009). Standard cutoff values for Cronbach’s alpha were used to evaluate the results, with α ≥ .70 indicating adequate reliability, α ≥ .80 indicating very good reliability, and α ≥ .90 indicating excellent reliability (Kline, 2005).

Results

Primary Model

When all factors were tested together in the first test of the primary model, only three of the five preestablished fit criteria were met (TLI, CFI, and RMSEA) (see Table 2). Modification indices indicated problems with many items. We used the jigsaw piecewise procedure (Bollen, 2000) to test the nine first-order factors individually and then in combination with each other. Obtaining good fit required the deletion of items for eight of the nine hypothesized factors. During the jigsaw piecewise analyses of the primary model using the calibration sample, the number of indicators in the primary model was reduced, using the procedures described earlier, from 50 to 31. The final factor structure of the nine-factor model is presented in Figure 2. In addition to the deletion of items, the error terms for two items on the Patient Parenting scale were allowed to be correlated to improve model fit. Both items assessed the caregiver’s tendency to listen to and talk to the child about misbehavior at the time of punishment, making the correlation theoretically sensible. Because of the nature of retained items in three of the scales, scale names were changed (Is Sociable with Other Children became Is a Good Playmate, Uses Good Social Skills became Uses Self-Control, and Positive Parenting became Patient Parenting). The final first-order model demonstrated good fit in the calibration sample (n = 329) according to all preestablished fit criteria, including a nonsignificant chi-square value (see Table 3). The final first-order model was also tested on the validation sample (n = 363). It again demonstrated good fit by all indices, as shown in Table 3.

Figure 2. Final Model, with Standardized Loadings for 31 Items from the Elementary School Success Profile for Families.

Notes: All latent factors were correlated in the confirmatory factor analysis. All loadings were significant at p ≤.05.

Table 3.

Fit Statistics for Four Models

			Fit Index
Model	n	df	χ2 (p)	TLI	CFI	RMSEA	WRMR
Initial first order (calibration sample)	329	8	26.07 (.00)	.97	.97	.08	1.28
Final first order (calibration sample)	329	8	15.12 (.06)	.99	.99	.05	0.85
Second order (calibration sample)	329	8	22.77 (.00)	.98	.98	.08	1.18
Comparison of first and second order	329	5	29.86 (.00)
Final first order (validation sample)	363	8	14.62 (.07)	.99	.99	.05	0.87

Notes: The chi-squares and degrees of freedom for weighted least squares means and variances adjusted were estimated according to formulas given in the Mplus technical appendices at http://www.statmodel.com. The comparison of nested models (primary versus alternative) also uses special chi-square and degree of freedom values and cannot be conducted in the usual manual way. TLI = Tucker-Lewis index; CFI = comparative fit index; RMSEA = root mean square error of approximation; WRMR = weighted root mean square residual. Recommended cutoffs: p > .05 (χ²); ≥.95 (TLI); ≥.95 (CFI); <.05 (RMSEA); ≤.90 (WRMR).

Alternative Model

The alternative model evaluated the hypothesis that the first-order factors of the primary model were better modeled as indicators of three second-order factors related to Home Education, Family Relations, and Social Behavior. The alternative model was nested in the primary model and was, therefore, expected to have worse fit. Consistent with expectations, the model met only three of our five fit criteria (TLI, CFI, and RMSEA), as shown in Table 3. According to the Mplus chi-square difference test for nested models, the fit of the alternative model was significantly worse than that of the primary model (χ² of difference = 29.86, df difference = 5, p = .00). Therefore, the less constrained model—the nine-factor, first-order model shown in Figure 2—was retained.

In the final model, the nine first-order factors had between three and five items each. As shown in Table 4, which lists the scale dimensions and a brief description of the items retained in each scale, internal consistency reliability scores (alphas) were adequate to very good for all scales, ranging from .72 to .88 (Kline, 2005).

Table 4.

Elementary School Success Profile for Families Child Behavior Scale Items and Reliability after Confirmatory Factor Analysis

Dimension	Description	α
Home Educational Environment	Three items assessing how often the caregiver talks to the child about what he or she is learning in school, what happens in school, and homework	.84
Home Learning Activities	Three items assessing the frequency with which the caregiver manages the child’s time on TV, play, reading, and learning activities	.72
Is a Good Playmate	Three items assessing how often the child plays with others and is accepted by and plays with other well- behaved children	.88
Uses Self-Control	Three items assessing how often the child thinks before acting, calms down when excited, and control temper	.81
Interacts Peacefully	Four items assessing how often the child fights, harms others, plays aggressively, or hits on purpose	.82
Warmth and Encouragement	Three items assessing how often the caregiver tells the child he or she is loved or has done a good job and makes the child feel special	.81
Family Who Care	Five items assessing how often family members support each other, play together, give each other attention, have open discussions, work together to solve problems, and feel loved	.87
Sibling Relationships	Three items assessing how often the child gets along with, shares with, or comforts other children in the home	.85
Patient Parenting	Four items assessing how often the caregiver praises the child, listens to the child before punishing, explains a punishment, and remains calm when punishing	.72

Discussion

Using CFA features in Mplus version 4.21 that accommodated the nonnormality, ordinal measurement level, missing values, and hierarchical structure of our data, we tested caregiver report data on 50 indicators of nine scales related to the home environment. To improve the rigor of our methods, we tested an alternative model and validated our findings with a validation sample randomly selected from the full data set. The results suggested that the hypothesized first-order model fit the data well. The analyses supported combining item scores according to the dimensions on reports to school staff on individual- and group-level ESSP scores from the family component. On the basis of both the model fit indices and the chi-square difference test, a first-order factor model demonstrated statistically significantly better fit than a second-order model. The inadequate fit of the second-order factor structure indicates that it is not appropriate to report higher order composite scores from the dimensions.

In addition to providing evidence of the quality of the scales on the ESSP for Families, the analyses allowed us to shorten the ESSP for Families by 19 items related to the home environment, increasing the feasibility of its administration to family members. The jigsaw piecewise procedure recommended by Bollen (2000) was useful for identifying items for deletion. The primary model maintained good fit following item deletion, and scale scores maintained reliability. For the ESSP for Families questionnaire, therefore, reducing item redundancy was a desirable course.

The results of the CFA contribute to evidence of the validity of data collected from family members about aspects of the home environment and children’s behavior outside of school. Because caregivers are critical reporters on experiences of their children in the home environment (Achenbach et al., 1987; Pepler & Craig, 1998), the positive findings regarding the quality of the ESSP for Families lend support to the overall quality of the ESSP as a tool to help school staff better understand students and how to help them succeed. The improved feasibility of the reduced ESSP for Families makes it more possible for school social workers to implement the ESSP in schools to guide intervention efforts. Generally, a shorter survey is likely to be perceived as less burdensome by participating families. As such, the shorter instrument is likely to increase caregiver response rates, which will improve the overall quality of the data obtained by providing a more representative picture of caregiver perspectives at a school. Because the ultimate purpose of the ESSP is to inform social work interventions, more representative data will be useful in selecting the most appropriate and effective intervention strategy for a particular school. Increased response rates will also help to reduce amounts of missing data, which can pose significant analytical problems.

Strengths and Limitations

The use of Mplus is a strength of the current study because of the program’s capacity to analyze hierarchical, nonnormal, ordinal data with missing values. These features enabled us to take full advantage of the data available. An additional strength of this study is the use of a validation sample to provide evidence that satisfactory fit of the final calibration model was not based on statistical chance that was amplified with modifications. Our sample was also of adequate size; was diverse in terms of race/ethnicity, grade level, and achievement level; and contained children and schools with characteristics consistent with the target population of the ESSP.

A limitation of the current study is that we did not obtain information about which caregivers completed the online versus the hard copy survey or which caregivers completed the Spanish version in either format. It is possible that differences in survey administration could change the nature of caregivers’ responses to survey items. This possibility will be tested in future research on the ESSP for Families. Another limitation of the current study is that we were unable to identify duplicate cases of caregiver report, such as when one caregiver has two children participating in the ESSP and fills out one survey with respect to each child. Therefore, some families may be represented more than once in the data set. We do not anticipate that many families were overrepresented in this manner, because several schools collected data on one grade level only, and several schools randomly sampled children for inclusion in the ESSP study. Finally, as mentioned in the description of the sample, there were statistically significant differences between caregivers who completed the ESSP for Families and caregivers who did not with respect to the academic achievement of their children. Such differences may be due to low-performing schools being less likely to have the sorts of trusting relationships with families that encourage participation. Stronger efforts to encourage survey participation by all families in schools, particularly those with children who are not achieving academically, may help to remedy this limitation in future projects.

Future work will also assess relationships among family-, teacher-, and student-report scales on the ESSP for evidence of construct validity and relationships between ESSP data and data from other sources for evidence of construct and criterion validity. The stability of constructs over time and across populations is also an important topic for analysis in the future.