Abstract
This study examines the psychometric properties and component structure of a newly developed observational system, the Aftercare and School Observation System (ASOS). Participants included 468 children drawn from a larger longitudinal intervention study. The system was utilized to assess participant children in school lunchrooms and recess and various afterschool environments. Exploratory factor analyses examined whether a core set of component constructs assessing qualities of children's relationships, caregiver involvement and monitoring, and experiences in school and aftercare contexts that have been linked to children's behavior problems would emerge. Construct validity was assessed by examining associations between ASOS constructs and questionnaire measures assessing children's behavior problems and relationship qualities in school and aftercare settings. Across both settings, two factors showed very similar empirical structures and item loadings, reflecting the constructs of a negative/aggressive context and caregiver positive involvement, with one additional unique factor from the school setting reflecting the extent to which caregiver methods used resulted in less negative behavior and two additional unique factors from the aftercare setting reflecting positivity in the child's interactions and general environment and negativity in the child's interactions and setting. Modest correlations between ASOS factors and aftercare provider and teacher ratings of behavior problems, adult-child relationships, and a rating of school climate contributed to our interpretation that the ASOS scores capture meaningful features of children's experiences in these settings. This study represents the first step of establishing that the ASOS reliably and validly captures risk and protective relationships and experiences in extra-familial settings.
Keywords: Child adjustment, Behavior problems, School context, After school context, Observational data
Introduction
As children develop, they spend increasing amounts of time in multiple contexts outside of their home, including school, afterschool, and community environments. By age 7, approximately half of children's daytime hours are spent in school, and for many children, afterschool programs or activities away from family also fill a significant portion of their day. In fact, time use studies suggest that most school-age children spend less time in the presence of parents than other adults (e.g., teachers; Hofferth and Sandberg 2001). The child's expanding ecology allows for the development of key relationships with others outside of the child's family, including peers, teachers, and other adults. These extra-familial relationships and experiences have powerful effects on children's social, emotional, and behavioral development. For example, positive relationships with teachers and peers in school have been shown to build children's self-esteem and social competence and to confer protection for children at risk for behavioral or learning issues (Pianta et al. 1995; Baker 2006). Conversely, the quality of school and afterschool environments, as well as having negative relationships with teachers and peers in these settings has been associated with long-term risk for problem behavior and school failure (Kellam et al. 1998; Ialongo et al. 2001).
Although all children spend time in school, afterschool, and community contexts, the structure and quality of activities and relationships children experience in these contexts may vary considerably (Fauth et al. 2007). Importantly, the quality and impact of features may vary across school or afterschool context, and for individual children in the same setting. In addition, some children may experience multiple adverse environments and relationships in multiple settings, whereas other children may be exposed to a combination of adverse and supportive experiences within or across settings. For example, children may be exposed to aggressive peers in an afterschool program, but have positive and healthy relationships with peers in school, the latter of which may mitigate the effects of exposure to aggressive peers in relation to the child's own aggressive behavior (Eccles 1999). It is critical to explore links between features in children's multiple contexts to fully capture patterns of risk and protective influences on children's developmental outcomes (Youngblade and Theokas 2006).
Which factors in these extra-familial contexts are most important to assess? A number of components of children's out-of-home experiences have been associated with child adjustment. Reviews of research literature examining links between features of out-of-home settings and developmental outcomes for children (Ingoldsby et al. 2012; Pierce et al. 2010) and adolescents (Mahoney et al. 2009) suggest that key components of settings may be organized along a few domains: relationships with others in the setting (e.g., positive adult-child and child-peer relationships), organizational features (e.g., structuring of activities for positive engagement, level of supervision), and broader contextual characteristics (e.g., positive emotional climate, indication of risk for or tolerance of child deviant behavior). For example, features contributing to positive school climate, including the presence of clear school-wide regulatory and discipline practices, and warm relationships between students and teachers, have been associated with lower rates of conduct problems (Leff et al. 2003; Nelson et al. 2002). Studies of afterschool programs have shown that positive staff-child relationships and availability of diverse, age-appropriate activities positively impact children's academic and social outcomes (Pierce et al. 2010).
The majority of studies of out-of-home contexts have examined one or only a few features within a specific setting (e.g., quality of the school or afterschool program). Some of these investigations have utilized observational systems to measure contextual features. This observational method results in a relatively objective assessment of behavior and interpersonal processes in a naturalistic setting. Observational ratings are often favored over questionnaire-based report measures because they protect against informant bias (e.g., parents with their own mental health concerns typically rate children's behavior more negatively than others; Ferguson et al. 1993) and can involve observing children in their own natural environments. However, observational systems that are typically used in school and afterschool contexts tend to focus on measuring individual children's behavior rather than a more comprehensive set of contextual characteristics, including attributes of the setting and interactions children have with adults and peers. For example, Leff and Lakin (2005) reviewed six well-established empirically-supported school-based observational measures of playground behavior, and found that coding for four of the six systems primarily resulted in scores of child and peer aggressive and prosocial behavior. The Interpersonal Process Code (IPC, Rusby et al. 1991) assesses child affect and activities in addition to child behavior. However, the IPC is complex and coding affect is noted to be difficult and to have variable reliability. The student interaction in specific settings (SISS, Cushing and Horner 2003) assesses school climate, but results in school-level scores and is noted to only assess negative behaviors. In addition, Leff and Larkin noted that few of the coding systems could be used in multiple types of out-of-home contexts, with only one system, the IPC, usable in multiple settings.
Observational methods also have been employed in afterschool settings, and these tend to be more comprehensive in their measurement of multiple features in the environment. For example, Vandell et al. (2005; Pierce et al. 2010) have conducted a set of studies of afterschool programs in which they explored features that are most reflective of high quality programming and most strongly associated with child outcomes, including positive relationships between children and caregivers and diversity of activities available in the setting in relation to social skills and achievement measures of child adjustment. Yet, in large part, these systems have not been designed to be used outside of afterschool settings. Designing an observational system that can evaluate the quality of structural characteristics in children's out-of-home settings and the quality of interactions children have with adults and peers across settings offers the potential for evaluating how specific contextual factors are similarly experienced by children in multiple settings. In addition, a cross-setting observation system could also be applied to examine whether there are distinct patterns in how experiences in particular settings are associated with different types of children's adjustment, including both prosocial and antisocial behavior with peers and adults (Pierce et al. 2010).
Review of existing measures indicated that a comprehensive observation system that could be employed in multiple types of children's contexts and across different ages was needed. To fill this need, the Aftercare and School Observation System (ASOS) was created. The ASOS assesses the frequency and quality of important components of contexts in which children spend their time that are hypothesized to impact children's development: relationships with caregivers and peers in the setting, caregiver involvement and monitoring of behavior, and quality of environmental features and experiences in the setting. In addition, the system measures various aspects of child behavior, such as deviancy, aggression, and prosociality. The objectives of the current paper are to report on the psychometric properties and component structure of the ASOS. The system was utilized to assess participant children in school lunchrooms and during school recess, and in a variety of afterschool environments (e.g., at home with caregivers, in afterschool programs, recreation programs, which we collectively refer to as “aftercare” settings). It was expected that exploratory factor analyses would support a core set of component constructs assessing qualities of children's relationships, caregiver involvement and monitoring, and experiences in the context that have been linked to children's behavior problems. Construct validity was assessed by testing for positive associations between ASOS constructs and a set of questionnaire measures assessing children's behavior problems and relationship qualities in school and aftercare settings (i.e., teacher and aftercare provider/parent reports).
Method
Participants
Participants included 468 children drawn from a larger longitudinal intervention study that involved home assessments from ages 2 to 7.5 (n = 731). Data for these analyses were collected as part of the age 7.5 assessment. Families were recruited for participation at Nutritional Supplement Centers for Women Infants and Children (WIC) programs in three distinct geographic communities: Pittsburgh, PA (urban), Eugene, OR (suburban), and primarily outside of Charlottesville, VA (rural) when their children were between 2.0 and 2.9 years old. The primary caregiver and his/her child were invited to participate in the Early Steps Multisite Study if they met criteria by having socioeconomic, family, and/or child risk factors for early conduct problems. For more information about recruitment procedures, see Dishion et al. (2008).
Children in the sample (51 % male) were predominantly European American (49.6 %), African American (27.1 %), biracial (14.5 %), and other (8.8 %). In terms of ethnicity, 13 % reported their children as being Hispanic. Two-thirds of the sample had an annual income less than $20,000. Most primary caregivers (97 % female) reported their highest level of education as high school or a high school equivalency degree (40 %) and 57 % of children lived in a two-parent household (i.e., married or living together). Of the primary caregivers who participated at the age 2 assessment, 97.1 % were mothers, 1.9 % fathers, and 1 % other caregivers.
Retention
Of the original 731 families who initially participated at age 2, 560 (77 %) were available at the follow-up at age 7.5. Selective attrition analyses from ages 2 to 7.5 revealed that families with significantly lower levels of parental education were more likely to drop out of the study at subsequent assessments. There were no selective attrition effects by project site, intervention status, children's race, ethnicity, or gender, maternal depression, or externalizing problem behavior (Chang et al. 2012). For age 7.5 school observations, 356 were available, primarily due to difficulties in obtaining cooperation at two of the largest school systems, which significantly reduced retention in school data in those sites. The completion of aftercare visits was slightly higher, with 404 observations completed. Of the 560 children retained at the age 7.5 home assessment, 64 % had school observational data and 55 % had teacher data. Seventy-two percent had aftercare observational data and 71 % had aftercare provider data.
Design and Procedure
During the home assessments, primary caregivers provided written consent for his/her child to participate in the school and aftercare components. The school and aftercare setting observations occurred a few months following the age 7.5 home assessment. The aftercare observation took place in the child's most structured or most frequently attended setting. For 43 % of families, the observed setting was at their own home with the primary caregiver. Although the system is designed to capture behavior in structured (e.g., classrooms) and less structured school settings, these observations occurred during the child's recess and lunch periods for two reasons: in order to ensure variation in adult monitoring and surveillance for validation purposes; and to capture child-peer interactions and behaviors (e.g., physical fighting) most closely linked to conduct problems. We also collected data from the child's primary teacher to assess the child's behavior at school, including conduct and emotional problems, social skills, quality of teacher-child and peer-child relationships, and information pertaining to experiences in, and the quality of, the broader school environment. The aftercare provider completed questionnaires about the child's behavior, quality of their relationship with the child, the child's relationship with peers, and level of monitoring occurring in the setting.
Measures
Demographics Questionnaire
When the children were 2 years old, demographic data were collected from the primary caregiver during a home assessment. This measure included questions about family structure, parental education and income, parental criminal history, and areas of familial stress and strengths.
Child Behavior Checklist/Teacher Report Form
Using a 3-point Likert scale (0 = “not true” to 2 = “very true or often true”), aftercare providers (which were at times the Primary Caregivers) completed the Child Behavior Checklist (CBC, Achenbach 1991a) and teachers completed the Teacher Report Form (TRF, Achenbach 1991b). Items from the following subscales were summed to create the Aftercare Provider-reported and Teacher-reported scores: Rule-breaking, Aggressive Behavior, Internalizing Behavior, and Externalizing Behavior.
Student–Teacher Relationship Scale/Adult-Child Relationship Scale
Teachers completed an abridged 15-item version of this questionnaire (Pianta 1994). Items are derived from attachment theory and have been widely-used to capture child-teacher relationship quality (Burchinal et al. 2002). Aftercare providers completed an adapted version, with the language of some items modified to reflect parent- or other adult-child relationships (Pianta and Steinberg 1991). Items were summed into two subscale scores for each reporter: adult-child conflict and openness.
School Quality Survey
Teachers completed this measure to assess the level of structure, organization, and social climate of the school environment. The measure includes items from two measures, one of which is the Playground and Lunchroom Climate Survey (Leff et al. 2003). One factor score consisting of 10 items rated on a four-point frequency scale, Structure for Activities, was utilized in these analyses. This score demonstrates good internal consistency (α = .84) and test–retest reliability. Sample items include “there are many games for children to play on playground,” “there are playground rules for children.”
Teacher-Peer Social Skills
This 14 item measure assesses the teacher's perceptions of the child's relationships with, and qualities of peers (Dishion and Kavanagh 2003). Three scale scores, derived from items rated on a 5-point frequency scale from “1 = very few (less than 25 % of peers)” to “5 = almost all (more than 75 % of peers)” were used in these analyses: Peers Dislike/Reject Child (1 item); Deviant Peers (4 items, example items include “What percent of child's peers…misbehave in school?” “…experiment with drugs/alcohol?”); and Prosocial Peers (2 items; “How often has this student associated with others who…take school seriously and complete their homework?” “…are involved in positive school or community activities?”). This measure demonstrated adequate psychometric properties in prior research (Dishion and Kavanagh 2003).
Aftercare and School Observation System (ASOS)
The newly created Aftercare and School Observation System (ASOS) draws from similar behavioral and observer impression systems (Vandell et al. 2005). The ASOS is a comprehensive observation system developed to assess key contextual characteristics and children's behavior, and was employed in lunchrooms, recess, and aftercare settings (e.g., daycares, sports practices). Trained observers rated 12 caregiver, child, and peer (if applicable) behaviors and qualities related to their interactions and activities in 30-s intervals in two 10-min periods. Observers then assessed the environment to make 20 global impression ratings assessing characteristics of the overall context. The ASOS was designed to be utilized in diverse social settings and includes ratings of the child's behavior (e.g., aggression, isolation), qualities of the caregiver-child relationship (e.g., caregiver warmth, responsivity, behavior management) and child-peer relationship (e.g., child-peer deviancy, peer exclusion), characteristics related to structure, monitoring, and caregiver involvement (e.g., caregiver surveillance/tracking; interactive involvement), and the child's activities.
ASOS Reliability
The strategy for obtaining initial reliability involved calibration with a master coder at an average of 70 % across all codes. Assignments consisted primarily of video tapes containing peer activities with a caregiver present (i.e., summer camp activities, lunch groups) or detailed written scenarios of peer, caregiver, and target child interactions. Reliability assignments continued in monthly intervals, reviewed by site lead coders. Lead coders met monthly via video-conference with the master coder. Coders completed a total number of reliability assignments that met or exceeded 10 % of the total sample. The percent agreement across 32 raters across three sites ranged from 83 to 98 % for interval codes and 62–100 % for global codes.
Results
The primary goal of the study was to explore the empirical factor structure of the ASOS to develop valid and reliable factor scores and test whether similar scores emerge in different settings. To address this goal, exploratory factor analyses (EFA) were conducted on the full sample of children observed in school lunchrooms and recess periods (n = 356) and in diverse aftercare settings (n = 404). All analyses were conducted using Statistical Package for the Social Sciences (IBM SPSS, version 18.0).
The first step of the analysis involved determining the best set of variables to submit to exploratory factor analysis. Ratings (0 = not observed, 1 = observed) were first summed across the 10 intervals in each 10-min observation and then averaged across the two observations for each of the 12 behavioral codes. Each of the 20 global ratings were also averaged across the two observation periods. Next, the distributional properties of the ASOS were examined. Some codes demonstrated significant skew patterns, low or zero variance, or very low base rates; these codes were eliminated from further analysis (e.g., child isolated from caregivers, peer excluded child, child prosocial behavior to caregivers). A few codes where interval and global impression ratings assessing similar behaviors were highly correlated (i.e., .60 or above) were combined [e.g., peer negative behavior; caregiver negative management (use of physical or verbal aggression with child)]. This step resulted in the identification of 18 codes with adequate distributional properties.
Next, separate EFA models were computed using data from each setting, one for school and one for aftercare, with the 18 codes submitted to the model. Initial models were computed with no constraints on the number of factors to be extracted. As recommended by Costello and Osborne (2005), models were computed with oblique rotations (direct oblimin) using maximum likelihood estimation with examination of scree plots and interpretability of pattern matrices to determine best fit. Subsequent models were computed by eliminating items that had low (under .25) communalities and/or loadings on factors and re-computing EFA with varying numbers of set factors (i.e., forced factor solutions).
In the school setting, a five factor solution emerged as the best fitting model, accounting for 48 % of the variance. However, two of the factors in this model only had two items each with loadings over .25. Thus, those two factors are not considered further in this report. Table 1 presents the final factor loadings for school observation data. In the aftercare setting, a four factor model that accounted for 57 % of the variance showed the best fit. Table 2 presents the factors and loadings for the final model for aftercare setting data. To examine whether a different factor structure emerged among children who were in home settings (i.e., with primary caregiver or siblings in own home, with other adult caretaker in child's or other home) versus more formal aftercare settings (i.e., sports practice, after school tutoring), additional exploratory models were computed on this subset (n = 309) of the aftercare data. As results were similar, all aftercare cases were included in the final aftercare setting EFA.
Table 1.
EFA factor loadings of ASOS items measured in school settings
| Scale and observation items (type of code) | Loadings |
|---|---|
| Negative/aggressive context: school | |
| Child physical or verbal aggression towards peers (C) | 0.84 |
| Peer negative acts or aggression (C) | 0.79 |
| Negative emotional climate (G) | 0.79 |
| Child angry affect towards peers (G) | 0.62 |
| Child verbal and physical aggression towards school caregivers (C)a | 0.49 |
| Child angry affect towards school caregivers (G) | 0.49 |
| Context conducive to deviant behavior (G)a | 0.33 |
| School caregiver positive involvement | |
| School caregiver active involvement (G) | 0.82 |
| School caregiver other talk (I) | 0.78 |
| School caregiver responsivity and sensitivity (G) | 0.68 |
| School caregiver warmth (G) | 0.66 |
| School caregiver positive behavior management (G) | 0.58 |
| Child positive affect to school caregiver (G) | 0.41 |
| School caregiver active surveillance (G)a | 0.39 |
| School caregiver applies appropriate structure | |
| Appropriate structure in context (G) | 0.70 |
| Context conducive to deviant behavior (G)a | −0.69 |
| School caregiver negative management (use of physical and verbal aggression) (C) | −0.25 |
I interval code, G global code, C combined interval and global code
Item loads on multiple factors
Table 2.
EFA factor loadings of ASOS items measured in aftercare settings
| Scale and observation items (type of code) | Loadings |
|---|---|
| Aftercare caregiver positive structuring of context | |
| Aftercare caregiver active involvement (G) | 0.84 |
| Aftercare caregiver responsivity and sensitivity (G) | 0.82 |
| Aftercare caregiver active surveillance/tracking (G) | 0.80 |
| Aftercare caregiver other talk (I) | 0.76 |
| Aftercare caregiver warmth (G)a | 0.44 |
| Aftercare caregiver positive behavior management (G) | 0.40 |
| Context conducive to deviant behavior (G)a | −.32 |
| Appropriate structure (G) | 0.30 |
| Child positive affect to aftercare caregiver (G)a | 0.28 |
| Positive emotional climate (G) | 0.27 |
| Child-aftercare caregiver negative interaction | |
| Child physical and verbal aggression towards aftercare caregiver (C)a | 0.83 |
| Aftercare caregiver negative management (use of physical and verbal aggression (C) | 0.76 |
| Child angry affect towards aftercare caregivers (G) | 0.57 |
| Negative emotional climate (G)a | 0.42 |
| Context conducive to deviant behavior (G)a | 0.25 |
| Positive emotional climate | |
| Positive emotional climate (G) | 0.73 |
| Child positive affect towards peers (G) | 0.65 |
| Child positive affect towards aftercare caregivers (G)a | 0.59 |
| Aftercare caregiver warmth (G)a | 0.38 |
| Negative-aggressive context: aftercare | |
| Child physical or verbal aggression towards aftercare caregivers (C)a | 0.89 |
| Peer negative act or aggression (C) | 0.89 |
| Child angry affect towards peers (G) | 0.75 |
| Negative emotional climate (G)a | 0.58 |
| Context conducive to deviant behavior (G)a | 0.39 |
I interval code, G global code, C combined interval and global code. For 43 % of families, the observed aftercare setting was at their own home with the primary caregiver
Item loads on multiple factors
Across both settings, two of the factors showed very similar empirical structures and item loadings, reflecting the constructs of a negative/aggressive context and care-giver positive involvement. Five of the 18 items loaded highly on the factors labeled “Negative/Aggressive Context,” with an additional two items also loading on that factor in the school setting. A set of seven common items loaded on factors assessing the extent of “Caregiver Positive Involvement”, although an additional three items loading on the aftercare setting factor suggested that the factor also captures the level of appropriate structure and monitoring provided by caregivers in that setting. Thus, this factor was titled “Aftercare Caregiver Positive Involvement and Structuring.” The third factor emerging from the school observational data contains three codes assessing the extent to which caregiver methods used in the setting result in less negative behavior (“School Caregiver Appropriate Structure”). In the aftercare setting, the third factor is comprised of codes measuring the positivity in the child's interactions and general environment (“Positive Emotional Climate”), and the fourth factor reflects negativity in the child's interactions and setting (“Child-Aftercare Caregiver Negative Interactions”).
Intercorrelations Among Factor Scores
Initial exploration of indicators of validity was conducted by examining intercorrelations among factor scores emerging within each setting (see Tables 3, 4). For this analysis, we expected that factor scores reflecting similar features (e.g., negative-aggressive context and child-care-giver negative interaction) would be modestly but not highly correlated, as we expected each factor score to account for unique information. In general, factor scores within settings were modestly to moderately intercorrelated, demonstrating expected patterns and suggesting that factor scores are capturing unique features in each setting. In the school setting, only one of the three intercorrelations was statistically significant (r = −.55, p < .01), with lower Negative-Aggressive Context scores associated with higher levels of appropriate structuring and less negative management by school caregivers. In the aftercare setting, five of the six intercorrelations were significant (rs ranging from −.20–.70, p < .05). Intercorrelations among the two positively and the two negatively-scaled scores tended to be higher than correlations between factors assessing positive versus negative behaviors.
Table 3.
Intercorrelations among ASOS school setting factor scores
| Measure | 1 | 2 | 3 |
|---|---|---|---|
| 1. Negative-aggressive context: school | – | .06 | −.55* |
| 2. School caregiver positive involvement | – | – | −.00 |
| 3. School caregiver applies appropriate structure | – | – | – |
p < .05
Table 4.
Intercorrelations among ASOS aftercare setting factor scores
| Measure | 1 | 2 | 3 | 4 |
|---|---|---|---|---|
| 1. Aftercare caregiver positive involvement and structuring | – | −.25* | .67* | −.34* |
| 2. Child-aftercare caregiver negative interaction | – | – | −.20* | −.70* |
| 3. Positive emotional climate | – | – | – | −.10+ |
| 4. Negative-aggressive context: aftercare | – | – | – | – |
p < .05;
p < .10
Correlations Between Factor Scores and Ratings of Child Behavior, Relationships, and Setting Features
To further explore construct validity, correlations between factor scores and ratings of child behavior, relationships, and setting characteristics were analyzed. Overall, the pattern of relationships was consistent with expectations. These correlations are displayed in Tables 5 (school) and 6 (aftercare). For example, in the school setting, factor scores that reflect higher child aggression (Negative-Aggressive Context) were modestly to moderately correlated with higher teacher ratings of Externalizing Behavior, Teacher-Child Conflict, and higher Peer Rejection and Deviant Peer teacher rating scores. In the aftercare setting, correlation patterns also suggested good construct validity. For example, the Child-Aftercare Caregiver Negative Interaction factor score was negatively associated with aftercare providers' ratings of Externalizing Behavior and Adult-Child Conflict.
Table 5.
Correlations among ASOS school scores and teacher ratings of child behavior and setting characteristics
| ASOS scores | Teacher report form |
Student–Teacher Relationship Scale |
School quality |
Teacher peer social skills |
||||
|---|---|---|---|---|---|---|---|---|
| Internalizing | Externalizing | Positive relationship with child | Conflictual relationship with child | Structure for activities and monitoring | Peers dislike/reject child | Deviant peers | Prosocial peers | |
| Negative-aggressive context: school | .00 | .32* | .00 | .27* | −.09 | .32* | .42* | −.20* |
| School caregiver positive involvement | .13* | .19* | .02 | .17* | .11+ | .14* | .15* | −.17* |
| School caregiver applies appropriate structure | .03 | −.17* | .01 | −.09 | .06 | −.14* | −.26* | .08 |
< .05;
< .10
Table 6.
Correlations among ASOS aftercare scores and aftercare provider ratings of child behavior
| ASOS scores | Child Behavior Checklist |
Adult-Child Relationship Scale |
||
|---|---|---|---|---|
| Internalizing | Externalizing | Positive relationship with child | Conflictual relationship with child | |
| Aftercare caregiver positive involvement and structuring | −.16* | −.15* | −.12* | −.08+ |
| Child-aftercare caregiver negative interaction | .02 | .21* | .03 | .15* |
| Positive emotional climate | −.20* | −.23* | −.12* | −.18* |
| Negative-aggressive context: aftercare | −.06 | .13* | .04 | .02 |
< .05;
< .10
In the school setting the direction of significant associations between Caregiver Positive Involvement factor scores and child outcomes was in the opposite direction from that found in the aftercare setting. In the school setting, Care-giver Positive Involvement scores demonstrated positive relationships with teacher ratings of conduct problems and teacher-child conflict, and ratings of fewer positive and prosocial peers. In the aftercare setting, Caregiver Positive Involvement scores were negatively associated with after-care provider ratings of child externalizing and internalizing problems. It is important to note that Caregiver Positive Involvement is coded for both “positive” behaviors such as engaging in a game with a child and more neutral (or less valent) behaviors, such as engaging in a conversation with a child about misbehavior. Additional analyses explored the hypothesis that the Caregiver Positive Involvement factor score may be sensitive to different frequencies of typical behavior and reasons for adult responses to child behavior in each setting (e.g., in school settings caregivers may be more frequently called upon to respond to negative behavior). Initial exploration of differences in the aftercare and school contexts revealed differences in the ratios of caregivers to children in these settings. Specifically, in the aftercare setting, there was a relatively high ratio of caregivers to children (.8) whereas in the school setting, the ratio was much smaller (.1). Further, there was no significant difference in caregiver involvement for children who scored one standard deviation above the mean on externalizing problems and those below this level in the aftercare setting. Whereas in the school setting, caregiver involvement was higher for children who scored one standard deviation above the mean on externalizing problems.
Discussion
This study addresses some key issues regarding existing behavioral observation systems raised by Leff and Lakin's (2005) review. Leff and Lakin concluded that it was important for the field to develop observational systems that: (a) have applicability across multiple contexts, both structured and unstructured; (b) include both positive and negative behaviors; (c) carefully consider the research and clinical applications of findings (e.g., manualized; well-defined codes and factors); and (d) have strong, established psychometric evidence supporting their use. Moreover, the ASOS also addresses a key concern that has been raised by Vandell et al. (2005) regarding afterschool coding systems; namely, that coding systems in these settings rarely account for the characteristics of the setting that may also have important contributions to child developmental outcomes, such as the extent to which activities in the setting are structured and support learning, and the positive and negative emotional climate of interactions in the setting.
The goals of the current study were to: describe the development, utility, and reliability of the ASOS; examine its underlying component factor structure; and explore the reliability and construct validity of the system. The ASOS was designed to assess the frequency and quality of key characteristics and experiences across children's diverse settings; in particular, to capture these characteristics from a dyadic perspective rather than a static count of individual behaviors. We hypothesized that reliable factors assessing qualities of children's relationships with caregivers and peers, caregiver involvement and monitoring, and features of the context that have been associated with children's behavior problems (e.g., support for deviant behavior) would consistently emerge in exploratory factor analysis of observational data collected in school and afterschool settings. The hypotheses were tested in a study of low-income children observed in school lunchrooms and recess settings and in aftercare. Results from exploratory factor analysis generally supported the hypotheses. In school and after-school contexts, three and four factors, respectively, were reliably established. Two factors had similar empirical structures and item loadings across both settings. These factors assessed negative and aggressive behavior in the setting, and levels of caregiver involvement and methods of encouraging positive child behavior. In general, the pattern of correlations among factor scores within settings also supported the hypothesized ASOS factors, as modest to moderate correlations were demonstrated in expected patterns (e.g., Negative-Aggressive Context score scores associated with lower levels of caregiver structuring and monitoring).
Further, the modest correlations between ASOS factors and aftercare provider and teacher ratings of behavior problems, adult-child relationships, and a rating of school climate also contributed to our interpretation that the ASOS scores capture meaningful features of children's experiences in these settings. Notably, teacher ratings of child and peer relationships and behaviors were moderately associated with the analogous scores from the ASOS, even though the ASOS was applied in unstructured school settings (lunchroom and recess) where teachers do not tend to be the adult caregivers. Aftercare provider ratings of the relationship quality with the child and their assessments of the child's behavior problems also tended to align with ASOS ratings in that setting, although these associations were low in magnitude. Interestingly, the direction of significant associations between Caregiver Positive Involvement factor scores and child outcomes was in the opposite direction in the school and aftercare settings. This suggests that the Caregiver Positive Involvement factor score may be tapping into different reasons for caregiver involvement across settings. In school settings, perhaps caregivers are likely to be highly and frequently involved in trying to shape children's negative behavior, utilizing positive behavior management techniques. Thus at school, care-giver involvement ratings may best capture the frequency and intensity of caregiver interactions with the child. In aftercare settings, this factor score may better capture the quality of the caregiver's interactions with the child and the positive impact of warm and sensitive care giving on children's behavior. As there are fewer caregivers to respond to children in the school setting, providing responsiveness to behavioral concerns may be a greater priority than proactively providing support to promote prosocial behavior. These secondary hypotheses were supported by the fact that Caregiver Positive Involvement scores in the school setting were significantly higher for children who scored one standard deviation above the mean on externalizing scores, compared to those with the lowest externalizing scores; the same was not found in the aftercare setting. Further exploration of the items and patterns of frequency and intensity are needed to tease apart these findings.
Limitations
This study has several methodological limitations. First, the behaviors and contextual experiences that the ASOS assesses occur relatively infrequently, resulting in low base rates that could affect item loadings and result in different findings emerging in different samples. This base rate of certain important behaviors (e.g., aggression) is an issue that plagues many behavioral coding systems, particularly for behaviors that require the coding of multiple actors and/or are subtle in nature (e.g., relational aggression, deviancy training chains; Leff and Lakin 2005). The original purpose of the ASOS was to design a coding system that could be utilized in multiple settings and would result in identical structures. Although identical factor structures did not emerge across settings, very similar patterns emerged in each setting. Despite this limitation, finding similar factor scores suggests that this system has practical value, in that one could train observers with one system that has application in multiple contexts, activities, and persons.
In addition, the ASOS factors were constructed based upon observations of children in first and second grade. It is unclear whether similar factor patterns would emerge for older or younger children, although the system was developed with the expectation that the behavior ratings would apply to a wide ranging developmental span. The current study is notable for recruiting children at increased risk for problem behavior and included a sample of ethnically diverse children from urban, suburban, and rural communities; it is unclear if the current results would be generalizable to lower-risk, predominantly middle- or upper-middle-class children.
Another limitation is that the available measures for establishing construct validity did not include clear, analogous assessments of contextual features of the environment. For example, the Negative-Aggressive Context factor scores were modestly associated with aftercare and school provider perceptions of child externalizing behavior, but this is not a direct measure of general negativity in the environment. It will be important in future applications of the ASOS to include other validation measures that comprehensively align with the established factor scores. Finally, it is important to note the concurrent nature of the study in validating the use of the ASOS. Ideally, ratings on the ASOS also would be found to predict patterns of child behavior across time using multiple informants and methods based on evaluations of child behavior in multiple contexts.
Future Directions
Findings from this study suggest some next steps. Within the next couple of years, ASOS data will have been collected on multiple waves of study participants, which will allow us to conduct confirmatory factor analysis to substantiate the factors that emerged from the present study. In addition, the ASOS was developed in large part to be able to examine known risk and protective factors associated with children's out-of-home contexts and their independent and interactive contributions to children's outcomes. This study represents the first step of establishing that the ASOS reliably and validly captures risk and protective relationships and experiences in extra-familial settings. It will be important to explore the independent contribution of out-of-home factors in conjunction with other key family and individual characteristics in predicting children's social, emotional, and behavioral outcomes. As noted earlier, some children may experience adverse or negative events or relationships in multiple settings, whereas other children may be exposed to a combination of risk and protective experiences within or across settings (Fauth et al. 2007). By understanding the relative relationships among these factors, it may be possible to identify ways in which children's experiences in specific arenas could be modified or enhanced to mitigate against negative impacts experienced in other contexts.
References
- Achenbach TM. Manual for the child behavior checklist/4–18 and 1991 profile. University of Vermont, Department of Psychiatry; Burlington, VT: 1991a. [Google Scholar]
- Achenbach T. Manual for the teacher's report form and 1991 profile. University of Vermont, Department of Psychiatry; Burlington: 1991b. [Google Scholar]
- Baker JA. Contributions of teacher-child relationships to positive school adjustment during elementary school. Journal of School Psychology. 2006;44:211–229. [Google Scholar]
- Burchinal MR, Peisner-Feinberg E, Pianta R, Howes C. Development of academic skills from preschool through second grade: Family and classroom predictors of developmental trajectories. Journal of School Psychology. 2002;40:415–436. [Google Scholar]
- Chang H, Shaw DS, Dishion TJ, Gardner F, Wilson MN. Direct and indirect effects of the Family Check-Up on self-regulation from toddlerhood to early school age. 2012. Manuscript submitted for publiation. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Costello AB, Osborne JW. Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation. 2005;10:1–9. [Google Scholar]
- Cushing LS, Horner R. Student interaction in specific settings (SISS) measure, coding procedures, and definitions. 2003. Unpublished coding manual. (Available from Peabody College at Vanderbilt University, Box 328, Nashville, TN 37203) [Google Scholar]
- Dishion TJ, Kavanagh K. Intervening in adolescent problem behavior: A family centered approach. Guilford; New York: 2003. [Google Scholar]
- Dishion TJ, Shaw D, Connell A, Gardner F, Weaver C, Wilson M. The family check-up with high-risk indigent families: Preventing problem behavior by increasing parents' positive behavior support in early childhood. Child Development. 2008;79(5):1395–1414. doi: 10.1111/j.1467-8624.2008.01195.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Eccles JS. The development of children ages 6 to 14. The Future of Children. 1999;9:30–44. [PubMed] [Google Scholar]
- Fauth RC, Roth JL, Brooks-Gunn J. Does the neighborhood context alter the link between youth's after-school time activities and developmental outcomes? A multilevel analysis. Developmental Psychology. 2007;43:760–777. doi: 10.1037/0012-1649.43.3.760. [DOI] [PubMed] [Google Scholar]
- Ferguson DM, Lynskey MR, Horwood LJ. The effect of maternal depression on ratings of child behavior. Journal of Abnormal Child Psychology. 1993;21(3):245–269. doi: 10.1007/BF00917534. [DOI] [PubMed] [Google Scholar]
- Hofferth SL, Sandberg JF. How American children use their time. Journal of Marriage and the Family. 2001;63:295–308. [Google Scholar]
- Ialongo N, Poduska J, Werthamer L, Kellam S. The distal impact of two first-grade preventive interventions on conduct problems and disorder in early adolescence. Journal of Emotional and Behavioral Disorders. 2001;9:146–160. [Google Scholar]
- Ingoldsby EM, Shelleby EC, Lane T, Shaw DS. Extrafamilial contexts and children's conduct problems. In: Malhomes V, King R, editors. The Oxford handbook of child development and poverty. Oxford University Press; New York: 2012. pp. 404–422. [Google Scholar]
- Kellam S, Ling X, Merisca R, Brown CH, Ialongo N. The effect of the level of aggression in the first grade classroom on the course and malleability of aggressive behavior into middle school. Development and Psychopathology. 1998;10:165–185. doi: 10.1017/s0954579498001564. [DOI] [PubMed] [Google Scholar]
- Leff SS, Lakin R. Playground-based observational systems: A review and implications for practitioners and researchers. School Psychology Review. 2005;34:475–489. [Google Scholar]
- Leff SS, Power TJ, Costigan TE, Manz PH. Assessing the climate of the playground and lunchroom: Implications for bullying prevention programming. School Psychology Review. 2003;32:418–430. [Google Scholar]
- Mahoney J, Vandell D, Simpkins S, Zarrett N. Adolescent out-of-school activities. In: Lerner RM, Steinberg L, editors. Handbook of adolescent psychology. 3rd ed. Vol. 2. Wiley; Hoboken, NJ: 2009. pp. 228–267. [Google Scholar]
- Nelson JR, Martella RM, Marchand-Martella N. Maximizing student learning: The effects of comprehensive school-based program for preventing problem behaviors. Journal of Emotional and Behavioral Disorders. 2002;10:136–148. [Google Scholar]
- Pianta RC. Patterns of relationships between children and kindergarten teachers. Journal of School Psychology. 1994;32:15–31. [Google Scholar]
- Pianta RC, Steinberg M. Relationships between children and kindergarten teachers: Associations with home and classroom behavior. Paper presentation at the biennial conference of the Society for Research in Child Development; Seattle, WA. Apr, 1991. [Google Scholar]
- Pianta RC, Steinberg MS, Rollins KB. The first two years of school: Teacher-child relationships and deflections in children's classroom adjustment. Development and Psychopathology. 1995;7:295–312. [Google Scholar]
- Pierce KM, Bolt DM, Vandell DL. Specific features of after-school program quality: Associations with children's functioning in middle childhood. American Journal of Community Psychology. 2010;45:381–393. doi: 10.1007/s10464-010-9304-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Rusby J, Estes A, Dishion TJ. Interpersonal process code. Oregon Social Learning Center; Eugene, OR: 1991. Unpublished technical manual. [Google Scholar]
- Vandell DL, Shumow L, Posner J. After-school programs for low-income children: Differences in program quality. In: Larson RW, Mahoney JL, editors. Organized activities as contexts of development: Extracurricular activities, after-school and community programs. Erlbaum; Mahwah, NJ: 2005. pp. 437–456. [Google Scholar]
- Youngblade LM, Theokas C. The multiple contexts of youth development: Implications for theory, research, and practice. Applied Developmental Science. 2006;10:58–60. [Google Scholar]