A Small-Scale Randomized Efficacy Trial of Carescapes: Enhancing Children’s Social Development in Child Care Homes

Abstract

The quality of the child care environment and caregiver practices can potentially have significant, lasting impact on children’s social development. This study involves the development and a small-scale efficacy trial of the Carescapes program, a video-based training program that focuses on promoting positive social development in young children attending family child care. Fifty-seven caregivers who provided child care in their homes were randomly assigned to immediate intervention or waitlist control groups. Random coefficients analyses showed significant increased use of effective behavior management practices and decreased overall children’s problem behavior for the intervention group. A mediation model demonstrated that increases in effective behavior management practices were associated with decreases in problem behavior. A medium intervention effect was found for caregiver’s monitoring and a small effect for use of positive attention. These effects declined 5 months following the intervention. Implications for future efficacy and effectiveness studies in family child care settings that involve strategies to facilitate maintenance are discussed.

Children experience the greatest rate of development during the first 5 years of their lives, and besides the family environment, child care is the setting in which this growth predominately occurs (Shonkoff & Phillips, 2000). Child care is typically the first experience young children will have with a group of other young children, and thus plausibly can facilitate positive interactions and the formation of positive relationships among children, as well as the development of socially skilled behavior in young children. Evidence supports positive influence of child care quality on social outcomes (Loeb, Fuller, Kagan, & Carrol, 2004; NICHD Early Child Care Research Network, 2001; Peisner-Feinberg & Burchinal, 1997; Phillips, McCartney, & Scarr, 1987). Yet the NICHD large-scale study of Early Child Care and Youth Development brings into question whether child care quality has a lasting positive influence on children’s social development (Belsky et al., 2007; NICHD Early Child Care Research Network, 2003, 2004). Differential findings likely depend on the child’s age when the care is provided, the way in which child care quality is defined, and what social outcomes are assessed.

Much less is known about the extent to which the quality of care in family child care settings influences children’s social development. In family child care settings, the care of children is provided in the caregiver’s home. It is difficult to articulate empirical evidence that is solely pertinent to family child care as many studies have combined data from family and center-based care, or equivocal results have been found in the studies of family child care, likely due to low sample sizes or poor research designs (Kontos, 1992). A number of more recent studies have demonstrated that training for caregivers who provide child care in their homes was related to greater quality of care giving practices. Care providers in family child care settings with higher levels of training have been found to provide higher overall quality of child care (Norris, 2001; Weaver, 2002). Both level of education and training were shown to predict quality of caregiving in child care homes (Burchinal, Howes, & Kontos, 2002) and these predictions were stronger than those of adult-child ratio and caregiver experience.

Specific caregiver practices that have been associated with higher levels of training are more frequent engagement with children (Bordin, Machida, & Varnell, 2000), less detachment from children (Burchinal et al., 2002), and a greater likelihood for planning of child activities (Kontos, Howes, & Galinski, 1996). Although these caregiver practices have been shown to promote positive social development in children in center-based child care (e.g., Dunn, 1993; McCartney, Scarr, Phillips, & Grajek, 1985; Peisner-Feinberg et al., 2001), the studies in family child care settings did not directly investigate the impact of these caregiver practices on children’s behavior. A meta-analysis of caregiver training studies confirms positive impact of caregiver training on caregiver practices in regular child care and preschools, with fewer studies and weaker effects found for children’s skills and behavior (Fukkink & Lont, 2007). These results were found to generalize to child care providers with varying levels of education.

The Importance of Facilitating Positive Social Development in Young Children

A number of social skills in young children have been identified as important to overall social competence and peer acceptance. These include effective initiation of interactive social play, effective social responses, and cooperative interactions such as offering help and sharing (Odom & Ogawa, 1992). Children who lack such social skills are likely to experience a range of social problems and difficulties, such as peer rejection and aggression from playmates (Olson, 1992). Children lacking in social competencies are also at risk for developing internalizing problems, such as social withdrawal and anxiety (Olson & Rosenblum, 1998).

Social problems first identified during the preschool years tend to be stable (Richman, Stevenson, & Graham, 1982), and are likely to increase in severity over time (Walker, 1995). Children who enter school with social behavior problems, such as oppositional behavior and aggression, are at elevated risk for continued social difficulties throughout elementary school (Campbell & Ewing, 1990; Mesman, Bongers, & Koot, 2001) and for exhibiting antisocial behavior during adolescence (Loeber, 1990; Moffitt, 1993; Patterson, Reid, & Dishion, 1992). Attending to ways in which caregivers of young children in community settings can facilitate social competencies and prevent the development and escalation of such problems is essential.

Although a few studies have investigated longitudinal effects of early childhood intervention programs on social adjustment problems in youth, these studies have mainly been conducted in more formal preschool and center-based child care settings, such as the Perry Preschool Program (Schweinhart, Berrueta-Clement, Barnett, Epstein, & Weikart, 1985) and the Abecedarian Project (Campbell, Ramey, Pungello, Sparling, & Miller-Johnson, 2002). To develop a broader reach of these effective practices, it is important to intervene in additional forms of community settings in which young children spend a substantial amount of time.

Variable Quality of Child Care in Home Settings

About 3.7 million toddlers and preschool-age children in the United States spend an average of 26 hours per week in family child care settings (Mulligan, Brimhall, West, & Chapman, 2005). Compared to center-based care, the quality of care in family child care varies greatly (Goelman & Pence, 1990). Caregivers in family child care are likely to have lower levels of education and training compared to those providing center-based care (Howes & Hamilton, 1993) and the environments are less likely to be safe and designed for age-appropriate activities (Howes, 1983). Children from low-income families are more likely to attend family child care than center-based child care (Frankel, 1994; Hofferth, Shauman, Henke, & West, 1998), which may be due to availability and affordability. Unfortunately, it is the low-income children who are likely to attend the family child care settings with the lowest quality ratings (Goelman & Pence, 1990). Thus, the young children who are at risk of developing social and behavioral problems (Brooks-Gunn, Duncan, & Aber, 1997; Dodge, Pettit, & Bates, 1994) are the ones who are more likely to spend a substantial amount of time receiving low quality child care. Improvements in the quality of family child care, particularly in the ways in which social development is nurtured and child behaviors are managed, would likely have long-term positive influence on many at-risk children.

Purpose of This Study

The purpose of this study was to develop and evaluate the efficacy of Carescapes, a video-based training program to give family child care providers strategies in facilitating children’s social development, proactive approaches for managing children’s behavior, and ways to promote positive behavior in children who exhibit frequent challenging behaviors. Video-based instruction is particularly useful in disseminating effective behavioral intervention strategies in community settings because of its efficiency and potential for wide reach. Although empirically validated programs involving teacher training, parent training, and children’s social skills training exist, few applied settings have implemented them (Carnine, 1997; Kazdin, 1991). A key reason appears to be that there are practical obstacles for implementing programs that require substantial resources. Video-based programs can offer a more efficient and affordable format for providing quality training to child care providers.

In a prior study, an instructional video-based training module on ways to set up the child care home to promote social development was developed and evaluated (Rusby, Taylor, & Marquez, 2004). This training module was delivered to caregivers in an interactive 3-hour workshop session. The content included an overview of social development, importance of social skills, and typical social skills for preschool-age children, and also demonstrated ways to promote social development by a) arrangement of physical space, b) selection of materials and toys, and c) setting schedules and routines. Overall, caregivers reported that they learned new practical information, and promising trends in improvements of the child care environment and positive behavior in children were found.

In the present study, two more training modules were developed for the Carescapes program, one on proactive approaches for managing children’s behavior and the other on understanding and dealing with problem behavior. Each of the three training modules were provided to caregivers in interactive 3-hour workshop sessions. This small-scale efficacy trial is the first evaluation of all three modules using a randomized waitlist control design. Most interventions are first evaluated in efficacy studies carried out by the developers (Flay et al., 2005). In efficacy studies, interventions are carried out in optimal conditions with highly trained interventionists and ample resources. Subsequent, experimental effectiveness studies in which interventions are carried out in community settings (e.g., schools, clinics) under natural conditions of that setting are needed to provide evidence for empirical support (Chambless & Ollendick, 2001; Flay et al., 2005). This efficacy trial is the first step in the evaluation process.

The Carescapes program was primarily expected to improve the use of proactive behavior management practices, such as monitoring and providing clear expectations and positive attention. Increases in these practices were expected to reduce children’s problem behavior in the child care homes. Secondary hypotheses were that caregivers in the intervention group would make improvements in the child care environment and that children attending child care homes in the intervention group would exhibit increased social skills.

Method

Participants

Participants were drawn from the Oregon Child Care Resource and Referral Network agencies in two counties (total population approximately 640,000). Certified and registered family child care homes were considered eligible for participation. At the time of the study, in Oregon, caregivers with child care homes licensed as a registered child care home could have 3–10 children, and were required to have a) training in infant and child CPR and first aid, b) a food handlers permit, c) training on child abuse and neglect, d) an on-site health and safety inspection, and e) 8 hours of additional child care related training every 2 years. A certified child care home could have up to 12 children (or up to 16 with prior approval) and the facility was primarily educational for children 36 months old and older. In addition to the requirements for registered child care, the provider and assistants must have 15 hours of training related to child care every year. To be eligible for the study, at least two children who were 2½ to 6 years old needed to attend the child care home. Also, caregivers had to speak and understand English and could not have participated in a previous study that involved a portion of the intervention. There were 544 eligible care providers.

To increase minority participation, random stratified sampling of participants from each ethnic and race category was originally planned in the county that had this information in their database. The goal was to invite half the sample from minority populations, but since there were fewer minority caregivers than anticipated, all eligible minority caregivers from the county were invited to participate. In the other county, ethnicity and race information of the neighborhood schools was collected (Oregon Department of Education, http://www.ode.state.or.us/sfda/reportstudents.htm) and two thirds of the child care homes invited to participate in this study were randomly selected from 7 of the 36 neighborhoods with the greatest ethnic diversity (1 with the greatest Hispanic population and 2 neighborhoods each representing the greatest percentage of each racial category—African-American, Asian/Pacific Islander, and American Indian/Alaskan Native). One third of child care homes were randomly selected from the other neighborhoods in the county.

Caregivers were found inactive if they had moved out of the area, the phone had been disconnected without new contact information, or they had shut down or planned to shut down their child care soon. During the recruitment phase, 60 caregivers were found “inactive” and 23 were ineligible for participation. Out of 188 active and eligible child care homes selected, 22 (12%) did not respond after initial contact, 96 (51%) declined participation (primarily because they were too busy), and 70 (37%) agreed to participate. Based on U.S. Census data, the median household income of the neighborhoods in which the child care homes were located did not substantially differ between care providers who consented and those who did not consent to participate (the median household income was $39,937 for consenters’ neighborhoods and $39,140 for nonconsenters). Thus, the participating sample does not appear biased regarding the economic status of the neighborhoods in which they were located. By the end of the 2-year study, 57 caregivers remained in the study (81% retention).

Participants were offered $20 for filling out baseline questionnaires, $25 for filling out postintervention questionnaires, and $25 for each workshop attended. Participating caregivers were also provided with three credit hours per workshop toward their child care professional development. All the participants were female, 73% were Caucasian, 10% were Hispanic/Latino, 4% Asian or Hawaiian/Pacific Islander, 3% American Indian, 1% African-American, and 9% were of other or unknown ethnicity/race. Other demographic information on participating caregivers collected at baseline (such as caregiver age, education, years of experience, percent of homes with one or more than one caregiver, and number of children) is presented in Table 1. There were no significant differences between the intervention and waitlist control groups on these variables.

Table 1.

Participating Caregivers and Child Care Home Demographics

	Total Sample	Intervention	Waitlist Control
Sample Size	63	33	30
Caregiver Age
1. 18–21 years old	0%	0%	0%
2. 22–29	17%	18%	13%
3. 30–39	32%	30%	36%
4. 40–49	26%	27%	26%
5. 50 and older	25%	24%	26%
Mean	3.59	3.55	3.63
Caregiver Education
1. High school or GED	29%	33%	22%
2. Some college	43%	36%	52%
3. AA degree	6%	3%	10%
4. BA/BS	14%	18%	10%
5. Some graduate school	8%	9%	6%
Mean	2.26	2.35	2.15
Total number of adults providing care
1. One	73%	74%	70%
2. More than one	27%	26%	30%
Mean	1.28	1.26	1.30
Mean years of experience	9.3	9.5	8.9
Mean total number of children	8.7	8.7	8.7

Note. All mean differences between intervention and waitlist control groups were not statistically significant.

Individual children were not participants in this study. Parents were informed about the evaluation study and the site visit assessments of the child care home, ways in which the child care provider promotes social development, and overall, what children do.

Research Design and Procedures

Baseline questionnaires completed by the primary caregiver were collected prior to randomization of child care providers into intervention and waitlist control conditions. In cases where there were more than one caregiver, the caregiver who provided the most care to children was considered the “primary” caregiver who participated in the study. All caregivers at each child care home (not just the “primary”) were invited to the intervention or waitlist workshops. Approximately 1 month after the intervention group participated in the last of three workshops, all participating caregivers completed the questionnaire again (post).

Assessments also involved three site visits per phase in which research assistants conducted observations and completed ratings of caregiver practices and overall child behavior. On average the three assessment times within each phase (pre and post) were about 9 weeks apart (Figure 1 shows the design and assessment timeline). The collection of postdata began 4 weeks after the intervention group completed the third workshop, so the last assessment data point (T6) occurred an average of 22 weeks (or about 5 months) post intervention. During the first presite visit (T1) and the first postsite visit (T4) assessors completed a rating on the child care environment of each child care home. Random reliability checks were assigned on 18% of the site visits, in which two research assistants visited the site to collect observation data. By baseline data collection, 65 caregivers continued participation in the study and 100% completed the baseline questionnaires. The first site visit assessments were completed on 98% of participating child care homes in baseline, the second on 94%, and the third on 92%. By the end of the baseline observation phase, 63 caregivers were participating (two caregivers dropped out of the study because they were closing down their child care business).

Figure 1.

Design and assessment timing. Caregiver questionnaire and Childcare Ecology Checklist was collected at T1 and T4.

After the preintervention assessment data were collected, family child care sites were randomly assigned to participate in the intervention workshops immediately or after completion of the posttest assessments (33 child care homes were assigned to immediate intervention and 30 to waitlist control condition). Random assignment to condition controls for causally relevant variables (Rubin, 1974) and reduces the likelihood that something other than the intervention would account for observed differences between conditions at the conclusion of the study. The intervention workshops were about 2 weeks apart. After each workshop, participating caregivers filled out a feedback form on the usefulness of the workshop and barriers to implementation of the practices learned. The beginning of the postdata collection began about 1 month after the third intervention workshop.

Intervention

The development of the Carescapes intervention content was drawn from research from the early intervention field and included studies in preschools and child care. The research base for the content of each intervention module is described below.

Module 1: Setting up to support social development

This training module demonstrated how to enrich the child care experience by organizing the child care space into activity areas, providing a variety of toys and materials, and offering a variety of activities that promote positive social interactions and socially skilled behavior. This training module also showed how to effectively lead children through transitions between activities. Having an enriched and organized child care environment is critical for promoting social engagement and preventing problem behavior (Hilton, 1987; McEvoy, Fox, & Rosenberg, 1991; Nordquist & Twardosz, 1990; Twardosz & Risley, 1982) and helps prepare children for school (Love et al., 2003). When caregivers plan and organize experiences and routines that are manageable and predictable for young children, the emotional demands of the daily experiences are reduced, enabling the child to focus on learning new skills (National Research Council & Institute of Medicine, 2002), including important social skills. Greater details about the content and research base for this module is reported in Rusby et al. (2004).

Module 2: Proactive approaches for managing children’s behavior

The content of the second training module focused on proactive behavior management practices. This module included (a) building a positive relationship with children, (b) developing and teaching clear rules and expectations, (c) getting and maintaining children’s attention, (d) providing clear directions before an activity begins, (e) monitoring, and (f) providing positive attention and encouragement. Although much of the research demonstrating positive effects of these behavior management practices has occurred in school settings (e.g., Hawkins, von Cleve, & Catalano, 1991; Nafpaktitis, Mayer, & Butterworth, 1985; Walker, Schwarz, Nippold, & Irvin, 1994), the research in child care and preschool settings has consistently matched the findings (e.g., Dunn, 1993; Howes, Phillips, & Whitebook, 1992; Webster-Stratton, Reid, & Hammond, 2001).

Module 3: Understanding and dealing with problem behavior

The content of the third training module focused on ways to deal with ongoing behavioral challenges. The module described and provided examples of (a) the coercive process, (b) effective strategies for dealing with challenging behavior, (c) understanding the function of recurring problem behavior, and (d) building a support plan for individual children who display ongoing behavioral challenges. It is expected that even with the environmental and proactive practices in place, a small proportion of children will continue to exhibit challenging behavior. Research in school settings indicates that students who exhibit elevated rates of problem behavior need a more individualized approach for intervening (Sprague, Sugai, & Walker, 1998). This module demonstrates appropriate consequences (promoting use of the least intrusive strategy) for varying challenging behaviors. It also offers a Behavior Support Guide, which helps child care providers understand the function of a child’s challenging behaviors and guides them through the development of a plan for teaching and supporting positive replacement behaviors. This module is based on functional behavior assessment and support—an individualized approach for clearly defining the problem behaviors, defining when and in what situation the problem behaviors tend to occur, and analyzing the contextual factors that maintain those behaviors (O’Neill et al., 1997). Experimental evaluations have shown that behavior support plans based on functional behavior assessment in school settings result in decreased aggressive and disruptive behaviors among a variety of children with behavior problems (Carr & Durand, 1985; Lalli, Browder, Mace, & Brown, 1993; Taylor & Romanczyk, 1994). Although developed in work with children with developmental disabilities, the approach has been shown to be effective with students with average to above-average intelligence with behavioral problems (Broussard & Northrup, 1995; Dunlap et al., 1993; Kern, Childs, Dunlap, Clarke, & Falk, 1994; Lewis & Sugai, 1996), with socially withdrawn behaviors (Lewis & Sugai, 1993), and with attention deficit hyperactivity disorder (Northrup et al., 1995; Umbreit, 1995).

Intervention development

Evidence supports that video-based modeling approaches can be both effective and efficient for training parents, teachers, and child care providers (Aguirre & Marshall, 1988; Taylor, Schmidt, Pepler, & Hodgins, 1998; Webster-Stratton, Hollinsworth, & Kolpacoff, 1989; Webster-Stratton et al., 2001). The video presentation was designed to feature “real-life” demonstrations of examples in family child care settings. Documentary footage was gathered from numerous family child care sites to provide diversity, and scripted vignettes were used whenever specific examples or negative examples were needed. The video and manual were designed for interactive use, providing a number of breaks for interactive opportunities (a viewer pauses the tape to complete an activity described in the manual). Feedback was provided on the drafts of the scripts and practice activities for each intervention module in separate focus groups of seven to eight family child care providers (59% Caucasian, 14% Hispanic/Latino, 18% Asian or Hawaiian/Pacific Islander, and 9% African-American). The feedback from focus groups was incorporated and then materials were reviewed by expert consultants. Four of the consultants had Ph.D.’s and were faculty or researchers with expertise in early intervention, child care environments, proactive management practices, functional behavior assessment and support, family interventions, and/or video-based training programs. One consultant was the co-director of the local Child Care Resource and Referral Network and had experience in training and resource development for child care providers. Based on consultant feedback the video script, manual, handouts, and child activities were finalized and the script provided a blueprint for the production of the video.

Intervention procedures and participation

Caregivers in the immediate intervention group were invited to participate in the set of three 3-hour workshops that were scheduled about 2 weeks apart. Caregivers were given a choice of whether they would like to attend the workshops on a weekday evening or a Saturday morning. Workshops were conducted in the caregivers’ local county Child Care Resource Referral agency or research institute meeting rooms and were provided by the program developer, the research scientist who was the Principal Investigator of this study. Caregivers who missed a given workshop were invited to attend the workshop on the alternative day in their community or to view the video and do the practice activities in the manual at home before the next workshop session. For caregivers who did a module through home study, the interventionist checked their activity work in their manual, answered questions, and checked for understanding of the content at the next workshop. Good participation rates were achieved—90% of the caregivers were exposed to module 1 (71% attended the workshop and 19% completed the home study), 90% received module 2 (81% attended the workshop and 9% did home study), and 87% received module 3 (80% attended the workshop and 7% did home study). A total of 97% of the caregivers in the intervention group received some exposure to the intervention and 90% received at least two modules of content.

Measures

Many studies on evaluating the effects of child care interventions have utilized global outcome measures of child care quality. Reviews of existing research suggest that several important dimensions of child care quality are not being adequately measured (Friedman & Amadeo, 1999; Lamb, 2000). Studies using more global measures of child care quality have had mixed results in predicting children’s developmental outcomes, particularly in the social domain. Indeed, some researchers hypothesize that divergent research results regarding the impact of child care quality are due, at least in part, to differences and inadequacies in measurement of child care quality (Hagekull & Bohlin, 1995; Lamb, 2006). We have therefore avoided using global measures of child care quality for this study, and have selected multiple assessment methodologies that specifically measure empirically supported child care environmental features and caregiver practices and that measure specific targets of the intervention content.

Measures for this study include caregiver questionnaires, ratings of the child care environment, and observations of caregiver practices and the overall behavior of preschool age children in the child care. Table 2 displays the items and reliability alphas for each composite score of caregiver reports and of observer ratings used in the present study.

Table 2.

Items and Reliability Alphas for Each Composite Measure

	Cronbach’s Alpha
Measure and items	Baseline	Post
Caregiver Report of Proactive Behavior Management	.79	.65
Taught or reminded children of rules
Prepared children for transitions
Taught social skills to children
Praised children for good behavior
Gave clear positive directions
CCEC: Enriched Environment	.91	.89
Activity areas are designed to promote social interaction
Toys and materials match children’s developmental level
Materials encourage children to engage in a variety of play
Variety of materials encourage cooperative and social play
Adequate amount of materials provided
Children are actively engaged with materials in all play areas
CCEC: Organized Environment	.88	.90
Physical space promotes intended activities
There are clearly defined activity areas
Compatible activity areas are next to each other
Materials are organized in a way so children can put them away
CCEC: Planned Activities and Schedule	.85	.86
Activities are planned ahead of time
Materials are prepared ahead of time
Children are playing throughout the child care home (reversed)
There is a sequence of activities, a schedule is followed
There is a balance of structured and unstructured activities
There is a balance of active and calm activities
CCEC: Monitoring	.82	.84
Activity areas defined by low boundaries, children are visible
Caregiver circulates through child care and/or area
Caregiver scans entire play area at least once per minute
Caregiver maintains proximity to children needing assistance
Observer Rating of Caregivers’ Effective Behavior Management	.85	.86
Supportive of children’s efforts
Encourages active participation
Intrusiveness (reversed)
Clear instructions or directions
Prepared children for transitions
Made unreasonable requests (reversed)
Monitored children
Gave reminders of expected behavior
Had good control or influence on children
Modeled positive behaviors
Problem solved with children
Was positive and reinforcing toward children
Consistent rules and consequences
Had little or no influence over children
Observer Ratings of Children’s Positive Behavior	.95	.85
Level of children’s compliance
Handled transitions well
Followed rules and expectations
Level of children’s cooperation
Engaged in problem solving
Expressed feelings appropriately
Level of interest, enthusiasm, and involvement
Level of focus and engagement

Child Care Provider Survey

Caregivers filled out a questionnaire at baseline and postintervention on the strategies they have used to promote child development and to manage children’s behavior in their child care and the usefulness of those strategies, including the enrichment of the environment, planned activities, and their behavior management practices. A composite score of the caregivers’ self-report of proactive behavior management practices was computed by taking an average of items (see Table 2 for items and reliability alpha). This survey also included items on the caregiver’s perceptions of the density of problem behavior in the child care setting. Demographic information was also included in the survey, such as caregiver experience and education, child–adult ratio, and age range of children. This survey was developed and used in the preliminary study (phase 1) on family child care providers (Rusby et al., 2004) and includes salient demographic and provider strategies presented in the literature cited above.

Direct observations of caregiver practices and child problem behaviors

Direct observations of the child care providers’ behavior management practices and of overall child problem behavior occurred during three separate visits at baseline and post intervention. The observations of child behavior involved tallying all behaviors exhibited during the observation period by any of the children in the child care home. On each visit, caregiver and children’s behavior were observed for 20 minutes during free play, when the children tend to be more active. Observers were masked regarding the intervention condition of the child care site. The Assessing School Settings: Interactions of Students and Teachers (ASSIST; Rusby, Taylor, & Milchak, 2001) observation frequency coding system was adapted for use in child care settings. For this study, child care providers’ frequency of approvals (positive attention for children’s behavior) and criticisms per minute were calculated as well as a ratio of approvals to criticisms for each measurement time point. Interrater reliability was computed using a one-way random effects intraclass correlation coefficient (ICC; McGraw & Wong, 1996; Shrout & Fleiss, 1979). Good interrater reliability was achieved—the ICC for approval was .94 at baseline and .95 at postdata collection and for criticism was .86 at baseline and .91 at post. The approval-to-criticism ratio was significantly associated with observed fewer problem behaviors and more positive behaviors at baseline (see Table 3).

Table 3.

Correlations Between Child Care Constructs and Children’s Behavior at Baseline

	Problem Behavior2	Positive Behavior3
Enriched Environment1	−.14	.48***
Organized Environment1	−.12	.40***
Planfulness1	−.22	.49***
Monitoring1	.00	.27*
Approval:Criticism2	−.28*	.46***
Effective Behavior Management3	−.41***	.82***

¹Child Care Ecology Checklist.

²Observation.

³Obs Rating.

^*p < .05

^***p < .001.

During the three visits to each of the child care sites at baseline and at post test, project staff also observed the overall problem behavior of preschool-aged children at the child care. Observers tallied the rates of noncompliance, physical aggression, and potentially dangerous behavior that children exhibited during a 20-minute observation. Noncompliance was coded whenever a child did not follow a caregiver’s directive for a behavior change (to either start doing something or to stop doing something; for example, if the caregiver says it is time to put away the toys and a child continues to play with blocks). Physical aggression was coded for whenever a child engaged in aversive physical contact with another person; for example, when a child hit, pushed, or grabbed a toy away from another child. Potentially dangerous behavior was coded for behavior that was potentially dangerous to self or another child and for misuse of equipment. Examples include throwing a toy against the wall, running down the slide, rough and tumble play such as play wrestling, misuse of playground equipment. A composite score for problem behavior rates per minute was created by summing the counts of each of these observed behaviors and then dividing by the total minutes observed. The ICC reliability for the problem behavior composite was .90 at baseline and .94 at post. Finally, although the number of children in the child care home might influence the rate of problem behavior, a preliminary analysis found this not to be the case. The number of children in child care and the rates of problem behavior were not significantly associated. Therefore, it was not necessary to account for the number of children when calculating the rate of problem behavior score.

Observer ratings of effective behavior management and children’s positive behavior

During the three visits to each of the child care sites at baseline and at post test, project staff also completed an observer rating of their global impressions of caregiver practices for facilitating social development and managing behavior and of the children’s positive behavior. The observer ratings of caregivers behavior management practices were rated on a 5-point scale: 1 = “did not occur,” 2 = “occurred once,” 3 = “occurred 2 to 3 times,” 4 = “occurred multiple times (> 3),” and 5 = “constantly occurred.” Items were adapted from the coder impression inventory on preschool teacher behavior management skills (Webster-Stratton et al., 2001) which were originally derived from the Oregon Youth Study observer impressions of parent monitoring and discipline (Capaldi & Patterson, 1989). See Table 2 for items and reliability alpha for the present study. The items for the measures of overall child positive behavior were derived from the Classroom Atmosphere Rating Scale (Greenberg & Wehby, 1995) and were previously adapted for use in family child care settings (Rusby et al., 2004). These items were rated by the observer on a 5-point Likert scale, from 1 “very low” to 5 “very high”; see Table 2 for items and reliability. Significant associations were found between effective behavior management practices and fewer observed problem behaviors and more positive behaviors at baseline (see Table 3).

Child Care Ecology Checklist

Project staff completed the Child Care Ecology Checklist (CCEC) at each site, once before and once after intervention. Assessment staff observed the child care and provided ratings on the (a) physical space, (b) materials, (c) scheduling, (d) planning, (e) teaching routines and expectations, (f) monitoring, and (g) managing attention, each on a 4-point scale. The CCEC was adapted from the Early Childhood Classroom Ecology Checklist (Project STAR, 2002; Kaminski & Stormshak, 2006). Items included on this measure are based on the literature on effective practices in preschool and child care settings, and thus align with the practices promoted in the intervention described previously. This measure has been a useful consultation tool in Head Start classrooms (Kaminski & Stormshak, 2006; Yoshikawa & Knitzer, 1997). Good item correlations and adequate interrater reliability were found for a priori composites from this measure: organized environment, enriched environment, and planned activities; and the measure of toy variety, a subset of enriched environment, was sensitive to intervention (Rusby et al., 2004). Items and reliability alpha for the organized environment, enriched environment, planned activities/schedule, and monitoring composites used in the present study are in Table 2. These child care setting composite scores were positively associated with observer ratings of children’s positive behavior at baseline (see Table 3).

Caregiver Interaction Scale (CIS; Arnett, 1989)

During the three child care site visits at baseline and at post test, research staff also completed the Caregiver Interaction Scale. The CIS measures caregiver sensitivity and involvement. Arnett (1989) described four factors: positive interaction, punitiveness, permissiveness, and detachment; items on each factor had a minimum loading of .49. Other researchers have combined some of these factors into a caregiver sensitivity score (de Kruif, McWilliam, Ridley, & Wakely, 2000). The CIS predicted children’s secure attachment to their child care provider (Galinsky, Howes, Kontos, & Shimm, 1994). For this sample of family child care providers at baseline, alphas for the factors of the CIS ranged from .75 to .89.

Caregiver ratings of program usefulness

Following each workshop session, caregivers provided a report of their satisfaction with the workshop and materials and also were asked to identify any barriers they expected to find in trying to implement changes in the family child care environment and in their behavior management practices to facilitate social development. A fourth satisfaction questionnaire was mailed to caregivers in the intervention group about a month following the set of workshops to assess their continued satisfaction and whether they actually made changes in their child care based on the training. These caregiver satisfaction reports have been used to provide a summary of whether family child care providers learned new information, gained practical skills, and planned to make changes in their child care based on a workshop on promoting social development (Rusby et al., 2004).

Knowledge test on the Behavior Support Guide

This knowledge test was developed to assess whether caregivers understood the concepts about the function of problem behavior and developing effective behavior support plans presented in the third workshop module. During the workshop, caregivers completed a sample Behavior Support Guide based on a video example of a child’s problem behavior in a home child care setting. These were scored for how well each caregiver defined the problem behavior and its function, and developed a support plan.

Analytical Procedures

For data collected once before and once after intervention (i.e., the caregiver reports and the CCEC), a multivariate analysis of variance was conducted to identify effects by condition (intervention vs. waitlist control), time (baseline vs. post), and interaction between group and time (Cohen, Cohen, West, & Aiken, 2003). For data collected three times in each phase—three assessments before intervention and three assessments after intervention, including direct observations, observer ratings, and CIS—a longitudinal, random coefficients analysis (RCA) was conducted. This analysis is also called a growth model or multilevel model for change (Fitzmaurice, Laird, & Ware, 2004; Nich & Carroll, 1997; Singer & Willett, 2003). The RCA provides a more rigorous treatment for the analysis of repeated measures data than traditional methods (Fitzmaurice et al., 2004; Gueorguieva & Krystal, 2004; Nich & Carroll, 1997). RCA properly accounts for autocorrelation among repeated assessments of the same participant and, in this study, employed maximum likelihood estimation, which allowed for the use of all available data from all assessments. Such an analysis supplies unbiased results even in the face of substantial attrition, provided any missing data were missing at random (Laird, 1988; Shafer & Graham, 2002). In the present study, attrition or other missing data did not appear to represent a meaningful departure from the missing-at-random assumption, meaning that missing data were not likely to depend on unobserved determinants of the outcome of interest (Little & Rubin, 2002).

We fit models to our data with SAS PROC MIXED 9.1 (SAS Institute, 2004) using the restricted maximum likelihood method (REML) generally recommended for multilevel models (Hox 2002; Verbeke & Molenberghs 2000). Variance parameter estimates were not constrained to values at or above zero, and MIXED provided negative variance estimates for some models. Negative variances often result from computational limitations for estimates very near zero and do not represent problems with the models or analysis procedures (Kreft & de Leeuw, 1998; Singer & Willett 2003). Forcing non-negative estimates can lead to depressed Type I error rates and reduced statistical power in some situations (Murray, Hannan, & Baker, 1996). Finally, RCA models estimated with maximum likelihood procedures are relatively robust to violations of normality (Fitzmaurice et al., 2004; Maas & Hox, 2004).

Unlike standard regression software, PROC MIXED provides no standardized beta estimates. This effect-size formula transforms the t-value and degrees of freedom from a statistical test into a correlation coefficient (Rosenthal, Rosnow, & Rubin, 2000; Rosenthal & Rubin, 2003). In simple models with one outcome and one predictor, this provides an effect-size estimate interpreted just like the usual Pearson correlation coefficient. When squared, it gives the proportion of overlapping variance or variance explained and equals a usual R² statistic. In standard regression models with multiple predictors or covariates, the r-value computed from this formula equals the partial correlation coefficient, which gives the magnitude of the relationship between two variables controlling for all other variables in the model.

The RCA approach also offers considerable flexibility (Fitzmaurice et al., 2004; Singer & Willett, 2003). A basic model, for example, would estimate an intercept or level and a slope for all participants. A more sophisticated model estimates separate intercepts and slopes by condition across the three pretest time points, change in level for intervention participants at the first postintervention time point, and change in slope for intervention participants across the three postintervention time points. Although this latter model was appropriate for the data collected, it was first compared to a somewhat simpler model to determine the best representation of the data. That is, before comparing conditions for each outcome measure, models that included the same preintervention intercepts and slopes for both intervention and waitlist control conditions were compared with models that fit separate preintervention slopes for intervention and waitlist control conditions. This comparison provided, essentially, a more sophisticated test of pretest differences than typically conducted, as it allowed for a comparison of both intercepts and slopes before the introduction of the intervention. The best fitting model was selected using the Akaike Information Criterion (AIC), an index of the relative fit of competing models to the same data (Akaike, 1974; Burnham & Anderson, 2002). Built upon the principle of Occam’s razor, the AIC leads to the selection of a more generalizable final model than does the reliance of significance tests of individual parameter estimates (Myung, 2000; Zucchini, 2000). Lower AIC values indicate greater empirical support, given the number of parameters estimated and the underlying data. For each of the outcome measures presented, the model that estimated the same preintervention intercept and slope across conditions fit the data best.

The final analysis provided estimates of (a) normative intercept or level at the first time point for all participants, (b) normative slope across the three pretest time points for intervention participants and all time points for waitlist control participants, (c) change in level for intervention participants at the first postintervention time point, and (d) change in slope for intervention participants across the three postintervention time points (Fitzmaurice et al., 2004). The change in level and slope represents differences from those expected from normative growth, which was estimated from waitlist control condition participants across all assessments and treatment condition participants from their preintervention assessments. These change parameters represent the hypotheses of interest, that is, the immediate and longer-term effects of intervention on each outcome measure.

Due to the relatively small sample, power to detect effects was low. Although power calculations for the model tested are very complicated, power for a simpler analysis of covariance, with a pretest covariate of r = .50, should provide a reasonable estimate, if not a slight underestimate (Muthén & Curran, 1997; Venter, Maxwell, & Bolig, 2002). Such an analysis has power of .64 to detect a medium effect of condition (d = .50; Cohen, 1988) at the .05 level. To balance the likelihood of Type I errors with the chance of Type II errors, Cohen (1990) and Rosnow and Rosenthal (1989) recommend an adjustment to alpha, the Type I error rate. Thus, results with p < .10 are interpreted, which increases power to .75. Furthermore, effect sizes for tests of condition are also presented to aid interpretation. An effect size equivalent to Cohen’s d (Cohen, 1988; Rosenthal & Rubin, 2003) was computed, which represents the proportion of the pooled standard deviation accounted for by the parameter estimate.

Effects on caregiver outcomes were expected to mediate child outcomes. For example, we hypothesized that the intervention would have effects on the child care environment and caregiver practices which would then lead to improvements in child behavior and promote positive social interactions among children. Thus, when significant intervention effects were found for measures of children’s behavior and for measures of caregivers’ practices, mediation models were tested to determine whether the change in caregivers’ practices were functionally related to the changes in children’s behavior (Baron & Kenny, 1986; Judd & Kenny, 1981). Demonstration of mediation requires three steps. First, changes in child outcomes must be associated with intervention condition. Second, the intervention must be shown to predict changes in caregiver practices. Third, the measure of caregiver practices are added into the model that tests the condition on child outcomes. In this third step, caregiver practices should “explain” the condition effect on child outcomes. That is, caregiver practices are expected to predict child outcomes and the condition effect is expected to decrease or disappear. Mediator effects with intervention exposure were also tested, as well as moderator effects for years of experience providing child care, using procedures suggested by Jaccard and Turrisi (2003). Hours of intervention exposure were calculated (3 hours for each workshop attended and 2 hours for each home study completed), ranging from 0 hours (no exposure) to 9 hours.

Results

Feedback from Child Care Providers on Satisfaction and Barriers to Implementation

A large majority of the child care providers reported that the Carescapes workshops were useful (see Table 4). For modules 1 and 2, 90% of the caregivers said that they learned “some” to “very many” new things and practical skills. For module 3, 100% of the caregivers said that they learned “some” to “very many” new things and practical skills. A large majority of caregivers reported that they planned to make “some” to “very many” new changes or try new things based on the workshops (86% for module 1, 82% for module 2, and 92% for module 3). Barriers to implementation were reported mostly for making changes in the child care environment (module 1)—30% said that changes would cost too much, 20% said that they did not have enough time, and 17% said that their place was too small. Only one caregiver reported that no changes or improvements were needed in her child care home. Time was a barrier for at least 10% of the caregivers for each of the modules. No caregivers reported needing more information before they could implement the practices presented in modules 1 and 2. Only 7% of the caregivers reported needing more information on dealing with challenging behaviors (module 3). The average score on the knowledge test on the Behavior Support Guide from module 3 was 80% (the median was 83% and range was 46% to 96%). Only one caregiver scored below 60% and four scored between 60% and 70%.

Table 4.

Caregiver Feedback on Program Usefulness

Feedback questions for each workshop	Percent of Caregivers’ Responses
Workshop 1: Setting up to Support Social Development	Nothing new	A few new things	Some new things	Many or very many new things
How much did you learn?	3%	7%	52%	38%
How many practical skills did you gain?	0%	10%	62%	24%
Do you plan to make changes or try new things based on this workshop?	0%	14%	62%	24%
Workshop 2: Approaches for Guiding Children’s Behavior
How much did you learn?	0%	10%	38%	52%
How many practical skills did you gain?	0%	10%	34%	52%
Do you plan to make changes or try new things based on this workshop?	0%	17%	41%	41%
Workshop 3: Understanding and Dealing with Challenging Behavior
How much did you learn?	0%	0%	15%	85%
How many practical skills did you gain?	0%	0%	35%	65%
Do you plan to make changes or try new things based on this workshop?	0%	8%	46%	46%

Note. Thirty caregivers in the intervention group provided program usefulness data.

Intervention Outcomes

Outcomes for observed caregiver practices

Random coefficients analyses were performed on observed caregiver approvals and criticisms, observer ratings of caregiver use of effective management practices, and on the CIS. No significant outcomes were found for observed rate of criticisms or for the approval-to-criticism ratio. A medium effect was found for increased rates of approvals in the intervention group (intercept t = 1.77, p = .081, d = .45). Rates of approval especially increased at immediate post (T4); however, decreases in slope for approvals were greater at post (T4–T6) for the intervention group compared to the waitlist control (slope t = −1.68, p = .094) as shown in Figure 2. Initial increases in caregiver’s use of approval were found for the intervention group, but over time approvals decreased at a greater rate for the intervention group than for the waitlist control group. Parameter estimates for the RCA for caregiver approval and effective behavior management are provided in Table 5. Significant improvements were found in caregivers’ effective behavior management for the intervention group (intercept t = 2.82, p = .006). The level-change represents a medium-to-large effect size (partial r = .34, d = .72; Rosenthal & Rubin, 2003). The intervention and control group slopes at post did not significantly differ (see Figure 3). Caregivers in the intervention group increased their use of effective behavior management practices compared to those in the waitlist control group and the rate of change in use of effective behavior management practices across the three postdata timepoints did not differ for the intervention and control groups.

Figure 2.

RCA for observed caregiver rates of approval.

Table 5.

Estimates from the Random Coefficients Analyses for Caregiver Approval, Caregiver Effective Behavior Management, and Child Problem Behavior

Effect or statistic		Caregiver Approval	Caregiver Effective Behavior Management	Child Problem Behavior
Fixed effects	Intercept	.339*** (.033)	3.602*** (.056)	.138*** (.024)
	Slope	−.008 (.009)	−.042** (.015)	.026* (.011)
	Condition × posttest intercept	.093* (.053)	.220** (.078)	−.130* (.052)
	Condition × posttest slope	−.061* (.036)	−.078 (.052)	.045 (.037)
Variance components	Residual	.0631	.1249	.0593
	Intercept	.0346	.1246	.0008
	Slope	−.0007	−.0005	.0008

Note: Entries for fixed effects include parameter estimates with standard errors in parentheses. To compute t-values, divide the estimate by the standard error; t-values may differ from text due to rounding.

^*p < .10.

^**p < .01.

^***p < .001.

Figure 3.

RCA for caregiver effective behavior management.

Outcomes for caregiver reports of their practices and of child behavior

Pre–post increases in self-reported improvements in environmental enrichment (physical layout and toy variety) were found in both the intervention and control groups (p = .011), yet no significant differences between the groups were found. No significant findings occurred for self-reported planfulness. A significant group-by-time interaction was found for self-reported proactive behavior management practices; the waitlist control group reported increased use of proactive practices, whereas the intervention group reported decreased use.

Outcomes on Child Care Ecology and Caregiver Interaction Scale

No significant pre-post differences were found between the intervention and waitlist control groups on the CCEC scores for organized environment, enriched environment, or planned activities (see Table 6). Monitoring was found to decrease in both groups over time; however, the decrease was smaller for the intervention group (p = .087) and a medium effect size was found for this difference (Cohen’s d = .51; Cohen, 1988). No significant outcomes were found for the CIS factors.

Table 6.

Multivariate Analysis of Variance to Identify Pre–Post Changes by Group

Construct	Waitlist Control Group		Intervention Group		Analysis of Variance
	Baseline M (SD)	Posttest M (SD)	Baseline M (SD)	Posttest M (SD)		F	p
Child Care Ecology Checklist
					Group	0.08	.78
Organized Environment	1.70 (.70)	1.71 (.65)	1.68 (.73)	1.83 (.80)	Time	0.56	.46
					G X T	0.43	.52
					Group	0.77	.38
Enriched Environment	2.05 (.65)	2.14 (.63)	2.12 (.72)	2.33 (.56)	Time	2.66	.11
					G X T	0.41	.53
					Group	0.17	.69
Planfulness	2.25 (.48)	2.08 (.57)	2.10 (.72)	2.12 (.67)	Time	0.63	.43
					G X T	0.85	.36
					Group	0.04	.85
Monitoring	2.30 (.41)	1.96 (.56)	2.14 (.53)	2.08 (.62)	Time	6.11*	.02
					G X T	3.06	.09
Caregiver Report
					Group	1.81	.18
Enriched Environment	2.11 (1.02)	2.25 (1.07)	2.30 (.81)	2.72 (1.05)	Time	6.98*	.01
					G X T	1.73	.19
					Group	0.75	.39
Planfulness	3.58 (1.59)	3.77 (1.42)	3.85 (.91)	4.03 (.78)	Time	1.71	.20
					G X T	0.01	.95
					Group	0.17	.68
Proactive Behavior Management	6.01 (.81)	6.23 (.47)	6.32 (.43)	6.04 (.84)	Time	0.10	.75
					G X T	5.31*	.03
					Group	0.15	.70
Children’s Problem Behavior	3.40 (1.35)	3.42 (1.61)	3.48 (1.37)	3.07 (1.48)	Time	0.94	.34
					G X T	1.09	.30

Note. G X T = group-by-time interaction.

^*p < .05.

Because monitoring decreased over time, we conducted post hoc analyses and determined that a greater proportion of observations took place outdoors at baseline (43%) compared to posttest (14%; χ² = 35.26, p < .001). The change in observation settings across time, however, did not differ by condition. We also tested whether monitoring depended on the setting and found no significant differences in monitoring between outdoor and indoor observations.

Outcomes for observed overall child behavior

Random coefficients analyses were conducted on observed children’s problem behavior and on observer ratings of children’s positive behavior. Overall, the rate of child problem behavior increased across time (t = 2.30, p = .022). Figure 4 shows this in the line for the control group across all six data points and in the first three data points for the intervention group. The children in the intervention child care homes showed a decreased level of problem behavior immediately after the intervention (t = −2.48, p = .016). This change in the level of problem behavior just after intervention represents a medium-to-large effect size (partial r = −.30, d = .63). The difference in the slopes after the intervention was not statistically significant between the intervention and control groups. In other words, the rates of change in problem behaviors following the intervention did not differ. RCA parameter estimates for child problem behavior are shown in Table 5. No significant outcomes were found for observer ratings of children’s positive behavior.

Figure 4.

RCA for children’s problem behavior.

The rate of problem behavior was not related to the number of children in the child care home, and therefore the calculation of the rate did not account for the number of children. The number of children in each child care site could nonetheless moderate differences in the impact of the intervention. To test for moderation, we added the number of children in child care and its interaction with condition into the random coefficients model of the rate of child problem behavior. Neither term was statistically significant. The number of children in the child care home did not impact the intervention outcomes found for overall children’s problem behavior.

Mediation model of caregiver effective management and children’s problem behavior

To test whether the change in caregivers’ practices due to intervention was functionally related to the changes in children’s behavior, we tested a mediation model (Baron & Kenny, 1986; Judd & Kenny, 1981; MacKinnon, Lockwood, Hoffman, West, & Sheets, 2002), shown in Figure 5. Two of the three steps to demonstrate mediation were shown previously: the intervention produced differences between conditions in caregivers’ effective behavior management (path a in Figure 5) and children’s problem behavior (path c). In the third step, we tested whether caregivers’ effective behavior management predicted children’s problem behavior (path b) and simultaneously reduced the condition effect on children’s problem behavior (path c’). Caregivers’ effective behavior management explained a significant portion of the intervention effect on children’s problem behavior (t = −6.38, p < .001). The effect size for this mediation effect was medium (partial r = −.37). The mediation model, with caregivers’ effective behavior management predicting children’s problem behavior, produced a decrease in the direct effect of condition onto children’s problem behavior from d = .63 (t = −2.48, p = .016) to d = .48 (t = −1.89, p = .064). The causal steps alone, however, are insufficient to demonstrate mediation (MacKinnon et al., 2002). A test of the product of paths a and b, a b, however, demonstration of mediation (MacKinnon et al., 2002; Sobel, 1982), and has been shown equivalent to a test of the reduction in effect of condition on the outcome, c – c’ (MacKinnon, Warsi, & Dwyer, 1995). This test was statistically significant (t = −2.58, p = .012). Thus, although caregivers’ effective behavior management did not completely reduce the effect of condition on children’s problem behavior, it did account for some of the statistically significant reduction in problem behavior associated with the intervention.

Figure 5.

Mediation of caregiver effective behavior management on children’s problem behavior.

Effects of intervention exposure and years of experience providing child care

The extent of intervention exposure had no effect on caregivers’ classroom management practices or children’s rates of problem behavior. Experience providing family child care was associated with effective behavior management practices, where greater experience was associated with more effective practices (partial r = .25, t = 2.04, p = .046), but experience had no moderator effect. Child care experience also had no moderating effect on children’s problem behavior.

Discussion

This study had three primary goals. The first was to develop a research-based intervention for small groups of family child care providers serving preschool-aged children. The second was to evaluate the feasibility and acceptability of offering this intervention for the intended recipients. The third was to evaluate the immediate effects of the intervention in a small randomized efficacy trial on caregiver practices, the child care environment, and children’s behavior.

This project developed three research-based workshops with practical advice on promoting social development and managing challenging behavior. The workshops included presentations of video examples that were followed by discussion and practice activities. The development of the video-based program and a written manual increases the likelihood that the workshops can be readily replicated.

This study demonstrated that the intervention was highly appealing to family child care providers. In a previous survey about training needs, family child care providers reported that one of their biggest priorities for training is how to manage and reduce disruptive behavior (Rusby, 2002). In the present study, family child care providers confirmed this priority by the low attrition rate and good participation in the assessment and intervention activities, as well as through their positive evaluations of the workshops. Eighty-one percent maintained their participation in the study over the course of 2 years. Workshop attendance actually increased after the first workshop (from 71% to 81%). Those who missed a workshop watched the video and completed activities on their own time, resulting in 90% intervention participation for the first two modules and 87% for the third module. Between 90% and 100% of participants reported that they learned “some” to “very many” new things from the workshops and 86% to 100% reported learning “some” to “very many” practical skills. Collectively this demonstrates that this intervention was perceived by family child care providers as being useful and meeting an important need.

The third goal of this study was to conduct an initial experimental evaluation of the intervention. Perhaps the most important findings of the efficacy trial were that caregivers in the intervention group improved their observed effective behavior management practices (p = .006, d = .72). Reductions in overall problem behavior were found in the intervention child care homes compared to the waitlist control group (p = .016, d = .63). The medium-to-large effect sizes found for both these outcomes represent a change of approximately two-thirds of a standard deviation for intervention participants. The effective behavior management practices included supporting children’s efforts, encouraging active participation, providing clear instructions, preparing children for transitions, monitoring children, providing reminders of expectations, modeling positive behaviors, providing positive reinforcement to children, and being consistent with rules and consequences. A mediation model for these effects was tested, demonstrating that the improvements in behavior management practices were associated with reductions in children’s problem behavior (p < .001). A medium effect size was found for the mediation effect.

We adjusted the p-value to .10 to balance the likelihood of Type I and Type II errors in small-scale studies such as this (Cohen, 1990; Rosnow & Rosenthal, 1989). With the adjusted p-value, medium effects favoring the intervention group were found on the composite measure of caregiver monitoring, which included not only caregiver behavior (i.e., scanning, circulating, and maintaining proximity to children needing assistance) but also a rating of the extent to which the physical environment was set up to enhance visibility. A small effect was also found for observed positive attention, such as praise and approving gestures or physical contact. Caregivers in the intervention group increased the amount of positive attention provided to the children compared to the waitlist control group.

This study also detected evidence that the improvements in child care provider behavior faded over time. The greatest increases in effective behavior management and positive attention were found for the first observation, which occurred about a month following the intervention. Use of positive attention by child care providers in the intervention group decreased over time, indicating that this increase was not maintained 5 months following the intervention. Similarly, the improvements in effective behavior management practices and children’s behavior found in the intervention groups declined over time.

Maintaining effects in studies aimed at improving young children’s behavior and interpersonal skills has been a common challenge. For example, in school-based settings proactive behavior management practices for promoting positive behavior have been empirically tested and found to be effective; however, the difficulty is in maintaining the use of these practices over time (Sugai & Horner, 1999). In a review of randomized controlled trials aimed at improving children’s interpersonal skills, Taylor, Eddy, and Biglan (1999) also found evidence for short-term effects, but little evidence for maintenance of the effects. The authors surmise that the programs that do result in positive outcomes tend to be lengthy. This concurs with the results of a meta-analysis of preschool prevention programs (Nelson, Westhues, & MacLeod, 2003), in which the length of the intervention program significantly accounted for a portion of the variance in the maintenance of positive social outcomes into elementary and middle school. Perhaps breaking the intervention content into smaller “chunks” (e.g., provide a series of six workshop sessions that are 1½ hour long, rather than three workshop sessions that are 3 hours long) would allow caregivers to focus on and practice each strategy learned over a longer period of time. Attention to helping child care providers maintain their implementation of effective behavior management practices throughout the year would likely impact the maintenance of child outcomes. Such maintenance of effective practices could be promoted through booster sessions throughout the year. Booster sessions that include a review of main ideas, involve problem solving, and emphasize accomplishments and positive changes are more successful in achieving maintenance (Whisman, 1990). It also is possible that caregivers did not understand how to generalize the techniques learned to new problems or challenges that came up for them after the workshops were over. A follow-up booster session may help in learning how to apply what was learned to new situations.

Another approach for promoting generalization and maintenance of training outcomes involves behavioral consultation. Much of the research on the effectiveness of behavioral consultation has been done for school-age children in special education (e.g., Sheridan, Welch, & Orme, 1996). A consultation model of “coaching” that involves the caregiver in goal setting and intervention planning has been applied to child care (Hendrickson, Gardner, Kaiser, & Riley, 1993) and preschool settings (Kaminski & Stormchak, 2006). Such consultation approaches have also been described as “technical assistance” and resulted in preschool teachers’ generalization of instructional strategies (Peck, Killen, & Baumgart, 1989). Providing multiple examples of applying strategies to different situations via coaching facilitates maintenance and generalization (Stokes & Osnes, 1986). Positive outcomes were also found for preschool-age children who exhibited social and/or behavioral problems when consultation was provided that utilized assessment feedback (Schill, Kratochwill, & Elliott, 1998). Providing an on-site coaching method of consultation in addition to the workshop training would likely assist caregivers in understanding how to apply what was learned in the workshops to a variety of situations in their child care home and to get hands-on practice with goal setting and intervention planning, both of which would promote maintenance of their use of effective practices.

This study also found that, although intervention participants engaged in significantly more monitoring than waitlist control participants after intervention, both intervention and waitlist control groups showed a reduction in monitoring over time. In post hoc examination of the data, it appeared that baseline observations occurred more frequently on sunny days when children were outside and post observations occurred more frequently on rainy days when children were inside. Although it is plausible that higher levels of monitoring occurred in outdoor settings than inside the child care homes, that was not the case. The extent to which observations were conducted in indoor or outdoor settings did not differ by condition, and thus did not bias the outcomes. It is also possible that caregivers decreased monitoring to typical levels as they got more accustomed to the observers’ presence. In future studies, it may be important to increase the number of initial observations to acclimatize subjects to observers.

There was a lack of intervention outcomes for the child care environment from the Child Care Ecology Checklist (CCEC), which measured the organization of the environment, enrichment of the environment, and the planned activities. It is plausible that the CCEC was not sensitive to intervention change. A major change may only be reflected in one variable that comprised a composite score. Without sensitive measures, interventions may have beneficial effects that go unmeasured, resulting in potentially important interventions being discarded due to lack of evidence. Feedback from participants offered useful insights on barriers to change for these environmental features. Caregivers reported that the greatest barriers were in making changes to the physical environment compared to any other intervention component. These barriers were due to cost, time, and limited space. To facilitate such change, a combination of individualized consultation and financial support may be necessary. Given the associations between the environment in center-based child care and child behavior (Clarke-Stewart, 1987; Dunn, 1993), it seems reasonable to expect similar relationships in family child care settings. Modest investment in stimulating resources for child care homes would likely have a positive impact on many children.

One finding contrary to what was hypothesized was that caregivers in the intervention group reported decreased use of effective behavior management practices and the waitlist control group reported increased use. These results did not concur with observations of effective behavior management. Child care providers may not be valid reporters of their own behavior. Perhaps an assessment effect occurred for the waitlist control group where providers in the waitlist control condition became sensitized to behavior management concepts and increased their reporting of their use. It is also possible that once caregivers in the intervention group learned more about effective caregiver practices, they became more aware and possibly more critical of their own behavior.

Limitations of the Study and Future Research

An important limitation of this study is that it involved a relatively small sample. This small-scale efficacy study is the first step in examining the value of the developed intervention program for family child care providers. Since the intervention program was developed for caregivers of preschool-age children, the study sample was limited to child care homes with preschool-age children in attendance. The study sample also was limited to caregivers who had a good understanding of English. Study results therefore can be generalized solely to family child care providers who speak English and care for preschool-age children. A larger efficacy study is needed to test the extent to which outcomes found in this study are replicated. Depending on results of a larger replication efficacy trial, effectiveness studies in which the program is delivered in natural community settings would be the next step towards providing empirical evidence for the intervention program (Flay et al., 2005).

Another limitation of the study is that the effects are short-term, and there is reason to believe the benefits of the training would continue to fade with time without booster sessions or other strategies to promote maintenance. The data for this study are limited in that the last assessments were collected only five months following the intervention. It is important to recognize that outcomes of interventions for young children may occur at later developmental stages (e.g., Ialongo, Poduska, Werthamer, & Kellam, 2001; Olds et al., 2004), demonstrating the importance of long-term follow up. Replication studies with larger samples that incorporate maintenance strategies and evaluate them with long-term follow-up assessments would offer greater empirical evidence for the Carescapes program.

Another limitation of the efficacy trial was that all measures were aimed at the child care home, the care provider, and overall behavior of all children attending the child care home. No individual data on children were collected. The observation on children’s behavior involved the behavior of any children who were attending the child care home. Longitudinal measures at the individual child level would allow the examination of outcomes for children through different developmental stages. Moreover, with a larger sample size, it would allow for a study design in which children are nested in child care homes.

Summary

This study involved the development of a video-based intervention program for family child care providers aimed at promoting positive social development in preschool-age children and the first randomized controlled trial on its efficacy. This research highlights the importance of specific, proactive caregiver practices that encourage and support children and that prevent problems through monitoring and clearly defined expectations. These practices were found to be socially valid and useful for practitioners who provide child care in their home. However, results of the small-scale efficacy trial are mixed, as the positive outcomes on caregiver practices and child behavior that were found did not last. Clearly, further steps in evaluation are needed, such as larger-scale longitudinal efficacy and effectiveness studies to provide the empirical evidence needed for dissemination.

Given the dearth of interventions specifically designed for family child care providers, future early intervention and prevention work focused on family child care settings is called for. There are very few randomized controlled trials in preschools to promote social skills and reduce problem behaviors (Taylor et al., 1999) or in child care settings (Fukkink & Lont, 2007). This study stands as an example for conducting randomized controlled trials in child care settings, particularly in family child care in which experimental studies are lacking (Kontos, 1992). This study has shown that caregiver practices that have been found effective in preschool and group-based child care settings are also useful for home child care providers. It demonstrates the feasibility of such endeavors. Home child care providers were engaged in the training and they implemented some of the strategies learned in their child care home. Although increased effective behavior management was found immediately following the intervention and was related to decreased overall problem behavior of children in the child care home, the use of these strategies did not maintain over time. Further work on how to provide on-site consultation for family child care providers is needed to promote maintenance and generalization of effective caregiving practices. Also, further work is needed to address the barriers of the cost and time needed to implement environmental changes in child care homes. Such improvements imply the need for sources of funding for consultation and resources (such as age appropriate toys, equipment, furniture) for home child care settings. The many children who attend family child care are likely to benefit when the breadth of good quality early childhood practices and research extends to those settings.