Estimating the “Impact” of Out-of-Home Placement on Child Well-Being: Approaching the Problem of Selection Bias

Abstract

This study used data on 2,453 children age 4 to 17 from the National Survey of Child and Adolescent Well-Being and 5 analytic methods that adjust for selection factors to estimate the impact of out-of-home placement on children's cognitive skills and behavior problems. Methods included ordinary least squares (OLS) regressions and residualized change, simple change, difference-in-difference, and fixed effects models. Models were estimated using the full sample and a matched sample generated by propensity scoring. Although results from the unmatched OLS and residualized change models suggested that out-of-home placement is associated with increased child behavior problems, estimates from models that more rigorously adjust for selection bias indicated that placement has little effect on children's cognitive skills or behavior problems.

Developmental researchers have long been interested in understanding how various caregiving experiences, both within and outside of children's homes, impact child development and well-being. An important extension of this line of inquiry focuses on changes in caregiving contexts due to child maltreatment and the subsequent out-of-home placement of children by Child Protective Services (CPS). Knowing whether out-of-home placement by CPS mitigates or heightens developmental risk for maltreated children is crucial for assessing the efficacy of child welfare policies and interventions, as well as for understanding the relative risks and benefits of major discontinuities in children's care. Despite the importance of this issue for both child welfare policy and developmental theory, however, studies of the effects of out-of-home placement on child well-being have yet to overcome important challenges related to selection bias in who enters state custody; that is, children who remain in the care of their parents and those who are placed out-of-home are likely to differ on a host of observable and unobservable factors, including socioeconomic characteristics and the types and severity of maltreatment they have experienced. Such differences pose a considerable barrier to producing unbiased estimates of the effects of out-of-home placement on child well-being. As such, it remains unclear whether placement is generally beneficial, harmful, or inconsequential for the development and well-being of maltreated children (Courtney, 2000; McDonald, Allen, Westerfelt, & Piliavin, 1996), especially those at the margin of placement (Doyle, 2007).

This study used data on 2,453 children age 4 to 17 from the National Survey of Child and Adolescent Well-Being (NSCAW) and five analytic methods that adjust for selection factors—including ordinary least squares (OLS) regressions with extensive controls and residualized change, simple change, difference-in-difference (a repeated measures analysis of group differences in change over time), and fixed effects models—to estimate the impact of out-of-home placement on children's cognitive skills and behavior. All models were estimated using both the full sample and a matched sample generated by propensity scoring. Comparing estimates produced by each of the methods may provide insight into whether associations between out-of-home placement and child well-being are likely to be causal.

The study extends prior research in its use of multiple methods to adjust for selection factors and in two additional ways. First, the NSCAW data enabled us to account for a wide range of confounding factors—including child and family background characteristics, CPS case characteristics, and children's pre-placement levels of cognitive skills and behavior problems—many of which have been omitted from previous studies. The NSCAW sample is also larger, more geographically diverse, and includes a wider age-range of children than other existing child welfare datasets. Coupled with the longitudinal design of the study, this allowed us to follow a larger sample of children over time on multiple indicators of well-being than has been possible in most prior work. Second, our approach improved upon recent research using NSCAW in that we defined our sample to include only children for whom we had in-home (i.e., pre-placement) data. Prior research with NSCAW has utilized samples that included children already placed out-of-home at the baseline interview (Barth, Guo, Green, & McCrae, 2007; Stahmer et al., 2007). As such, “baseline” measures of family characteristics and child well-being were actually assessed during an out-of-home placement for at least some children, thereby reflecting the characteristics of these children's placement families and their levels of well-being while in placement. By excluding children who were never observed in their home of origin, our sample enabled us to better isolate changes in child outcomes that may result from out-of-home placement.

Background

An estimated 3.6 million children were investigated or assessed by CPS agencies in the United States in 2006; more than a quarter were substantiated or indicated for child abuse or neglect (U.S. Department of Health and Human Services [USDHHS], 2008a). In all, more than 300,000 U.S. children entered and approximately 510,000 were residing in some form of out-of-home placement as a result of CPS involvement (USDHHS, 2008b). The traditional objectives of the U.S. child welfare system have been to ensure safety and promote permanency for children. Although improving child well-being has often been viewed as an implicit goal of the system, it has only more recently, with the passage of the Adoption and Safe Families Act of 1997 (ASFA), been made explicit (Wulczyn, Barth, Yuan, Harden & Landsverk, 2005). Indeed, since 2001 state CPS agencies have been required to undergo federal Child and Family Service Reviews which assess and monitor their progress towards promoting child safety and permanency, as well as meeting children's educational, physical, and mental health needs. Whether safety, permanency, and well-being are better served by removing children from their homes or keeping family units intact is likely to depend, at least in part, on the relative risks and benefits of continuity versus discontinuity of care for children. Out-of-home placement may provide children with physical safety and opportunities to develop nurturing relationships with adults, but its abrupt and indefinite nature may also place additional burdens on already vulnerable children. It may therefore present both opportunities for resilience and new stressors (c.f., Rutter, 2000).

Early work by Maas and Engler (1959) indicated that children in substitute care fared less well than community samples of youth on a range of developmental outcomes, a finding that has been repeatedly replicated (Blome, 1997; Clausen, Landsverk, Ganger, Chadwick, & Litrownik, 1998; Shin, 2004). Research also indicates, however, that children who come to the attention of CPS have experienced a variety of other adversities that alone might jeopardize functioning. These adversities often include parental substance use and mental health problems, poverty, and abuse or neglect (Barth, Wildfire, & Green, 2006; Child Welfare League of America, 2001; Reams, 1999), each of which may challenge development (Barnard & McKeganey, 2004; Brooks-Gunn & Duncan, 1997; Cicchetti & Toth, 2005; Glaser, 2000; Stone, 2007) and thereby partially or fully account for differences in outcomes between children experiencing out-of-home placement and those in community samples. Indeed, an early large-scale longitudinal study of children in out-of-home placement found that many showed improvement with regard to physical, cognitive, and school-related outcomes during their first 6 months in care (Fanshel & Shinn, 1978), suggesting that some of the adverse outcomes associated with placement may reflect pre-placement experiences rather than the effect of placement itself (Waldfogel, 2000).

Further complicating efforts to understand the unique impact of out-of-home placement on child well-being, children who are placed out-of-home are also likely to differ from those who are substantiated for maltreatment but are not removed from home: They are likely to have experienced a greater severity of maltreatment and higher levels of prior contact with CPS (USDHSS, 2005). The two groups also tend to differ on factors such as parental cooperation with CPS, parental stress, parenting skills, social support, substance abuse, domestic violence, and criminal justice involvement (Shlonsky, 2007). As such, isolating the impact of out-of-home placement on well-being poses a number of analytic challenges.

Analytic Challenges and Identification Strategies

Because it is not possible to simultaneously observe a child both in his or her home and in an out-of-home placement, or to randomly assign children to out-of-home placement, researchers must rely on statistical methods to adjust for selection bias in who enters state care. Four strategies have been used in existing studies. These include controlling for correlates of both out-of-home placement and child outcomes, employing matching techniques, estimating change models, and utilizing instrumental variables methods. Below, we discuss the benefits and disadvantages of each of these approaches and highlight the value of applying additional methods to the study of relations between out-of-home placement and child well-being.

The most commonly used strategy for attempting to adjust for pre-placement ecological adversity and maltreatment experiences is to control for confounding covariates while comparing outcomes for children who have experienced out-of-home placement and those who have not. Most existing studies of this type have compared children experiencing placement to those who were not maltreated (or were not reported to CPS) but had similar levels of socioeconomic disadvantage (Bilaver, Jaudes, Koepke, & Goerge, 1999; Blome, 1997; Buehler, Orme, Post, & Patterson, 2000; Fantuzzo & Perlman, 2007; Pears and Fisher, 2005a; 2005b; Viner & Taylor, 2005) or to children who were maltreated (or reported to CPS) but remained in their homes (Johnson-Reid & Barth, 2000; Leslie, Gordon, Ganger, & Gist, 2002). Studies using this approach have found adverse associations of placement with child and adult outcomes in domains such as health, emotional and behavioral adjustment, cognitive development, criminal justice involvement, educational attainment, and economic status. Although this strategy is useful for adjusting for the confounding effects of observed variables that are associated with both out-of-home placement and child outcomes, estimates are subject to bias due to unobserved factors. Studies of this type are further limited in that they do not account for baseline differences in children's scores on the outcome(s), despite that children placed out-of-home and those remaining in-home are likely to differ in this regard. The detailed background information available in NSCAW allowed us to control for an extensive set of potentially confounding variables; we also employed a series of additional methods that accounted for children's baseline scores on the outcome measures and adjusted for some forms of unobserved heterogeneity.

A second strategy utilizes propensity score matching methods (Rosenbaum & Rubin, 1983) to identify treatment and comparison groups that are statistically equivalent on observable background characteristics and differ only in terms of whether children have experienced out-of-home placement. Associations between out-of-home placement and child outcomes are then estimated for the matched groups. Matching methods offer a distinct advantage over simply controlling for confounding covariates in that they can ensure appropriate overlap in the covariate distributions of the treatment and comparison groups, such that the models are not extrapolating over portions of the covariate distributions in which there is no support. Like comparison group studies that control for background characteristics, however, they adjust only for measured selection factors; estimates continue to be subject to bias due to unobserved factors.

Although several studies (Barth et al., 2007; Rubin, O'Reilly, Luan, & Localio, 2007) have used matching methods to estimate group differences in well-being among children in out-of-home care, we are aware of only one that directly compared outcomes for children experiencing placement and those remaining in-home. Berzin (2008) compared young adult outcomes for youth who had experienced foster care placement at some point during childhood with those of matched and unmatched samples of youth who had not. Results using the unmatched data indicated that youth who experienced placement had lower levels of educational attainment and higher rates of public assistance use, teen parenting, and criminal justice involvement. In contrast, results using one-to-one nearest neighbor propensity score matching, the matching scheme that produced the most similar treatment and comparison groups, revealed no associations between placement and any of these outcomes. This suggests that the associations identified in the unmatched data likely reflected differences in the characteristics and experiences of youth who did and did not experience placement, rather than the experience of placement itself. As noted above, however, matching methods are subject to bias due to omitted variables. Indeed, Berzin (2008) notes that several important background factors, such as parental substance abuse and criminal justice involvement, as well as the nature of maltreatment experienced by sample children, were omitted from her models. In addition, her analyses utilize a relatively small treatment group sample of 136 youth who experienced foster care placement at some point during childhood and were observed between the ages of 17 and 24. The study is unable to account for the timing of out-of-home placement during childhood or to assess the influence of placement on proximal outcomes. In contrast, we assessed the influence of out-of-home placement on children's cognitive and behavioral development over approximately the 2 1/2 years encompassing placement. Furthermore, our sample included considerably more children who experienced placement (N = 342), and our matching models accounted for a wider range of background factors, including pre-placement assessments of the outcome measures.

A third approach to reducing selection bias involves using longitudinal data to estimate changes in well-being over time as a function of out-of-home placement. Models for doing so can take different forms, including the residualized change, simple change, difference-in-difference, and fixed effects models that we utilized in the current study. In general, however, they reduce bias by accounting for children's baseline (or mean) levels of well-being when predicting later levels of well-being or changes in well-being over time. They thereby adjust—to varying degrees—for pre-existing (unmeasured) differences between children who are subsequently removed from home and those who are not. For example, change models may be useful for adjusting for prior maltreatment severity, which likely influences children's baseline and final levels of cognitive skills and behavior problems, as well as whether they are placed out-of-home, but is difficult to measure (Litrownik et al., 2005) and often omitted from empirical analyses.

Existing studies of this type have generally estimated either what we refer to as “residualized” change models (National Institute of Child Health and Human Development Early Child Care Research Network [NICHD] & Duncan, 2003), in which a given outcome is modeled as a function of an earlier score on that outcome and a set of background characteristics and experiences (Stahmer et al., 2007), or repeated measures analyses using MANOVA (Davidson-Arad, 2005; Davidson-Arad, Englechin-Segal, & Wozner, 2003) or ANCOVA (Lawrence, Carlson, & Egeland, 2006). By directly modeling the baseline score, such methods adjust for factors that are not directly observed, but that determine initial functioning on the outcome of interest and thereby influence the follow-up score. Several such studies have found placement to be associated with improvements over time on physical (physical activity, orientation, health, and safety) and psychological (self-actualization, relaxation, mental health, and identity) dimensions of quality of life (Davidson-Arad et al., 2003; Davidson-Arad, 2005). In contrast, others have reported associations of out-of-home placement with increases in behavior problems (Lawrence et al., 2006) and decreases in cognitive and language development (Stahmer et al., 2007).

Existing studies using MANOVA or ANCOVA are limited in that they have adjusted for few observable covariates and have utilized very small samples. A noteworthy shortcoming of the residualized change approach is that it does not adjust for factors that differentially impact an outcome at baseline and follow-up. As such, NICHD and Duncan (2003) emphasize the value of comparing results from multiple types of change models when attempting to estimate causal relations. Thus, in addition to OLS and residualized change models, we also employed simple change, difference-in-difference, and fixed effects models to estimate associations of out-of-home placement with child well-being. Our simple change and difference-in-difference models both used between-child variation in placement status to assess whether changes in well-being over time differed for children who did and did not experience placement. The simple change model adjusted for initial differences in well-being at the individual level, whereas the difference-in-difference model did so at the group level. Our fixed effects model used within-child variation in placement status to identify effects of out-of-home placement on intra-child change in well-being for children observed both in-home and during or after out-of-home placement. As discussed in the Method section, each of these strategies is likely to reduce bias to a greater extent than OLS and residualized change models, albeit under different assumptions. To further adjust for selection factors, we also estimated all of our models utilizing both our full and matched samples.

A fourth strategy for estimating causal effects involves the use of instrumental variables. We are aware of only one existing study of out-of-home placement that utilized this approach. Doyle (2007) used unique data on Illinois CPS caseworkers' propensities to remove children from home as an instrument to isolate the exogenous component of out-of-home placement and, thereby, to estimate causal effects. Results suggested that children assigned to caseworkers with higher propensities for removal were more likely to be placed out-of-home and, subsequently, to exhibit higher levels of delinquency and teen childbearing, and lower earnings. Although instrumental variables techniques can be used to estimate causal relations by purging an independent variable of the bias that results from unobserved heterogeneity (e.g., omitted variables) or endogeneity (Gennetian, Magnuson, & Morris, 2008), identifying a valid instrument is difficult (Currie, 2005). In the present case, we would have required an instrument that produced random variation in the likelihood that a child was placed out-of-home, but that was otherwise uncorrelated with child outcomes. We could not utilize state level policies regarding child removal as instruments, for example, because such policies are likely to be correlated with other state level factors that may independently influence child outcomes. Thus, we did not utilize instrumental variables methods in our analyses.

Placement Characteristics

Associations between out-of-home placement and child well-being are likely to vary by the length, stability, and type(s) of placements children experience. Placement length and stability may moderate these associations by influencing the consistency of care to which children are exposed and whether children experience disruptions in care (James, Landsverk, & Slymen, 2004). Children who spend a short period of time out-of-home may be affected differently than those who spend months or years in care. Likewise, children experiencing multiple placements may be impacted differently than those with stable placement experiences. Results from prior research on these factors have been inconsistent. Several studies demonstrate links between placement instability and adverse child outcomes (Newton, Litrownik, & Landsverk, 2000; Rubin et al., 2004; Rubin et al., 2007); other work finds no variation in outcomes by placement length or number of placements (Pears & Fisher, 2005a, 2005b).

Research on placement type has consistently indicated that children placed in residential facilities experience greater developmental adversity than those placed in family foster care, even after adjusting for initial levels of functioning (Davidson-Arad, 2005; Davidson-Arad, et al., 2003; McDonald et al., 1996). Among children placed in family foster homes, developmental outcomes may be influenced by the kinship status of foster parents. Residing with kin is thought to minimize the trauma of out-of-home placement and thereby promote child well-being (Chamberlain et al., 2006; Chapman, Wall, Barth, & the NSCAW Research Group, 2004; Ehrle & Geen, 2002; Holten, Rønning, Handegård, & Sourander, 2005; James, 2004). At the same time, kin foster parents tend to be less advantaged (Berrick, Barth & Needell, 1994; Ehrle & Geen, 2002) and to exhibit lower quality caregiving behaviors than non-kin foster parents (Gaudin & Stuphen, 1993; Harden, Clyman, Kriebel, & Lyons, 2004). As such, the extent to which developmental outcomes may be differentially affected by kin and non-kin placements is theoretically ambiguous. Empirical results have been similarly mixed. Findings from several studies suggest that children placed in non-kin foster homes exhibit poorer outcomes than those placed with kin (Holten et al., 2005; Keller, Wetherbee, Le Prohn, Payne & Lamont, 2001; Landsverk, Davis, Ganger, Newton, & Johnson, 1996; Lawrence et al., 2006; Leslie et al., 2002), whereas others provide little evidence of differences (Barth et al., 2007; Benedict, Zuravin & Stallings, 1996; Shore, Sim, Le Prohn, & Keller, 2002; Zimmer & Panko, 2006).

Although multiple existing studies have examined associations of placement characteristics with child outcomes, it is important to recognize that placement length, stability, and type are likely to be endogenous (i.e., determined by similar factors or processes) with children's cognitive skills and behaviors. As such, it is unlikely that estimates of associations between placement characteristics and child outcomes will reflect causal relations even when rigorous methods of adjusting for selection factors are employed. Nonetheless, to be consistent with prior work, and also in the hope of shedding further light on the potential moderating roles of placement length, stability, and type, we estimated supplemental models that allowed associations of out-of-home placement with child outcomes to vary by each of these factors. We note, however, that these estimates should not be interpreted as reflecting causal relations.

Method

Participants

Our analyses necessitated the use of data with highly detailed information on children's out-of-home placement experiences, background characteristics, and pre- and post-placement levels of well-being. NSCAW, the first national probability study to collect data directly from children and families coming into contact with CPS in the U.S., is the only available data source that provides such information on a national level. The full NSCAW sample includes 5,501 children from birth to age 14 (at baseline), who were investigated by CPS between the fall of 1999 and the winter of 2000. The study attempted to follow all sample children for 36 months, regardless of case disposition. Data were collected via interviews with children, caregivers, teachers, and CPS investigators at baseline and 12-, 18-, and 36-months after enrollment.

We used data provided by children, parents, current caregivers, and CPS caseworkers via the baseline, 18- and 36-month interviews (child assessments were not conducted at the 12-month interview). Our sample included only those children who entered the child welfare system due to a new CPS investigation during the initial NSCAW sampling period and were observed in-home at either the baseline interview (conducted on average about 14 weeks after the initial investigation) or, for children placed out-of-home at the baseline interview, the 18-month interview. We excluded children who were placed out-of-home at the time of both their baseline and 18-month interviews because we had no in-home (baseline) data on them or their families of origin. As such, we focused on estimating associations between subsequent out-of-home placement and child well-being over a two and a half year period (on average) for children who were between the ages of 4 and 14 at the time of NSCAW sampling and were observed living in their home of origin at the time of the baseline assessment used in this study.

From the full NSCAW sample of 5,501, we excluded 2,288 children because they were younger than age 4 at baseline and, therefore, were not assessed on the outcome measures of focus. We excluded an additional 694 children who entered the NSCAW sample as a result of a new CPS investigation but were not observed in-home at either the baseline or 18-month interview. Finally, we excluded 66 children for whom contradictory living arrangement data provided by caseworkers and caregivers precluded us from determining whether a child was in-home or out-of-home during the NSCAW assessments. Our final analysis sample consisted of 2,453 children. All of these children were retained in our sample throughout the observation period, regardless of their subsequent placement experiences. Given our exclusion criteria, our analysis sample was no longer representative of all children entering the child welfare system, nor of the original NSCAW sample; however, about 66% of children entering and 71% of children residing in out-of-home care in 2006 were between the ages of 4 and 17 (USDHHS, 2008b). Furthermore, descriptive statistics (not shown) indicated that our analysis sample was similar to the full NSCAW sample of children age 4 to 14 at baseline on most background characteristics, with the exception that the mean family risk score (described below) was higher in the full sample. Given that children excluded from our sample were out-of-home at both baseline and 18-months, it makes sense that their homes of origin were considerably less safe than those of children included in our sample. As such, our results may not be generalizable to children who are most likely to be immediately removed from home and kept in care for a long period of time. Although we selected the most appropriate sample with which to estimate causal effects of out-of-home placement on child well-being, our strategy increased the internal validity of our study at the expense of external validity.

Measures

Out-of-home placement

Our primary predictor variable was an indicator (1 = yes) that a child was removed from home by CPS between his or her baseline and follow-up assessments. This measure included placements in kin and non-kin family foster homes as well as other types of care (i.e., group homes, emergency shelters, psychiatric hospitals, residential treatment facilities, detention centers, transitional living arrangements, or other settings). Because children experiencing these ‘other’ types of placement may substantially differ from those experiencing only family foster homes, we conducted supplemental analyses that excluded the former group.

We designated the initial NSCAW interview as the baseline assessment point for all children observed in-home at that time (94%) and the 18-month interview as the baseline assessment point for the 6% of children who were not observed in-home at baseline, but were observed in-home at 18-months. We designated the 36-month interview as the follow-up assessment point for all children observed (either in-home or out-of-home) at 36 months (91%). We designated the 18-month interview as the follow-up assessment point for the 9% of children who were observed in-home at baseline and were assessed (either in- or out-of-home) at 18-months, but not at 36-months. In all models, we controlled for the number of months between a child's baseline and follow-up observations. We also controlled for whether a child experienced placement prior to their (in-home) baseline assessment. Because children who experienced placement prior to their baseline assessment are likely to differ from those who did not, we also estimated supplemental models which excluded the former.

Utilizing a dichotomous indicator of whether a child experienced placement to predict cognitive skills and behavior problems (and changes therein) provides evidence of the average association between placement and these outcomes, regardless of the length of time spent in care and the number and types of placements experienced. However, each of these factors—though likely endogenous to children's well-being—may moderate the associations of interest. Thus, in supplemental analyses, we also estimated models that used as the key predictors: (1) the total proportion of time a child spent out-of-home between the baseline and follow-up interview (logarithm), as well as the number of placements the child experienced during the observation period; and (2) the proportion of time (logarithm) a child spent in each placement type, as well as the number of placements the child experienced during the observation period.

Cognitive skills and behavior problems

Cognitive skills and behavior problems were measured at both baseline and follow-up. Children's cognitive skills were assessed using the Kaufman Brief Intelligence Test (K-BIT; Kaufman & Kaufman, 1990). The K-BIT measures verbal and non-verbal intelligence for individuals ranging in age from 4 to 90. The test is comprised of vocabulary (word knowledge and concept formation) and matrices (ability to perceive relationships and complete analogies) subtests and has demonstrated adequate internal consistency and test-retest reliability among children and adolescents (Kaufman & Kaufman, 1990). Among the NSCAW sample, internal consistency was .76 for the vocabulary subtest and .79 for the matrices subtest. Behavioral problems were assessed by children's raw scores on the internalizing and externalizing behavior problems subscales of the Child Behavior Checklist (CBCL) for children ages 4 to 18 (Achenbach, 1991). The CBCL was completed by the child's primary caregiver (usually the child's biological or foster mother) at each time point. The internalizing subscale (α = .91) measures withdrawn behaviors, somatic complaints, and anxious/depressed behaviors; the externalizing subscale (α = .92) measures delinquent and aggressive behaviors. Each of these instruments has favorable psychometric properties and has been widely used in prior research.

Control variables

Our models controlled for a rich set of covariates. Baseline demographic characteristics included continuous measures of child age, caregiver age, and the family's income-to-poverty ratio, as well as indicators (1 = yes) for whether the child was female; the child was Black, Hispanic, or of another race/ethnicity (with White children serving as the reference category); the primary caregiver was single (versus married); the primary caregiver was born outside of the U.S.; the primary caregiver's education was less than high school, high school (reference category), or more than high school; and a grandparent was present in the household. Baseline child maltreatment and CPS involvement measures included a 24-item family of origin risk score that was completed by the caseworker in the initial CPS investigation (α = .71); indicators (1 = yes) for whether the initial CPS investigation included physical abuse, sexual abuse, neglect-failure to provide, neglect-failure to supervise, and other maltreatment; an indicator for whether the initial investigation was substantiated; and an indicator for whether the focal child experienced out-of-home placement prior to the baseline assessment. The family of origin risk score was comprised of a series of indicators at the child (prior maltreatment reports, poor ability to self-protect, special needs), primary or secondary caregiver (substance abuse or mental health problems, criminal justice involvement, cognitive impairments, poor physical health, poor parenting, unrealistic expectations for the child, history of maltreatment, excessive discipline, recognition of problems in the home and motivation to change them, reasonable level of cooperation with CPS), and family (high stress, low social support, trouble accessing basic necessities, domestic violence) levels. We standardized this measure to have a mean of 0 and standard deviation of 1. Finally, we controlled for the number of months between a child's baseline and follow-up assessments in all models.

Missing Data

Like most longitudinal studies, NSCAW contains a considerable amount of missing data. Although nonresponse analyses suggested that missing data in NSCAW “is unlikely to be consequential for most types of analyses” (USDHHS, 2005, p. 2-12), complete case analysis can nonetheless lead to biased estimates (Little & Rubin, 1987). Furthermore, had we performed complete case analyses based on all of the variables included in our study, our sample size would have been limited to 1,610 children (249 of whom experienced out-of-home placement). Were we to conduct complete case analyses for each outcome, our sample sizes would have ranged from 1,626 to 1,794 children. Doing so, however, would have impeded comparisons across analyses as each would be based on a different sample. Thus, we imputed missing data for our full analysis sample using multiple imputation techniques (Allison, 2002; Rubin, 1987; Schafer, 1997). Multiple imputation is based on more plausible assumptions than complete case analysis. Complete case analysis assumes that the subsample of complete cases essentially comprises a random draw from the original sample such that each case has an equal probability of missing data; multiple imputation assumes only that data are missing at random such that the probability of missing data on a variable is unrelated to the value of that variable after taking into account the other variables in the analysis. Analyses of missing data in our analysis sample (not shown) revealed that grandparent presence in the household of origin, having a non-U.S. born caregiver, and experiencing an out-of-home placement prior to the baseline assessment were associated with an increased probability that a child had missing values on one or more variables; having an initial report for neglect-failure to provide and being observed for a slightly shorter duration between baseline and follow-up were associated with a decreased probability of missing data. Neither out-of-home placement experiences during the period of observation, nor any of the baseline or follow-up measures of cognitive skills or behavior problems was associated with the likelihood that a child had missing data, increasing our confidence that multiple imputation was an appropriate strategy. We used Stata's ice program to impute 5 datasets using all variables included in our analyses, and its mim program to conduct our analyses.

Empirical Strategy

We estimated the effects of out-of-home placement on cognitive skills and behavior problems for children over (on average) a two and a half year observation period utilizing five analytic methods: (1) OLS regressions with a rich set of controls, (2) residualized change models, (3) simple change models, (4) difference-in-difference models, and (5) fixed effects models. Each model was estimated using our full analysis sample and a sample generated by propensity score matching. As discussed above, adjusting for unobserved differences between children experiencing placement and those remaining in-home, and thus ensuring comparable treatment and comparison groups, poses a major challenge to identifying causal effects of out-of-home placement. Because OLS regression adjusts only for observable factors, results from the (particularly unmatched) OLS models will reflect more bias than those from the change models. As such, we present unmatched OLS results primarily for comparison with those of the matched OLS models and the change models. Although each of the additional methods should operate to reduce omitted variable bias in a similar way, each relies on different assumptions with associated shortcomings, such that there is likely to be variation in results across methods. Furthermore, none of the methods can be assumed to independently produce causal estimates; however, a comparison of estimates across models may provide insight as to whether associations between out-of-home placement and child well-being are likely to be causal.

We first estimated OLS regressions that related the level of well-being at follow-up to whether a child experienced out-of-home placement between baseline and follow-up, net of a host of background factors. The general form of these models was:

$${\text{CW}-\text{F}}_i=\alpha +{\beta }_1{\text{OOH}}_i+{\beta }_2{\mathbf{\text{X}}}_i+{\epsilon }_i$$ (1)

where CW-F_i is a cognitive skills or behavior problems measure for child i at follow-up; OOH_i is the key independent variable, whether child i experienced an out-of-home placement between baseline and follow-up; X_i is a vector of covariates; and ε is a disturbance term. β₁ is the “treatment effect” of out-of-home placement on well-being at follow-up. We first estimated this model controlling only for the number of months between the baseline and follow-up assessments. We then added baseline demographic characteristics, followed by child maltreatment and CPS involvement factors (in two steps) in order to assess the extent to which the associations of interest varied with the inclusion of more extensive controls.

Despite the inclusion of a host of controls, estimates produced by the OLS models are subject to bias due to unmeasured factors that are correlated with both placement and child well-being. These factors may partially be reflected in children's baseline scores on the outcomes. To attempt to account for such bias, the second step of our analysis consisted of what we refer to as residualized change models (NICHD & Duncan, 2003), but which have also been referred to as lagged dependent variable and regressor variable methods (Allison, 1990), of the form:

$${\text{CW}-\text{F}}_i=\alpha +{\beta }_1{\text{OOH}}_i+{\beta }_2{\mathbf{\text{X}}}_i+{\beta }_3{\text{CW}-\text{B}}_i+{\epsilon }_i$$ (2)

where CW-B_i is the baseline measure of the outcome (CW-F_i). In equation (2), well-being at follow-up is modeled as a function of out-of-home placement, baseline well-being, and the full set of controls. The initial well-being score functions as a proxy for unobserved pre-existing differences between children. Its inclusion in the model serves to adjust for the average influence of baseline well-being on later well-being, assuming that the baseline measure (and any associated unobserved factors) has an identical effect on the follow-up measure for children who did and did not experience placement. That is, the model adjusts for persistent child characteristics (e.g., genetic factors) that have consistent effects on CW-B_i and CW-F_i for children in both groups. Resulting estimates are less subject to bias than those produced by the standard OLS model. The treatment effect is interpreted as the effect of out-of-home placement on child well-being at follow-up, net of initial well-being; the model does not directly estimate the effect of placement on changes in well-being between baseline and follow-up.

The third step in our analysis was to estimate simple change models (Allison, 1990; NICHD & Duncan, 2003; Rogosa, Brandt, & Zimowski, 1982; Rogosa & Willett, 1983) that assessed the influence of out-of-home placement on changes in well-being, net of a host of covariates. We did so using the following equation:

$$\Delta {\text{CW}}_i=\alpha +{\beta }_1{\text{OOH}}_i+{\beta }_2{\mathbf{\text{X}}}_i+{\epsilon }_i$$ (3)

where ΔCW_i is the difference between a child's follow-up and baseline well-being scores. This model uses variation between children to identify differences in the average change in well-being over time for those children who did and did not experience placement. Coefficients are interpreted as the effects of the predictor variables on changes in the outcome. The model provides a significant advantage over the OLS and residualized change models by reducing bias associated with unobserved factors that have the same effect on well-being at both time points, while also allowing observed fixed characteristics to have time varying effects (i.e., to directly affect the change in well-being). Although the simple change model should reduce bias to a greater extent than the OLS and residualized change models, it has less precision to detect associations and does not adjust for unobserved variables that differentially affect well-being at different points in time.

Our fourth and fifth steps consisted of estimating difference-in-difference and fixed effect models. Both required that our data be organized in child-wave observations. As such, for these analyses we utilized 4,906 observations in which each child was represented twice (once at baseline and once at follow-up). The difference-in-difference model allowed us to compare the average effect of placement between non-identical treatment and comparison groups while accounting for both within group differences over time and between group differences at each time point. The model operates on the assumption that the relative difference in well-being between groups at each time point will be statistically equivalent in the absence of out-of-home placement. The approach removes bias that could be due to stable differences between the two groups, as well as that which could be due to time trends. We used these models to assess whether the average change in well-being between baseline and follow-up was equivalent for children who did and did not experience placement. The treatment effect (difference-in-difference estimator) is computed by subtracting the average change in the outcome between the two time points for the control group from the average change in the outcome for the treatment group (Murray, 2006). The model is similar to the simple change model except that it adjusts for average initial differences at the group, rather than individual, level. We estimated models of the form:

$${\text{CW}}_{it}=\alpha +{\beta }_1{\text{OOH}}_i+{\gamma }_1{\text{FU}}_{it}+{\delta }_1{\text{OOH}}_i\ast {\text{FU}}_{it}+{\beta }_2{\mathbf{\text{X}}}_i+{\epsilon }_{it}$$ (4)

where CW_it is a well-being measure for child i at time t (baseline or follow-up); OOH_i is an indicator that a child experienced placement, and is equal to 1 at both time points for children who did so; FU_it is an indicator (1 = yes) of whether a given observation occurred at follow-up; and OOH_i*FU_it is the interaction between out-of-home placement and the observation occurring at follow-up. Here, β₁ is a group difference in well-being that is assumed to be constant over time, such that it reflects the initial difference in well-being between children who were subsequently placed out-of-home and those who remained in-home (it does not reflect the effect of out-of-home placement on well-being); γ₁ is the average change in well-being between baseline and follow-up for all children (regardless of out-of-home placement status); and δ₁ is the treatment effect or difference-in-difference estimate, which reflects the extent to which the average change in well-being between baseline and follow-up differs for children who did and did not experience out-of-home placement. The difference-in-difference estimate is computed as $(\overline{\text{CW}}(\text{OOH}=1,\text{FU}=1)-\overline{\text{CW}}(\text{OOH}=1,\text{FU}=0))-(\overline{\text{CW}}(\text{OOH}=0,\text{FU}=1)-\overline{\text{CW}}(\text{OOH}=0,\text{FU}=0))$, where $\overline{\text{CW}}$ is the adjusted group mean on the outcome. We corrected the standard errors produced by the difference-in-difference models for intra-cluster correlation due to multiple observations of each child. In the tables that follow, we report estimates for only δ₁, which reflect the (treatment) effects of out-of-home placement on changes in well-being over time; we do not present estimates for β₁ or γ₁.

Although the difference-in-difference model reduces bias by accounting for pre-existing group differences in well-being and assessing whether the treatment and control groups experience similar changes in well-being over time, estimates will be biased to the extent that unobserved factors (e.g., a CPS policy change during the observation period that resulted in additional services for only one group of children) differentially affect changes over time in well-being for the two groups. Like the simple change model, the difference-in-difference model allows permanent characteristics to have time varying effects on the outcome (i.e., such characteristics are not differenced out of the model).

All of the change models described thus far use between-child variation in placement status to identify differences in the average change in well-being over time for children who did and did not experience placement. In contrast, fixed effects models identify effects via intra-child change for children observed both in-home and during or after out-of-home placement (Duncan, Magnuson, and Ludwig 2004). The method expresses each variable as a deviation from a child's mean value (across time) on that measure and differences the regression equation across time periods. This eliminates all time invariant observed and unobserved variables from the model. However, fixed effects estimates are subject to bias if unobserved variables (or their effects on the outcome) are time-varying, or if permanent characteristics have time-varying effects. Our fixed effects models took the following form:

$$\Delta {\text{CW}}_i=\Delta {\beta }_1{\text{OOH}}_i+\Delta {\beta }_2{\mathbf{\text{X}}}_i+\Delta {\epsilon }_i$$ (5)

where β₁, the treatment effect, is identified only for children who experienced placement.

Because 4 of the 5 models presented above identify effects through between-child variation in placement status, ensuring that our models are estimated for similar treatment and comparison groups with adequate overlap on their covariate distributions is a pressing concern. For this reason, the final step in our analysis consisted of re-estimating all of the models described above after using propensity score matching methods (Rosenbaum and Rubin, 1983) to construct treatment and comparison groups that were statistically equivalent with regard to all background characteristics. The primary advantage of this strategy is that it restricts inference to treatment and comparison samples with adequate overlap in the covariate distributions, thereby avoiding unwarranted model extrapolation. In addition, inferences drawn from the matched sample are specifically applicable to children on the margin of removal—the most relevant group among which to assess the impact of out-of-home placement on well-being. For models that compare well-being across children (all but the fixed effects models), this strategy allowed us to more rigorously adjust for selection bias by estimating differences between comparable groups of children. For the fixed effects models, it allowed us to estimate within-child change in well-being for those children who experienced placement and were also most like those who did not.

To construct the matched treatment and comparison groups, we first used a probit regression to estimate each child's conditional probability (i.e., “propensity score”) of being placed out-of-home based on all of our covariates (child age, race, and gender; caregiver age, marital status, education, and nativity; grandparent presence; family income-to-poverty ratio and risk score; types of maltreatment alleged; whether the initial report was substantiated; and whether the child experienced an out-of-home placement prior to the initial assessment), as well as children's baseline vocabulary, matrices, internalizing behavior problems, and externalizing behavior problems scores. We then used one-to-one matching without replacement to match each treatment (out-of-home placement) group child to the comparison group child with the closest propensity score, thereby limiting the sample to treatment and control children for whom there was sufficient overlap in propensity scores (common support). Unmatched children were discarded from the sample. Additionally, we trimmed the 10% of treatment observations at which the propensity score density of the comparison group was the lowest. After matching the treatment and comparison groups, we conducted additional analyses (not shown) to ensure adequate balance across their covariate distributions and found no significant differences on any of the background characteristics between the groups. The resulting matched sample (N = 616) was comprised of 308 children who experienced placement and 308 children who did not. These analyses were performed using Stata's psmatch2 program.

Because our analysis sample was not representative of the full NSCAW sample, the sample weights were not applicable and, therefore, not used; this should not influence our results aside from limiting their generalizability (Barth, Gibbons, & Guo, 2006), as discussed above.

Results

Descriptive Statistics

Table 1 presents descriptive statistics for out-of-home placement characteristics for children who experienced placement, as well as baseline demographic characteristics and child maltreatment and CPS involvement measures for children who did and did not experience placement. Approximately 14% of the children in our analysis sample experienced out-of-home placement between their baseline and follow-up assessment. On average, children who were removed from home spent 48% of the observation period out-of-home and experienced slightly more than 2 placements; they spent 17% of the observation period in non-kin foster homes, 17% in kin foster homes, and 13% in other types of placement.

Table 1.

Sample Characteristics

	No Out-of-Home Placement		Out-of-Home Placement
	M (SD)	%	M (SD)	%
Placement characteristics:
Proportion of time in out-of-home placement			0.48 (0.30)
Number of out-of-home placements			2.30 (1.78)
Proportion of time in non-kin foster home			0.17 (0.25)
Proportion of time in kin foster home			0.17 (0.29)
Proportion of time in other placement			0.13 (0.24)
Baseline demographic characteristics:
Child age (months)	113.42 (38.73)		117.32 (41.93)
Caregiver age (years)	34.06 (7.65)		35.86*** (9.63)
Family income-to-poverty ratio	1.21 (0.81)		1.08** (0.84)
Child is female		0.53		0.48
Child is white		0.48		0.48
Child is Black		0.25		0.28
Child is Hispanic		0.18		0.16
Child is another race		0.08		0.07
Caregiver is single		0.66		0.68
Grandparent present		0.09		0.10
Caregiver not US born		0.11		0.05***
Caregiver education is less than HS		0.28		0.34*
Caregiver education is HS graduate		0.45		0.45
Caregiver education is more than HS		0.26		0.21
Child maltreatment and CPS involvement:
Family risk score (standardized)	-0.17 (0.91)		0.33*** (0.95)
Initial investigation for physical abuse		0.34		0.38
Initial investigation for sexual abuse		0.21		0.14**
Initial investigation for neglect, failure to provide		0.22		0.26
Initial investigation for neglect, failure to supervise		0.37		0.39
Initial investigation for other maltreatment		0.22		0.26
Initial report was substantiated		0.53		0.64***
Out-of-home placement prior to baseline		0.06		0.03*
Other sample characteristics:
Baseline assessment at wave 3		0.07		0.03**
Follow-up assessment at wave 3		0.09		0.10
Months between baseline and follow-up	29.69 (6.28)		31.23** (5.75)
Observations	2,111		342

Note: 2,453 observations. Means (M) and standard deviations (SD) or percentages (%) presented. 14% of children experienced out-of-home placement.

^*p<0.05;

^**p<0.01;

^***p<0.001.

The raw data also indicated that children who experienced placement differed from those who did not on several important background characteristics. At baseline, children who were removed from home were more likely to have had a U.S. born caregiver. They also had older caregivers with lower levels of educational attainment, as well as lower family income-to poverty ratios and higher family risk scores. Finally, they were less likely both to have had an initial investigation that was for sexual abuse and to have been removed from home prior to their baseline assessment, but more likely to have had their initial CPS investigation substantiated.

Table 2 presents mean baseline and follow-up scores on the outcome variables for the full sample of children and for those who did and did not experience placement. The raw data indicate that, on average, children in our sample experienced increases in vocabulary and matrices skills, and decreases in internalizing and externalizing behavior problems between baseline and follow-up. Mean vocabulary and matrices scores did not significantly differ at either baseline or at follow-up for children who were removed from home and those who were not; children in both groups experienced significant (and similar) gains in cognitive skills between the two time points. Raw data for behavior problems, however, indicated that children who were not removed from home had significantly fewer internalizing and externalizing difficulties at both baseline and follow-up than children who experienced placement. In addition, children remaining in-home exhibited decreases in both internalizing and externalizing behavior problems between baseline and follow-up, whereas children who were removed from home displayed similar levels of internalizing behavior problems at both time points, but fewer externalizing behavior problems at follow-up than at baseline.

Table 2.

Descriptive Statistics for Child Cognitive Skills and Behavior Problems Measures

	Full Sample		No Out-of-Home Placement		Out-of-Home Placement
	Baseline	Follow-up	Baseline	Follow-up	Baseline	Follow-up
	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)
Outcomes:
Vocabulary	37.08 (14.66)	45.21a (11.95)	37.06 (14.59)	45.19a (11.87)	37.16 (15.07)	45.32a (12.43)
Matrices	23.34 (8.33)	27.87a (7.10)	23.39 (8.31)	27.87a (7.05)	23.03 (8.50)	27.85a (7.42)
Internalizing behavior problems	9.73 (8.31)	8.82a (7.68)	9.33 (8.06)	8.42a (7.48)	12.16b (9.38)	11.29b (8.41)
Externalizing behavior problems	15.54 (11.05)	13.74a (10.47)	14.66 (10.42)	13.10a (10.03)	21.00b (13.10)	17.72a,b (12.09)
Observations	2,453		2,111		342

Note: Means (M) and standard deviations (SD) presented. 14% of children experienced out-of-home placement.

^aSignificantly different from baseline mean at p < 0.05.

^bSignificantly different from “no out-of-home placement” group mean at p < 0.05.

To further examine associations of baseline cognitive skills and behavior problems with out-of-home placement, we estimated a series of models in which an indicator (1 = yes) for whether a child experienced out-of-home placement was regressed on a baseline cognitive skills or behavior problems measure and the full set of baseline covariates. Results (not shown) suggested that, after accounting for these selection factors, lower baseline cognitive skills and higher baseline behavior problems were associated with an increased likelihood of placement. This finding underscores the importance of adjusting for pre-existing differences when estimating effects of placement on child outcomes. We next present results from a series of regression models in which we employed multiple strategies of adjusting for selection factors.

Regression Results

Full sample

Table 3 presents regression results when the full (unmatched) sample was employed. Models 1 through 3 are the OLS regressions controlling for an increasingly detailed set of covariates. Models 4 through 6 are the residualized change, simple change, difference-in-difference, and fixed effects models. With regard to cognitive skills, results from Model 1, in which we controlled only for the number of months between the baseline and follow-up assessments, were consistent with the patterns indicated by the raw data: We found no significant differences in cognitive skills at follow-up between children who experienced placement and those who did not. This pattern held across all 7 models. Results for behavior problems were also consistent with the pattern found in the raw data: Model 1 revealed associations of placement with higher levels of both internalizing and externalizing behavior problems at follow-up. Although these associations were modestly attenuated with the addition of baseline demographic and child welfare case characteristics to the models, they retained statistical significance even after the full set of controls was included (Model 3). Results from the residualized change model (Model 4) suggested that, after controlling for baseline scores on the outcome measures, placement continued to be associated with higher internalizing behavior problems at follow-up, but was not associated with externalizing behavior problems. In contrast, results from the simple change, fixed effects, and difference-in-difference models indicated that placement was not associated with internalizing behavior problems, whereas results from the simple change and fixed effects models indicated that children experiencing placement exhibited a decrease in externalizing behavior problems between baseline and follow-up (the negative coefficient from the difference-in-difference model for externalizing behavior problems was also marginally significant at p = .066). Effect sizes, computed by dividing the relevant coefficient by the standard deviation (SD) of externalizing behavior problems for the full sample at follow-up (10.47), suggested that placement was associated with a 0.14 to 0.15 SD decrease in externalizing behavior problems between baseline and follow-up. On the whole, these results imply that, once unobserved time invariant selection factors have been taken into account, placement has a modest protective effect with regard to externalizing behavior problems and a neutral effect with regard to both cognitive skills and internalizing behavior problems.

Table 3.

OLS and Change Models of the Effects of Out-of-Home Placement on Subsequent Cognitive Skills and Behavior Problems, Unmatched Sample

	Vocabulary	Matrices	Internalizing	Externalizing
	B (SE B)	B (SE B)	B (SE B)	B (SE B)
Model 1: Controlling only for number of months between baseline and follow-up assessments
Out-of-home placement	0.11 (0.71)	-0.03 (0.43)	2.88** (0.45)	4.71** (0.62)
Model 2: Add baseline demographic characteristics
Out-of-home placement	-0.95 (0.47)	-0.33 (0.35)	2.58** (0.46)	4.18** (0.62)
Model 3: Add maltreatment and CPS involvement measures
Out-of-home placement	-0.99 (0.48)	-0.29 (0.35)	2.17** (0.46)	3.78** (0.62)
Model 4: Residualized change model
Out-of-home placement	-0.16 (0.35)	0.12 (0.29)	1.17** (0.41)	0.90 (0.52)
Model 5: Simple change model
Out-of-home placement	0.11 (0.37)	0.45 (0.32)	0.02 (0.48)	-1.61* (0.60)
Model 6: Difference-in-difference model
Out-of-home placement*Follow-up	-0.75 (0.42)	0.08 (0.35)	0.04 (0.59)	-1.51 (0.82)
Model 7: Fixed effects model
Out-of-home placement	-0.50 (0.43)	0.10 (0.35)	0.18 (0.47)	-1.45* (0.60)

Note: 2,453 observations. Regression coefficients (B) and standard errors (SE B) presented. Model 1controls only for the number of months between the baseline and follow-up observations. Model 2 includes additional controls for child age, caregiver age, family income-to-poverty ratio, child gender, child race and ethnicity, caregiver marital status, grandparent present, caregiver not US born, and caregiver education. Models 3 through 7 include additional controls for family risk score, maltreatment types at initial investigation, whether the initial report was substantiated, and whether the child experienced an out-of-home placement prior to his or her baseline assessment. Note, however, that time invariant measures (child gender, child race and ethnicity, caregiver not US born, family risk score, maltreatment types at initial investigation, whether the initial report was substantiated, whether the child experienced an out-of-home placement prior to his or her baseline assessment, and number of months between the baseline and follow-up assessments) were differenced out of the fixed effects models (Model 7) such that these effects were not directly estimated under this specification. Standard errors from the difference-in-difference models (Model 6) were corrected for intra-cluster correlation due to multiple observations of each child.

^*p<0.05;

^**p<0.01;

^***p<0.001.

Matched sample

Table 4 presents results from models 3 through 7 when estimated using the matched sample. These results are quite different from those shown in Table 3. Indeed, regardless of the method used to estimate associations of out-of-home placement with children's cognitive skills and behavior problems, we found no significant estimates when the matched sample was employed. A potential concern here is that we lacked the statistical power to detect effects in the matched sample given its considerably reduced sample size. However, an examination of tables 3 and 4 revealed that the differences in estimates were not solely due to imprecise estimation (larger standard errors) when the matched sample was employed; in many cases the coefficients themselves differed substantially across the tables suggesting that adjusting for differences in background characteristics via propensity score matching accounted for all remaining associations of out-of-home placement with cognitive skills and behavior problems.

Table 4.

OLS and Change Models of the Effects of Out-of-Home Placement on Subsequent Cognitive Skills and Behavior Problems, Matched Sample

	Vocabulary	Matrices	Internalizing	Externalizing
	B (SE B)	B (SE B)	B (SE B)	B (SE B)
Model 3: OLS model with full set of controls
Out-of-home placement	-0.55 (0.71)	-0.03 (0.52)	1.25 (0.71)	1.19 (1.04)
Model 4: Residualized change model
Out-of-home placement	-0.33 (0.61)	0.03 (0.50)	1.26 (0.63)	1.22 (0.89)
Model 5: Simple change model
Out-of-home placement	-0.25 (0.66)	0.09 (0.59)	1.27 (0.77)	1.28 (1.04)
Model 6: Difference-in-difference model
Out-of-home placement*Follow-up	-0.66 (0.73)	-0.05 (0.59)	1.20 (0.79)	1.54 (1.09)
Model 7: Fixed effects model
Out-of-home placement	-0.60 (0.73)	0.11 (0.56)	1.36 (0.77)	1.59 (1.08)

Note: 616 matched observations. Regression coefficients (B) and standard errors (SE B) presented. All models control for the number of months between the baseline and follow-up observations, child age, caregiver age, family income-to-poverty ratio, child gender, child race and ethnicity, caregiver marital status, grandparent present, caregiver not US born, caregiver education, family risk score, maltreatment types at initial investigation, whether the initial report was substantiated, and whether the child experienced an out-of-home placement prior to his or her baseline assessment. Note, however, that time invariant measures (child gender, child race and ethnicity, caregiver not US born, family risk score, maltreatment types at initial investigation, whether the initial report was substantiated, whether the child experienced an out-of-home placement prior to his or her baseline assessment, and number of months between the baseline and follow-up assessments) were differenced out of the fixed effects models (Model 7) such that these effects were not directly estimated under this specification. Standard errors from the difference-in-difference models (Model 6) were corrected for intra-cluster correlation due to multiple observations of each child. Treatment and comparison group children were matched on all covariates including the baseline measures of cognitive skills and behavior problems.

^*p<0.05;

^**p<0.01;

^***p<0.001.

Extensions

We conducted a series of extensions (results not shown) to our primary analyses. First, we re-estimated the internalizing and externalizing behavior problems models using the clinical cutoffs for these measures as outcome variables. Results from both the OLS model with the full set of controls and the residualized change model revealed associations of out-of-home placement with clinical levels of both internalizing and externalizing behavior problems when the unmatched sample was employed. However, we found no associations between out-of-home placement and clinical levels of behavior problems when estimated via any of the other methods in the unmatched sample. Likewise, we found no significant associations when the matched sample was employed, regardless of the estimation method utilized.

Second, because children experiencing what we have categorized as ‘other’ types of placements (i.e., other than family foster homes) are likely to differ from those placed in family foster homes on a range of background characteristics (Davidson-Arad et al., 2003; McDonald et al., 1996), we estimated supplemental models that excluded this group of children. Likewise, we estimated models that excluded the 156 children (6%) who experienced out-of-home placement between the time of their initial CPS investigation and the time of their baseline assessment. In both cases, results were qualitatively consistent with our primary findings.

Finally, we exploited our matched sampled to examine whether associations between out-of-home placement and child well-being may vary by a child's propensity for removal. Here we divided the matched sample into subsamples of children whose propensities for removal were above and below the median. We then re-estimated all of our models (separately) for each subsample. We found no significant associations in any of the models when estimated using either subsample. We note, however, that this may reflect a lack of statistical power through which to detect effects given the limited size (N = 308) of each subsample.

Placement characteristics

Finally, to examine whether relations of out-of-home placement and child outcomes differed by placement length, stability, and type, we estimated models (results not shown) that (1) allowed these associations to vary by the proportion of the observation period that a particular child spent out-of-home (logarithm), as well as the number of out-of-home placements a child experienced, and (2) allowed them to vary by the proportion of time a child spent in each type of placement (non-kin foster home, kin foster home, or other), as well as the number of out-of-home placements a child experienced. We found no associations of placement length or stability with any of the outcomes when the matched sample was employed.

Turning to placement type, when the matched sample was employed, we found that time spent in ‘other’ (non-foster home) placements was associated with increases in both internalizing and externalizing behavior problems when estimated by the OLS model with the full set of controls. We also found associations of time in ‘other’ placements with increases in internalizing behavior problems when estimated by both the residualized and simple change models, and with decreases in vocabulary skills when estimated by the difference-in-difference model. Finally, results from the fixed effects models revealed that time spent in both kin and non-kin foster placements was associated with increased vocabulary and math skills, and that time spent in non-kin foster homes was associated with decreased externalizing behavior problems.

Discussion

This study used NSCAW data and five analytic methods of adjusting for selection factors to estimate the impact of out-of-home placement on child well-being among 4 to 17 year old children using both matched and unmatched treatment and comparison samples. We first demonstrated that children who were removed from home differed from those who were not on a host of background factors. This confirmed that there was differential selection into placement and highlighted the importance of adjusting for selection factors, including baseline well-being scores, when attempting to estimate effects of out-of-home placement on child well-being.

We then estimated a series of models that increased in methodological rigor and ability to adjust for selection factors. We found no significant effects of out-of-home placement on cognitive skills in any of the models. With regard to behavior problems, when the unmatched sample was employed, our basic OLS results indicated that children who experienced placement exhibited significantly more internalizing and externalizing behavior problems at follow-up than children who did not. However, estimates produced by the change models did not support these findings. Results from the residualized change models suggested an association between placement and increased internalizing behavior problems, but no relationship between placement and externalizing behavior problems. Results from the simple change and fixed effects models, both of which are likely to reduce bias to a greater extent than the OLS and residualized change models, revealed associations between placement and decreases in externalizing behavior problems, thus raising considerable doubt that estimates produced by OLS and residualized change models lend themselves to causal interpretation.

Next, we used propensity score matching methods to construct treatment and comparison samples of children who were similar on all measured background characteristics and differed only in terms of whether they experienced out-of-home placement. We then re-estimated all of our models using the matched sample and found no significant associations between placement and either children's cognitive skills or their behavior problems. This did not appear to primarily reflect reduced statistical precision (increased standard errors) when the matched sample was employed; rather, many of the point estimates differed substantially when estimated using the matched and unmatched samples. This pattern is consistent with Berzin's (2008) findings which, though based on a sample of young adults, also revealed associations between out-of-home placement and adverse outcomes using unmatched data, but none using matched data.

Finally, to examine the ways in which experiences during out-of-home placement may have influenced child outcomes, we estimated supplemental models that accounted for variability in placement length, stability, and type. We found no associations of placement length or stability with children's cognitive skills or behavior problems once the matched sample was employed. Notably, however, our measure of placement stability, defined only by number of out-of-home placements, is quite crude; future research should more fully explore the potential moderating role of placement stability using more refined measures of stability (see, e.g., James et al., 2004; Rubin et al., 2007).

Our supplemental analyses did reveal several associations between placement type and child outcomes, the most consistent of which suggested links between non-foster home placements and increased behavior problems. However, as noted above, placement length, stability, and type are likely to be endogenously determined with children's cognitive skills and, in particular, their behavior problems. For example, if out-of-home placement was associated with reduced externalizing behavior problems (as suggested by our simple change and fixed effects results when estimated using the unmatched sample), then longer placements would likely be associated with larger reductions in this outcome. At the same time, if placement were effective at reducing externalizing behavior problems, and such reductions caused children to return home more quickly, then shorter placements would likely be associated with greater reductions in externalizing behavior problems. Counteracting effects of this sort may help to explain our lack of findings with regard to placement length. Similarly, children's levels of cognitive ability and behavior—both prior to and during placement—are likely to influence the types of placements to which they are assigned as well as the quality of those placements. As such, including placement characteristics in models that attempt to estimate causal relations between out-of-home placement and child outcomes may not be appropriate; we have little confidence that these estimates reflect causal relations.

On the whole, our results suggest that, on average, out-of-home placement appears to neither place additional burden on the already vulnerable children who enter State custody, nor contribute to improved well-being for these children, at least in terms of short-term changes in cognitive skills and behavior problems. Lack of findings for cognitive skills may, perhaps, reflect a growing emphasis on neighborhood-based foster care (Berrick, 2006) which may prevent children who enter placement from also experiencing changes in schooling that might engender significant improvements or decrements in cognitive skills. More generally, that our overall pattern of results suggests few impacts of placement on child well-being may not be particularly surprising given that previous research comparing children placed out-of-home to similar children who remained in-home has reported both positive (Davidson-Arad, 2005; Davidson-Arad, et al., 2003) and negative (Doyle, 2007; Lawrence, et al., 2006; Stahmer et al., 2007) effects. It is possible that our analytic methods have reduced bias that may be operating in different directions across prior studies depending on the characteristics of the treatment and comparison groups, as well as the particular child outcomes and placement measures utilized.

The current study differs from prior research in several important respects, most notably in its use of multiple methods of adjusting for selection factors that may influence placement decisions and child outcomes, but also in its use of national survey data that includes children experiencing CPS involvement in a variety of child welfare policy and practice contexts. To the extent that differences in our results and those of prior research can be explained by differences in analytic methods, the results of this study may suggest that findings of earlier work likely reflected bias due to unobserved heterogeneity across groups of children who did and did not experience placement. To the extent that differences in results reflect differences in samples utilized (i.e., local versus national), findings from prior studies may have been influenced by idiosyncratic aspects of the (local) macro-level environments in which children experienced out-of-home placement.

This study also has several limitations. First, our sample excluded infants and toddlers, as well as children who were never observed in their homes of origin. This ensured that baseline child and family assessments for the entire sample were conducted while children were in-home and also enabled us to assess changes in well-being using identical measures at baseline and follow-up. However, children who were excluded from our analysis sample because they were never observed in-home (i.e., were out-of-home at both the baseline and 18-month assessments) likely came from homes that were considerably less safe than those of the children included in our sample. As such, our findings may not be generalizable to children with the highest likelihood of being removed from home and spending a long period of time in care. Moreover, associations between out-of-home placement and child well-being may be very different for infants and young toddlers than for older children, an issue that warrants attention in future research.

Second, our outcome measures reflect only limited aspects of well-being. We focused on children's cognitive skills and behavioral problems, but did not consider the extent to which out-of-home placement affects other domains of functioning. Additionally, both the K-BIT and CBCL have been constructed such that they are appropriate for use with a broad age range of children. They may therefore lack the sensitivity or specificity to precisely detect some effects of placement on children's cognition or behavior during various stages of development. Furthermore, our behavior problems measures were drawn from the caregiver-reported version of the CBCL. As such, a large proportion of reports on children who experienced placement were provided by different reporters at baseline and follow-up (about 84%, as compared to 35% for children who did not experience placement). Notwithstanding the favorable psychometric properties and widespread use of the CBCL, utilizing reports from multiple informants at different time points may be problematic (Achenbach, McConaughy, & Howell, 1987). However, this is a common problem in studies of the effects of out-of-home placement on child behavior (Newton et al., 2000) and we have no reason to expect that it would bias our results more so than those of prior work. Nonetheless, future research would benefit from observational measures of child behavior provided by the same reporter(s) at multiple points in time. Future research that considers a wider range of outcomes with regard to major developmental tasks will also help to provide a more complete picture of how placement may influence children.

Third, our results are based on only a two and a half year observation period (on average). This may be too short a window to observe the full range of effects of placement on child outcomes, as such effects may manifest over time. Existing studies using rigorous methods to adjust for selection factors have produced mixed evidence regarding the long-term effects of placement. For example, Doyle (2007) used instrumental variables models to adjust for selection bias and found strong links between placement during childhood and a range of adverse outcomes in adulthood, whereas Berzin (2008) used propensity score matching methods and found none. Thus, additional research on the long-term effects of placement is warranted.

Fourth, there is likely to be considerable heterogeneity in associations of placement with children's cognitive skills and behavior problems that is obscured in our analyses. In particular, there may be differences in these associations by child age and gender (Horowitz, Balestracci, and Simms, 2001), as well as race and ethnicity (Jonson-Reid and Barth, 2000; Keller et al., 2001) and the types and severity of maltreatment children have experienced (Myers et al., 2002). In supplemental analyses, we attempted to examine whether any associations of placement with child cognitive skills and behavior problems varied by each of these factors using both the full and matched samples. To do so, we estimated a series of separate models in which child age at the time of removal, gender, race and ethnicity, maltreatment type, and (caseworker reported) severity of maltreatment risk were individually interacted with out-of-home placement. Although the models revealed several significant interaction terms, we found no consistent or theoretically relevant pattern by any of these factors. However, our confidence in these estimates was limited both because the cell sizes for these analyses were quite small and because we conducted a large number of statistical tests, potentially increasing the likelihood that we would identify significant effects in error. Furthermore, given the inconsistent pattern of these results and the lack of prior theory or evidence to support them, we could not rule out that they were due to chance. Future work would benefit from further exploring potential heterogeneity in associations of out-of-home placement and child well-being by demographic and maltreatment-related factors.

Finally, this study focused on estimating the “full” associations of out-of-home placement with children's cognitive skills and behavior problems, but did not explore the mechanisms through which such associations may operate. For example, we did not investigate whether placement influenced the cognitive stimulation, warmth, or support children received. Yet, these factors are important pathways through which any associations of placement with child outcomes are likely to occur. Future research on potential mechanisms, such as the quality of various substitute care arrangements and the degree of continuity in other aspects of children's lives, including peers, schools, and neighborhoods, is vital for understanding the circumstances under which placement may mitigate or heighten children's well-being.

Conclusion

The results of this study suggest that, once selection factors are taken into account, out-of-home placement has little influence on child outcomes, at least in the short-term. That placement does not appear to help or harm children in the domains of cognitive functioning or behavior problems may reflect the continued influence of other ongoing risks or of early maltreatment experiences and other ecological adversities on child well-being. It may also reflect that the child welfare system has historically focused on promoting safety and permanency for children, rather than child well-being. Our findings both that estimates produced by OLS models frequently differ from those produced by change models, and that estimates produced using unmatched observations frequently differed from those produced using matched observations, underscore the importance of rigorously adjusting for selection factors when estimating associations of out-of-home placement with child well-being.

Though based on a relatively limited set of well-being indicators, our findings also have implications for child welfare policy and practice. Although the primary focus of the child welfare system continues to be promoting safety and permanency, CPS is now also explicitly charged with improving child well-being (Wulczyn et al., 2005). To the extent that out-of-home placement does little to positively influence children's development, CPS agencies should ensure that decisions to remove children from home continue to be driven solely by concerns for child safety. Out-of-home placement is an integral and necessary means through which CPS aims to protect children from harm in their own homes. Our results should not be interpreted to suggest that CPS should move away from this form of intervention, nor that placement decisions should consider aspects of child well-being beyond assuring that children are protected from abuse and neglect. Furthermore, it is important to acknowledge that, although out-of-home placement may, on average, have little influence on children's well-being, there is likely to be considerable variation in how individual children respond to placement, which should be reflected in CPS guidelines and practices with regard to children who have been removed from home due to safety concerns. As such, if the child welfare system intends to actively engage in promoting child well-being, CPS agencies should carefully assess and balance the needs of individual children when designating supportive services for children in out-of-home care. Developmental research can play an important role in this effort by specifying mechanisms of risk and resilience for children in out-of-home placement.