Decision-making in Child Protective Services: Influences at multiple levels of the social ecology

Abstract

Decision-making in the child protection system is influenced by multiple factors; agency and geographic contexts, caseworker attributes, and families' unique circumstances all likely play a role. In this study, we use the second cohort of the National Survey of Child and Adolescent Well-Being to explore how these factors are associated with two key case decisions—substantiation and removal to out-of-home care. Analyses are conducted using weighted hierarchical linear models. We find that substantiation is strongly influenced by agency factors, particularly constraints on service accessibility. Substantiation is less likely when agencies can provide services to unsubstantiated cases and when collaboration with other social institutions is high. This supports the concept that substantiation may be a gateway to services in some communities. Agency factors contributed less to the probability of removal among substantiated cases, though time resources and constraints on decision-making had some influence. For both substantiation and removal risks, county, caseworker, and child characteristics were less influential than agency characteristics and family risk factors.

Rates of confirmed child maltreatment and out-of-home care placement vary across every level of geography. In 2012, the incidence of substantiated (confirmed) child maltreatment ranged from a low of 1.2 victims per 1,000 children in the state of Pennsylvania (a state which routes the majority of neglect cases to a separate system that is external to CPS) to a high of 19.6 in the District of Columbia (U.S. Department of Health and Human Services, 2013a). That same year, the number of children in out-of-home care on September 30 ranged from 2.5 per 1,000 children in Virginia to 14.2 per 1,000 in the District of Columbia (U.S. Department of Health and Human Services, 2013b). Prior research suggests that variation in child maltreatment rates may partly reflect differences in the characteristics of different geographic regions, including poverty (Ben-Arieh, 2010; Coulton, Korbin, & Su, 1999; Drake & Pandey, 1996), concentration of different racial minorities (Freisthler, Gruenewald, Remer, Lery, Needell, 2007; Fromm, 2004; Molnar, Buka, Brennan, Holton, Earls, 2003), population size (Ben-Arieh, 2010; Deccio, et al., 1994), and others. However, differences in maltreatment substantiation and out-of-home care rates may also be influenced by variation in policy, given that state legislators are able to define child maltreatment as broadly or narrowly as they choose (Child Welfare Information Gateway, 2011). Moreover, differences in practices across agencies may contribute to variation in decision-making. Specifically, agencies face different constraints on their time, resources, and decision-making autonomy. Although these factors have received little attention to date, the organizational context warrants consideration as federal and state governments exercise explicit control over the policies of child welfare systems. That is, organizational factors can be directly altered by policy decisions to a greater degree than community or individual factors. This study contributes to the current research on risks for maltreatment substantiation and entry into out-of-home care by focusing not only on individual or family risk factors, but also on caseworker, county and agency factors. Specifically, we use multi-level modeling and a dataset that is nationally representative of child welfare investigations to examine the extent to which individual, agency and community factors contribute to the risk of substantiation and removal to out-of-home care among a sample of child protective services (CPS) investigations.

Theoretical Framework

A dedicated body of research has sought to understand how CPS workers make their decisions (e.g., Baumann, et al., 2011; Davidson-Arad & Benbenishty; 2010; Munro, 2005; Wells, Fluke, & Brown; 1995). In the current study, we seek to extend this work by relying on the Decision-Making Ecology (DME) framework to guide our understanding of how child welfare professionals make decision within the context of actual CPS operations (Baumann, et al., 2011). As shown in Figure 1, the DME framework consists of three separate components: the factors that influence decisions, the decision-making process itself, and the outcomes of the decision. In the first part of the Figure 1, the model stipulates that there are multiple influences for CPS decisions, including factors related to the individual case, the specific CPS agency (their policies and procedures, time and resource constraints, caseload size, and organizational culture), the CPS worker him/herself (education, background, personal experiences, and attitudes), and external forces (laws and attitudes; characteristics about where the family resides) (Baumann, et al., 2011).

Figure 1.

Decision-Making Ecology Framework (Baumann, et al., 2011)

The diamond that is labeled “decision-making” in Figure 1 includes both the decision-making continuum (which includes the range of decisions CPS workers make, beginning with intake and ending at case closure) as well as the psychological process of decision-making (Baumann, Fluke, Dalgleish, and Kern; 2014; Fluke, Baumann, Dalgleish, & Kern, 2014). Applied to the field of child welfare, the General Assessment and Decision-Making Model proposes that individual CPS workers have their own personal threshold for a required amount and weigh of evidence to transform a judgment into an action (e.g., the decision to substantiate) (Dalgleish, 1988). This threshold may change over time in response to different influences, such as a policy that specifies an age requirement for cases that must be accepted (organizational factor) or the experience level of the worker, with newer workers likely being more cautious (decision-maker factor) (Baumann et al., 2014; Fluke et al., 2014).

The final component of this model is the actual outcome of the decision, which, in turn, exerts influence back onto the factors that will impact the next decision. Within the DME framework, outcomes are assumed to have consequences to the family, the CPS worker, and the CPS agency (Baumann, et al.; 2014). For example, if the outcome of a decision to keep a child in the home results in maltreatment recidivism, this outcome may impact the child through experiencing additional maltreatment (case factor), the worker through experiencing distress (decision maker factor), and the CPS agency through increased public scrutiny (organizational factor) (Fluke et al., 2014).

Literature Review

Understanding how children come to be victims of child maltreatment and end up in out-of-home care has been the subject of a large body of research. Many researchers seeking to understand the etiology of child maltreatment have relied upon an ecological framework (Bronfenbrenner, 1979) to consider factors at multiple levels of the social ecology that might contribute to child maltreatment outcomes. Studies examining the associations between individual parent, child, and familial characteristics and maltreatment (e.g., Kempe, 1962; Steele & Pollack, 1968) dominated the research on maltreatment risks until the late 1970s, when researchers began to also consider aspects of the social environment (e.g., Belsky, 1980; Garbarino, 1976; Gil, 1975; Pelton, 1978). Since that time, research on the causes of maltreatment has continued to proliferate, with a noticeable emphasis on individual-level contributors.

Child Protective Service Agency Characteristics

In addition to the factors driving differences in actual maltreatment behaviors, it is critical to consider the role of variation in CPS policies and practices. Official child maltreatment occurrences lie at the intersection of both a behavior by a caregiver and the decision of a CPS caseworker or supervisor. CPS workers are tasked with making difficult decisions with limited time, resources, and information. These decisions are “high stakes” in that an error in decision-making can result in preventable harm or even death of a child. On the other end of the continuum, decisions that are incorrectly conservative can result in the avoidable removal of a child causing unnecessary trauma inherent to being separated from the family of origin. CPS workers face a variety of constraints in making their decisions, and these organizational variables must be considered in understanding CPS decisions (Gambrill, 2008).

Additionally, maltreatment decisions may be affected by the current climate of the agency. Workers in agencies who are under a consent decree may be more likely to substantiate maltreatment and to remove children. That is, consent decrees tend to result from well-publicized tragedies, often involving a child’s death, which contributes to a culture of fear of liability (Mezey, 1998; Smith et al, 2003). This may lead to lower thresholds for substantiation and removal by CPS in attempt to eliminate the possibility of a false negative (not intervening when intervention is necessary) (Camasso & Jagannathan, 2013; Fluke et al., 2014).

Time

Through the federal Child and Family Services Review, CPS agencies are required to meet a number of benchmarks related to the timing of individual cases. These include timelines for assessments, termination of parental rights, reunification, and adoption. However, federal standards still allow for flexibility to set shorter or longer timelines for the conclusion of an investigation, and for the time between removals and initial court hearings. Additional time to gather information may lead to more substantiations or removals, but the evidence is unclear (Child Welfare Information Gateway, 2003).

Service accessibility

Caseworkers must rely on existing community structures to provide needed services to clients. The presence of high-quality, voluntary services in the community may prevent the need for removal or ongoing case monitoring. Maguire-Jack and Byers (2014) find that having maltreatment prevention services within the county may influence CPS workers’ decisions to substantiate maltreatment and provide ongoing services, with some workers providing more services when community services were not available and others being more likely to substantiate services when families would not voluntary take up community services. Similarly, Fluke, Chabot, Fallon, MacLaurin, and Blackstock (2010) found that a lack of community resources was associated with the decision to place children into care, and was a contributing factor to placement disparities among aboriginal groups in Canada. However, caseworkers face many barriers to aiding clients, with service availability and accessibility limited (Geen & Tumlin, 1999). Even when services are available, agencies may lack the funding to pay for needed services or the staffing needed to provide adequate attention to each case (Geen & Tumlin, 1999).

Decision-making tools

Despite the proliferation of decision-making tools for CPS in recent decades, there remains a great deal of subjectivity in maltreatment screening, investigation, and substantiation decisions (DePanfilis & Girvin, 2005; Wells, Lyons, Doueck, Brown, & Thomas, 2004) (DePanfilis & Girvin, 2005; Wells, et al., 2004). CPS workers and supervisors must make a maltreatment determination based on the limited information they are able to gather during their investigation/assessment process, and using statutory definitions of maltreatment that may be vague and overarching. Often, structured decision-making and other standardized assessment tools have been used in an effort to reduce errors and improve consistency in decision making. However, there is limited evidence to suggest that such efforts are successful. An ethnographic study suggests that caseworkers do not use the tools to inform their decisions to the extent intended and that the tools undermine development of critical assessment skills (Gillingham & Humphreys, 2010).

County Characteristics

Within the small, but growing, body of research on the role of context in child maltreatment, a number of community variables have been found to be associated with maltreatment substantiations and foster care entry, including poverty (Coulton, Korbin, Su, & Chow, 1995; Freishtler, Bruce, & Needell, 2007; Freisthler, Midanik, & Gruenewald, 2004; Freisthler, Needell, & Guenewald, 2005; Fromm, 2004; Irwin, 2009; Lery, 2009), concentration of single, female-headed households (Coulton et al., 1995; Freisthler et al., 2007; Freisthler et al., 2004; Lery, 2009; Zhou, 2006), residential instability (Coulton et al., 1995; Freisthler et al., 2007; Freisthler et al., 2005; Fromm, 2004; Irwin, 2009; Lery, 2009), and concentration of minority children in investigations (Fallon, Chabot, Fluke, Blacstock, MacLaurin, & Tonmyr, 2013). These studies relied on neighborhoods as the geographical unit, whereas the current study focuses on counties. Many human services, including CPS, are organized through county systems, making the county context potentially more relevant in the current study. One prior multilevel study using county as the geographic unit of interest found that the availability of maltreatment prevention services was associated with substantiations, but did not find support for an association between county-level disadvantage or residential instability and substantiations (Maguire-Jack, 2014).

Caseworker Characteristics

Since assignment of cases to caseworkers generally occurs on a rotation, and thus is generally not a function of the family’s characteristics, the probability of a given outcome would be approximately equal across caseworkers if there were no unmeasured tendencies of caseworkers themselves. Yet, it is generally understood that caseworkers’ decision-making falls on a spectrum, with some caseworkers having a higher or lower propensity to substantiate or remove (Child Welfare Information Gateway, 2003; Doyle, 2007). Fluke and colleagues (2014) suggest that caseworkers who are newer may be more likely to err on the safe side and screen cases in or substantiate. Chabot and colleagues (2013) found that agencies with a greater proportion of CPS workers with a formal social work education was associated with a decrease in the likelihood that a child would be placed in out-of-home care, but this association was only marginally significant (p=.053). There is also evidence to suggest that large caseloads are associated with decreased likelihood of removal (Baumann et al., 2010). Generally speaking, large caseloads can make it difficult to meet time requirements for case disposition, and may result in “tunnel vision,” in which the worker considers only a narrow range of options to save time and effort (Munro, 2008). Finally, individual caseworkers’ attitudes have been found to drive their decision-making. Davidson-Arad and Benbenishty (2010) found that more positive attitudes toward removal contributed to more intrusive intervention recommendations and higher risk assessments.

Child and Family Characteristics

Child maltreatment substantiations and placement into out-of-home care has been linked to socioeconomic characteristics at the individual level, including family income (Berger, 2004; Berger & Waldfogel, 2004; Detlaff, Rivaux, Baumann, Fluke, Rycraft, & James, 2011; Horwitz, Hurlburt, Cohen, Zhang, & Landsverk, 2011; Rivaux, et al., 2008), family structure (Berger, 2004; Berger & Waldfogel, 2004), and unemployment (Berger & Waldfogel, 2004). In addition, substantiation is linked to a variety of mental and behavioral health factors, including intimate partner violence (Horwitz et al., 2011; Rumm, Cummings, Krauss, Bell, & Rivara, 2000), prior incidence of maltreatment, substance abuse, and mental illness (Child Welfare Information Gateway, 2003; Zuravin & DePanfilis, 1997). These factors are typically identified by CPS workers in risk assessments completed during an investigation, and the scores of those assessments are also associated with substantiation and foster care placement (Horwitz, et al., 2011). One child characteristic that appears to be important is child age, with young children (0 to 2) and teenagers at higher risk for substantiation than other age groups (Child Welfare Information Gateway, 2003). The evidence regarding race is mixed, with some studies finding no associated with substantiation (Font, Berger and Slack, 2012) or removal (Zuravin & DePanfilis, 1997) and others finding associations with both (Detlaff et al., 2011; Rivaux, 2008).

Method

Data

This study uses the second cohort of the National Survey of Child and Adolescent Well-Being (NSCAW II). NSCAW II, when weighted, comprises a nationally representative sample of CPS investigations. Data for the baseline survey (Wave 1) of NSCAW II began in 2008 and 2009, and included 5,873 investigations that were closed with a 15-month period (for a more indepth overview of the sample design, refer to Dolan, Smith, Casanueva, & Ringeisen, 2011a). These investigations were located throughout 88 agencies in 83 counties. Although follow-up interviews were conducted at later points, this study uses data from Wave 1 because that is when substantiation and removal consequent to the index investigation are measured. We make no exclusions to the original sample. Missing data are multiply imputed using chained equations. Due to multiple levels of measurement, data on the individual level are imputed separately from data on the agency level, and these sets are merged post-imputation (Gelman & Hill, 2009). We also note that this research was approved as part of an expedited review from the Internal Review Board at the University of Wisconsin-Madison.

Measures

This study focuses on two dependent variables –substantiation and removal to out of home care. Both are measured at baseline (Wave 1) and are dichotomous indicators, with 1 indicating yes, and 0 otherwise. Three groups of independent variables are included: agency level variables, county characteristics, and child and family factors. Agency level variables include 3 time-related measures, 5 service accessibility measures and 3 decision-making measures. All agency level variables are reported by the local agency director.

Time

First, we include an indicator of the time allotted between removal and an initial court hearing, which was dichotomized at 3 days, due to a skewed variation. Second, we include an indicator of length of time allotted for investigations, dichotomized at 30 days. Lastly, we include a measure of increased workload. This measure is equal to 1 if the agency reports any increase in the number of cases over the past 12 months, relative to prior years. Together these three items approximate whether caseworkers have adequate time to thoroughly investigate each case.

Service accessibility

Items in this group include service availability, collaboration, services for unsubstantiated cases, presence of a system of care, and funding cuts. A scale of 17 items is used to approximate service availability (indicating whether specific types of services are present in the area, such as domestic violence or transportation services), and a scale of 6 items is used to approximate how much agencies collaborate with other social institutions (e.g., “What types of collaborations does your agency have with family courts?”). Both scales are created based on the average across included items. (For a full list of the items in each scale and internal reliability information, refer to Appendices A and B.) Service availability is intended to capture the breadth of services that are offered in the area; whereas collaboration focuses on the degree of cooperation between the CPS agency and other relevant institutions like schools, law enforcement, and courts. Services for unsubstantiated cases is a single dichotomous item indicating whether services are able to be offered when an investigation is unsubstantiated. The fourth item is a dichotomous indicator of whether the agency director says there is a system of care in the community in which the agency is set. Lastly, there is an indicator of whether the agency lost more than one quarter of its funding in the past 12 months.

Decision-making

The third set of agency factors, decision-making constraints, focuses on aspects that structure the way in which caseworkers are supposed to carry out their jobs. Specifically, we include two dichotomous measures–whether the agency (1) operates under a consent decree, and (2) uses a structured decision-making model—as well as a count measure of the number of standardized assessment tools an agency uses during investigations.

County characteristics

We include 5 county factors. First, a measure of logged county population (in 2008) is used to assess population density. Second, to identify disadvantaged communities, we include measures of child poverty (the percent of children falling under the federal poverty line) and crime (arrest rate per 100,000), both dichotomized as equal to 1 if the community falls in the top quintile of the distribution. These are dichotomized due to a non-normal distribution of values.

Finally, ethnic heterogeneity is measured using two variables: percent of the county population that is Black and percent that is Hispanic. These variables are included in NSCAW II in the form they are used in the analysis.

Caseworker characteristics

We consider 4 caseworker variables. First, for education, we include dichotomous indicators of whether the caseworker has (1) a social work degree or (2) an advanced degree (i.e., masters or above). We also consider two continuous measures: years of experience in child welfare, and average number of new investigations per months over the past three months.

Family risk factors and child demographics

Family risk factors are dichotomous indicators as assessed by the caseworker at the time of the investigation. Specifically we include 5 risk factors: history of CPS involvement, mental health or substance abuse problems, domestic violence, poor parenting skills, economic hardship, and child safety/special needs. Child demographics include age (years) and race (black, Hispanic, or other race: reference white).

Analytic Approach

We use hierarchical linear models (HLM) to estimate the associations between agency, county, child, and family characteristics and two outcomes, substantiation and removal to out of home care. We note that the models predicting removal are conditional on having been substantiated. Thus, whereas the full sample is used in the substantiation models (N=5,872), the sample for the removal models is all substantiated cases (N=3,635).

HLM is the approach of choice with nested data; in this case, investigations (level 1) are nested within agencies and counties (level 2). Notably, NSCAW II samples primarily 1 agency per county (81 of the 83 counties are represented by a single agency). Thus, we must consider agency and county to occur at the same level of estimation, although, in reality agencies are clustered within county. Similarly, despite instances where there are multiple cases assigned to a single caseworker, there are over 5,000 caseworkers sampled in Wave 1 (Dolan, Smith, Casanueva, & Ringeisen, 2011b), indicating that very few cases involved the same caseworker. Thus, caseworker variables are considered as level 1 variables.

HLM assumes that the level 2 units have their own intercept; meaning, net of all other characteristics, the probability of a given outcome (substantiation or removal) will differ by agency and county. Thus the equation for level 2 is represented as:

$${\alpha }_{0j}={\alpha }_{00}+{\delta }_{01}{Z}_j+{\mu }_{0j}$$

Where the probability of an outcome for county j is a function of a general intercept, a set of agency and county level characteristics (Z) and the unique effect of each individual county (µ). This intercept α_0j then functions as the intercept in the level 1 equation:

$$\mathit{\text{PR}}(y)={\alpha }_{0j}+{\beta }_{1i}{X}_{\mathit{\text{ij}}}+{\epsilon }_{\mathit{\text{ij}}}$$

Where the probability of outcome Y for person i in county j is a function of the county intercept, and child and family characteristics (X) and an unstructured error term (ε). We estimate 4 models for each of our outcomes. These models begin with only agency level characteristics (Model 1), and then add county characteritics (Model 2), caseworker characeristics (Model 3) and finally, family risk factors and child demographics (Model 4). Adding groups of variables in a nested progression allows us to examine the relative contributions of each set of factors, and to observe how the coefficients for the agency and county variables change once lower level variables are controlled. All models are weighted using multi-level weights (separate weights for the agency and case levels) that were provided to us by the parties responsible for the NSCAW study at our request. The weighted sample constitutes a nationally representative sample of investigations; weights adjust for factors such as the oversampling of infants and children/families receiving services and non-response. These analysis were conducted in Stata Version 13, using the mixed effects model commands for multiply imputed data.

Lastly, we note that, given our use of multiply-imputed data, which results in larger standard errors, we note coefficients at significance levels up to .10. Although we are less confident in estimates with p values between .05 and .10, we consider these results to be marginally significant and believe they warrant addiitonal examination. Given our relatively large sample, particularly of level 2 units, we do not believe statistical power is substantially hindering our analyses. In HLM, although there is no “golden rule” for the number of level 2 units required to conduct analyses, some have recommended that 20 should be used as the minimum number for adequate statistical power (Kreft & de Leeuw, 1998). In the current analyses, we use information from 83 counties, suggesting a relatively high level of statistical power.

Results

Descriptive Statistics

Approximately 25 percent of cases were substantiated, and of those, about 24 percent resulted in removal. A description of the sample by substantion and removal can be found in Table 1. Compared with unsubstantiated cases, substantiated cases came from agencies with longer timelines for completing investigations and higher service availability. Substantiated cases were also more likely to come from agencies that lacked a system of care in the area, that were operating under a consent decree and used more standardized assessment tools. On the county level, substantiated cases were more likely than unsubstantiated cases to be from communities with more black residents. Substantiated cases were also more likely to have been investigated by caseworkers with an advanced degree and more years of experience. On the family and child level, all risk factors except CPS history and economic problems were more common in substantiated cases than in unsubstantiated cases. Child demographics did not differ by substantiation status.

Table 1.

Descriptive statistics

	Substantiated			OHC (if substantiated)
	No	Yes	Sig	No	Yes	Sig
Time Constraints
Allows more than 30 days to complete investigation	42.14	54.36	***	55.87	49.48	+
Allows 3+ between removal and initial hearing	35.25	31.82		28.83	41.42	***
Increase in agency workload	37.80	41.10		39.89	44.99
Service Accessibility Constraints
Collaboration with other social institutions	1.00 (.02)	.98 (.04)		.98 (.05)	.98 (.04)
Can provide services for unsubstantiated cases	86.40	85.37		85.01	86.51
Service availability	.74 (.02)	.78 (.01)	*	.79 (.02)	.76 (.03)
Community has system of care	86.29	77.70	***	77.19	79.35
Agency lost 25% or more of funding in last 12 months	5.70	4.36		3.30	7.78	*
Decision-Making Constraints
Consent decree	29.76	36.02	*	35.81	36.69
Structural decision making model	74.82	73.65		71.96	79.07	*
Number standardized assessments used	3.91 (.07)	4.37 (.09)	***	4.36 (.11)	4.41 (.13)
County Characteristics
% of county black 2008	11.67 (.34)	13.31 (.45)	**	13.33 (.55)	13.26 (.67)
% of county Hispanic 2008	18.06 (.64)	17.41 (.75)		17.12 (.94)	18.36 (.90)
County population 2008	3.58 (.06)	3.64 (.07)		3.63 (.09)	3.68 (.09)
High arrest rate	18.70	22.74		22.13	24.72
High poverty rate	27.07	28.74		27.88	31.51
Caseworker Characteristics
Social Work degree	26.64	27.65		28.95	23.45	+
Advanced degree	23.57	30.10	*	31.48	25.67	+
Years of experience	5.62 (.24)	6.53 (.29)	*	6.56 (.36)	6.44 (.44)
Average number of new investigations per month - Past 3M	16.00 (.52)	15.51 (.64)		15.40 (.78)	15.86 (.88)
Family Risk Factors
CPS history	31.25	21.48	***	20.61	24.25
Caregiver mental health/substance abuse problems	12.36	27.60	***	24.55	37.40	***
Domestic violence	7.00	15.70	***	16.63	12.72
Poor parenting skills	7.75	16.64	***	16.82	16.05
Economic hardship	3.83	3.54		3.36	4.14
Child safety/Child has special needs	35.29	43.19	**	43.94	40.76
Child Demographics
Black	22.38	23.90		22.01	29.98	*
Hispanic	27.21	30.13		31.36	26.15
Other race	7.53	7.60		8.47	4.77	*
Age	7.35 (.17)	6.91 (.21)		7.07 (.26)	6.42 (.30)	+

Notes: Descriptive statistics are weighted. Cases=5,872. Agencies=85. Counties=83.

⁺p<.1

^*p<.05

^**p<.01

^***p<.001

Substantiated cases resulting in removal were more likely to come from agencies allowing 30 or fewer days to complete an investigation and 3 or more days between removal and initial hearing. They were also more likely to come from agencies that lost funding and used a structural decision making model. Removal cases were marginally less likely to have been investigated by a caseworker with a social work degree or an advanced degree. The only family risk factor positively associated with removal was caregiver mental health and substance abuse problems. Additionally, removal cases were more likely than non-removal cases to involve black children or younger children, and less likely to involve non-Hispanic children of a race other than white or black.

HLM Results

Results of our HLM estimates predicting substantiation are found in Table 2. We find no statistially significant associations between time factors and substantiation. However, for service availability, we find that two factors, collaboration and ability to provide services for unsubstantiated cases, are associated with a significantly lower probability of substantiation, even after controlling for county, family and child characteristics. These factors predict a 16 (collaboration) and 20 (services for unsubstantiated cases) percentage point (PP) lower probability of substantiation. For decision-making factors, we find that use of a structural decision-making model predicts a large decrease in the probability of substantiation, whereas each additional standardized assessment used predicts a (marginally significant) 2.1 PP increase in the probability of substantiation. Being under a consent decree was marginally significant in models 1 through 3, predicting increased probability of substantiation, but became nonsignificant in model 4. Joint significance tests confirm that decision-making and service accessibility factors are both important sets of predictors for substantiation.One county characteristic was associated with substantiation risk –a 1 percent increase in the proportion of Hispanic residents predicted a 0.7 PP decrease in the probability of substantiation.

Table 2.

Estimated Probability of Substantiation

	M1		M2		M3		M4
	B	SE	B	SE	B	SE	B	SE
Time Constraints
Allows more than 30 days to complete investigation	.092	.084	.085	.083	.087	.083	.069	.073
Allows 3+ between removal and initial hearing	−.036	.053	.004	.050	.003	.049	−.018	.045
Increase in agency workload	−.008	.060	−.014	.056	−.000	.055	−.001	.050
Service Accessibility Constraints
Collaboration with other social institutions	−.162*	.065	−.155**	.056	−.152**	.055	−.155**	.050
Can provide services for unsubstantiated cases	−.237**	.086	−.242**	.083	−.232**	.083	−.199*	.078
Service availability	.084	.083	.110	.089	.106	.086	.093	.079
Community has system of care	.021	.095	.030	.096	.027	.096	−.005	.092
Agency lost 25% or more of funding in last 12 months	.140	.155	.159	.120	.158	.124	.158	.132
Decision-Making Constraints
Consent decree	.160+	.083	.178+	.097	.175+	.094	.138	.086
Structural decision making model	−.238**	.078	−.230**	.072	−.222**	.071	−.200**	.064
Number standardized assessments used	.023+	.012	.017	.013	.017	.013	.021+	.012
County Characteristics
% of county black 2008			.003	.002	.003	.002	.002	.002
% of county Hispanic 2008			−.006+	.004	−.006+	.004	−.007*	.003
County population 2008			.009	.033	.008	.033	.003	.031
High arrest rate			−.048	.102	−.040	.099	−.033	.090
High poverty rate			−.047	.057	−.042	.056	−.044	.052
Caseworker Characteristics
Social Work degree					.001	.032	−.002	.030
Advanced degree					.052*	.025	.056*	.023
Years of experience					.002	.002	.001	.002
Average number of new investigations - Past 3M					.000	.001	−.000	.001
Family Risk Factors
CPS history							−.003	.019
Caregiver mental health/substance abuse problems							.227***	.035
Domestic violence							.195***	.041
Poor parenting skills							.186***	.040
Economic hardship							.055	.053
Child safety/Child has special needs							.101***	.022
Child Demographics
Black							.029	.026
Hispanic							.047	.037
Other race							.015	.037
Age							−.000	.002

Notes: Models estimated using multi-level sampling weights. Cases=5,872. Agencies=85. Counties=83.

⁺p<.1

^*p<.05

^**p<.01

^***p<.001

Caseworker factors were largely insignificant, with the exception of advanced degree, which predicted a 5.6 PP increase in substantiation. Family risk factors are the strongest predictors of substantiation, with all risk factors except economic problems and CPS history predicting increased probability of substantiation. Lastly, neither child age nor child race predicted substantion.

Turning to the risk of removal among substantiated cases (Table 3), 2 factors are consistently and significantly associated with increased risk. Allowing more than 3 days between removal and initial hearing and use of a structured decision-making model predicted increases in the probability of removal. Use of standardized assessment tools was marginally significantly associated with higher risk of removal. Lastly, allowing services in unsubstantiated cases was associated with a higher risk of removal among substantiated cases..

Table 3.

Estimated Probability of Out-of-Home Placement (Conditional on Substantiation)

	M1		M2		M3		M4
	B	SE	B	SE	B	SE	B	SE
Time Constraints
Allows more than 30 days to complete investigation	.020	.041	.009	.043	.006	.041	.009	.042
Allows 3+ between removal and initial hearing	.127***	.033	.108***	.032	.110***	.033	.119***	.032
Increase in agency workload in past year	−.018	.033	−.002	.040	−.019	.040	−.031	.041
Service Accessibility Constraints
Collaboration with other social institutions	.025	.031	.017	.030	.014	.031	.022	.031
Can provide services for unsubstantiated cases	.081*	.040	.068+	.035	.058	.037	.072*	.037
Service availability	−.036	.042	−.079*	.034	−.074*	.037	−.058	.038
Community has system of care	.001	.043	−.028	.041	−.029	.040	−.054	.042
Agency lost 25% or more of funding in last 12 months	.152	.107	.160	.109	.153	.107	.133	.117
Decision-Making Constraints
Consent decree	.025	.044	.004	.039	.009	.039	−.004	.041
Structural decision making model	.118**	.041	.092*	.042	.091*	.041	.092*	.042
Number standardized assessments used	.006	.007	.011+	.007	.013+	.007	.012+	.008
County Characteristics
% of county black 2008			−.001	.001	−.001	.001	−.002	.002
% of county Hispanic 2008			−.001	.002	−.001	.002	−.001	.002
County population 2008			.030	.019	.034+	.020	.025	.020
High arrest rate			.106*	.048	.097+	.050	.094+	.049
High poverty rate			.062	.040	.057	.041	.051	.042
Caseworker Characteristics
Social Work degree					−.043	.032	−.041	.031
Advanced degree					−.054	.035	−.051	.032
Years of experience					−.001	.003	−.001	.003
Average number of new investigations - Past 3M					.000	.002	.000	.002
Family Risk Factors
CPS history							.081*	.040
Caregiver mental health/substance abuse problems							.141***	.036
Domestic violence							.000	.044
Poor parenting skills							.019	.049
Economic hardship							.056	.078
Child safety/Child has special needs							.009	.032
Child Demographics
Black							.060	.048
Hispanic							−.038	.048
Other race							−.093*	.043
Age							−.004+	.003

Notes: Models estimated using multi-level sampling weights. Cases=3,635.

⁺p<.1

^*p<.05

^**p<.01

^***p<.001

One county characteristic was marginally predictive of removal—high arrest rate was positively associated with the probability of removal. However, the combination of the county factors are largely jointly significant, suggesting that there may be substantial correlation among these factors. CPS history and parental mental health and substance abuse problems were associated with a higher probability of removal. Child age predicted a (marginally significant) slightly lower probability of removal. Lastly, non-Hispanic children of a race other than black or white were at lower risk of removal, relative to white non-Hispanic children.

Discussion

This study sought to identify the respective contributions of family, agency, caseworker, and county factors in predicting substantiation and removal to out-of-home care. Our findings suggest that agency factors are important predictors of substantion and, conditional on substantiation, predictive of children’s removal to out of home care.. However, our study has some limitations that must be considered. First, whereas the agency factors we focus on are measured at the agency level, some of them likely are the product of state-level policies. Thus, some of what we are attributing to between-agency variation is actually reflective of between-state variation. Second, we are unable to look at factors on the neighborhood level, and instead can only measure county-level characteristics. The limitations of this approach are documented by Dark and Bram (2007). Third, there are important agency level factors that are not measured in the data that are likely to effect outcomes. For instance, the availability of foster homes may influence the risk of out-of-home placement. Similarly, we were unable to consider caseworker burnout, which may affect decision-making.

Agency-Level Influences

Despite these caveats, our findings have many implications for child welfare practice. Though substantiation is intended to reflect a confirmation that maltreatment occurred, as demonstrated by a preponderance of the evidence, it is widely understood that substantiation rates differ dramatically across and within states. The same is true for rates of removal to out-of-home care. The results of this study suggest that agency factors, specifically constraints on service accessibility and decision-making, are associated with the probability of substantiation. Although family risk factors are still significant predictors of substantiation and removal, the importance of agency factors suggests that substantiation may be problematic as an indicator of maltreatment. That is, if many factors unrelated to the family or child’s circnumstances are associated with substantiation, then perhaps substantiation indicates something quite different across locales. This concept is bolstered by evidence suggesting little differences between substantiated and unsubstantiated cases, in terms of children’s outcomes (Hussey et al., 2005; Leiter, Myers, & Zingraff, 1994). Nevertheless, states have the right to set their own standards for maltreatment, and whereas federal standards for maltreatment would create uniformity, there is not a clear consensus on what those standards should be.

Yet, differences in substantiation rates may not simply reflect differences in definitions of maltreatment. Cases assigned to agencies that provide services to unsubstantiated cases had a significantly lower probability of substantiation. This is consistent with the oft-heard anecdote about substantiation acting as a gateway to services. That is, if a family presents with problems that are on the line of being substantiated or not, whether that case is substantiated may depend on whether that family needs services that can only be accessed through CPS channels.

Use of a structured decision-making model was differentially associated with substantiation and removal—it was associated with lower risk of substantiation but a higher risk of removal among substantiated cases. Constraints on decision-making appear to result in a stricter threshold for substantiation. In turn, those cases which are substantiated under a structured decision-making model may be especially high risk, thus resulting in a higher risk of removal among substantiated cases.

The strongest predictor of removal among substantiated cases was whether the agency allowed three or more days between removal and the initial court hearing. One possible explanation for this is that laxer time restrictions could allow caseworkers to remove children on less solid evidence, under the assumption that they will have time to gather additional evidence prior to the court hearing. Another explanation is that, because removal cases take up more of caseworkers’ time than non-removal cases, less time to prepare for an initial hearing perhaps acts as a deterrent to removal. However, this is speculative and these results suggest a need for additional research on how time constraints impact case decisions.

County-Level Influences

The proportion of the county that is Hispanic was associated with a small decrease in the probability of substantiation. The proportion of Hispanic residents was also found to be a protective factor in an HLM study of Chicago neighborhoods (Molnar et al., 2003), which was attributed to higher levels of social support and social networks among Hispanic residents. A high arrest rate in the county had a marginally significant association with the probability of removal among substantiated cases. One explanation for this could be that a high rate of arrests reflects a higher invidivual risk of arrest, which may make parents temporarily unavailable. That is, if a parent is arrested and placed in jail, and a second parent or appropriate family member is not immediately identified or available to care for the child, then removal may be the only alternative. A high arrest rate might also be indicative of a high level of crime or a more punitive attitude at the county-level, both of which might create an atmosphere that makes it more difficult to keep or return a child home.

We found no evidence of an association between substantiation of maltreatment and nearly all of the county-level characteristics, namely, proportion of county that is black, county population, arrest rate, or poverty rate. Although the geographic area of a county is ideal in many ways for this type of analysis because CPS systems are typically organized at the county level, the role of contextual characteristics might be washed out at that level, due to variation across neighborhoods within a county. That is, the effects found in the prior literature connecting maltreatment substantiations or foster care entry with poverty (Coulton, Korbin, Su, & Chow, 1995; Freishtler, Bruce, & Needell, 2007; Freisthler, Midanik, & Gruenewald, 2004; Freisthler, Needell, & Guenewald, 2005; Fromm, 2004; Irwin, 2009; Lery, 2009), and other contextual variables may not be visible at this large unit of geography.

Caseworker-Level Influences

Caseworkers with an advanced degree (a masters degree in any field) were more likely to substantiate. It is unclear why this would be the case, though it is possible that more educated caseworkers would be able to identify more subtle risks or problems in the home environment that lead them to substantiate, or are better skilled in interviewing and thus are more likely to elicit disclosures of maltreatment from children. Aside from that finding, however, we found no association between caseworkers’ education, experience, or caseload and the probability of substantiation or removal. This may reflect several factors. First, it is important to note that it is generally expected that factors like more relevant educational training and more years of experience will result in better decision-making. Yet, it is not clear that, for example, more educated workers should substantiate or remove children less often. Moreover, what constitutes a better decision is inextricable from agency and community conditions. For instance, all else equal, services may alleviate immediate threats to safety in a highly-resourced county, and in a less-resourced county, removal may be necessary because there are no local services available to address safety concerns.

Nevertheless, there are several other reasons that caseworker characteristics would not be associated with case outcomes. First, the pre-employment training and annual training that agencies provide to caseworkers may be more influential in their decision-making than their pre-employment education. In addition, the effects of individual caseworkers’ education may spill over to other caseworkers. It is not uncommon for caseworkers to consult one another or their supervisors on cases, or to accompany one another on home visits (particularly where safety concerns are present). Thus, any impact of a caseworker’s education on their decision-making may be difficult to identify given that decision-making may also be influenced by the education of their peers. Turning to experience, this is a factor that could have both positive and negative impacts on decision-making. Experience is expected to bring more knowledge and better skills with regard to collection of information, observation of family environments, and evaluation of available evidence. At the same time, a longer time working in child welfare may result in higher symptoms of burnout, which can impair decision-making. A longer tenure also may mean that a caseworker was passed over for promotion or advancement, which may indicate other aspects of caseworker quality. Lastly, it was an unexpected finding that caseload was not associated with decision-making. A decision to substantiate, and especially a decision to remove, is likely to result in increased paperwork, court hearings, and other demands on time. Thus we would perhaps expect an overburdened worker to be less likely to subtantiate and remove. The null finding may reflect an inadequate measure of case burden. The number of new investigations (the available measure of caseload) does not account for the fact that, especially in rural counties, caseworkers may carry investigations in addition to ongoing (open) cases, the latter of which may not be counted in the measure. In addition, the burden of any individual case may vary extensively based on factors such as household size (which corresponds to the number of interviews a caseworker must complete as part of the investigation) or the involvement of outside agencies like law enforcement (which may happen when CPS investigations run concurrent with a criminal investigation). These sorts of case factors may be randomly distributed across caseworkers but if not, this could distort the use of caseload as a measure of workload.

Child and Family Factors

Child and family characteristics have been the most widely studied in the decision-making literature. We found that several family risk factors were associated with substantiation – substance abuse and mental health, domestic violence, poor parenting skills and child disability/special needs. Surprisingly, economic hardship was not associated with increased substantiation or removal risk. Although prior studies have suggested that income is predictive of substantiation and removal (Berger, 2004; Berger & Waldfogel, 2004; Detlaff, Rivaux, Baumann, Fluke, Rycraft, & James, 2011; Horwitz, Hurlburt, Cohen, Zhang, & Landsverk, 2011; Lindsay, 1991; Rivaux, et al., 2008), we are only able to assess the effects of caseworker-reported economic hardship. The economic hardship item asked in the risk assessment is about families’ ability to meet basic household needs. Given the low reported rates, it is likely that caseworkers were construing this question quite narrowly, and perhaps subjectively. Future research would benefit from more explicit economic hardship items in the risk assessment.

Whereas many of the risk factors were predictive of substantiation, only mental health and substance abuse problems were significantly predictive of removal, and poor parenting skills approached statistical significance. It is not perhaps surprising that domestic violence would predict substantiation but not removal. Although a threshold of maltreatment may have been met in a domestic violence situation, the perpetrator of the violence (typically the father or male partner) can be court-ordered to leave the home, which often resolves the immediate safety issue and allows the non-offending parent (typically the mother) and child to remain together. Caregiver mental health and substance abuse have been noted in prior studies as predictive of substantiation and removal (Child Welfare Information Gateway, 2003; Zuravin & DePanfilis, 1997). This may reflect the difficulty in accessing mental health and substance abuse services without substantiation, given that they can be costly, particularly for parents without insurance. Similarly, these services may be in short supply, resulting in wait lists and other barriers to addressing immediate safety concerns; thereby increasing removal risk.

Research studies linking child demographics to substantiation and removal have produced somewhat mixed results. However, consistent with some prior research findings (e.g., Font et al., 2012; Depanfilis & Zuravin, 1997), we find that black children are not at significantly higher risk of either substantiation or removal risk. (However, we do find that non-Hispanic children of a race other than black or white are at lower risk of removal in substantiated cases.) We also find that age is (marginally significantly) negatively associated with removal. Although we cannot be certain why this is the case, this may reflect a feeling that there is less CPS can do to alter the course for older children, and it may reflect the reality that older children are more difficult to place in suitable foster homes.

Conclusion

In sum, we find that family risk factors and agency factors, specifically service accessibility and use of decision-making tools to be most predictive of substantiation, net of other agency, county, and child characteristics. This may suggest that substantiation is not a clear indication of maltreatment occurring or even the severity of maltreatment risks. Furthermore, our findings indicate that states and local agencies should consider disentangling services from substantiation, such that families need not have their case substantiated in order to access useful services. Fewer factors overall were predictive of removal, suggesting that much remains unknown about removal decisions.