The Organizational Context of Substantiation in Child Protective Services Cases

Abstract

Substantiation of child maltreatment is among the most important decisions made by Child Protective Services, and may have wide-reaching implications for child and family well-being. Yet, relatively little research has been undertaken to understand the organizational context of substantiation. Using national population data from the U.S., this study examined the associations between state and county contexts with county substantiation rates using multilevel negative binomial regression. The results show that organizational context (policy and practice characteristics) influences substantiation rates. In particular, standards of evidence, alternative options for investigation and disposition of allegations, and workload burden were all significant prediction of substantiation rates. However, the associations of organizational factors and substantiation varied across types of maltreatment allegations; neglect and physical abuse allegations were more heavily influenced by organizational factors than sexual abuse or multi-type maltreatment allegations. Implications for child protection policy and practice are discussed.

Millions of families each year have contact with child protective services (CPS), with approximately 1 in 3 children subject to a CPS investigation between birth and their 18^th birthdays (Kim, Wildeman, Jonson-Reid, & Drake, 2017). Given the wide reach of CPS, and its power to intervene in or disrupt family life, its organizational behavior warrants critical examination. The overwhelming majority of child maltreatment allegations made to state authorities is handled by CPS rather than the criminal justice system. Whereas criminologists have paid significant and longstanding attention to the behavior of police and criminal prosecutors, relative little research in this area focuses on the decision to substantiate child maltreatment.

Substantiation is a decision typically made by a caseworker (alone or in consultation with a supervisor) following an investigation that involves interviews with alleged victims, perpetrators, and others, along with review of ancillary evidence (e.g., police reports, medical records). Those who are substantiated as perpetrators of child maltreatment may be placed on a list known as the central registry, which can be used to screen candidates for certain types of employment (positions working with children) and to screen prospective foster and adoptive parents (Child Welfare Information Gateway, 2014b). It is also used in many localities to determine the level and voluntariness of services the family may receive. Despite these potential consequences for perpetrators and the family unit, there has been a robust and ongoing debate among researchers about whether substantiation is a meaningful indicator of maltreatment (Chiu, Ryan, & Herz, 2011; Drake, 1996; Hussey et al., 2005; Kohl, Jonson-Reid, & Drake, 2009; Leiter, Myers, & Zingraff, 1994; Snyder & Smith, 2015). In particular, because families investigated by CPS are predominantly low-income and disproportionately Black and Native American (Dolan, Smith, Casanueva, & Ringeisen, 2011; Kim et al., 2017), the decision-making process is especially relevant for understanding the experiences of vulnerable populations. This study examines variability in the decision to substantiate at the county level (the typical organization of child welfare services). Specifically, we examine the state- and county-level factors associated with substantiation rates (among investigated or assessed children) for cases dispositioned in 2014.

Background

Substantiation has been used in a number of studies (and is often described in general media) as an indicator of the incidence of maltreatment victimization. However, many scholars have disputed this characterization, based on three interrelated arguments. First, there is variability in state statutes defining maltreatment and the level of evidence required for substantiation (U.S. Department of Health and Human Services, 2017), which may provide between-states variability. Second, unsubstantiated is not synonymous with untrue (Drake, 1996). Though false allegations do occur, substantiation is not an unimpeachable assessment of truth. A relatively large body of work documents that substantiation decisions (determinations of whether maltreatment occurred) are influenced by a range of factors unrelated to the factual basis of the allegations and interpretation of evidence varies across caseworkers (Bartelink et al., 2014; Child Welfare Information Gateway, 2003; English, Brummel, Graham, & Coghlan, 2002; Font & Maguire-Jack, 2015). The third criticism is that because of the first and second criticisms (lack of a consistent national definition and the opacity of decision-making processes), substantiation is unlikely to distinguish children at risk , either for recurrence of maltreatment (Kohl et al., 2009) or adverse development (Hussey et al., 2005; Snyder & Smith, 2015), with exceptions (Chiu et al., 2011).

If not an indicator of the veracity of an allegation, what is the meaning of substantiation? Understanding substantiation requires positioning it in the context of CPS. Many have argued that CPS parallels other institutions of social control, such as the criminal justice system, because parents experience its interventions –even in-home services—as coercive, intrusive, and threatening (Dumbrill, 2006; Pelton, 2016). In this context, perhaps substantiation is better understood one form of sanction that can be handed down by CPS. Other “sanctions” that can be instituted by CPS include placement on a central registry (a state-maintained database of known maltreatment perpetrators), as well as (with family court authorization) court-ordered services and monitoring, removal of children of the home, and termination of parental rights (legal severance of the parent-child relationship). Importantly, whereas CPS does not consider its actions to constitute sanctions, per se, its actions are largely perceived as both coercive and punitive by media (Clifford & Silver-Greenberg, 2017) and researchers (Edwards, 2016; Roberts, 2012, 2014) alike. Most prior work in this area has focused on removal of children from the home as the most severe or intrusive sanction imposed by CPS (Edwards, 2016; Roberts, 2012). However, all forms of intervention, including removal, cannot be fully understood outside of the context of substantiation. Substantiation typically precedes placement of an alleged perpetrator on the central registry and directly informs or justifies the imposition of monitoring, services, or removal of children. Among victims with unsubstantiated cases, few of their families received services (about 30% overall, though fewer than 10% in at least 19 states) and very few (under 2%) were removed from the home, whereas among substantiated victims, 61% of their families received services and about 23% were removed from the home (U.S. Department of Health and Human Services, 2017). In addition, the services provided tend to be more intensive and longer-term for substantiated versus unsubstantiated cases. Thus, if we consider substantiation as a gateway to potential system “sanctions,” it is important to understand the role of organizational context in the usage of substantiation.

States vary in their approach to child protection in ways pertinent to understanding substantiation. First, although there are federal definitions for maltreatment, states have the ability to add to those definitions as long as they do not drop below the floor set forth in the federal definition. Although all states’ definitions include physical abuse, sexual abuse, emotional abuse, and neglect, the breadth of these definitions may vary, particularly for neglect. For example, only 23 states addressed children’s exposure to domestic violence in their statutes, and only 14 addressed prenatal exposure to drugs or alcohol. Other areas of difference include educational neglect (included in 25 states’ statutes), and parental substance abuse (8 states). Of course, states can intervene in situations even if they are not explicitly addressed in statute by framing the situation in the context of existing definitions. For example, parental substance abuse could be framed as neglect under the claim that an intoxicated parent is incapable of appropriate supervision, even if parental substance abuse is never mentioned in statute. However, what is contained in state statutes may nevertheless matter, in that it indicates the states’ priority concerns, may affect caseworker training, and guides mandatory reporters in determining when to make a report of suspected maltreatment.

Second, states determine how cases are dispositioned, both by setting the standard for evidence and by allowing for alternatives to the binary of substantiated and unsubstantiated. State standards for substantiation range from probable cause to clear and convincing evidence, though most states use a “preponderance” standard (meaning, more likely than not). Several states also allow for cases to be dispositioned as “indicated’ rather than substantiated or unsubstantiated, which means that evidence exists, but may be insufficient for the evidentiary threshold to substantiate. This allows workers to identify concerns and motivate intervention without making a strong claim about the level of evidence. Similarly, many states now have alternative response programs, which are supposed to place “low-risk” (based on the intake information) cases on a voluntary track, where substantiation is typically not used. We hypothesize that these three factors (high evidentiary burden, “indicated” disposition options, and alternative response) will be negatively associated with substantiation rates. Two of these factors (high evidentiary burden and indicated disposition) have been linked with lower substantiation rates in previous studies (Child Welfare Information Gateway, 2003).

Lastly, states vary in the centralization of the CPS system. Only 9 states (CA, CO, MN, NC, ND, NY, OH, PA, VA) have county-administered CPS system; 3 states (MD, NV, and WI) are hybrids (meaning, partially county-administered), and the remaining states’ systems are state-administered (Child Welfare Information Gateway, 2012b). State-administered means that the state has centralized control over issues such as caseworker training, policy-making, and funding. It is not clear what, if any, effect state administration is likely to have on substantiation rates. It may be simply that county-administered systems produce greater within-state variance. Yet, even in state-administered systems, CPS is largely organized at the county level. Counties carry out the everyday activities of the system, including case-level decision-making and supervision, which may mean that county-level contexts and constraints are most important.

Caseworkers’ behavior and characteristics has been a large emphasis in prior research on decision-making. The composition of counties’ CPS workforces may influence substantiation rates. Issues such as caseworker retention and workload are well-recognized as problems for CPS agencies across the U.S. (Blome & Steib, 2014). They are also interrelated: vacancies generated by turnover may increase the workload for remaining employees. We expect workload to be associated with lower rates of substantiation, and retention to predict higher rates. Caseworkers with high workloads have less time to conduct a thorough investigation and may ultimately find sufficient evidence to substantiate in fewer cases. In addition, it often requires greater effort on the part of a caseworker to substantiate a case – they are more likely to be expected to provide services, and there are sometimes additional assessments or reports that are required with substantiated cases. High workload may discourage caseworkers from decision-making that generates more work for themselves.

The composition of children subject to investigation by CPS is likely to vary across counties based on their overall population characteristics and other dynamics. Because this study is concerned with the within-system dynamics of investigative decision-making, we focus on the characteristics of investigated children, specifically age and racial composition. Age is relevant because younger children are more vulnerable to severe maltreatment, particularly death (U.S. Department of Health and Human Services, 2017); thus, they are likely to be perceived as in greater need of intervention. Racial composition is likely to be relevant given differences in the prevalence of risk factors, such as poverty, across racial groups. We also consider whether the proportional composition of allegations and source of referrals varies across counties. We suspect that certain forms of maltreatment, such as sexual abuse, may be viewed more seriously than others and thus counties with a higher proportion of sexual abuse investigations may have a higher substantiation rate. Similarly, particular sources of information may be more credible or have more evidentiary backing (for example, referrals from law enforcement or medical personal are likely to include police reports or medical records to support their claims).

Method

The National Child Abuse and Neglect Data System (NCANDS) is a federally-supported data repository to which states submit records annually. Through a special pilot program through the National Data Archives on Child Abuse and Neglect, we were able to securely access a version of NCANDS that has complete county identifiers (identifiers for small counties are not included in the generally available version). The pilot data allowed us to examine patterns in both rural and urban counties.

NCANDS, though voluntary, receives data from the majority of states in most years. Notably, what a state submits in a given year is not everything that may have occurred in that year: cases reported in one year may be dispositioned the next year, or may not be approved until the next year, resulting in a reporting lag. Thus, we use all cases dispositioned in 2014, irrespective of the year in which the data were submitted to NCANDS, in order to ensure the most complete numbers. For 2014, there were 3,030 unique counties. (For the purpose of analysis, the District of Columbia is considered to be a stand-alone county.) This equates to 96% of the 3,142 counties or county-equivalents in the U.S. However, we excluded states or counties that did not consistently include caseworker IDs (necessary for calculation of workload, turnover, and worker experience metrics), information about allegation type, child age, or child race. This resulted in a total sample of 2,440 counties (78% of total) in 44 states plus the District of Columbia. (The states excluded in their entirety due to these restrictions were Hawaii, Missouri, New York, Pennsylvania, Vermont, and West Virginia).

Measures

Outcome Variable

Our outcome measure is the maltreatment substantiation rate overall and by alleged maltreatment type. Our primary measure included cases that were substantiated through traditional investigation in addition to cases with an alternative response victimization determination. Alternative response victimization determinations are a form of substantiation that applies only to cases that receive an “alternative response,” meaning an assessment instead of an investigation. Typically, alternative response is used for cases that are deemed low-risk at intake and involves voluntary services. Not all states have alternative response programs that have distinct determinations and states have introduced these at different time points. Across states, substantiation rates ranged from less than 6% to above 50%, with most states substantiating between 10% and 30% of cases. Five states had substantiation rates of less than 10%, versus eight states with substantiation rates greater than 30%.

Type of maltreatment was coded as physical abuse, sexual abuse, neglect (including medical neglect), emotional abuse, multi-type, and other. Because other was not defined and no exemplars were provided, in cases where both other and a primary type of maltreatment (physical abuse, sexual abuse, emotional abuse, or neglect) was identified, the investigation was coded as the primary type rather than multi-type. Lastly, there were some states (n=22) that coded some investigations as “no alleged maltreatment” but nevertheless investigated and sometimes substantiated some form of maltreatment. Akin to those coded other, designations of no allegations have no conceptual meaning. Nevertheless, they are included in our analyses because their inconsistent use across counties and states could be pertinent to substantiation patterns.

Key Independent Variables

Our key independent variables include both county- and state-level measures (See Table 1). We calculated county-levels measure of caseworker workload and retention using the NCANDS data. Individual caseworkers were identified by unique codes within state. Retention was measured as the proportion of caseworkers active in the first quarter of the year who were also active in the last quarter of the year. Workload was equal to the monthly number of new investigated children per caseworker averaged across the year. The monthly rate was used instead of the annual average because high rates of turnover would underestimate typical workload over the course of the year.

Table 1.

Definitions and Means/SDs for Key County and State Variables

	M	SD
County (n=2,440)
Retention	50.2	28.3
Percent of active investigators in Quarter 1 of 2014 who were still active in Quarter 4
Workload	8.5	5.7
Average number of investigated children per active caseworker per month, 2014
Alternative Response Usage	51.2	31.2
Percent of alleged victims who received an alternative response	17.9	26.9
State (n=45)	n states	%
Evidentiary Burden
Level of evidence required for substantiation
High (Preponderance/clear and convincing)	33	73%
Moderate (Credible)	7	16%
Low (Probable cause/reasonable)	5	11%
Centralization
State vs. County Administered
State	34	76%
County or Hybrid	11	24%
Non-Binary Disposition Options
Allows for "Indicated" as a dispositional option for cases for which there is some evidence but not sufficient for substantiation	6	13%
Breadth of Civil Maltreatment Definitions
Number of "optional" expansions of maltreatment definitions explicit in state law, of the following: Educational neglect, Exposure to domestic violence, Prenatal substance exposure, Parental substance abuse (range: 1-4)	M=2.1, SD=.89

We also constructed a county-level measure of the use of alternative response. Although the presence of alternative response is generally dictated at the state level, there may be variability in the use of this approach. The measure was equal to the number of children who received an alternative response divided by the total number of children who received any response (investigation or alternative).

We constructed 4 state-level variables using government reports on state system characteristics (specific sources are cited for each measure). We measured centralization based on whether the state’s CPS system was state-administered, county-administered, or a hybrid. Nine states (CA, CO, MN, NC, ND, NY, OH, PA, VA) have county-administered CPS system; 3 states (MD, NV, and WI) are hybrids (meaning, partially county-administered), and the remaining states’ systems are state-administered (Child Welfare Information Gateway, 2012b). Given the small number of hybrid states, we created a binary indicator where 1=state-administered, 0=county-administered or hybrid.

We measured the required level of evidence for substantiation using states’ reported standards for 2014 (U.S. Department of Health and Human Services, 2015). In 2014, six states used a “reasonable evidence” standard; 8 used a “credible evidence” standard; and 34 used the “preponderance of evidence” standard. Only two states did not use one of those three: Kansas, which required “clear and convincing” evidence, and Arizona, which required “probable cause.” We categorized these into three groups: low evidentiary burden (reasonable evidence and probable cause); moderate burden (credible); and high burden (preponderance and clear and convincing).

Third, we used a binary indicator of whether the state allows “indicated” as an alternative disposition to substantiated. Six states included this option in 2014: AZ, MD, MI, MT, OH, TN (U.S. Department of Health and Human Services, 2015).

Fourth, we created an indicator of the breadth of state maltreatment statutes, focusing on the areas where there is greater disagreement: exposure to domestic violence, parental substance abuse, prenatal exposure to substances, and educational neglect (Child Welfare Information Gateway, 2012a, 2014a). Each indicator was equal to 1 if the state’s civil statutes addressed it, 0 otherwise. The sum of the four indicators was used to indicate the breadth of civil definitions, with higher values indicator greater breadth.

Covariates

These variables were based on characteristics of the county’s investigated children in a given year (i.e., percent with x characteristic divided by the total number of investigated children). These included reporting source: (law enforcement, education/child care, other mandatory reporters; reference = voluntary or unknown); alleged maltreatment type (physical abuse, sexual abuse, multi-type, emotional abuse, other/unknown, or none; reference = neglect); child’s race/ethnicity (Black non-Hispanic, Hispanic, American Indian, Asian/Pacific Islander, multiracial; reference = White non-Hispanic); and child’s age (3 and under, 11 to 17; reference = 4 to 10).

In Table 2, we show the means and standard deviations of the county-level (aggregate) investigation characteristics. In the average county, the majority of investigated children were White (M=60.7), followed by Black (M=12.9) and Hispanic (M=12.4). The largest group of investigated children in the average county was ages 4 to 10. The average county received 18% of its investigated referrals from law enforcement, 17% from education/child care personnel, 21% from other mandatory reporters, and 43% from voluntary or unidentified sources. Neglect was the sole allegation in the majority of referrals for the average county.

Table 2.

Aggregate Characteristics of Counties’ Alleged Victims (n=2,440 counties)

	M	SD
Pct. Black	12.9	18.9
Pct. American Indian	2.6	9.8
Pct. Multiracial	4.0	5.3
Pct. Asian/Pacific Islander	0.4	1.0
Pct. White	60.7	24.1
Pct. Hispanic	12.5	17.7
Pct. Under 3 years old	26.3	6.2
Pct. Ages 4 to 10	43.6	5.9
Pct. Ages 11 to 17	29.1	6.8
Pct. Law enforcement	18.1	9.9
Pct. Education/Child care	17.1	8.7
Pct. Other mandatory	21.4	9
Pct. Voluntary/Unknown	43.4	15.5
Pct. Neglect	55.4	16.6
Pct. Physical abuse	12.3	9.3
Pct. Sexual abuse	5.5	4.9
Pct. Emotional maltreatment	2.9	5.1
Pct. Multiple forms	13.1	10.3
Pct. None	8.7	13.5
Pct. Other/unknown	2.1	5.6

Analytic Approach

We used negative binomial regression with a random intercept for state to estimate the associations of county and state characteristics with county substantiation rates. Specifically, we used menbreg in Stata 14. Negative binomial regression is used with count data (in our case, the number of substantiations); it is an alternative to the poisson model that is appropriate when the data are over-dispersed. The negative binomial model includes an overdispersion parameter (α); when α>0, the model has overdispersion; when α=0, it is the same as a poisson model. We tested both poisson and negative binomial models of our data and found that our overdispersion parameter was non-zero and a comparison of the log-likelihoods indicated the negative binomial model was more appropriate to our data.

The range of possible values for the count of substantiations varies across counties. All counties have a lower limit of 0, and an upper limit equal to the number of children investigated in the county. Thus, all models included an exposure variable that is equal to the number of investigated children. The exposure variable was log-transformed and constrained to have a coefficient of 1. We also examined the dispersion assumptions for our model, and selected the traditional (mean) dispersion model based on the BIC statistic. Because counties are nested with state and we were interested in the effects of state-level system structures, we included a random intercept for state. For the regression analysis, all county-level variables were standardized to have a mean of 0 and standard deviation of 1.

Results

In Table 3, we show the results of random-intercept negative binomial regressions predicting the number of substantiations (adjusted for the number of investigations) in 2014. We produced stepwise models that added blocks of covariates, but the associations of our key independent variables with the substantiation rate were not altered by the inclusion of covariates, so we show only the complete model. We found that two state characteristics were predictive of the substantiation rate. First, the rate of substantiation in counties with lower evidentiary standards was 1.7 times higher. Second, counties in states that allowed the use of indicated as an alternative to substantiation had lower substantiation rates (at a ratio of .58 to 1) than in counties without an “indicated” disposition option. The centralization of the state’s CPS system was not predictive, nor was the breadth of state maltreatment statutes.

Table 3.

Multilevel Negative Binomial Regression Model

	b(se)	IRR
State variables
Reasonable/probable cause evidentiary standard	0.56 (0.15)***	1.76
Credible evidentiary standard	0.12 (0.14)	1.12
County/Hybrid administration	0.25 (0.19)	1.29
Indicated disposition option	−0.54 (0.19)**	0.58
Breadth of state definitions	−0.05 (0.08)	0.95
County variables
Retention	0.00 (0.01)	1.00
Workload	−0.05 (0.01)***	0.96
Use of alternative response	−0.26 (0.09)**	0.77
Racial/Ethnic group
Pct. Black	−0.03 (0.01)*	0.97
Pct. American Indian	0.02 (0.01)	1.02
Pct. Multiracial	−0.02 (0.02)	0.98
Pct. Asian/Pacific Islander	0.02 (0.01)	1.02
Pct. Hispanic	−0.01 (0.01)	0.99
Age group
Pct. Under 3 years old	0.05 (0.01)***	1.05
Pct. Ages 11 to 17	0.01 (0.01)	1.01
Referral source
Pct. Law enforcement	0.11 (0.02)***	1.11
Pct. Education/Child care	0.02 (0.01)	1.02
Pct. Other mandatory	0.07 (0.02)***	1.07
Allegation type
Pct. Physical abuse	−0.09 (0.03)**	0.91
Pct. Sexual abuse	0.02 (0.02)	1.02
Pct. Emotional maltreatment	0.05 (0.02)*	1.05
Pct. Multiple forms	−0.02 (0.02)	0.98
Pct. Other/unknown	−0.04 (0.05)	0.96
Pct. None	−0.04 (0.02)*	0.96
Constant	−1.66 (0.23)***
ln Alpha (dispersion parameter)	−2.54 (0.09)***
var(_constant, state)	0.17 (0.05)***

Note: N=2,440 counties. IRR= incidence rate ratio (exponentiated coefficients). Reference group for racial group is % White; reference group for age if % ages 4-10; reference group for referral source is % voluntary, anonymous, or unknown; reference group for allegation type is % neglect.

^*p<.05

^**p<.01

^***p<.001

Turning to county characteristics, we found that a one standard deviation (SD) increase in average workload was associated with a 5% decrease (IRR=0.95) in the substantiation incidence rate. Caseworker retention rate was not predictive of substantiation rates. The percentage of cases receiving an “alternative response” was associated with fewer substantiations: specifically, a one SD increase in assignment to alternative response was associated with a 23% decrease (IRR=0.77) in the substantiation incidence rate. Our county-level covariates indicated the proportion of investigated children who were Black was associated with a 3% decrease (IRR=0.97) in substantiation rate, and the proportion under the age of 3 was associated with a 5% increase (IRR=1.05) in substantiation rate. The proportions of investigated children referred by law enforcement or other mandatory reporters, or with multi-type maltreatment allegations, was associated with a higher substantiation rate. Lastly, the proportions of investigated children with physical abuse allegations and no allegations were associated with lower substantiation rates.

In Table 4, we show our results separately by maltreatment type. We only estimate these models for the most common categories (neglect, physical abuse, sexual abuse, and multi-type). In these models, the percentages for race, child age, and referral source composition are specific to that allegation type (i.e., the percentage Black for the neglect model refers to the percent of neglect allegations involving a Black child). The findings with regard to state and county organizational characteristics were similar for neglect, physical abuse, and multi-type maltreatment, compared with our overall substantiation rate models. One exception was that the percent of children receiving an alternative response was not predictive of substantiation for multi-type maltreatment (though the coefficient was not significantly different in the multi-type models versus the neglect or physical abuse models). For sexual abuse allegations, however, none of the state or agency organizational characteristics had statistically significant coefficients, and in most cases, the coefficients were also small in magnitude relative to the coefficients for the same variables in other models. Significantly, coefficients for evidentiary burden and alternative response usage were significantly (at p<.05) smaller in the sexual abuse model than in the neglect or physical abuse models.

Table 4.

Multilevel Negative Binomial Regression Models by Alleged Maltreatment Type

	Neglect		Physical abuse		Sexual abuse		Multi-type
	b(se)	IRR	b(se)	IRR	b(se)	IRR	b(se)	IRR
State variables
Reasonable/Probable cause evidentiary standard	0.58 (0.16)***c	1.78	0.52 (0.21)*	1.67	0.07 (0.17)ad	1.08	0.62 (0.15)***c	1.86
Credible evidentiary standard	0.27 (0.18)	1.31	0.13 (0.22)	1.14	−0.13 (0.20)	0.87	0.22 (0.16)	1.24
County/Hybrid administration	0.24 (0.19)	1.27	0.00 (0.32)	1.00	0.05 (0.14)	1.06	0.30 (0.19)	1.35
Indicated disposition option	−0.48 (0.18)**	0.62	−0.47 (0.21)*	0.62	−0.29 (0.19)	0.75	−0.76 (0.26)**	0.47
Breadth of state definitions	−0.08 (0.09)	0.92	−0.01 (0.12)	0.99	0.00 (0.07)	1.00	−0.03 (0.09)	0.97
County variables
Retention	0.00 (0.01)	1.00	−0.01 (0.02)	0.99	0.03 (0.02)	1.03	0.01 (0.02)	1.01
Workload	−0.04 (0.01)***	0.96	−0.05 (0.02)**	0.95	−0.03 (0.01)	0.97	−0.06 (0.01)***	0.95
Alternative response usage	−0.29 (0.10)**c	0.75	−0.24 (0.11)*c	0.79	0.01 (0.03)ab	1.01	−0.10 (0.07)	0.91
Racial/Ethnic group
Pct. Black	−0.05 (0.01)***	0.95	0.03 (0.02)	1.03	−0.02 (0.01)	0.98	−0.03 (0.01)**	0.97
Pct. American Indian	0.02 (0.01)	1.02	0.00 (0.02)	1.00	−0.03 (0.01)*	0.97	0.02 (0.01)	1.02
Pct. Multiracial	−0.02 (0.02)	0.98	−0.02 (0.02)	0.98	−0.03 (0.02)	0.97	0.01 (0.02)	1.01
Pct. Asian/Pacific Islander	−0.00 (0.02)	1.00	−0.01 (0.01)	0.99	0.01 (0.02)	1.01	−0.01 (0.02)	0.99
Pct. Hispanic	0.02 (0.01)	1.02	−0.03 (0.01)**	0.97	0.02 (0.02)	1.02	0.01 (0.01)	1.01
Age group
Pct. Under 3 years old	0.05 (0.02)*c	1.05	0.09 (0.03)*c	1.09	−0.03 (0.02)	0.97	0.08 (0.02)**c	1.08
Pct. Ages 11 to 17	−0.02 (0.02)c	0.98	−0.07 (0.02)**c	0.94	0.13 (0.02)***	1.13	0.01 (0.02)c	1.01
Referral source
Pct. Law enforcement	0.15 (0.02)***	1.16	0.17 (0.04)***	1.19	0.12 (0.04)**	1.12	0.20 (0.02)***	1.22
Pct. Education/Child care	0.02 (0.02)	1.02	0.03 (0.05)	1.03	−0.01 (0.04)	0.99	0.05 (0.02)**	1.05
Pct. Other mandatory	0.14 (0.02)***	1.15	0.08 (0.05)	1.09	0.05 (0.04)	1.06	0.11 (0.02)***	1.11
Constant	−1.59 (0.25)***		−2.13 (0.33)***		−1.39 (0.17)***		−1.36 (0.23)***
ln Alpha (dispersion parameter)	−2.25 (0.13)***		−2.36 (0.12)***		−2.94 (0.16)***		−2.66 (0.14)***
var(_constant, state)	0.21 (0.06)***		0.40 (0.14)**		0.13 (0.03)***		0.22 (0.05)***

Note: IRR= incidence rate ratio (exponentiated coefficients). Reference group for racial group is % White; reference group for age if % ages 4-10; reference group for referral source is % voluntary, anonymous, or unknown.

^*p<.05

^**p<.01

^***p<.001

^a=significantly different from neglect model at p<.05

^b=significantly different from physical abuse model at p<.05

^c=significantly different from sexual abuse model at p<.05

^d=significantly different from multi-type model at p<.05

In addition, there were some differences in county investigation composition characteristics. For neglect and multi-type maltreatment, percent Black was associated with lower substantiation rates, but was not associated with sexual or physical abuse substantiation rates. The percent of children under 3 was associated with higher substantiation rates for neglect, physical abuse, and multi-type maltreatment but not sexual abuse. The percentage of children ages 11 and older was associated with lower physical abuse substantiation rates and higher sexual abuse substantiation rates. Differences in coefficients between the sexual abuse model and the other models were statistically significant at p<.05 for percent black, percent under 3, and percent ages 11-17. For all forms of maltreatment, percent of law enforcement referrals predicted higher substantiation rates; the percent of other mandatory reporters was associated with higher substantiation rates for neglect and multi-type maltreatment only, and the percent of education/child care reporters was associated with higher substantiation rates for multi-type maltreatment only.

Discussion

The extent to which substantiation is perceived to be an objective and meaningful indicator of maltreatment or related risk characteristics is important for the actual and perceived legitimacy of the child welfare system, as well as for contextualizing research on the causes or consequences of maltreatment. Importantly, geographic variability alone does not per se indicate a problem with substantiation as an indicator. In a federalist system, it is expected that states craft systems and policies that allow them to best serve their respective constituencies, within the bounds of a general federal framework. However, because national statistics on maltreatment victimization are often used as a point of reference, it is important to understand how states differ. Consistently, we found that states with lower evidentiary burdens had higher substantiation rates, and that states that allowed ‘indicated’ as an alternative disposition to substantiation had lower substantiation rates. This suggests that substantiation may be better understand as an indicator of what could be proven, rather than the veracity of an allegation. CPS caseworkers do not have the same powers to investigate as police, nor are they rigorously trained in the collection of evidence. This may also help to explain why cases referred by law enforcement have higher rates of substantiation for all maltreatment types: investigations carried out by law enforcement provide evidence that can be used by the caseworker without further validation.

Importantly, we cannot ascertain from the data which unsubstantiated cases involved true incidences of maltreatment, and which substantiated cases did not. There are no standardized means of reporting on the quality of evidence or rationale for the disposition decision. More significantly, the line between low-quality parenting and child abuse or neglect is not as clear as the dichotomous decision-making process would suggest. We make no claims about what constitutes a “correct” standard of evidence or what level of harm is adequate to label someone a perpetrator of maltreatment. However, families with unsubstantiated cases are likely to be re-reported (Kohl et al., 2009), and re-investigating comes at a significant cost (in staff time and agency resources). We assert that investigating cases, in itself, provides no public good – that is, identifying maltreatment or related risk factors is useful only when it leads to the provision of aid. Thus, to focus investigations on the identification of families’ risks and needs, in addition to the veracity of allegations, and to provide services irrespective of the latter, may be a more efficient use of limited state and local resources. Certainly, some states provide services to some families with unsubstantiated cases, but most families with unsubstantiated cases receive nothing, and among those that do, services are likely to be less intensive and shorter-term. We also found that the percentage of children whose cases were handled through an alternative response was associated with lower substantiation rates. This is likely because many states with alternative response programs do not ask the caseworker to make such a determination. Thus, a case that might otherwise have sufficient evidence to substantiate, if handled through alternative response, would likely have no such disposition. If so, it is problematic for estimating or tracking the prevalence of maltreatment, especially at the national level where changes in state policy may be indistinguishable from true changes in maltreatment incidence. However, the implications for family and child well-being are less clear. Alternative response is one avenue through which agencies have sought to intervene and provide supports without the adversarial nature of a CPS investigation. To the extent that alternative response is providing services without substantiation, the reduction in substantiations that accompanies alternative response is not necessarily concerning. Some studies have suggested that alternative response does not compromise child safety, though differences in the structure and processes of alternative response programs may result in different implications for children (Kyte, Trocme, & Chamberland, 2013; Loman & Siegel, 2015). Notably, because some alternative response programs only implement services on a voluntary basis, which families—particularly higher risk families—may refuse (Bartholet, 2014), it is also possible that children who were indeed subjected to maltreatment are neither being identified as such nor are their families receiving services to prevent future harm. Third, we found that workload was associated with lower substantiation rates. This may suggest that caseworkers with high workloads lack adequate time to dedicate to the collection of information and thus are less likely to uncover sufficient evidence. If overburdened, caseworkers may forego activities that could uncover clearer or more compelling evidence of maltreatment but are not required in policy. Notably, it is not apparent that in-depth investigation is the best expenditure of caseworker time, particularly in cases where families are interested in voluntary services to address presenting risks; in those cases, time spent in direct provision of services may better support the mission of child safety than investigative activities. Regardless, it is also possible that counties where workloads are higher have other characteristics that reduce substantiation rates. For example, if those counties screen out relatively few reports, this may result in higher workloads and more low-risk cases being investigated. Additional research is needed to validate and understand this finding.

Lastly, there were differences by type of alleged maltreatment in how state and county factors were associated with substantiation rates. In particular, sexual abuse substantiation was not strongly linked with any organizational factors. This may reflect that sexual abuse investigations have clearer parameters in that any sexual contact with a child constitutes sexual abuse. To the contrary, legal definitions allow for more uncertainty in other forms of maltreatment—for example, hitting a child is not always physical abuse and leaving a child unattended is not always neglect. In addition, whereas caseworkers may weigh multiple factors in determining the risk of future harm in physical abuse and neglect cases, continued exposure to a perpetrator of sexual abuse alone is sufficient to determine a risk of future harm. In addition, workload may have a lesser impact on sexual abuse investigations because much of the evidence collection burden falls on, or is shared by, others, including doctors who complete the medical exam, child advocacy centers that conduct the videotaped interview of the child, and law enforcement officers who share information from their concurrent investigations.

We also found that, whereas the use of alternative response was associated with lower rates of substantiation for physical abuse and neglect, this was not the case for multi-type maltreatment. Although states and counties exercise various standards in determining which cases are eligible for an alternative response, and we are not privy to those decision-making processes, it is likely that more complex cases are least likely to involve an alternative response.

In considering these findings, readers should take note of a few limitations. Most importantly, there are other factors for which we could not account that may explain variability in substantiation rates. Perhaps most importantly, in many states, there is a statewide hotline for reporting maltreatment but the decision of whether to investigate is left to the county of responsibility. This can create significant variability in the risk composition of investigations – counties that screen out a large proportion of referrals are likely to have higher substantiation rates (as a proportion of investigated cases) because the cases they assign for investigation are a relatively narrow and (likely) higher risk pool. However, states do not report screened out referrals at the county level (or sometimes at all) in any current database, making it highly difficult to understand the variability in screening decisions. As a national standard for data collection, tracking of screened out referrals at the individual level is politically infeasible due to privacy and due process concerns, but it would be advantageous for researchers and policymakers alike to track aggregate measures of screening decisions at the county level. Second, state policies are constantly evolving and changes in substantiation rates could reflect changes to definitions of maltreatment, standards for screening or investigation, or other practices that are difficult to measure. We also acknowledge that our method of analysis cannot determine causality of state or county factors, especially given the use of cross sectional data. In addition, the aggregation of data to the county level does not allow for examine of case-level factors which weigh heavily on the substantiation decision.

Implications and Conclusions

For a system to be perceived as fair and legitimate in the eyes of reported families as well as the general public, the decision to substantiate (which carries the label of perpetrator) must be made based on fair, objective, and consistent criteria. Our findings suggest that this important decision is, at least in part, driven by organizational factors that are unrelated to the facts or circumstances of a case. Concerns about erroneous decision-making has sparked calls for increased use of structured risk assessment approaches (Camasso & Jagannathan, 2013) to guide decision-making and for accountability through increased performance measurement and data collection requirements (Geen & Tumlin, 1999). However, these responses are unlikely to result to the desired changes without a clear understanding of the decision-making context and may have unintended consequences. Although caseworker-level subjectivity is a justifiable target for reform, counties have a great degree of influence on how decisions are made by individual caseworkers, through shaping the agency culture and expectations, mentoring and supervision, and the time available to caseworkers for investigating.

It is immensely improbable that, in the average county, more than 80% of allegations are false. Rather, as others have long argued, substantiation is a metric of the severity of allegations and quality of evidence, among other factors, and it may bear no relation to future risk or need for services (Drake & Jonson-Reid, 2000; Kohl et al., 2009). Indeed, some have argued that most unsubstantiated investigations involve the occurrence of maltreatment, or minimally, a need for preventive services (Drake, 1996); yet, because substantiation is used as an indicator of truth, a designation of “unsubstantiated” is itself a justification for not providing services. As a result, reliance on substantiation is likely to leave families (and children) at risk. However, even if substantiation and service provision were fully decoupled, it remains important that substantiation determinations be objective, consistent, and meaningful. Because substantiation can determine eligibility for employment or volunteering at schools or day cares, or licensure as a foster parent, these determinations have widespread implications for the safety of children, in addition to the fair and equal treatment of the accused.

How should states handle substantiation? The answer to this question largely depends on the goal – is substantiation used for tracking maltreatment rates, to guide decision-making about intervention, to identify individuals who should be excluded from employment involving children, or for some other purpose? We suggest that, for tracking maltreatment rates, states could uniformly adopt a multi-category disposition that focuses only on the quality of evidence and has no relation to the need for services or perceived severity or harmfulness of the alleged maltreatment. For example, caseworkers could rate whether the allegations contained in the report were true, likely true, unclear, likely false, or demonstrably false. Some states use this type of approach already, which can provide “confirmed victims” and “likely victims” estimates to better understand how rates of maltreatment may be changing over time. For determining whether to directly intervene, caseworkers could rate – if true or likely true –whether the allegations are sufficiently severe to warrant CPS-funded and monitored intervention. Lastly—with consideration of the allegations and an assessment of family and child risks—caseworkers would evaluate whether the child or family has need for voluntary services that would not be funded or monitored by CPS. This last step would ensure that cases with weak evidence, or strong evidence but low severity, are nevertheless considered for voluntary services that could prevent escalation of maltreatment risk. Although some states do more clearly separate the provision of services from substantiation, few are transparent about how severity factors into the substantiation or service provision decisions.