It's not as simple as it sounds: Problems and solutions in accessing and using administrative child welfare data for evaluating the impact of early childhood interventions

Abstract

In recent years, there has been increasing interest in using administrative data collected by state child welfare agencies as a source of information for research and evaluation. The challenges of obtaining access to and using these data, however, have not been well documented. This study describes the processes used to access child welfare records in six different states and the approach to combining and using the information gathered to evaluate the impact of the Early Head Start program on children's involvement with the child welfare system from birth through age eleven. We provide “lessons learned” for researchers who are attempting to use this information, including being prepared for long delays in access to information, the need for deep understanding of how child welfare agencies record and code information, and for considerable data management work for translating agency records into analysis-ready datasets. While accessing and using this information is not easy, and the data have a number of limitations, we suggest that the benefits can outweigh the challenges and that these records can be a useful source of information for policy-relevant child welfare research.

In the past 10 years, policy makers, researchers, and funders have increasingly called upon state human service agencies to share data and information as a means to both improve services for families and to support research and evaluation of policies and programs (Academy of Medical Sciences, 2006; Lee, Warren, & Gill, 2015; Council of Professional Associations on Federal Statistics, 2014; Goerge & Lee, 2013). Administrative data that are collected and compiled by state and local agencies have the potential, it has been argued, to serve as an existing source of information that could be useful for answering a variety of important research and evaluation questions (Brownell & Jutte, 2013; Coalition for Evidence-Based Policy, 2012). For example, in 2013, the federal agency that oversees child welfare interventions, funding, and research, the Children's Bureau, issued an Information Memorandum (ACYF-CB-IM-13-02) to state child welfare agencies urging them to work with program evaluators to facilitate access to child welfare administrative data for research purposes, noting that sharing this information provides a broad benefit to the field of child welfare. Specifically, by providing information on policy-relevant outcomes such as incidents of abuse and neglect and episodes of foster care, states can support relevant and rigorous evaluation to contribute to the much-needed evidence base of successful interventions to prevent maltreatment and ameliorate its negative consequences.

Despite the logic of using administrative data to evaluate intervention effectiveness, the process of obtaining, manipulating, analyzing, and interpreting this information, which is typically not collected for research purposes, is complex (Lee et al., 2015). This article presents an example of lessons learned from accessing and combining child welfare administrative data across six states to evaluate an early childhood preventive intervention. We describe the steps we took to develop information access agreements, to match and ensure accuracy of data, and to define and operationalize key child welfare-related indicators across agency databases, as well as the challenges we encountered and the solutions generated.¹ We also provide recommendations for both researchers planning to use administrative data, as well as for the design and improvement of state agency data systems.

1. Using administrative data to evaluate child abuse prevention programs

Efforts to implement and rigorously evaluate child maltreatment prevention programs have expanded considerably over the past three decades. While a number of these programs have shown promise in terms of promoting positive parenting and reducing risk factors for mal-treatment, relatively few programs have examined the impact of services on rates of child maltreatment directly (Howard & Brooks-Gunn, 2009; MacMillan et al., 2009). One commonly cited reason for the absence of studies directly examining child maltreatment outcomes is the difficulty of obtaining reliable information about child abuse and neglect occurrence (Fallon, Trocmé, MacLaurin, Sinha, & Black, 2011). While a variety of measures, including parental self-report, emergency room records, and service provider reports, have been used in evaluation studies (with varying levels of success), state child welfare agency records remain the most direct and widely available source of information about child maltreatment for researchers (Brownell & Jutte, 2013).

Brownell and Jutte (2013) provide a strong rationale for using administrative data as a resource for research related to child abuse and neglect. They note a number of advantages of using administrative child welfare records as a source of outcome information for research. First, these records are not subject to the social desirability bias likely to be present in self-report measures of harsh/abusive parenting behavior (Cichetti & Carlson, 1989; MacMillan, Jamieson, & Walsh, 2003). Compared to parental self-report of their own behavior (the most commonly used outcome measure in most program evaluation studies), documented child maltreatment bears the stamp of ‘objectivity’ at least in contrast to parents' reports of their own abusive/neglectful behavior. Further, while administrative records almost certainly under-represent actual incidence rates, as shown in studies comparing self-reported abuse compared to agency records (Brown, Cohen, Johnson, & Salzinger, 1998; MacMillan et al., 2003), documented maltreatment incidents do provide some externally validated information that maltreatment likely occurred. Reports that are substantiated through agency investigation at a minimum meet that state's criteria for abuse or neglect, although the levels of harm, types of neglect/maltreatment, age of victims, and other factors, as well as the subjective influences brought to bear by agency investigators are also likely to influence the decision to substantiate a maltreatment incident (or not).

Other administrative sources of information about child abuse and neglect, such as hospitalizations and childhood injuries, are likely to under-estimate actual incidence even further by focusing only on those cases that result in physical harm (O'Donnell, Nassar, Leonard, Hagan, Mathews, Patterson, & Stanley, 2009; Spivey, Schnitzer, Kruse, Slusher, & Jaffe, 2009). Third, child welfare administrative records provide information about highly relevant outcomes such as length of stay in foster care that can be linked to service system costs and potential cost-savings of program interventions. Fourth, administrative records provide case level data on a population (within a given jurisdiction) that can be tracked longitudinally without the attrition and loss to follow-up that can plague researchers utilizing longitudinal survey or interview data (Macmillan, Jamieson, Wathen, Boyle, Walsh, Omura, Walker, & Lodenquai, 2007). Thus, there are potential benefits in terms of the level of rigor that can be maintained in studies that utilize administrative records. Further, because these data are available over extended periods of time, researchers can collect data retrospectively and examine patterns of maltreatment for children across a number of years at a significantly lower cost than original-source longitudinal studies (Brownell & Jutte, 2013; Coalition for Evidence-Based Policy, 2012). Indeed, the availability and perceived efficiency of conducting research that uses data that are already collected and compiled is a key factor in the growing interest in using administrative records for research purposes.

At the same time, the challenges of accessing administrative data and using these records for understanding child maltreatment prevention efforts have been noted. As noted above, the most frequently cited problem with the use of state agency records is the likelihood that these documented reports underestimate the actual prevalence of child maltreatment (Fallon et al., 2011; MacMillan et al., 2003). Additionally, there have been concerns about using documented child maltreatment records in evaluating intervention program outcomes due to heightened surveillance by mandated reporters (e.g., program staff) for children in the treatment group (Howard & Brooks-Gunn, 2009; Reynolds, Mathieson, & Topitzes, 2009). Another concern with utilizing child welfare records in research is the variability in definitions of child maltreatment, and in state and local processes for investigating and recording it. States differ considerably in how investigators decide which cases to investigate, the threshold or level of harm required to substantiate the incident, and the types of evidence that are elicited and used in decision-making (Fallon et al., 2011; Runyan et al., 2005). To the extent that child welfare records are combined across different states or data sources, researchers must be cautious in interpreting and synthesizing this information. Goerge & Lee (2002) note that having to access data on a state-by-state basis is a major deterrent to the use of administrative data for research purposes; however, federal datasets do not typically contain the identifying information needed to use those data sources for program evaluation. Moreover, privacy concerns and concerns with confidentiality of child maltreatment records, especially at the individual child or case level, are often used to create unnecessary barriers to research access to this information (Academy of Medical Sciences, 2006; Brownell & Jutte, 2013). Finally, Brownell and Jutte (2013) note that while administrative data sources have long been used successfully in the field of health research and epidemiology, many of the variables important to understanding child abuse and neglect (such as family risk factors, parenting, and even basic socio-demographic information) are often not reliably available in administrative datasets. For this reason, researchers interested in understanding child development, family risk and protection, and child maltreatment have not typically used administrative data, and are therefore unfamiliar with the processes for accessing, linking, and manipulating these data for research purposes. While administrative data records provide a potentially useful source of information, they are not usually developed or structured for individual level, longitudinal data analysis that is standard in developmental and evaluation research.

This paper provides a description of the approach and methodology used to address some of these challenges in using administrative data to evaluate long-term child welfare outcomes for an early childhood intervention program. We provide a detailed example of the processes used to access, link, and compile and combine child welfare records obtained from six different states as a means of evaluating a large-scale randomized controlled trial study of the Early Head Start program, a prevention program for low income families with infants and toddlers (see Green et al., 2014, for results of this study). By highlighting methodological issues, providing detailed descriptions of how we operationalized maltreatment variables, and suggesting areas in which state agencies might improve the quality of administrative records, we provide researchers with a template that can be used to facilitate more opportunities for accessing and using child welfare administrative records for program evaluation purposes. Additionally, we aim to increase opportunities for cross-project comparison and synthesis by providing specific techniques for operationalizing administrative child welfare data elements that can be adopted in other research studies.

2. Methodology

This research was initiated by federal agency staff from the Centers for Disease Control and Prevention and the Administration for Children, Youth, and Families who worked together to identify large-scale early childhood prevention programs that might benefit from additional research focused on maltreatment outcomes. Federal staff first identified the peer-reviewed literature for large scale (n > 1000) randomized controlled trials (RCTs) conducted in the U.S. that showed positive impacts on risk factors for child maltreatment but which had not assessed the effect of the intervention on child maltreatment. Of particular interest were interventions delivered through public policy mandates; parent education/training programs were intentionally excluded, as these were the focus of a different effort (see Kaminski, Valle, Filene, & Boyle, 2008). This search identified RCTs with effects on the following risk factors for child maltreatment at various levels of influence: neighborhood social disorder and parental depression (Katz, Kling, & Liebman, 2000; Leventhal & Brooks-Gunn, 2000; Rosenbaum & Harris, 2000), harsh parenting (Eldred & Zaslow, 1998; USDHHS, 2002), family stress (Huston et al., 2003), family poverty (Eldred & Zaslow, 1998; Foley et al., 2002; Huston et al., 2003; Knox, Miller, & Gennetian, 2000), and partner violence (Knox et al., 2000).

Based on a number of factors, the Early Head Start (EHS) program was eventually selected as an appropriate model for further longitudinal research. A long-running randomized study of EHS showed positive impacts on parenting behaviors, maternal depression, and children's social behaviors, particularly aggressive behavior (USDHHS, 2002). These behaviors are potentially linked to child maltreatment, as children who are more socially competent and less aggressive are less likely to be physically abused and neglected (Stith et al., 2009). Children who received EHS services were also less likely to have been to the emergency room for accidents or injuries. Finally, the existence of national standards and infrastructure for the EHS program increased the scalability and sustainability of this particular intervention.

Early Head Start is a two-generation early intervention program for low-income infants and toddlers and their families. Early Head Start was authorized in 1994 with the first 68 grantees funded in 1995 and now serves over 110,000 children per year in over 1000 programs, making it one of the largest programs serving low-income infants and toddlers in the United States. EHS aims to promote positive development in children directly, by providing services to children from birth to three years of age, and indirectly, by providing supports to parents in their role as primary caregivers and promoting parent self-sufficiency and healthy family functioning. EHS programs use two primary service approaches: (1) home visiting, in which weekly 90-minute home visits are provided to families, coupled with group socialization activities; or (2) center-based child development services with at least two home visits per year. Many programs provide EHS services using both models.

Original Congressional authorization of EHS services mandated that the program be rigorously evaluated, and a randomized controlled trial (RCT), referred to as the Early Head Start Research and Evaluation Project (EHSREP), was launched in 1996, at the same time the program began. Across 17 sites in different geographic regions of the U.S., 3001 low-income families with a pregnant woman or an infant (age < 12 months) were enrolled in the study between July 1996 and September 1998. Families in these sites were randomly assigned to receive Early Head Start services or to be in a control group who could utilize any community services except Early Head Start. Extensive data, including parent interviews, direct child assessments, observations of parent–child interactions and of the home environment, but not reports of child maltreatment, were collected at ages 14, 24 and 36 months during the program and again at age 5 and grade 5 follow-ups.

Funding for accessing retrospective child welfare data was allocated through a competitive grant contract to Northwest Professional Consortium (NPC) in Portland Oregon, and led by researchers with long-term involvement in the EHSREP national study. The project was designed as a feasibility study to determine the following: (1) whether child welfare data could be accessed from multiple sites and linked to EHS study data at the case level; (2) whether child welfare data from multiple sites could be meaningfully combined; and if so, (3) whether, and to what extent, EHS study participants were present in state child welfare records. To the extent that these goals could be successfully implemented, a final goal was to explore whether EHS participation was associated with differences in the type or frequency of child welfare involvement.

An initial sample of six of the 17 EHS study sites in five states were selected for inclusion in this preliminary study based on the following: (1) the presence of a local EHSREP researcher with a history of working with the local or state child welfare agency; (2) geographic representation of sites in the United States; (3) ethnic/racial diversity in EHS populations served; (4) representation of both home-based and center-based EHS program models; and (5) availability of locally collected data that might be particularly useful in informing child abuse prevention outcomes.

3. Facilitating access to child welfare data

3.1. Confidentiality and protection of human subjects

While data collection from the original EHS study participants has continued since the start of the study in 1996, significant study attrition has compromised the randomized nature of the original RCT. Thus, an advantage to using administrative child welfare records was the ability to retain all originally randomized participants in the study sample. Further, we sought to link individual child welfare administrative records with the rich longitudinal child and family data collected through the EHSREP. To do this, we needed to secure agreements from state child welfare agencies that would allow them to link their individual child-level records with EHSREP study participants without obtaining informed consent.

Despite growing concerns with privacy and data security, provisions in current federal laws allow this type of data access (Lee et al., 2015). Specifically, the study team used a provision of the HIPAA legislation that allows for research to be conducted using existing administrative records if certain conditions are met. The researchers' affiliated Institutional Review Boards were asked to review the study purpose and methodology and to approve a Waiver of Authorization of Informed Consent that would allow us to move forward with the study. To obtain such a Waiver, four general conditions must be met: (1) the research must pose minimal risk to participants; (2) the research has no adverse effects on the rights/welfare of participants; (3) it is not practical or feasible to obtain direct consent; (3) the research is not possible without disclosure of identifiable information; and (4) identifiable records will be adequately protected from improper use and disclosure. This retrospective data collection met these criteria in that (a) there was no direct contact with study participants; (b) security procedures for protecting confidential information were comprehensive and met high level security standards; (c) the assessment of the outcomes for the entire (randomized) study sample would not have been possible if direct participant consent had been required; and (d) a comprehensive data security plan would be in place for identifiable records with the priority of protection of confidentiality. Portland State University and Harvard University's Institutional Review Board (IRB) granted the Waiver of Informed Consent, as well as IRB approval.

3.2. Establishing relationships with child welfare agencies

Next, the study team contacted the research/data office within the child welfare agency for each of the six states involved in the study to determine the availability of electronic data for the proposed study period (January 1, 1996–December 31, 2009), their initial willingness to share individual-level data, and to identify procedures for developing a data-sharing agreement. In some states, this first step in establishing a relationship with key individuals at the state child welfare agency was facilitated by local EHS researchers with existing relationships to child welfare agency administrators. These local research contacts helped the team to identify the key agency staff to work with, increased the level of trust between the child welfare agency and the external research team, and helped convey the local importance of the project to the state. However, in other states, simply identifying and contacting the appropriate person to work with on questions regarding data sharing was more challenging, and we relied on web-based searches and multiple phone calls to identify the appropriate individuals. The ability to identify a key contact person, establish good communication with this person or persons, and to develop a procedure for identifying the process for moving forward on there search process were key to the success of the project in every state, and typically involved a sometimes lengthy series of emails, phone calls, conference calls, and follow-up communications to move the process forward.

3.3. Negotiating agreements with state agencies

During these initial conversations, it became clear that our institutional IRB reviews and approved Waivers of Informed Consent did not guarantee that states would agree to provide access to the necessary child welfare information. Protocols for obtaining approvals and developing data sharing agreements differed considerably across the six states. In addition to approval by the university IRBs at both primary research institutions (Portland State University and Harvard University), most states required an additional application to a state-operated IRB and/or additional internal data request review teams, as well as formal data-sharing agreements in the form of contracts and/or memoranda of agreement between the contractor (NPC Research) and the state. Two states imposed additional data security procedures and security software that exceeded University standards and required changes in software and security practices. A further complication were the differences in how states and state Departments of Human Services interpreted federal statutes regarding identifiability of data, as well as states' own specific statutes surrounding sharing of client data. In these cases, review of the data sharing protocols by state Attorneys General offices was required, and often required numerous discussions over details regarding the methods for data security and the type of data that could be shared for research purposes. One state ultimately did not agree to share information with the study team, because of ongoing litigation facing the child welfare agency that involved interpretation of data sharing statutes at the state level.² Two additional sites were added as replacement sites, for a final target sample of seven EHS programs in six states (one state had two EHSREP sites).In all, it took between three and 14 months of working with these six states to obtain approvals for data sharing; receipt of data files took up to two years from initial contact with states.

Throughout this process, we made every effort to minimize burden on state agency staff, recognizing that almost all state child welfare agencies, and in particular, research and data offices, were short-staffed. This project was implemented during the height of the financial recession, and many state agencies had undergone budget and staffing reductions. The resulting demands on child welfare research staff time slowed the process of establishing data sharing agreements in several cases. Although we had originally budgeted for funds that could be used to offset state agency staff time (e.g., by paying for staff time to work with our research team or to provide data), assigning staff to this task proved impractical for most states. Typically, the issue was not funding per se, but rather the lack of qualified staff time to devote to working with the team.³ Further, one state's child welfare agency was impacted by a natural disaster causing flooding and relocation of state agency offices, and resulting in additional delays in the data sharing process.

4. Data availability, data definitions, and data comparability

4.1. Availability of historical data

Most of the study states began implementing electronic administrative data systems near the time of initial randomization of participants for the EHS RCT, although most had also significantly revised their system at some point between this study's start (1996) and end (2009) dates. These system revisions (typically implementation of new or revised versions of Statewide Automated Child Welfare Information Systems, which in many states began in the early 2000s⁴) often involved ‘conversion’ of historical data, a process through which some data were archived and made unavailable. Since 1988, states have been asked to submit data on a variety of child welfare-related variables to the Children's Bureau (the federal agency overseeing federal funding and regulation of child welfare services). The Children's Bureau maintains two primary data systems related to child welfare involvement (U.S. Department of Human Services, 1992–2012): (1) the National Child Abuse and Neglect Data System (NCANDS;), a voluntary reporting system that includes a variety of information about child welfare investigations and reports; and (2) the Adoption and Foster Care Analysis and Reporting System (AFCARS), which includes case-level but de-identified information about children in foster care placements. We began our process with each state by requesting information that was specified in either the NCANDS or the AFCARS data files, and that was most important to our purposes of understanding the type and frequency of child welfare involvement among EHS study participants. Specifically, we requested (1) dates of all reports made on the EHS child or parent; (2) disposition of these reports (founded/substantiated or unfounded/unsubstantiated); (3) the type of abuse reported; (4) the perpetrator of the abuse; (5) start and end dates of any out-of-home placements; (6) placement types (e.g., foster care and kinship care); (7) reasons for placement changes or ending; (8) case start and end dates; and (9) case disposition dates and types (e.g., reunification and termination of parental rights).

We chose these data elements for a number of reasons. First, as key variables in the federal reporting system, we hoped that most states would have relatively reliable and consistent information for each of these indicators. Second, based on our experience conducting program evaluation of child maltreatment prevention programs, we believed these to be the most important for constructing key variables likely to be impacted by these interventions. Further, building on recent research that suggests that it is important to “unpack” the causes and consequences of maltreatment of different types (e.g., physical abuse vs. sexual abuse), and which differs in the developmental timing, chronicity, and severity of maltreatment (Cicchetti & Valentino, 2006; Manly et al., 2001) our goal was to create a set of child maltreatment variables that would provide this level of detail for analysis.

We quickly learned that, despite efforts by the Children's Bureau to increase consistency in variable definitions and reporting guidelines, states differ widely in the way that child welfare cases are processed and in how key information such as type of abuse, perpetrator relationship to child, types of foster care placements, and case dispositions are recorded. States also varied considerably in terms of which of these data elements were maintained by the state agency, and which were able to be disclosed to external researchers. Further, as noted by both state agency staff as well as in federal reports (Government Accountability Office, 2003), early data contained in SACWIS databases was not always consistent or reliable. This necessitated detailed conversations between the study team and child welfare research staff to determine which data fields were likely to be entered reliably, and for which periods of time. These conversations were invaluable to better understand how data elements from each state would be able to be later combined by our research analyst.

5. Data matching processes

In order to link the EHS study participants with state child welfare data, each state needed to match the list of participants with their electronic data records. To do this, we established secure data transfer protocols, and each state received a site-specific data file containing the following identifiers for matching with child welfare records: mother and EHS study focus child name (first and last), date of birth, sex, and race/ethnicity. In addition, about two-thirds (68%) of the sample also had a valid social security number, which were provided when available. Using this information, state agency affiliated research staff matched the EHS study sample participants with child welfare case records. In most cases, the specifics of the matching protocols the states used was not shared with our research team. Some agencies employed relatively sophisticated computerized matching software, while others had to search individual child names and identifiers on a case-by-case basis (the average number of children provided for matching in each state was relatively small, about 150–200).

6. Operationalizing and coding child welfare variables

6.1. Availability of information in administrative records

Although research has suggested that the frequency and type of unfounded reports to child welfare are important indicators of risk for child maltreatment (Fallon et al., 2011; Putnam-Hornstein, Cleves, Licht, & Needell, 2013), data related to unsubstantiated reports were not available in most sites due to expunging of these records. Additionally, in one state, child welfare case processing guidelines had been changed to an alternative, non-investigative process by which many cases are not formally substantiated if the child welfare system determined that services are needed. In this instance, we worked with that state's administrative data analyst to operationally define reports that likely would have been substantiated as those in which a report resulted in either opening a child welfare case or providing child welfare services.⁵ Thus, although “substantiated” cases were used in order to increase consistency, understanding differences in the process by which cases are substantiated (or not) was important to ensuring greater cross-state comparability of data.

Further, most states were unwilling to share information that relied upon identifying specific EHS parents. Such information would be useful in understanding the impacts of a family-centered program, such as EHS, on parents' abusive or neglectful behavior more broadly. This was due to concerns about providing information related to children other than EHS study child to the research team. Child welfare data systems typically maintain information about reports and foster care placements at the child level; therefore, information about specific EHS child participants could be clearly identified. Records involving EHS parents, however, might involve children not involved in the original EHS study. Therefore, while we attempted to collect information about whether the perpetrator of maltreatment events involving the EHS focus child was the EHS parent (see below) we were largely unable to collect data about whether specific EHS parents had been involved in maltreatment events other than those involving the EHS focus child.

As described previously, our goal was to obtain a sufficient level of detail in child maltreatment data to allow us to understand types of neglect and abuse experienced by study children, the developmental timing of maltreatment, and the severity and frequency of the maltreatment. However, this level of detail proved difficult to obtain through administrative datasets. All states were able to provide dates of substantiated reports as well as foster care placement start and end dates. While states were also able to provide information about the type of abuse allegations, the relationship of the perpetrator to the child, types of foster care placements, and reasons for placement changes, the way that this information was coded and the level of detail available varied considerably from state to state.

6.2. Differences in allegation types

One major difference across states was whether child welfare agencies provided us with multiple allegation types associated with child maltreatment reports or provided only a single type for all reports, regardless of whether more than one type of maltreatment had occurred. Recent research has suggested that information about the type of maltreatment (e.g., physical abuse vs. neglect vs. sexual abuse) is extremely important in terms of understanding precursors and impacts of maltreatment as well as what interventions may be most effective (Erickson & Egeland, 2002; Manly, Kim, Rogosch, & Cicchetti, 2001). In all but one state, we were provided with multiple allegations for each child maltreatment report (e.g., a single maltreatment report might include allegations of physical abuse, neglect, and emotional abuse). However, one state provided only the “primary” allegation type. Because of this, we could not examine, in all states, differences in frequency of abuse only versus neglect only versus a combination of abuse and neglect. In order to combine the information across states, we needed to assign a “primary” allegation type to each maltreatment report. This necessitated developing decision rules for coding type when multiple maltreatment types were provided for a given report. We decided to designate as sexual abuse a report that had any allegation of sexual abuse. In the absence of sexual abuse, if a report had any allegation of physical abuse, the primary code was physical abuse (note however, that because these allegations were infrequent, they were ultimately combined for analysis). A third category contained reports that had only either emotional abuse or neglect (but had no allegation of sexual or physical abuse). Unfortunately, this precluded cross-site analysis of potential differences in types of abuse that involved multiple abuse types. These decision rules were meant to “rank” the abuse types in terms of severity or other dimensions, but rather to provide some information that differentiates physical forms of abuse from neglect and non-physical abuse. Studies have shown that neglect, for example, seems to have unique etiology as well as different long-term consequences, compared to physical forms of abuse (Stoltenborgh, Bakermans-Kranenburg, & van IJzendoorn, 2013).

As can be seen in Table 1, states varied considerably in codes used to describe maltreatment type.

Table 1.

Maltreatment type codes across states.

Maltreatment type category originally requested	State coding provideda
	Final cross-site code	A	B	C	D	E	F
Physical abuse	Physical abuse	Yes	Yes	Yes	Yes	Yes	Yes
				Physical abuse (substances)		Drug positive infant
Sexual abuse	Sexual abuse	Yes	Yes	Yes	Yes	Yes	Yes
Neglect	Neglect	Abandonment	Yes	Neglect–inadequate supervision	Yes	Failure to protect	Yes
		Lack of supervision		Neglect–failure to protect		Improper supervision	Substantial risk
		Physical neglect		Neglect–other		Medical neglect	Caregiver absence
						Physical neglect
						Abandonment
Threat of harm	Not used, combined with neglect	No	Yes	No	No	Threat of harm–abuse	No
Emotional/psychological abuse	Not used, combined with neglect	No	No	No	Yes	Mental injury	Yes

^aStates coded as follows: “Yes” = the cross site code was present in the original administrative data provided; “No” = the cross site code was not present. Other related codes that were provided by states and recoded into the final cross-site coding system are listed.

6.3. Differences in perpetrator types

As was the case for maltreatment types, information about the perpetrator of the maltreatment varied considerably across the states (see Table 2). Because of this variability, only two codes could be retained to describe the identified perpetrator on a case: (1) biological mother versus other caregiver. In states in which multiple perpetrators were identified for each report, we coded perpetrator as the biological mother if she was any one of the perpetrators, because of the availability of data regarding the mother in the EHSREP dataset. However, it is also important to understand that the ‘perpetrator’ may not be the person who is directly inflicting harm on the child — for example, in many states if there is domestic violence present in the home to which children are exposed, the mother may be indicated as a “perpetrator” by virtue of being present in the home as the father of the child may not be present in the home (Child Welfare Information Gateway, 2012).

Table 2.

Perpetrator codes across states.

Perpetrator type originally requested	State coding provideda
	Final cross-site codes	A	B	C	D	E	F
Mother	Yes	Yes	Primary caretakerb	Yes	Parentc	Yes	Yes
Father	Not used, combined with all other	Yes	No	Yes	Parentc	Yes	No
Father figure	Not used, combined with all other	No	No	Stepfather	Parent partnerc	No	No
Grandparent	Not used, combined with all other	Yes	No	Yes	Yes	Yes
Other relative	Not used, combined with all other	Aunt	Relative	Aunt	No	Step parent	No
		Cousin	Sibling	Cousin
		Step parent
Other non-relative	Not used, combined with all other	Yes	Friend	Yes	Child care provider	Yes	No
			Foster parent		Friend neighbor
			Neighbor
			Paramour
Unknown	Not used	No	Yes	Yes	Yes	No	Yes
All other caregivers NOT motherd	Yes	No	Yes	No	No	No	No

^aStates coded as follows: “Yes” = the cross site code was present in the original administrative data provided; “No” = the cross site code was not present. Other related codes that were provided by states and recoded into the final cross-site coding system are listed.

^bThis state provided guidance that the primary caregiver code was used for the mother; “other primary caregiver” was used to designate any parental figure other than the biological mother.

^cThis state had information that stated only “parent” or”parent partner”. To determine whether the “parent” was the mother or father required additional data collection through direct case file review at this site.

^dThis perpetrator type was created for final analysis as the only other perpetrator category that could be coded across state systems.

6.4. Differences in foster care placements types

In terms of information about foster care placements, the placement type codes also varied by state, as shown in Table 3. Different placement types of different implications for the potential impact of the placement on child well-being (e.g., kinship vs. stranger foster care) as well as for the cost of the out-of-home placement. State child welfare agencies typically provided placement data as a series of placements, some of which were in the form of consecutive placements constituting a single placement episode. We created dichotomous codes for each of the placement type variables, based on whether the child had ever been placed in each type of placement (yes/no). If the placement did not have an end date as of the end of our study period, we coded the placement as “still in care”.

Table 3.

Placement type codes across states.

Placement types originally requested	State coding provideda
	Final cross-site codes	A	B	C	D	E	F
Emergency shelter	No	No	No	Yes	No	Yes	No
Foster home	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Pre/adoptive home	Yes	Yes	Yes	Yes	Yes	Yes	No
Residential treatment or group home	Yes	No	Yes	Yes	Yes	Institution/detention	Yes
Therapeutic foster care	No	No	No	Yes	No	No	No
Guardianship	No	No	No	No	No	Unrelated caregiver	Yes
Kinship care	Yes	Yes	Yes	No	Yes	Yes	Yes
			Other parent
Other out-of-home	Yes	No	No	Acute CRT	Respite care	No	Court specified home
					Court-ordered placement		Small family home
Placed at home	No	Yes	No	No	No	No	No

^aStates coded as follows: “Yes” = the cross site code was present in the original administrative data provided; “No” = the cross site code was not present. Other related codes that were provided by states and recoded into the final cross-site coding system are listed.

Information about the outcome of an out-of-home placement (e.g., reunification with parent vs. freed for adoption) is important to understanding the effectiveness of interventions. States provided information about case disposition in several ways. Some states included a case disposition code; others provided a ‘placement discharge reason’; we interpreted both of these as providing information on what happened to the child at the end of an out-of-home placement episode. States cautioned us that these codes did not necessarily mean that the child was in a permanent placement, although most also stated that these codes represented the current, putatively permanent, placement for the child. Thus, for example, a given child might have more than one “final” disposition code within his/her record. For our purposes, we selected the last disposition or placement discharge code in our study time period for each child. These codes were somewhat more consistent than abuse, perpetrator, and placement codes. All states included codes indicating whether children were reunified with parent(s) or adopted (although adoption codes were sometimes indicated by termination of parental rights and sometimes by placement in an adoptive home). Two states provided codes indicating placement in guardianship, transfer of jurisdiction to another agency (e.g., juvenile justice), or some other final disposition; these additional codes were not used in final cross-site analysis.

6.5. Identifying timing of child maltreatment

A key question of interest was related to timing of child maltreatment events.We were interested in both whether the maltreatment occurred (and was reported) while the child was participating in the EHS program, as well as the development timing of the maltreatment, which may influence the child's well-being. To allow examination of the timing of child maltreatment in relation to program participation (before, during, or after participation in the EHS program) any child welfare report or placement that occurred between the date of random assignment and program exit date was coded as occurring “during” program participation. Of course, this could only be calculated for children in the EHS program group. To create a comparable variable in both program and control groups, we used events that occurred between randomization date and age 3.5 years as a proxy, as participation in the EHS program is limited to children aged birth through 3 and nearly all children had exited by age 3.5 years. This no doubt over-estimates the actual time spent by most children in the program, as the preponderance of children did not remain in services for the entire period; however, it provides a way to compare maltreatment during similar time periods for both control and intervention groups. The maximum possible length of time in the program was used instead of the average length of time in the program as it also creates a more naturalistic time-point at which children are likely to transition to group based preschool settings (around age 4). We were also able to create variables related to the age of the child at each child welfare encounter.

6.6. Defining frequency of maltreatment

To develop a variable that would allow us to understand whether a child experienced recurring maltreatment, we created a sum or count of (1) the number of substantiated reports and (2) the number of out-of-home placements that occurred for each child by summing the number of substantiated reports and the total number of out-of-home placement episodes for each child. Placement counts were related to the placement episode and not to changes in the placement type, with concurrent placements counted as a single episode. Placements were considered to be a single episode if the end date of one placement was within seven days of the start dates of another placement. For example, if data indicated that a child was placed in foster care on January 1, 2000, and remained there until June 30, 2000, at which point there was a new placement start date and end date (e.g., change to a therapeutic group home followed by return home), this was counted as one out-of-home placement episode, with two placements and two placement types. Length of time in out-of-home placement was calculated for the total study time frame as the total number of days spent in all out-of-home placements through the study end date.

Another complication in examining the incidents of maltreatment was that some children had out-of-home placements without associated substantiated reports. Several states clarified that under some circumstances children were placed out of home without a substantiated report. Further, in many states it was clear that once a report was made and investigated, especially if the child was receiving services through child welfare, subsequent reports were unlikely to be investigated and/or retained in the database (unless, for example, the report concerned a different perpetrator or circumstance). Thus, in order to know whether a child had ever been involved with the child welfare system was not as simple as looking at whether the child had ever had a report. To address this, we created a proxy variable indicating whether a child had ever had either a report or an out of home placement (if no reports existed for the child).

7. Final dataset & analysis

Ultimately, we were able to obtain, code, and link child welfare administrative data from six states and seven EHSREP study sites to EHSREP longitudinal data for 1247 EHSREP children. Results of the outcome study have been published elsewhere (Green et al., 2014). Ultimately, we were able to acquired child welfare records that could be used to construct consistent outcome variables reflecting the developmental timing and frequency of child welfare involvement for a longitudinal period spanning over 10 years. This information was used to estimate the impacts of the EHS program on child welfare involvement maintaining the original randomized design (albeit in a subset of sites) and with limited loss to follow-up (but see discussion of mobility, below). Further, we were able to do some limited analysis of the type of maltreatment experienced, as well as limited examination of the perpetrator of the abuse/neglect. In a subsample of sites, we were also able to confirm the accuracy of electronic administrative data in comparison to the case file of record. At the same time, we were not able to access information consistently about unsubstantiated reports of abuse or neglect, conduct more in-depth examination of the type or severity of maltreatment, nor were we able to identify whether parents were involved in maltreatment cases for children other than the EHS study child.

8. Discussion

One of the major purposes of this study was to assess the feasibility of gathering long-term historical child welfare administrative records for the original participants of an early childhood RCT. Our findings suggest that it is feasible, and can produce useful information, but with many obstacles and some significant limitations. Below we summarize some of the key lessons learned in addressing these obstacles, and recommend strategies for potentially improving both the process for accessing data as well asthe limitations inherent in using administrative data for research and evaluation purposes.

9. Lessons learned & recommendations

9.1. Accessing data: ensure adequate time and resources

One of the key lessons for researchers who seek to access child welfare data across multiple states is to plan for the amount of time needed to develop and finalize agreements, and to receive data from state agencies, a key issue noted by Lee et al. (2015). Moreover, there is a need to allocate resources for consistent and persistent follow-up throughout this phase of the work. There was considerable variability in the required procedures and processes for releasing child-level information as well as in the level of responsiveness from various state agency personnel. In one state, the process took as little as three months. In others, the process took close to two years due to reasons ranging from limitations in the availability of state agency personnel to facilitate the review and data sharing processes, unexpected issues such as of flooding of administrative offices (due to Hurricane Katrina) and requirements for additional data security procedures (e.g., purchasing and installing new software). Ultimately, persistent follow-up and a willingness on the part of the study team and funder to be flexible with due dates and responsive to the practical limitations within state child welfare agency research offices resulted in successful access to these important data – eventually.

9.2. Accessing data: overcoming confidentiality concerns

Goerge and Lee (2013) note that legal “statutes are often used by government officials to dissuade potential users of administrative data from pursuing access” (p. 435). While it is unclear whether agency staff that were initially approached for this study explicitly tried to dissuade us from pursuing access by citing concerns with various state-level statutes, certainly several had significant and justifiable concerns, at least at the outset, with confidentiality issues in sharing this information, especially in the absence of informed consent. We were largely able to overcome these concerns by applying a combination of strategies, including (1) providing examples from other states who agreed to share information, as well as providing example data-sharing agreements; (2) developing partnerships with local researchers who had successfully worked with the state child welfare agency on other projects (involving them in helping to ‘brainstorm’ ideas for facilitating access, for example); (3) identifying a ‘local champion’ within the state agency or in their local early childhood policy/services department who could help underscore our messages about the importance of the work; and (4) persistent, but respectful, messages that we could successfully develop a strategy that could address the states' concerns while meeting the project's needs.⁶

9.3. Linking data: ensuring accurate matching of records

One of the major limitations of collecting child welfare administrative data is that the quality and accuracy of data matching done by state agencies is largely unknown. To help increase the probability of successful linking, researchers should collect as many child and parent level identifiers as possible, including social security numbers (present for about two-thirds of the EHS study sample, and available in some, but not all, child welfare records). Further, the extent to which state agencies had access to software that allows probabilistic matching and personnel trained in these advanced techniques likely influenced the quality of data matching. To the extent that researchers can work with state agencies to use this probabilistic matching software, even providing the software and training, the quality of the match is likely to increase. At a minimum, researchers should request and document information about the agency's process for matching records, and be sure to carefully check the outcomes of the matches to ensure as much accuracy as possible.

Another clear limitation of the EHS study was the inability to match children who had moved out of state. Children and parents who moved out of the state in which the EHS programs were located could not be matched in the state's child welfare data system unless the child welfare encounter happened prior to the family's move, or if families moved out of state and then returned (although they may have had reports in other states). Due to limited resources, we were unable to access child welfare data in states other than the original study sites; doing so would no doubt increase the overall prevalence rates among the sample. In the EHSREP study, because we had information about families' location through the children's fifth grade year, we could analyze and describe mobility (although not for the full randomized sample). Results indicated that only 3% of EHS-CWS study children moved out of the original study state for all of the primary data collection points (through Grade 5); however, over a third had a pattern of moving in and out of the state in which they originally received services (41%). Fully one-third were in the original study state for all subsequent data collection periods (37%). Importantly, we found no evidence of either differential mobility across treatment and control groups, nor were there major differences in either baseline characteristics, or in likelihood of having a child welfare encounter, for families who were lost to EHSREP follow-up vs. those who were not (Green et al., 2014). Researchers seeking long-term data on child welfare system involvement must attend to these issues and carefully analyze patterns of attrition and mobility.

9.4. Understanding child welfare data: variability in child welfare system processes

Child welfare systems differ considerably in such basic areas as how they define and respond to different types of reports and allegations, their processes for investigating and substantiating (or not) reports, and the way that they record this information in their administrative data systems (Daro & McCurdy, 1994; Ocasio, Morton, & Simmel, 2013). As others have noted, these variations pose considerable challenges for investigators comparing maltreatment rates in different state sites especially in terms of reliance on ‘substantiated’ reports as a primary dependent variable (Paxson & Haskins, 2009). In our own work prior to this project, we had extensive history working within single states to access child welfare data; the issue of how to combine and interpret data from different systems raised a new level of complexity.

To the extent that researchers have used (and will continue to use) substantiated reports of maltreatment as a key outcome indicator, comparability across states will likely grow even more complicated as more states adopt what are known as Differential Response (DR) approaches (Fluke, Merkel-Holguin, Yuan, & Fuller, 2014). These approaches, which are designed to improve family engagement with needed services, involve a non-investigative, non-adversarial approach that is typically applied when a report reflects a need for services but not a significant safety concern for the child. Differential Response systems are likely to result in fewer substantiated reports, which may further reduce the usefulness of substantiated reports asan indicator of abuse and particularly, neglect (Child Welfare Information Gateway, 2014). While only one of the states in the EHSREP study had implemented a DR system, many more states have begun to implement these systems in recent years. In this state, we worked closely with state agency researchers to identify child welfare cases that, although not technically substantiated, were seen as meeting former/traditional criteria for substantiation. Being aware of each state's process for investigating, reporting, and substantiating is critical to the interpretability of information and consistency across states.

9.5. Variability in data: differences in availability of information

Another limitation in the EHSREP study was our inability to access historical records related to unsubstantiated reports consistently across states. Unsubstantiated reports are a critical source of information about child maltreatment, given the variability across states in how, when, and to what extent reports are investigated described above, as well as the evidence suggesting little or no difference between substantiated and unsubstantiated cases in regards to risk factors or future risk (Drake, Jonson-Reid, Way, & Chung, 2003; Jonson-Reid, Drake, Kim, Porterfield, & Han, 2004; Kohl, Jonson-Reid, & Drake, 2009; Leiter, Myers, & Zingraff, 1994). State agencies should be encouraged to maintain records for unsubstantiated reports so that this information can be utilized for research purposes.

Another potentially useful piece of information that was largely unavailable in child welfare administrative data was information about who reported the event. Information on who reported the event is important for program evaluation research because participation in an intervention may increase the visibility of events such as child maltreatment (“surveillance bias”) and obscure the intervention's positive impacts by elevating the rate of child maltreatment in the intervention group and not the control group (Howard & Brooks-Gunn, 2009). This is a serious methodological challenge for prevention research, and one that led Howard and Brooks-Gunn (2009) to posit that “the difference in surveillance between the treatment and control groups probably explains why so few home-visiting programs have measurable effects on rates of abuse and neglect” (p. 122). While many states do document this information, the level of detail is not sufficient to identify whether staff from particular programs that might be the focus of evaluation were involved in reporting. Even through case file review, it proved difficult to identify whether the EHS program, in this study, had been involved with either reporting or follow-up services. Again, states should be encouraged to record and retain this information; researchers involved in program evaluation may want to attempt to capture reporting done by interventionists in other ways so that surveillance effects can, at a minimum, be explored and/or explained.

9.6. Variability in data: inconsistency in information for coding across key variables

As described previously, there was substantial variability in how important descriptive information related to the maltreatment records was recorded. We provided extensive detail on our coding process in this paper to provide a possible template for other researchers doing work with child welfare administrative data. Using consistent definitions and coding across research studies would help promote comparability of results. More systemically, we would urge states to adopt uniform definitions, as has been proposed by the Centers for Disease Control and Prevention (Leeb, Paulozzi, Melanson, Simon, & Arias, 2008), for such key information as type of abuse, perpetrator relationship to child, and types of out-of-home placements. Meanwhile, researchers who desire deeper information about type, severity, perpetrators of abuse should seek access, and allocate resources to, case file information. In the current study, case file reviews provided insight into both issues as well as circumstances surrounding the maltreatment events; however, only four sites provided access to this information and doing so greatly increased the resources necessary for the research.

9.7. Data quality: understanding the reliability and accuracy of administrative records

A significant concern in using administrative data for research is that the data may not be recorded accurately and reliably (Brownell & Jutte, 2013). These data are collected by child welfare staff for a variety of purposes unrelated to research, and protocols for data entry and quality control are highly variable. In the EHSREP study,we used our somewhat limited access to case file information as an opportunity to cross-check the accuracy of the administrative data provided for these cases. Specifically, we cross-checked report and placement dates, abuse types, and perpetrator information at these sites. Overall, case file information confirmed the accuracy of the administrative data in these four sites to a somewhat surprising extent. The only corrections that were made were in three reports out 169 (2%) had been coded as general neglect that were corrected to physical abuse. While this does not mean that this information is reliably collected in all states, it provides some level of support for data accuracy; researchers who are able to access case files can use these data for this purpose as well.

10. Conclusions: weighing the benefits and costs of using administrative child welfare data

In conclusion, we found that collecting child welfare administrative data retrospectively, despite the many challenges, ultimately resulted in a rich longitudinal dataset to explore impacts of the EHS program on child welfare system involvement. Based on these experiences, we encourage other researchers to take advantage of past and current rigorous evaluations of preventive interventions, as well as other rigorous quasi-experimental studies, to explore their impact on documented child maltreatment. At the same time, pursuing administrative maltreatment records requires a great deal of communication, relationship building, tolerance for bureaucratic hurdles, persistence, and patience. Although the resources needed are likely to be significantly less than what is required for high-quality longitudinal direct data collection (e.g., interviews with parents), it is neither cheap nor easy to obtain and use this information. We would also encourage federal and state agencies to continue to the critical work of moving towards greater consistency in record keeping and in procedures and policies governing access to case-level administrative data. As such data systems improve and expand, their usefulness for research, as well as the ease with which such information can be obtained, will help make this information more valuable for multiple purposes, not the least of which is to support the evidence base related to understanding effective maltreatment prevention interventions.