Methodological Issues in Child Welfare and Children's Mental Health Implementation Research

Introduction to the Special Issue

Implementation research has been characterized as an emerging science with conceptual, methodological and educational challenges (Proctor et al. 2009). It has the potential to close the research to practice gap for evidence-based treatments, a key translational issue in the NIH Roadmap Initiative. However, given the recency of attention to implementation science, numerous conceptual and methodological issues must be addressed for this area to fulfill its promise. The five papers comprising this issue are intended to move implementation science forward by presenting an overarching implementation conceptual model clearly linked to implementation stages, identifying measurement and design challenges based on aspects of the model and offering suggestions for addressing these challenges.

The intellectual work for these papers is supported by a multidisciplinary NIMH-funded advanced center focusing on implementation methods research. The teams of investigators embrace the NIH Roadmap philosophy of strength and innovation through intellectual diversity and hail from the fields of mental health services research, treatment development research, design and statistics, and experience in the dissemination and implementation of evidence-based interventions. The investigators work within a virtual center framework with two organizational anchors: the Child and Adolescent Services Research Center (CASRC, http://www.casrc.org) at Rady Children's Hospital—San Diego, and the Center for Research to Practice (CR2P, http://www.cr2p.org) in Eugene, Oregon.

The focus for the five papers is on implementation research specific to two child service sectors; child welfare and child mental health. The focus on these sectors is historically and conceptually based. CASRC began over 20 years ago by conducting epidemiologic and services research on the need for and use of services, especially mental health services for children involved in the child welfare system and provided mental health care by the public child mental health system. The first decade of studies were observational in design and did not speak to issues related to effective approaches to implementing evidence-based practices in usual care settings.

In 2000, CASRC researchers began to develop intervention studies in collaboration with treatment developers at the Oregon Social Learning Center, the parent organization for CR2P. Subsequent partnerships have led to studies designed to test effective treatment models in community service settings and the current focus on implementation research including the development of the NIMH-funded Advanced Center for Implementation and Services Research.

Conceptually, the focus on both child welfare and child mental health has a clear rationale. First, studies have demonstrated that most mental health care for children involved in child welfare is delivered by the public child mental health system under Medicaid funding and that child welfare serves as a strong gateway into mental health care.

Second, the child welfare system can be characterized as a public health platform with a surveillance system, child protective services (CPS), and post-investigation services that identify a substantial portion of children at very high risk to develop behavior and emotional symptoms that require mental health interventions. Given that CPS is mandated to insure safety, promote stability and support child well being, the system can serve not only as a gateway into the child mental health system for addressing clinical problems, but also as a platform on which to deliver services that may prevent the development of behavioral and emotional disorders.

The first paper in the series, “Advancing a conceptual model of evidence-based practice implementation in public service sectors’ (Aarons et al. 2011) presents a comprehensive, multi-level implementation model that highlights elements hypothesized or empirically supported for each of four phases of the implementation process (Exploration, Adoption/Preparation, Implementation, and Sustainment). The rationale for developing a conceptual model specifically for public service sectors includes the observations that no consensus has yet coalesced around a single conceptual model among implementation scientists and that much of implementation science has its major roots in the business and medicine quality improvement literatures with no assurance that a model developed from these sectors can be readily applied to public mental health, social services, and alcohol/drug settings, especially for children. The framework proposed is based on a clear assumption “that implementation models arise through a lens that is shaped by the service contexts chosen for emphasis,” underscoring the potential utility of explicitly using specific service contexts as the primary driver for a framework. In addition, this framework argues that “different variables may play crucial roles at different phases of the implementation process”, and also provides a rich set of examples from the child welfare system for the first time in a conceptual discussion of implementation.

The next two papers address very different methods challenges in implementation research measurement, one dealing with the measurement of service delivery costs in the child welfare system, and the other addressing the vexing problem of fidelity measurement as systems and investigators move from efficacy studies to implementation research.

The paper on “A Strategy for assessing costs of implementing new practises in the child welfare system: Adapting the English Cost Calculator in the United States” (Chamberlain et al. 2011) reports on a research study designed to examine how well an innovative computerized unit cost calculator developed in England for use in child welfare systems fits with child welfare systems in the United States. This is the first article on the Cost Calculator published in a United States journal. This paper demonstrates the utility of looking at implementation processes and research from a service sector framework but underscores a marked distinction between measurement of service costs in child welfare as compared to the measurement of service costs in health care (including public child mental health). In general, services costs in health care have a well-developed measurement system for assigning monetary value to service activities while child welfare systems do not.

This paper also reports published data from England on how the Cost Calculator has been used to show the considerable difference in case worker service costs based on whether a placement is the first or second/later, especially for children who have behavioral and emotional problems. The example shows the importance of a tool that can assign costs to placement disruptions, and therefore, can provide cost comparisons between usual care practice and evidence-based practice interventions designed to impact placement disruptions.

The second measurement paper “Toward the effective and efficient measurement of implementation fidelity” (Schoenwald et al. 2011) tackles the enormous challenge of translating fidelity measurement, the hallmark of efficacy research, into fidelity measures suitable for use in non-research clinical settings where evidence-based interventions have been implemented. Following a review of the theory and practice of fidelity measurement, the authors use the important distinction between effective and efficient instruments to characterize the marked differences between the context of efficacy and the context of implementation research. Perhaps most helpful is a review of the approaches being used by practitioners of implementation research to address the fidelity measurement challenges. This paper is the first in a program of research that is being supported by the Advanced Center cooperatively with the NIMH-funded Developing Center led by Mark Atkins.

The final two papers discuss the unique challenges of implementation research from a study design perspective by presenting structured reviews of designs that have been used in implementation studies. The first of the design papers, “Mixed method designs in implementation research” (Palinkas et al. 2011), provides an introduction to designs that have become characterized as mixed method, those comprised of both qualitative and quantitative methods. Through the structured review of 22 mental health services research studies that employed mixed method designs, this paper explores the range of designs and serves as the first step in moving toward explicit standards in the methodological innovation.

The second design paper, “Design features in implementation research: A structured review of recent literature” (Landsverk et al. 2011), discusses current controversies in design choices such as number and type of levels, the use of randomization in usual service settings and alternatives to randomization. In addition, the paper presents a structured literature review of empirical studies of dissemination and implementation research since 1995.

The series is not meant to be a comprehensive examination of all methodological issues that challenge the field of implementation research. Rather, it provides an in-depth discussion of a number of critical methodological issues that must be addressed in the emerging scientific study of the dissemination and implementation processes. Each of these papers represents a program of research designed to generate solutions to the challenges identified.