Traveling without a map: an incomplete history of the road to implementation science and where we may go from here

Abstract

This editorial provides a brief history of mental health services research over the last 30 years and how findings from large-scale studies shocked the field and led to the lines of inquiry culminating in current implementation science research. I review the manuscripts published in this special issue of Administration and Policy in Mental Health in light of that history and usethese studies as a way to assess the state of the field. Finally, I present five takeaways extracted from these articles that may be useful in considering future directions for implementation research.

I wasn’t an IRI fellow but I am a wannabe. I represent the tail end of two generations of researchers that fumbled sideways into implementation science by necessity. Twenty-five years ago implementation science was not on the forefront of thinking in mental health research, and certainly wasn’t known by that name. Now it comprises a – if not the – core set of activities and concepts in mental health services research. The impetus for this seismic change came as the result of a substantial and possibly misguided financial investment and two landmark experiments with unexpected results.

A Brief and Selective History of How Mental Health Services Research Arrived at Implementation Science

In the 1980s, there was growing recognition that most children with mental health problems were not getting appropriate care (Knitzer & Olson, 1982; Tuma, 1989). Inpatient stays for youth were at a startling and unacceptable high (Behar, 1990). This problem was largely conceptualized as one of capacity within the mental health system to provide less restrictive levels of care, and families’ lack of access to care. In 1984, Congress appropriated substantial funds to implement the Child and Adolescent Service System Program (CASSP), envisioned as a comprehensive mental health system of care for youth and their families (Schlenger, Etheridge, Hansen, Fairbank, & Onken, 1992). In the CASSP model, services were provided in the community, included all ancillary services needed for the child and family to be engaged in treatment, coordinated across child-serving systems, and provided in the least restrictive setting possible. The services were to “wrap around” the child and family, addressing every need. Service systems across the country received federal funds to create these systems of care (Stroul & Friedman, 1988). CASSP principles rapidly became the core principles of many advocacy groups and service systems. These principles seemed intuitive, were based on commonly shared values among clinicians, researchers and advocates, and there was every expectation that their operationalization would result in better care – and therefore better outcomes – for children and their families.

The Stark County and Fort Bragg experiments were designed to test this Congressionally-mandated, unprecedented expansion of children’s mental health services. Fort Bragg comprised a quasi-experiment with almost 1000 children and their families. In a rigorous longitudinal analysis, there were no differences between groups on any of 10 child outcome measures of interest (Bickman, 1996a, 1996b). Criticism of the study was swift: it was quasi-experimental; the CASSP principles weren’t fully implemented; and military families’ experiences were not comparable to those of the general public (Hoagwood, 1997; Mordock, 1997; Saxe & Cross, 1997). A second study took place in Stark County, Ohio. This one comprised a randomized trial of 350 children and their families in the civilian population. For all intents and purposes, it replicated the findings from the Fort Bragg study (Bickman, Summerfelt, Firth, & Douglas, 1997; Bickman, Summerfelt, & Noser, 1997). Again there was much criticism; the two most common ones were that the outcome measures were wrong and that the CASSP principles already were widely implemented, meaning that there was little difference between the experimental and control conditions (Friedman & Burns, 1996). Sub-analyses suggested that neither of these criticisms was valid (Bickman et al., 1998). Treatment dose was not associated with outcome, and children in either group who dropped out of treatment had the same outcomes as those who did not.

The principle investigator of both studies, Len Bickman, was searingly honest in his examination of these data. I sat in quiet awe at the University of South Florida’s Children’s Mental Health Conference in 1998, listening to him present these results from five years of data (Bickman, Lambert, Andrade, & Penaloza, 2000) and fervently hoping that it wasn’t true, that an audience member would ask some question or make some perceptive observation pointing out Dr. Bickman’s mistake. If you weren’t around 25 years ago, you may not fully realize the gut punch these results represented. The brutal and not-so-subtle implication was that the best we could do wasn’t good enough to improve behavioral health outcomes for children.

Three Responses to a Crisis

At least three lines of thinking and research emerged in response to these findings. While this taxonomy is overlapping, not complete, and perhaps poorly named, we might categorize these areas as: 1) soul searching, 2) looking outward, and 3) looking inward. Soul searching comprised an in-depth, pre-theoretical and primarily qualitative/ethnographic exploration of service systems and the therapeutic encounter. The fundamental questions underlying much of this research could be construed as, “what did we miss when it comes to the needs of children and their families, the way they access and use services, the way those services are provided, and the construction of the systems in which those services are provided? (Hohmann, 1999; Ware, Tugenberg, Dickey, & McHorney, 1999)” The stakeholder engagement (Aarons, Wells, Zagursky, Fettes, & Palinkas, 2009; Fischer, Shumway, & Owen, 2002; Garland, Lewczyk-Boxmeyer, Gabayan, & Hawley, 2004; Israel, Schulz, Parker, & Becker, 1998), qualitative and mixed-methods inquiry (Palinkas, 2014), and broad compendia of implementation barriers and facilitators that comprise many implementation science frameworks could be considered direct descendants of this line of research, which sought to unpack the “black box” of mental health services and the context in which they are delivered (Bickman, 2000; Birken et al., 2017; Williams & Beidas, 2019).

A second line of inquiry, looking outward, was catalyzed by work of researchers like Charles Glisson, who wisely pointed out that services are provided through organizations, and characteristics of those organizations may affect both how care is delivered and the associated outcomes (Glisson, 1994; Glisson & Hemmelgarn, 1998). If we want to improve outcomes, Glisson and colleagues suggest, we must first change the often overburdened and variably managed organizations in which those services are delivered (Glisson, 2002). An important study conducted a decade ago providing evidence for this hypothesis still constitutes one of the most rigorous tests of an implementation strategy to date (Glisson et al., 2010). Implementation scientists borrowed models and theories from industrial psychology and organizational dynamics, often with a focus on leadership and organizational culture and climate (Novins, Green, Legha, & Aarons, 2013; Powell et al., 2012). To a large extent, most tested implementation strategies rely on this line of research (Powell et al., 2012).

A third line of inquiry, looking inward, was brought to the fore in a commentary by Weisz, Han and Valeri, entitled “More of What?” published right after the publication of the Fort Bragg experiment results. It comprised an insightful treatise on the need to measure use of evidence-based practice and treatment fidelity in these large-scale evaluations (Weisz, Han, & Valeri, 1997). As the authors state, “there is no evidence that any of the specific treatments used [in the Fort Bragg study] had any empirical support.” In fact, research suggested that assessments and treatments delivered in the community had little connection to the evidence base (Bickman, 2000; Garland, Hurlburt, Brookman-Frazee, Taylor, & Accurso, 2010; Lewczyk, Garland, Hurlburt, Gearity, & Hough, 2003). If we want to improve community outcomes, those researchers suggested, we need a suite of readily implementable, demonstrated effective interventions that we move with care into community settings. Researchers begin to wrestle with words like “transportability” and “dissemination” (Schoenwald & Hoagwood, 2001). Rigorous tests of modular (Weisz et al., 2012), trans-diagnostic, and principle-based (Farchione et al., 2012; Kennedy, Bilek, & Ehrenreich-May, 2019) approaches demonstrated the validity of this line of thinking, and have brought into question the population-level utility of complicated, multi-component treatments, which had been the ones most tested in university settings, but which community clinicians often have difficulty using the way they were designed (Beidas et al., 2019). Research on measuring and improving treatment fidelity (especially through direct-to-clinician strategies), and developing treatments that are designed for community implementation by a workforce with little time for training and supervision have developed from this line of thinking (Cheron et al., 2019; Ingersoll & Wainer, 2013).

As these three bodies of research evolved, so did funding mechanisms to support them. The National Institute of Mental Health published funding announcements requesting proposals on implementation, and then partnered with other institutes to form the Dissemination and Implementation Research in Health study section, which welcomed studies examining this problem. As of this writing, NIH and other entities have sponsored more than a decade of conferences to bring together people who develop and test models and strategies to address this set of thorny problems that do not fit neatly into any one discipline. In parallel, the field created training programs for the next generation of researchers who, unlike their predecessors, would be trained from early in their careers as implementation scientists (Baumann et al., 2019; Brownson et al., 2017; Proctor & Chambers, 2017; Tabak et al., 2017).

Evaluating the Implementation Research Institute

And that’s the purpose of the Implementation Research Institute (IRI). IRI itself represents an implementation strategy for the field, a carefully constructed program to create and support researchers devoted to building on the surprising findings of Fort Bragg, Stark County, and similar studies, and to developing methods to figuring out what went wrong and how to fix it. This issue of APMH represents a self-reflection on IRI’s value in advancing the field of implementation science. Bauman and colleague’s article most explicitly quantifies IRI productivity. While determining causality is challenging in these cases, applicants selected for IRI were more likely than those who were not to receive implementation science grants and to publish related articles. By the most proximal and academic metrics, therefore, IRI is successful in moving the field forward. The remaining articles represent perhaps more illustrative examples of IRI’s legacy. Each article comprises both a rigorous study by gifted and well-trained researchers, and a case study of what the field has accomplished. Each offers an important contribution to the field. In what follows I enumerate just a few observations about these contributions, the nice links they form with each other, and what they say about the state of implementation science.

To start, Lau and colleagues nicely lay out one of the fundamental challenges of implementation research: implementation happens all the time, usually doesn’t adhere nicely to a researcher’s schedule, and rarely is amenable to study using traditionally rigorous research designs. Community decision makers have more important problems than to support research; this is their world, we’re just trying to study it. Therefore we often must rely on observational and at best quasi-experimental studies rather than the randomized designs that are considered the gold standard. While randomization may increase internal validity, studying implementation in the wild may be much more ecologically valid and yield more important and actionable results than investigator-initiated trials. Lau et al. explicitly describe the tension between the ethnography and the RCT. They also point to the fact that the “participant” or unit of analysis in implementation studies is the practitioner, and that we must navigate the tension between our ideas as researchers regarding what therapists should do, and the resources and constraints those therapists face.

Saldana and colleagues neatly thread the challenge of rigorous study of the implementation that happens in the wild. The “Stages of Implementation Completion” measure offers a way to quantify implementation progress. Saldana points out the lack of validated, universal measurement tools to assess even the most basic constructs that comprise implementation. There are two aspects of the SIC measure that I think are particularly noteworthy. First, by laying out explicitly the stages of implementation, Saldana, in this and previous work, offers a specific, testable pathway for how implementation happens. One can disagree with her about the sequence or importance of various steps, but careful delineations like this are critical to moving our field forward. In some ways, it is a contribution similar to Proctor’s implementation strategy components of actors, actions, dose, and temporality (Proctor, Powell, & McMillen, 2013). Both cases represent a call to the field to delineate more rigorously what both implementation and implementation strategies mean. I hope that this type of specificity soon will be as expected in implementation studies as a CONSORT diagram is for randomized trials.

A second important aspect of the SIC is that it is adaptable to different implementations. Rather than create a different measure for every study, we should consider measures in which the question stems or general framework is validated, but that specifying the behavior of interest is flexible.

Gopalan and colleagues bring up a different but equally important point related to studying implementation: if the implementation we care about occurs in communities, then studying it requires close community partnerships. Yet scientists rarely are trained in skills needed to form and maintain these partnerships. Their paper nicely lays out these skills and two case studies. It’s important that they focus on early-stage investigators, because developing these skills and relationships often comes at the expense of traditional academic productivity. Teaching scientists to balance these two domains and getting university promotions committees to value the former will be critical to advancing implementation science. Their article nicely lays out the groundwork for an extension of implementation science training.

Hamilton and colleagues use their partnership with the Veteran’s Administration to give an excellent example of “looking inward.” They describe the many psychiatric disorders that frequently co-occur with each other, the VA’s reliance on collaborative care for most of the mental health care it provides, and the challenges that raises for providing evidence-based mental health care, particularly for PTSD, for women veterans. In this case, the requirements of evidence-based treatment and the way it is packaged interfere with its acceptability for patients and clinicians, the clinical profile of the patients, and the structure designed to deliver it. While the authors propose the need for efforts to increase clinician confidence in delivering treatment, I think that their more insightful and potentially actionable suggestion is that we explore other forms of transdiagnostic and modular treatments for this population and this setting.

Brookman-Frazee and colleagues examine system-level effects on implementation. They find that in both schools and community clinics, the role of partnership and leadership are critical. The role of funding and various central policies differ based on the autonomy of entities within the system. Their article provides an excellent example of the need to “look outward,” and the importance of applying the same measures to different settings so that we can have rigorous data on how different organizational missions and structures affect implementation.

Elwy and colleagues examined domains well established in social psychology that have been applied with varying degrees of rigor in implementation science: influence and normative pressure. They ask, how do we leverage it to effect successful implementation? Their conclusion is that behavior change often is most successful when it’s social, a finding that has been replicated in many domains other than implementation. Their study also links nicely to that of Brookman-Frazee and colleagues, in that to the extent organizational variables are important in increasing the use of evidence-based practice, it is likely that they are mediated through some change in practitioner behavior. Linking psychological theories that include construct proximal to behaviors of interest with organizational theories that describe constructs that act upon the practitioner will be critical to developing causal models predicting implementation, and offer a concrete set of levers by which to change implementation.

Five Takeaways from this Special Issue of APMH

In reading these studies through my admittedly biased lens, several themes emerged that may offer some next steps in this excellent work and in implementation research in general:

1. Others have described this in depth before, but central to almost all of these studies is the critical importance of community partnership. For implementation to be successful, those in charge and those on the front line must have input on problem of interest, and be involved in strategy design. We should place a heavy emphasis on reducing burden to those we expect to implement. The implementer is the client and we should address problems of immediate import to them that make their lives easier as well as make their practice more effective.

2. Understand the specifics of the thing you want implemented. Many interventions, such as the ones Hamilton, Saldana, and Brookman-Frazee describe, are multi-component. What drives successful implementation of one component may not be the same as what drives implementation of another. We must be prepared to modify or sacrifice some components in the service of successful implementation. Designs like multiphase optimization strategy (MOST) and Sequential Multiple Assignment Randomized Trials (SMART) will be critical to identifying, not just which components comprise the active ingredients of an intervention, but also which have a chance of success in community settings, paired with implementation strategies (Collins, Murphy, & Strecher, 2007).

3. Several studies in this issue mention or directly describe the role of leadership. A growing body of research suggests that when it comes to implementation, little starts or sustains for very long without committed leadership. While I agree completely, I have concerns about the “implementation leadership” construct. Sometimes leaders agree with us about what should be implemented, and may vary in the extent to which they can effectively implement it. On the other hand, many great organizational leaders may think very little of your intervention. They may think, quite legitimately, that it isn’t right for their organization. Again, the Hamilton and Brookman-Frazee studies provide great examples. We are in danger of equating “agreeing with us” with good leadership. This stands in sharp contrast to what many of the studies here report about valuing community voices and to the skill set Gopalan and colleagues advocate for as part of implementation science training.

4. Providers are human beings. This is articulated best in the Elwy study, which demonstrates that practitioners are subject to the same constraints and biases as all other human beings. The limited resources available in many community mental health settings may exacerbate these biases. It may be worth construing implementation science as a science of adult behavior change within organizations. As such we should take advantage of disciplines from social and industrial psychology and behavioral economics that have been successful in leveraging psychological constructs such as norms and cognitive biases, and in making the behavior of interest as easy as possible to complete.

5. Finally, nowhere more than in implementation science do we feel the tension between internal and external validity. This is Lau and colleague’s central thesis. Often implementation scientists interpret this tension as necessitating a trade-off between rigor and relevance. I think that this is a false dichotomy. I think it relates to the nascence of our field, the fact that constructs and measures are still in development, and that we have not yet agreed on causal theory. We must work as a field to the point that relevance and rigor go hand in hand.

The Elephant in the Room?

There is an unstated set of facts, to which some of the studies in this issue at least obliquely refer: we are working with underfunded, deprofessionalized, often disrespected systems, such as community mental health, schools, and primary care, that serve those who are least able to advocate for themselves. As implementation scientists, we must grapple with these larger issues that directly or indirectly influence every implementation decision. It is exciting to see in recent years that implementation scientists are embracing the study of related policies, because unless we address the fundamental issues of resources and accountability, the changes we attempt to affect won’t sustain.

Reading these manuscripts excites me about the direction our field is taking and gives me confidence in new generations of researchers steeped in ways of thinking and conducting studies directly tied to implementation. The field is in good hands, which is reassuring, because there is much left to accomplish.