Implementing evidence-based treatment protocols: Flexibility within fidelity

Abstract

Efficacious psychological treatments exist for a variety of mental health conditions, but many who could benefit from these treatments do not receive them. Increasing efforts have been made to disseminate effective protocols, and several approaches for implementing such treatments have been proposed, including the use of protocols, principles, practices, and policies. We discuss the relative merits of disseminating protocols, and highlight the importance of employing flexibility within fidelity. We describe the benefits of using protocols, including their empirical support, guidance for decision making, and structure to facilitate training and enhance treatment integrity. We also address several criticisms that have been offered against protocols, citing data that indicates that many of the criticisms are not warranted.

Although efficacious treatments have been developed, many individuals who could benefit from these interventions do not have access to them (President’s New Freedom Commision on Mental Health, 2003). Efforts to reduce the gap between research findings about effective treatments and the practice of providing effective services have been both long-standing and conscientious, yet not entirely successful. Additional strategies and efforts are needed to achieve optimal implementation.

As noted by the American Psychological Association (APA) Presidential Task Force on Evidence-Based Practice (EBP; 2006), EBP in psychology is an umbrella term that refers to “the integration of the best available research with clinical expertise in the context of patient characteristics, culture, and preferences” (p. 273). In contrast to EBP, empirically-supported treatments (ESTs) are typically developed to address a particular disorder and then evaluated in rigorous efficacy studies (i.e., randomized clinical trials). Even though EST is a narrower term, we argue that protocols that are considered “ESTs” can effectively be implemented using an EBP approach. Thus, we discuss the particular importance of employing “flexibility within fidelity” when implementation efforts are focused on protocols. Consistent with the definition of EBP, this approach includes making appropriate adaptations to protocols based primarily on available research evidence, but also on clinical expertise and patient features (APA Presidential Task Force on Evidence-Based Practice, 2006; Lilienfeld, Ritschel, Lynn, Cautin, & Latzman, 2013).

Research that effectively increases the availability of evidence-based treatments via large-scale dissemination and implementation efforts will benefit from development within a theoretical framework. One such framework is the Consolidated Framework for Implementation Research (CFIR; Damschroder et al., 2009), which incorporates five domains: the intervention, the outer setting (i.e., factors external to the organization), the inner setting (i.e., characteristics of the organization implementing the intervention), the individual (i.e., characteristics of individuals involved in implementation), and the process of implementation (e.g., planning, execution). Although each domain requires attention, a necessary first step is selecting the intervention to be disseminated and implemented. Examining and selecting the specific components of evidence-based care that are best suited for dissemination is a critical element of the CFIR. According to the CFIR (Damschroder et al., 2009), interventions include “core components” (i.e., key elements that cannot be changed) and an “adaptable periphery,” which are the parts of the intervention that can be adapted for the implementation context and setting. The present paper considers the strengths of implementing protocols, which have identifiable core components and adaptable peripheries that can be leveraged to optimize the successful implementation of evidence-based practices (EBPs). We also highlight specific advantages of using treatment protocols and discuss some of the criticisms and how they have been addressed.

Flexibility within Fidelity

A discussion of the merits of protocols first necessitates consideration of the importance of “flexibility within fidelity” (Kendall, Gosch, Furr, & Sood, 2008). Fidelity refers to therapist adherence to a protocol’s treatment components, therapist competence in delivering the intervention, and the degree to which the treatment differs from others (i.e., treatment differentiation; Schoenwald et al., 2011). Flexibility within fidelity refers to the implementation of an EST protocol in a manner that contains the core ingredients to attain fidelity, but that adapts its implementation to be in sync with individual client presentations. A core ingredient, for example, could be the use of homework. For “fidelity” (adherence), homework would need to be implemented; for flexibility, the implementation would allow for variation, such as different homework for different-aged youth (i.e., consideration of client characteristics). A core ingredient may be similar across protocols for different diagnostic groups, but in that case, flexibility within fidelity would warrant a different approach to implementing that “ingredient” (i.e., treatment differentiation). For example, protocols for both depression and anxiety have demonstrated benefits from taking a problem-solving perspective; this ingredient would be necessary for fidelity, but would be implemented differentially for depression and for anxiety. Whereas problem-solving for depression may focus on addressing barriers to behavioral activation, problem-solving for anxiety may involve a discussion of the types of exposures that could effectively target a feared stimulus. Similarly, flexibility within fidelity is relevant to the application of modular approaches to treatment, where core strategies are linked to specific problems and applied sequentially (see Chorpita, Daleiden, & Weisz, 2005; Weisz et al., 2012). In essence, at any point along the sequence, the strategies that are deemed essential for an empirically-supported protocol must be present for the intervention to be delivered with fidelity, but in the spirit of EBP, the application of the core strategy is accomplished in client-specific fashion (i.e., competently). Consistent with the CFIR, “core components” are preserved and presented but within an “adaptable periphery.” Criticisms directed toward evidence-based protocols are based in part on the misconception that protocols must be implemented rigidly. Proper implementation involves a flexible application of manual-based procedures (protocols) in which the therapist is sensitive and adaptive to client characteristics and needs. Thus, when implemented flexibly, protocols lend an ideal opportunity to cultivate an “evidence-based orientation to practice.” (APA Task Force on Evidence-Based Practice with Children and Adolescents, 2008).

Another consideration related to appropriateness for implementation is that treatment protocols/manuals can be of varying quality. At the least desirable end of the continuum are rigid manuals that lack in the communication of the guiding principles. In contrast, quality manuals are principle-driven, acknowledge and guide adaptations, and reflect a consistency between principles and procedures. Preferred manuals include an underlying theoretical framework, treatment and session goals, specific activities to meet the stated goals, as well as strategies for addressing and overcoming potential challenges. The use of treatment protocols does require some level of adherence to the protocol, but treatment goals can and should be met using a degree of flexibility and creativity. Several illustrative examples will be provided. Our examples involve cognitive behavioral therapy (CBT), a treatment approach that has been deemed efficacious (Butler, Chapman, Forman, & Beck, 2006; Hollon & Beck, 2013). Specifically, we will use CBT for youth anxiety to illustrate flexible applications for meeting goals.

Across several specific protocols, the manuals for CBT for youth anxiety incorporate cognitive restructuring (i.e., changing self-talk). Nevertheless, several factors, such as developmental level, insight, and ability to engage in abstract thinking, influence the specific ways in which cognitive restructuring for anxiety is implemented with children and adolescents. For younger children, we may assist with identifying cognitive distortions without mentioning “cognitive distortions,” but by distinguishing between their own thoughts and the things that the “worry monster” tells them. Externalizing anxiety as the “worry monster” often makes it easier for children to generate ideas of what to “say back to the worry monster” (i.e., coping thoughts). In addition, some children prefer to label “coping thoughts” with their own terms, such as “feel better thoughts.” Older children and adolescents learn about “thinking traps,” which they can recognize in themselves and be able to address relative to their own thoughts. Alternatively, adolescents can play a “matching game” to identify the thinking traps that apply to other peoples’ thoughts. Discussion of characters from books or TV shows can be used to facilitate this type of conversation. A protocol may state that the therapist should address cognitive distortions—but the manner in which this is accomplished varies as the manual “come to life” (Kendall, Chu, Gifford, Hayes, & Nauta, 1998). Fidelity would not permit a failure to address anxious self-talk, but flexibility would permit client-appropriate adaptions of the topic.

In his review of manual-based treatments, Wilson (2007) provided additional examples of ways in which therapists individualize protocols, such as formulating a treatment plan that is consistent both with the overall treatment model and the client’s presenting problem, identifying dysfunctional and unhelpful beliefs specific to the client, and maximizing time using strategies found to work for a given client. In short, the benefit of employing a protocol is most evident when it is used with flexibility and within fidelity (i.e., includes core ingredients) rather than as a set of strict requirements to which the therapist is expected to rigidly adhere. As emphasized by Stirman and colleagues (2017; 2015; 2013), there is a distinction between fidelity-consistent and fidelity-inconsistent modifications (and adaptations) to protocols. Modifications are defined as planned or unplanned changes to the content or delivery of a protocol (Stirman et al., 2015). Modifications are often made by therapists to address domains consistent with the CFIR (e.g., inner setting and the individual) that are not specifically incorporated into the treatment protocol. In some instances, these changes are appropriate, fidelity-consistent, and responsive to contextual factors. In other instances, these modifications are fidelity inconsistent and deviate far from the originally-designed intervention. As noted, modifications are most acceptable if they are made within the “adaptable periphery” and do not undermine the integrity of the intervention (Damschroder et al. 2009; Stirman et al. 2015)

In order to assess changes to intervention delivery, Stirman and colleagues (2013) developed a useful model for thinking about modifications to treatment protocols when they are applied in routine care settings. For example, contextual modifications can occur at the format (i.e., group vs. individual), setting (e.g., clinic vs. school), personnel (mental health professional vs. teacher), and population (e.g. anxiety vs. depression) levels. Protocols may be amenable to such modifications. Content modifications (e.g., tailoring, condensing, extending, re-ordering) are also examples of ways that protocols can be implemented flexibly, but only when the decision-making takes into account therapist knowledge, patient characteristics, and available research evidence. In their review, Stirman and colleagues (2017) found that the few studies that actually compared adapted and non-adapted treatments found that outcomes were generally comparable, suggesting that protocols may not be “fragile” as sometimes suggested. Relatedly, Goldstein and colleagues (2012) provide guidance for a step-by-step approach to appropriately adapt protocols for new populations. Additional work is needed to identify the specific types of protocol modifications that are appropriate and sufficiently meet the threshold for fidelity across the various treatment targets.

Overlap Between Protocols and Principles

The phrase “treatment protocol” can easily illicit a sense of required rule following. “Treatment principles” suggests a less rigid directive that is guided by theory. It is noteworthy that although empirically-supported protocols are distinct from evidence-based principles, there are meaningful similarities. EST protocols are generally developed based on a core set of underlying theoretical principles (Clark, 2004), and although certain features may vary across protocols, the underlying principles are similar. The consistency is in the adherence to the underlying principles. For example, versions of CBT for anxiety have been developed for young children (e.g., under age 7; Being Brave: Hirshfeld-Becker et al., 2010; Hirshfeld-Becker et al., 2008), school-age children (e.g. Coping Cat: Kendall & Hedtke, 2006), adolescents (e.g., C.A.T. project; Kendall, Choudhury, Hudson, & Webb, 2002), and emerging adults (e.g., L.E.A.P; Zakarin & Albano, 2016). Each of these protocols share similar ingredients (e.g., homework, cognitive restructures, exposure tasks) that have been tailored to be developmentally appropriate. Although a therapist may wish to follow the specific protocol for a specific age group, it is also reasonable for a therapist who is proficient with any one of the protocols to make the appropriate developmental adaptations.

A common and potentially accurate argument against using protocols is that therapists would be expected to learn a large number of different protocols in order to be prepared for clients with different presenting problems. Carried to an extreme, this argument has some merit. However, the actual number of EST protocols is fewer than one would think, and there are several common principles across them. In essence, one would not need to master a large number of protocols to treat a wide variety of presenting problems. Once therapists have an understanding of the underlying theory and principles of the core ingredients across several protocols, as well as an understanding of how to implement them flexibly, they are prepared to treat a wide range of disorders. Learning about the principles that are the underpinnings of the treatment, the factors that maintain disorders, and the flexible implementation of treatments would extend the usefulness of protocols to various ages and presentations. One could assert that, in many instances, there is a blurry distinction between using empirically-supported protocols and applying evidence-based principles. In our perspective, protocols provide structure for treatment, and provide a clear starting point for disseminating evidence-based treatments.

Strengths of Evidence-Based Protocols

Psychological treatments for mental health conditions have been written about for over a century, but until recently, the descriptions were largely theoretical, difficult to operationalize, and nonspecific. However, mental health treatments can be operationalized and described in sufficient detail to permit reliable application. The use of treatment protocols represents one approach for operationalizing mental health treatments, but this approach is not without shortcomings. Nevertheless, there are several positive features, including empirical support, decision-making, treatment integrity, and training.

Empirical Support

Empirically-supported protocols, by definition, have been evaluated empirically – usually in the context of methodologically rigorous randomized control trials (RCTs). In addition, they are typically developed with a strong theoretical rationale and address the factors that maintain the disorder. Protocols that qualify as empirically supported (Chambless & Hollon, 1998) have been evaluated at more than one site, by more than one investigator, with real patients and with rigorous and blind evaluation of outcomes. One benefit of using an empirically-supported protocol is being able to cite to a client (or a client’s parents) the percentage of people for whom the treatment has been efficacious in the past. Of course, individual variations in response to treatment should be acknowledged, but those varying responses may be addressed by applying the protocol flexibly.

What about treatment protocols that do not have empirical support, or have been found to be less efficacious? Such treatments are not only less likely to have a favorable outcome, but they can also be undermining to the client’s future efforts to change. One would not want a client to end treatment thinking, “I went to treatment and it didn’t help.” Less optimal treatments can also be damaging, as in the client who ends treatment and thinks “I must be really troubled; I went to treatment and I am still not any better.” In short, use of an empirically supported (i.e., found to be optimally effective) treatment is preferred not only for the mere probability of success, but also for the reduced likelihood of undermining client motivation and client treatment-seeking efforts.

Decision-Making

There are many variations of psychotherapy for mental health disorders. Some have been evaluated favorably, some have been evaluated and found to be less effective, and some have not yet been properly evaluated. Given the breadth and variety of interventions available, how does one decide what to implement in a “real-world” setting? As noted by the APA Presidential Task Force on Evidence-Based Practice (2006), an EBP approach requires clinical expertise (i.e., knowledge and proficiency across several domains) and knowledge of research evidence. The use of empirically-supported protocols minimizes this burden to some extent because of the use of clear guidelines to dictate research quality, as well as a clear outline for the clinical strategies to be used. Southam-Gerow and Prinstein (2014) noted that researchers have been attempting since the 1960s to guide provider decision-making when it comes to selecting appropriate treatments. These efforts, over the past several decades, have largely focused on assessing the evidence for protocol-based treatments. The decision-making process for selecting what treatment to implement is often aided by selecting the protocol with the most and strongest empirical support for the target problem. The Evidence-Base Update series (e.g., Dorsey, McLaughlin, et al., 2017; Evans, Owens, & Bunford, 2014; Freeman et al., 2014; Kaminski & Claussen, 2017) in the Journal of Clinical Child and Adolescent Psychology provides disorder-specific reviews of the evidence to guide decisions about psychosocial interventions.

In the Evidence-Base Update articles, treatment approaches are sorted by level of empirical support. The criteria (Southam-Gerow & Prinstein, 2014) include well-established, probably efficacious, possibly efficacious, experimental, and treatments of questionable efficacy. These criteria were adopted from previous work (e.g., Chambless, Baker, Baucom, Beutler, & Calhoun, 1998; Chambless & Hollon, 1998). As Southam-Gerow and Prinstein (2014) stated, “the goal of these various evaluation criteria are to inform professionals and consumers about the state of the science” (p. 3). Perhaps one of the greatest strengths in promoting the implementation of empirically-supported protocols is the high-quality scientific evidence to support their use. The Evidence-Based Update series is a useful resource for understanding the methodological rigor and degree of research support behind each protocol. It highlights the fact that protocols are not all “created equal,” but many have been subject to rigorous testing and comparison. In addition, treatments can be selected based on the level of evidence – those with the most evidence may be most appropriate for implementation efforts.

The effects of any given treatment can vary, and there may be pretreatment predictors that suggest that certain clients will benefit more than others. This information can help decision making. In addition, when there are two or more treatments with empirical support, outcomes may differ depending on moderators (e.g., age, gender, ethnicity; Chorpita et al., 2011). When one examines the evidence-base for a given protocol it is useful to consider predictors, potential moderators, and evidence of generalizability.

Enhances Treatment Integrity

One challenge facing service providers who seek to implement evidence-based treatments in real-world settings is ensuring treatment integrity, or adherence to the treatment protocol. Measuring and maintaining treatment fidelity can be difficult, especially if the criteria for fidelity are not well defined (Schoenwald et al. 2011). Protocols are often well suited for establishing criteria for fidelity, as the inclusion of core strategies are a near-universal feature of protocols. Importantly, fidelity does not “wipe out” attending to the client’s needs. Rigid adherence is likely associated with poorer outcomes (Castonguay, Goldfried, Wiser, Raue, & Hayes, 1996), highlighting again the need for measures of competence (Schoenwald et al. 2011), as well as flexibility within fidelity. For example, the Coping Cat manual provides several analogies relevant to sports to explain self-rating and self-reward. Although such examples resonate with many children and adolescents, other youth prefer analogies that relate to their favorite books, TV shows, or other extracurricular activities. Adapted analogies are considered adherent (i.e. meet treatment fidelity). In another example, a child who has a fear of vomiting will likely generate a different list of feared situations than a child who is afraid of the dark. The process of generating a list of feared situations would be the essential “core component” that would be evaluated for integrity, but the specific entries would vary by client. In each example, the protocol would be administered with integrity (i.e., adherent to the protocol) while also tailoring it for the individual (i.e., therapist competence).

Another potential advantage to using treatment protocols is that they may reduce the number of therapy sessions required. Unlike the gardener who set the hose on spray and seeks to try to water everything, protocols set their hose on stream and soak the target problem. Because protocols include sessions that are intentionally goal-directed and provide clear strategies for meeting these goals, they are likely to address a target problem after fewer sessions than an unstructured approach.

Facilitates Training

Having training and supervised experience in the delivery of an intervention is central to quality implementation (Schoenwald, Mehta, Frazier, & Shernoff, 2013). Similarly, having the intervention in written form (i.e., protocol) facilitates uniform and reliable training and supervision, which should be considered when implementing ESTs in usual-care settings (i.e., transportability; Schoenwald & Hoagwood, 2001). It is unlikely that multiple trainees with multiple trainers will learn the same program if the program is not written down.

A critical element in effective dissemination and implementation of evidence-based treatments is the means through which training is provided (Herschell, Kolko, Baumann, & Davis, 2010). Protocols are particularly well-suited to training because the intervention, in written form, details the core ingredients needed for fidelity. The simple structure of a protocol, too, can be helpful. Learning may be facilitated by the structure inherent in protocols, as trainees can easily be provided with an agenda and a specific set of goals to remain adherent to the treatment. As noted, clinical judgment remains relevant, but protocols have additional “built-in” supports to buttress therapists’ use of relevant strategies.

Efforts to train therapists in evidence-based protocols have been successful. For example, Smith and colleagues (2017) examined how therapists in practice and research settings delivered the Coping Cat intervention for anxiety disorders. Providers were trained via a workshop led by an expert and subsequent ongoing supervision. Findings indicated that delivery of the intervention was similar in both research and practice settings. Although therapists in practice settings delivered an increasing number of non-CBT interventions throughout the duration of the study, the prevalence of non-CBT interventions was relatively low. In another study (Villabo et al., 2017), therapists in clinics in Norway were also trained in the Coping Cat protocol. Although there was some initial hesitancy, the therapists were accepting of the approach and were reliable in their implementation. In addition, the outcomes (reduced youth anxiety) were comparable to the effects found in research clinics (Villabo et al., 2017). These studies provide evidence that community therapists can successfully be trained in a manual-based treatment, and that outcomes are favorable.

Using a manual-based treatment may directly address some other concerns about training and implementation. For example, less time may be required to train therapists about evidence-based protocols than about less structured treatments. Technology can also assist in the effort to be efficient in training. Kendall and Khanna (2008) provided therapists with a computerized training on treatment for youth anxiety (CBT4CBT) and found that participants gained more knowledge about the protocol than those who received manual-based training. Similar results have been found for a web-based training program for Trauma-Focused CBT (Cohen & Mannarino, 2008). Such training approaches may increase the feasibility of training in a protocol, and improve the speed with which such protocols can be disseminated.

Criticisms of Treatment Protocols

Of course, not all protocols are the same. A detailed protocol does not itself ensure efficacy or effectiveness. Until evaluated, a protocol is merely a protocol. However, once evaluated and once findings qualify the protocol as empirically supported, the protocol acquires the status of meriting dissemination and implementation. But even after proper evaluation, we would be naïve and uninformed to not acknowledge and recognize the several criticisms that have been leveled at protocol-based treatments. Of note, several of the misconceptions broadly applicable to EBPs and refuted by Lilienfeld and colleagues (2013) also apply to protocols specifically. In some instances, the criticisms to protocols are applicable, but less so than critics suggest. In other instances, the criticisms may have applied in the past, but they are no longer applicable. The times, they have been a changin’.

Protocols are “Cookbooks”

An early and somewhat common criticism of treatment protocols was that they were “cookbooks:” rigid and insensitive to individual clients’ needs. This criticism, if true and taken to extreme, would be legitimate. However, one treatment does not fit all; protocols are not intended to be implemented like a cookbook, but should instead be implemented in response to the individual client’s needs and presentation.

As the reader is likely aware, there are several disorders - each with their own theory-driven and empirically-supported protocols. And, even within a single protocol, there is room for “flexibility within fidelity.” Core elements of ESTs exist for a variety of disorders, and these elements must be present when assessing adherence to a protocol. Examples include behavioral activation for depression (Dimidjian et al., 2006), behavioral parent training for disruptive behavior problems (Kaminski & Claussen, 2017), and exposures for anxiety disorders (Ale, McCarthy, Rothschild, & Whiteside, 2015). As noted previously, core elements must be included, but they are administered flexibly. The selected specific activities for behavioral activation, for instance, vary from one individual to another, as might the frequency in which the client is expected to engage in these activities.

Many protocols for a target disorder include several nearly universal elements. For anxiety disorders, for example, these elements include homework, problem solving, cognitive restructuring, and exposure tasks. Protocols may not indicate the exact amount of time that needs to be spent on each element, but the protocol will indicate that the element needs to be incorporated. Clients who benefit from one element (e.g., exposure) but not another (e.g., cognitive restructuring) might spend differential amounts of time on these elements. The implementation of a protocol is not the ticking off of items on a list. Rather, adjustments are made based on knowledge of the principles of the treatment and features of the client, and this knowledge is used to flexibly implement the elements.

On the other hand, too much flexibility is problematic. Too much flexibility can “break” fidelity. If a key “ingredient” is not included (e.g., not using exposures for treating anxiety), then there is not treatment fidelity. In a study comparing CBT to usual care in community mental health clinics (Southam-Gerow et al., 2010), 41% of the community therapists did not use the exposure tasks that were intended elements of the protocol. In those cases, if fidelity to exposures is being assessed, not including this key element removes fidelity and may in turn lead to less than optimal outcomes.

Kung Pao Chicken, in restaurants across the US and Europe, has basic ingredients—but the dish nevertheless varies. Sometimes there are water chestnuts, sometimes there are mushrooms, and sometimes the peanuts are sautéed, yet other recipes have peanuts that are dry. Despite the variability in Kung Pao Chicken, there is always chicken and the dish is never Beef Wellington. Similarly, the specific ways in which the empirically-supported protocol is implemented varies, but remains true to the protocol.

Protocols Detract from the Therapeutic Alliance

For decades, with varying degrees of support, powerful attributions for treatment outcome have been made to the therapeutic alliance (i.e., therapist-client relationship). Some argue that the alliance is predictive of positive grains, others show evidence that gains precede the building of an alliance (DeRubeis & Feeley, 1990), and others report a reciprocal relationship between alliance and outcome (Marker, Comer, Abramova, & Kendall, 2013). Meta- analyses examining the alliance-outcome relationship have shown varying estimates of the degree to which alliance is associated with youth treatment outcomes (McLeod, 2011; Murphy & Hutton, 2018; Shirk & Karver, 2003). However, one might consider the following: alliance may be a necessary component, but it is not sufficient.

A criticism of treatment protocols is that the emphasis placed on adherence and delivery of the treatment elements strains or de-emphasizes the development of the therapeutic relationship. Despite this criticism, data indicate that a strong therapeutic alliance has been associated with better outcomes for manual-based treatments (Shirk, Karver, & Brown, 2011), and that youth receiving protocol-driven treatments in research and practice settings have equal or higher alliance relative to those receiving usual care (McLeod et al., 2016). Apparently, there is evidence that high fidelity to protocols and a strong therapeutic alliance can co-exist.

To address concerns about the therapeutic relationship, Addis et al. (1999) suggest strategies to implement during training in evidence-based protocols, such as showing videotaped vignettes and completing role-plays. Again, “flexibility within fidelity” is relevant; therapists can use varying degrees of warmth and directness to deliver the elements of the protocol while maintaining a strong alliance. Therapists may also benefit from the knowledge that the alliance is not as fragile as they may believe: even the introduction of exposures (to fear-evoking stimuli) in therapy does not rupture the therapeutic alliance (Kendall et al., 2009).

Protocols Fit only a Few (Not Broadly Applicable)

Treatment protocols have been described as not applying to diverse clinical presentations or clients. In his review of manual-based treatments, Wilson (2007) pointed to the “all-or-nothing thinking” inherent in this description. Thinking “flexibility within fidelity” broadens the scope. If therapists were to assume that they were restricted to only using the exact words/examples provided in a manual, protocols would be narrowly applicable. That said, protocols are not intended to treat all presenting problems: they are specific to identified/targeted mental health conditions.

Typically, protocols target the main problem (i.e., the most interfering concern or the presenting problem with the highest severity rating), but this does not assume the absence of co-occurring problems. There is often a misconception that RCTs exclude individual who would be difficult to treat or who have comorbidities. However, Wilson (2007) notes that several studies (e.g., Barlow, Levitt, & Bufka, 1999; Stirman, Derubeis, Crits-Christoph, & Rothman, 2005) found that RCTs do not have narrow exclusion criteria. In fact, modern RCTs often include individuals with more severe psychopathology and higher rates of comorbidity than in routine clinical practice (Westbrook & Kirk, 2005). Schindler et al. (2011) substantiated this claim with a study that divided a naturalistic sample into two groups: those who would meet typical RCT inclusion criteria and those who would not. Their data analyses indicated no meaningful differences in treatment outcomes between the two groups, though both groups had a smaller effect size than participants in most RCTs. Post and colleagues (2013) also reviewed the literature to assess how best to generalize efficacy results, and suggest that results of RCTs should be assumed to apply to wider populations unless there is a compelling reason to believe otherwise.

The value of using diagnostic categories in the context of treatment protocols is typically to develop strategies that will benefit a relatively homogenous group (i.e., people with similar clusters of symptoms). Variations in presentations due to comorbidities, level of severity, external stressors, or other factors fall within an expected range. People with similar presenting problems should still benefit from similar treatments, especially if those treatments are flexibly applied to the individual. Evidence for this point comes from several studies that have implemented evidence-based treatments in uncontrolled studies and in naturalistic settings and compared the outcomes to tightly controlled RCTs. For example, Peeters et al. (2013) allowed patients to select their preferred treatment for depression in a naturalistic setting. Several of the participants had high levels of comorbidity and recurrence of depression, and may not have met inclusion criteria for many efficacy studies. Yet, all groups had a significant decrease in depression symptoms. Similar results were found when administering a protocol for severe mood disorders in a partial hospitalization program (Bjorgvinsson et al., 2014). Although participants’ symptoms were more severe than typical in RCTs, the magnitude of symptom improvement was comparable to that found in RCTs. These findings suggest that protocols can successfully be applied to clients with severe and complex presenting problems.

When protocols are evaluated in modern research, the exclusion criteria are for those client features that would dictate an alternate treatment. In RCTs, severe cases and cases with comorbidities are included, but a case with a presenting problem that warrants a different treatment would be excluded. For example, in the Child/Adolescent Anxiety Multimodal Study (Walkup et al., 2008), a wide variety and severity of anxious youth (Kendall et al., 2010) met diagnostic criteria and received anxiety-focused treatments. Youth who met criteria for major depression, however, would be better treated with a program for depression and, thus, were excluded. Protocols do have broad applicability; the excluded clients are those whose presentation would dictate a different treatment.

One additional criticism of protocols is that they are not applicable to youth with diverse backgrounds. The research-to-practice gap is particularly large among racial and ethnic minorities in the United States, as indicated by lower utilization of mental health services and lower quality of care (e.g., Alegría et al., 2008; Marrast, Himmelstein, & Woolhandler, 2016). Although RCTs may be limited by a lack of participant diversity (e.g., race, ethnicity, socioeconomic status), some research has specifically examined the degree to which evidence-based protocols can be adapted for cultural and ethnic minorities. For example, Baumann and colleagues (2015) reviewed the ways in which empirically-supported parent training protocols have undergone cultural adaptation (see also Stirman et al., 2013). As cited by Baumann and colleagues (2015), cultural adaptations involve “the systematic modification of an evidence-based treatment (EBT) to consider language, culture, and context in such a way that is compatible with the client’s cultural patterns, meanings, and values” (Bernal, Bonilla, & Bellido, 1995, p. 362). Although relatively few studies on parenting interventions specifically focused on the effectiveness of cultural adaptations, the current evidence suggests that protocols can be appropriately modified using cultural adaptation frameworks. These results are promising, but addressing the adaptability of protocols for underserved minorities is an area for continued work in all implementation efforts, and does not apply only to protocols.

Resource Intensive

A final issue, which is not necessarily specific to protocols, is the amount of training and supervision required for therapists to become proficient in delivering a given intervention. In the implementation of an empirically-supported or evidence-based treatment, it does not appear sufficient to simply improve therapists’ knowledge about the protocol. To change behavior and increase adoption in practice, supervision/consultation appears to be a critical element of training (Beidas, Edmunds, Marcus, & Kendall, 2012; Edmunds et al., 2013). However, the amount of supervision provided in community clinics is often less than that provided in RCTs (e.g., Kolko, Cohen, Mannarino, Baumann, & Knudsen, 2009). Thus, efforts for implementation appear to benefit from an element of ongoing consultation (e.g., Kelly et al., 2000; Miller, Yahne, Moyers, Martinez, & Pirritano, 2004). Although it does not reduce the amount of resources required, Bearman and colleagues (2013) describe strategies that may optimize supervision and increase the likelihood that activities discussed in supervision will actually be implemented in practice. More work is needed to understand the best strategies for supervision/consultation, but Dorsey and colleagues (2017) have identified the importance of implementation climate as one predictor of the amount of time dedicated to clinical and EBT-related discussions in supervision.

As providers experience success in implementing protocols, attitudes toward them will likely improve. Research has consistently shown that having more positive attitudes toward EBPs is associated with greater likelihood of EBPs being implemented (Farrell, Kemp, Blakey, Meyer, & Deacon, 2016; Harned, Dimeff, Woodcock, & Contreras, 2013; Ruzek et al., 2015; Shafran et al., 2009). To support these arguments, there are several instances in which therapists in community mental health clinics (Foa et al., 2005) and with limited experience (Ost, Karlstedt, & Widen, 2012) have provided treatments and achieved comparable outcomes as therapists in academic settings and with more experience.

Demand for and Representation of ESTs

Efforts to disseminate and implement empirically-supported protocols can be said to be motivated by both sound science and a caring social concern. However, there are factors that can interfere with this good goal. We consider misuse of the claim of ESTs, the social forces that may facilitate eventual adoption of protocols, and the need to hold realistic expectations regarding treatment outcomes.

Using the Buzz Words

A treatment that qualifies as “empirically supported” has had a rigorous level of evaluation (e.g., RCT; use of protocols), but the descriptive use of the label “empirically supported” to claim a level of support without proper reflection on its accuracy is a real and growing concern. There are times when, for a variety of reasons, members of the mental health services professions are engaged in tasks that do not fit neatly into the typical roles for which they have been explicitly trained (e.g., assessment, research, therapy). One of these alternate tasks is fundraising, be it seeking research support, insurance coverage for a clinic, or state funding for an agency. It is often within the arena of financial support that a problem emerges. For example, when seeking continued support from a state, a representative of a clinical service may state that their agency uses ESTs. The state representative who controls the funding wants programs that employ ESTs and reports being impressed by the agency. Although the agency may not intentionally mislead patients or state representatives, this may inadvertently occur through the use of the “buzz words” that overstate the agency’s use of empirically-supported protocols. In this example, the agency representative may or may not have been aware of the important features of research necessary to justify the EST claim, but the fact was that protocols were not in place and a claim of providing ESTs was, at best, a stretch.

Why is the continued misuse of the buzz words (e.g. “EST”) a serious concern? The concern is simple: state officials, program administrators, and the public may be led to draw inaccurate conclusions. By claiming use of an empirically-supported protocol, and then not actually implementing such, any less than favorable outcomes will likely be mistaken as evidence of the ineffectiveness of the protocol. Jump ahead a decade. It will be less than an advancement to hear statements such as, “we tried that protocol or that EST and the outcomes were no better,” when in fact the claim that the protocol-driven EST was implemented and the conclusion of limited effectiveness are both inaccurate.

Forces that may get empirically-supported protocols implemented in the real world?

In the case of mental health treatment for youth, parents often seek services for their children, whether spurred by recommendations from school personnel, mental health professionals, or their own identification of need. Thus, perhaps parents are the best entry point for the promotion of protocol-driven ESTs (i.e., direct-to-consumer marketing; Becker, 2015). If a parent is informed about the effectiveness of a protocol and then seeks a professional trained and competent in providing it, chances are greater that the services provided will have fidelity to the protocol. Even better, perhaps, would be if parents were sufficiently informed about the specifics of the protocol-driven EST so that they would ask their provider if the core ingredients are being done. Not unlike insurance companies who pushed for more time-limited and effective treatments, and the related increase in attention to ESTs, parents may be the source for an effective push for professionals to provide protocol-driven ESTs with youth.

Training programs are an alternate and potentially effective approach. If and when graduate training programs provide ample and sufficient training in ESTs, graduates of these programs (i.e., future providers) will be able to implement the ESTs. One could argue that this process is slow (it would take decades to produce a community of therapists). However, the greatest magnitude of change in treatments, with regard to the development of ESTs, may have been accomplished in the past four decades. Some improvements may lie ahead, but a large portion of the “improvement variance” may have been accomplished. To the degree that this is accurate, the amount of change in ESTs may not be so great that the decades it takes to produce trained service providers would not be worthwhile.

Realistic Expectations

Even when ESTs are not provided, mental health services are not universally bad or without beneficial outcomes. In general, mental health service efforts have beneficial results. For example, ample percentages of youth with anxiety benefit from placebo, from non-protocol treatments, and from treatments not yet well evaluated. What is sought by encouraging the implementation of protocol-driven treatments is an improvement in the percentage of outcomes that are favorable. Going from 20% to 40% and to 60% is a worthwhile effort and goal. But even when a 60% positive treatment response is achieved following implementation of a protocol-driven EST, not everyone receiving the protocol improves to the degree that is sought, and not all gains are maintained. Implementing a protocol-driven EST can be said to maximize the likelihood of the higher percentage of improvement. Not perfect, not everyone, but better than the existing alternatives.

Conclusions

Evidence-based protocols can be effective in a variety of settings for a variety of clients, and they can be implemented flexibly and with fidelity. The critiques that protocols are rigid and overly directive are dated but nevertheless highlight the importance of flexibility in training and implementation – whether for therapists in graduate school or seasoned therapists attending continuing education courses. When protocols are implemented properly and successfully, the public perception of psychological services will be improved; nothing speaks louder than success. Improved outcomes will continue to increase the demand for empirically-supported protocols among consumers, which will spur therapists’ desire to receive training and to implement such interventions.