Commentary: Challenges and Opportunites in the Assessment of Fidelity and Related Constructs

Introduction

In this special issue, Bond and Drake (2019) capture some of the key challenges and considerations in evaluation of these measures. It is laudable that this issue focuses on a careful evaluation of the psychometric properties of fidelity measures, as many have not been closely evaluated. Fidelity is considered an implementation outcome, but it may also influence clinical outcomes. As such, fidelity measures have been developed and used in intervention process and outcome research, but the psychometric properties of many have not been examined closely. Reliable, valid measurement is critical to establishing what levels of fidelity are needed both to consider a program fully implemented, to understand factors that are necessary and sufficient for desired outcomes (Ruud, Hoifodt et al., 2020). While it is relatively easy to describe high, moderate, and low fidelity based on presence or absence of specific elements or on the quality of what was provided, it is more challenging to establish what specific elements, doses of exposure, and level of fidelity are sufficient to produce the outcomes we seek. It may also be important to distinguish between programs and interventions, while recognizing that nested within each are numerous specific factors that must be assessed. Many components are nested within a broader program or intervention, and often the elements that are most essential have not been empirically established. Determination of associations between intervention outcomes and intervention fidelity (to specific components or the entire intervention) careful measurement at multiple timepoints (Webb et al., 2010), under circumstances in routine care settings that make it challenging to isolate or experimentally manipulate specific elements of the program.

Levels of Fidelity

Fidelity to Program Characteristics. Programs comprise multiple components, and require fidelity measures to assess the presence of each of these elements. Such measures tend to assess the degree to which each component is in place, and the concept of adherence or extensiveness may be more central for programs than quality or competence. For example, presence or absence of key staff members, policies, or activities indicates whether these aspects of the program have been implemented as intended. In some programs, one or more specific intervention (e.g., Dialectical Behavior Therapy skills group, Seeking Safety) is a nested within the program. For example, in Illness Management and Recovery (Egeland et al., 2019), cognitive behavioral techniques, coping skills, and relapse prevention training are all required elements. These interventions, in turn, may consist of numerous elements that must be delivered competently to be considered to be fully or appropriately implemented.

Intervention fidelity. Fidelity assessments for interventions such as cognitive behavioral strategies are typically developed in clinical trials, and most commonly include assessments of both adherence (whether or not a component was delivered) and competence (the degree of skill with which it was provided). When the entire intervention, or elements of it, are nested within a program (e.g., Egeland et al., 2019), then, it is important to assess whether these interventions, as key components of the broader program, are present. However, there are several challenges to the feasibility and scalability of these components. Fine-grained assessments of intervention fidelity at this level can be challenging, as they often require time-consuming observation or reliable self-report, which can be elusive (which cognitive behavioral strategies or coping skills were emphasized? Were they taught, used and reinforced skillfully and appropriately?). They may take place in the context of scheduled group interventions or classes, but they may also be woven through and reinforced throughout the day (Riggs & Creed, 2017), which can make observer assessment difficult. However, some data suggest that observer ratings are more reliable than supervisor or provider self-report data (Caron et al., 2019); although other studies have found that providers may be able to accurately report on the less nuanced aspects of fidelity (Ward et al., 2013). Additionally, numerous observer ratings—more than are feasible on a large scale-- are required to ensure a stable estimate of the interventionist’s level of fidelity (Dennhag et al., 2012). As noted by Bond et al (2019), rater calibration may be particularly challenging for such items, as raters need to understand what competent delivery of the components looks like. Initial and ongoing calibration can be fairly labor intensive, but necessary to ensure consistent standards and accurate feedback. Such an investment may be important when high-stakes decisions are made—such as certifications of programs or therapists, funding, and policy decisions. Additionally, when there is a clearly established link between fidelity to the intervention and clinical outcomes, it may be particularly important to monitor and support fidelity. In fact, there is some evidence that observation and fidelity monitoring may improve clinical and implementation outcomes (Robbins et al., 2019; Aarons et al., 2010).

Supplementing Fidelity Assessment with Other Measures

Implementation and Quality Measures. Often the measures that are developed for clinical trials are used for assessment once interventions are implemented in routine practice. However, these scales often neglect additional factors that are baked into intervention research, which may themselves influence the degree of fidelity, or the clinical outcomes. Ruud, Hoifoldt, and colleagues’ (2020) finding that organizations are more likely to establish policies related to implementation than they are to fully implement new programs suggests the need for support and structure around the implementation itself. Heiervang and colleagues (2020) point out that fidelity assessment of specific practices does not include measurement of individualization and quality improvement that might influence program outcomes. These activities, which often accompany the intervention or program itself in clinical trials, are important for ensuring quality, consistency, and appropriate care (Lyon, Stanick, & Pullman, 2018). Considering their influence on, and interaction with fidelity, may advance the field’s understanding of how the process vs. content of implementation impacts program or intervention outcomes. In fact, programs may have better outcomes when we begin to consider these elements as essential as the elements of the specific program or intervention.

Adaptation. At both a program level, if applicable, and at the level of a specific psychological intervention, numerous factors may impact capacity and ability to provide the program as originally intended. Some circumstances will require adaptation. Adaptation can take many forms, ranging from changes in setting or format to the number or type of personnel who deliver an intervention. Changes to the content of interventions can range from minor tailoring to changing timing, or adding, removing, or substituting elements. Adaptations can be consistent or inconsistent with fidelity (Stirman et al., 2015; Marques et al., 2019). Some adaptations appear to enhance outcomes (Stirman et al., 2017; Marques et al., 2019). Others, particularly removal of key elements, are inconsistent with fidelity and may lead to decreases in the effectiveness of the program or interventions. Key to determining whether an adaptation is fidelity-consistent is whether core elements of the component are changed. However, more recently, implementation scientists have begun to look beyond the form of an element of the intervention or program, to its actual function or goal (Jolles, Legnick-Hall, & Mittman, 2019). If the function is preserved, program or intervention components can take many forms, as long as the key function has been preserved. For example, if the function or goal of psychoeducation in Illness Management and Recovery (Ruud, Hoifodt, et al., 2020) is to ensure that the consumer understands their condition and how to manage it, psychoeducation could in theory take many forms (peer-led groups, a provider-led orientation meeting, a game, or videos that are watched and then discussed) and could be adapted to accommodate local constraints and consumer preferences, as long as the goal is met. Supplementing fidelity assessment with a measure of adaptations that occur when provided in routine care settings, and examining it in conjunction with evaluation data provides opportunity for learning about what the core functions or elements of interventions actually are essential within different contexts, and which forms are feasible and effective (Stirman, Miller, & Baumann, 2019; Miller, Stirman, Baumann, 2020). As a result of such evaluation, fidelity measures (either decision rules for each item, or the items themselves) may require updating to reflect any new knowledge. This process will ensure that fidelity measures that were developed for the purposes of research reflect the realities and context of routine care.

Conclusion

This special issue presents exemplars of the type of rigorous evaluation that has been lacking for many fidelity measures. Collectively, the articles demonstrate the many considerations that must be made to understand whether key components interventions as they are implemented in communities. Ongoing program evaluation, refinement of these measures, and assessment of complementary constructs will allow the field to continue to advance our understanding of the role of fidelity in successful implementation.