Intervention fidelity: Ensuring application to practice for youth and families

Abstract

Purpose

Limited information on intervention fidelity is available in published studies with youth and families. The components of intervention fidelity, the complexity of measurement in these studies, and strategies for measuring intervention fidelity are described.

Conclusions

Strategies for ensuring intervention fidelity according to the Treatment Fidelity Workgroup of the Behavior Change Consortium in the areas of study design, provider training, treatment delivery, treatment receipt, and treatment enactment provide guidance for evaluating or developing intervention fidelity plans.

Practice Implications

Ensuring the quality of intervention fidelity in evidence-based reviews or when developing new interventions is essential for translating findings into practice.

Nursing intervention research involving youth and families has increased over the past decade, with greater emphasis on evidence-based practice for optimal clinical outcomes (Donelan-McCall, Eckenrode, & Olds, 2009; Leahey & Svavarsdottir, 2009; Purdy & Melwak, 2009). With the recent publication of the Institute of Medicine’s (2011) report on The Future of Nursing: Leading Change, Advancing Health, key recommendations include expanding opportunities for nurses to lead and manage collaborative efforts with transdisciplinary healthcare teams to conduct research for improving practice environments and health systems. As more nurses design and implement intervention studies, whether feasibility trials for the development of novel approaches to care or randomized clinical trials to examine the comparative effectiveness of interventions, ensuring intervention fidelity is paramount. The majority of interventions developed by nurse scientists address behavioral change to promote health or prevent disease and related complications. Intervention fidelity refers to the methodological strategies used to monitor and enhance the reliability and validity of behavioral interventions (Bellg et al., 2004). Implementation fidelity or treatment fidelity are terms that are frequently used interchangeably with intervention fidelity. The origin of the term fidelity is the Latin word “fidēlis,” meaning to be faithful or loyal (http://en.wikipedia.org/wiki/Fidelity). Hence, the emphasis is on being true to the developed protocol. Intervention fidelity should not be confused with overall trial fidelity that encompasses not only intervention fidelity but also sample selection, design, reliability, and validity of study measures. The purpose of this paper is to describe the components of intervention fidelity, the complexity of measurement when conducting research with youth and families, and strategies for measuring intervention fidelity. An example of an intervention fidelity plan developed for a feasibility trial to promote personalized exercise in adolescents with diabetes or obesity is described.

THE VALUE OF INTERVENTION FIDELITY

Planning and implementing research that can ultimately improve health outcomes requires considerable time and effort to develop theoretically sound and pragmatic delivery approaches for those intended to benefit from the intervention. Such investment deserves to undergo an assessment of the intervention fidelity so that credibility of the results is maintained. Unless intervention fidelity is explicitly maintained, the extent to which the theory-based intervention being tested is the major influence for the observed change in outcomes will remain unclear (Bellg et al., 2004). Intervention fidelity consists of two key components, the extent of adherence and competence in the interventionist’s delivery of an intervention as intended to effect desired change (Breitenstein et al., 2010a; Stein, Sargent, & Rafaels, 2007). Adherence involves how well the interventionist’s behaviors conform to the protocol (i.e., essential content), whereas competence refers to the skillfulness in the delivery of the intervention (i.e., quality of the implementation process). While efficacy trials include detailed fidelity checks during the establishment and testing of beneficial outcomes, dissemination projects to improve evidence-based practice (effectiveness in “real world” settings) should be regularly checked for adherence to the protocol, perhaps by using procedures that have been streamlined from the efficacy trial (Dumas, Lynch, Laughlin, Phillips Smith, & Prinz, 2001).

How do nurse researchers gain expertise in intervention fidelity measurement? All too often, this topic is not included in graduate research courses or is discussed in limited detail. There is scant reporting of intervention fidelity in journal papers, although the documentation of reliability and validity of study measures or instrumentation is expected and typically included (Bellg et al., 2004). In a study to evaluate intervention fidelity of behavioral health research, Borrelli and colleagues (2005) reviewed 342 papers published from 1990 to 2000 to determine the degree to which investigators reported strategies to maintain provider skills, check adherence to protocol, or report the use of a training manual. Major journals publishing health behavior change research were included, Annals of Behavioral Medicine, Health Psychology, American Journal of Health Promotion, American Journal of Public Health, and Journal of Consulting and Clinical Psychology. These researchers found that 54% of the publications did not include any of the standard fidelity measures, provider skill maintenance, protocol adherence, or use of a training manual. Although there is greater awareness of the value of intervention fidelity and inclusion of fidelity in published research over the past decade, Borrelli and colleagues provided rationale for greater consistency in reporting and indicated the types of intervention fidelity needed to enable accurate evaluation and replication of studies. Intervention fidelity measurement provides an opportunity for the research team to give interventionists appropriate feedback and ongoing education (Resnick et al., 2005b), and to adhere to the recent extension of the recommendations in the Consolidated Standards for Reporting Trials (CONSORT) for behavioral trials (Boutron, Moher, Altman, Schulz, & Ravaud, 2008). Without clear evidence of intervention fidelity, it is impossible for readers to judge whether two interventions were adequately compared or whether a valid study replication has occurred.

Prior to writing this paper, I conducted a literature search using PubMed to explore the current status of papers that addressed community-based pediatric intervention research. Interestingly, my search revealed 60 references from 1992–2011, with 70% published in the last decade. Only one reference was published in a nursing journal and included the work of Breitenstein and colleagues, who tested the reliability and validity of an instrument for measuring implementation fidelity for a group-based prevention intervention targeting low-income parents of young children (Breitenstein et al., 2010a). These investigators (Breitenstein et al., 2010a) acknowledged that few empirically supported parenting interventions have been adopted for use in community-based settings (Prinz & Sanders, 2007). Their rationale for this gap in moving interventions from controlled settings typical of clinical trials to community settings where larger scale adoption can occur is partially due to the failure of program developers to incorporate practical implementation strategies (Breitenstein et al., 2010a).

Behavioral interventions will only become part of clinical practice if they can be incorporated into usual care settings (Leventhal & Friedman, 2004). This integration requires a theory of the mechanisms underlying the behavioral change process that conveys various approaches for moving from implementation of theory, training, reception, and enactment to adherence. This includes understanding issues that affect the training of professionals (e.g., level of individual expertise, commitment, and motivation), how the professional’s style of delivery affects reception, and/or how and whether variation in reception affects differences in enactment and, ultimately, recipient adherence.

COMPLEXITY OF INTERVENTION FIDELITY MEASURES

Although behavioral change to improve health is possible, research evidence suggests that comprehensive intervention approaches at different levels (e.g., individual, family, practitioner, health system, and environment) are needed, tailored to specific settings and target groups. As a component of research dissemination, plans for change should be based on characteristics of the evidence, fidelity guidelines, and barriers and facilitators to change (Bellg et al., 2004; Grol & Grimshaw, 2003). Intervention fidelity measures necessitate a certain amount of complexity in order to be inclusive of all family members that are to receive the intervention and to consider the developmental level of the child and respective family members. Major questions to ponder during the creation of an intervention plan are:

1. What is the level of detail in the intervention (content)? How much flexibility in the delivery of the content (process) is allowed in order to be consistent with the theoretical foundation for the study? Content and process fidelity should be consistent with the conceptual model guiding the intervention (Dumas et al., 2001).

2. Who will deliver the intervention? Will the interventionists be blinded to their assigned study group? What amount of training will be needed for those providing the intervention? What is the length of the intervention and how will fidelity be maintained over time if there are personnel changes?

3. Who are all of the participants: young children, adolescents, parents, siblings, grandparents, or teachers? Will multiple family members be involved? Where will the intervention be delivered (e.g., clinic, home, school, community)?

4. Are there age, culture, or language variations to consider for either the interventionists or the recipients of the intervention? Intervention fidelity must embrace cultural and linguistic sensitivity through collaboration with participants during the planning process (Flores, 2009).

The risk of variance, or deviations from the planned study design, increases as the intervention design becomes more complex when multiple interventionists are needed to implement the intervention and if the intervention is offered multiple times to different groups of participants (Horner, Rew, & Torres, 2006; Santacroce, Maccarelli, & Grey, 2004). Variations across individual sessions can occur as the interventionist adapts the protocol based on assessments of previous sessions, or unplanned changes occur as the interventionist “drifts” from the protocol (Bellg et al., 2004).

Greater precision in developing an intervention fidelity plan that incorporates the “real world” influences mentioned in the above questions can facilitate one’s ability to translate the study evidence to practice settings. Achieving balance between standardization to support adherence and internal validity, and flexibility (customization of the intervention) to support competence and external validity is the goal of a well-designed fidelity plan (Santacroce et al., 2004).

STRATEGIES FOR INTERVENTION FIDELITY MEASUREMENT

Although few researchers have developed and published comprehensive, valid intervention fidelity plans in study reports (Breitenstein et al., 2010b; Stein et al., 2007), relevant strategies for assessing fidelity in behavioral interventions do exist (Resnick et al., 2005a). Due to the interactive nature of behavioral interventions, determining intervention fidelity can be challenging. The Treatment Fidelity Workgroup of the Behavior Change Consortium (BCC), comprised of representatives from the National Institutes of Health, the American Heart Association, and Robert Wood Johnson Foundation, provided recommendations intended to link theory and application in five areas in which to address intervention (treatment) fidelity: study design, provider training, treatment delivery, receipt of treatment, and enactment of treatment skills (Bellg et al., 2004).

Strategies for enhancing treatment fidelity related to study design should be thoroughly described prior to study implementation. How well is the intervention guided by an established theoretical foundation? In what specific ways are the major propositions of the theory reflected in the intervention? Procedures to monitor and minimize the potential for contamination between treatment groups or treatment and control and to measure dose and intensity (e.g., number and duration of sessions, frequency of sessions) need to be outlined prior to the initiation of the study. A major concern when comparing a primary intervention group to an alternative comparison group is that the groups receive exposure to similar treatments because of the poor design or drift in the treatment. Ensuring that there are valid differences in the treatment is an important aspect of intervention fidelity related to study design (Bellg et al., 2004).

Provider training and treatment delivery involve establishing performance criteria for both competence and adherence to the protocol. Standardized training manuals are essential, including measuring skill acquisition and making sure to budget for retraining to check for validity and reliability (consistency) over time and to prevent drift in providing the intervention. Using role playing during training and later observing or using audio or videotaping of the actual intervention to evaluate adherence to protocol using criterion checklists are practical strategies to check delivery (Bellg et al., 2004).

Standardized fidelity measures to assess treatment delivery exist. Examples include the Family-Focused Grief Therapy treatment integrity measure (Chan, O’Neill, McKenzie, Love, & Kissane, 2004), the Yale Adherence and Competence Scale (Carroll et al., 2000), various Motivational Interviewing adherence and competence measures (Miller & Rollnick, 2002), and the Fidelity of Implementation Rating System (Forgatch, DeGarmo, & Beldavs, 2005; Forgatch, Patterson, & DeGarmo, 2005). As with any measure, the validity and reliability must be adequate and reflect congruence with the specific intervention and underlying theoretical foundation.

In addition to verifying that an intervention was delivered as intended, proper interpretation of study results must address subjects’ receipt and enactment of behavioral skills learned though the intervention effects (Lichstein, Riedel, & Grieve, 1994). Receipt of treatment includes processes that monitor and improve subjects’ comprehension and ability to perform treatment-related behavioral skills. Methods of measurement included administering pre-tests and post-tests and structuring the intervention around goal-based objectives. Enactment is the final stage in implementing an intervention and involves subjects’ actual performance of treatment skills in the intended situations and at the appropriate time. How well subjects enact behavioral skills learned is termed intervention adherence, whereas efficacy relates to whether the intervention influences the research or clinical endpoint of interest (e.g., were average glucose levels decreased as reflected by A1C; Bellg et al., 2004).

AN EXAMPLE OF INTERVENTION FIDELITY: A FEASIBILITY TRIAL OF PERSONALIZED EXERCISE FOR TEENS WITH DIABETES OR OBESITY

To lead efforts in designing and implementing interventions for improving health outcomes for youth, particularly those with chronic conditions, feasibility trials that assess recruitment, retention, and intervention fidelity prior to conducting full-scale randomized clinical efficacy or effectiveness trials are greatly needed. The results of feasibility trials with children and adolescents are more common in recent years (Hazen et al., 2010; Long et al., 2011; Wagner, Smith, Ferguson, van Bakergem, & Hrisko, 2011). Despite the increased availability of the published results of feasibility trials with youth, challenges remain in garnering interest in participation on the part of parents who must provide informed consent (approval) for minors and retrieving details of intervention fidelity plans in current literature. Investigators report parental refusal rates as high as 50% (Hazen et al., 2010), and sample sizes for parent and youth participation in feasibility trials tend to be quite small, including seven to 12 parent–child dyads (Hazen et al., 2010; Long et al., 2011; Wagner et al., 2011). Rarely can a separate section entitled Intervention Fidelity be located in published reports. The most common information on fidelity focuses on parental and youth acceptability and satisfaction with the intervention (an aspect of treatment receipt), primarily obtained via questionnaire or survey of those participants who completed the intervention. One must also question the biases of those who remain in a study versus those who drop out. If possible, satisfaction or lack thereof and reasons for withdrawing from those who do not continue in an intervention are also needed for future reference when planning a subsequent trial.

Using the recommendations of the Intervention Fidelity Workgroup of the BCC to assess treatment fidelity in the key areas of study design, provider training, treatment delivery, receipt of treatment, and enactment of treatment skills, strategies employed in a community-based feasibility trial of personalized exercise for adolescents with diabetes or obesity are described. Final results of this study are forthcoming. For adolescents with either type 1 or type 2 diabetes or who were obese and not actively engaged in exercise, the primary aim of this investigation was to determine the level of their adherence to a personalized exercise intervention. The theoretical framework was based upon the integration of social cognitive theory (SCT; Bandura, 2001), family systems theory (Broderick, 1993), and the personalized exercise prescription (PEP) intervention model that has been previously reported along with the study protocol (Faulkner, Michaliszyn, & Hepworth, 2010). We also explored the influence of adolescents’ perceptions of exercise self-efficacy, exercise benefits and barriers, and family social support on adherence to personalized exercise. Therefore, the study design was built on a strong theoretical foundation for exploring exercise behavior change. Research evidence supports the association of individual cognitive processes such as self-efficacy in overcoming barriers to exercise for adolescents, as well as parental encouragement in promoting the benefits of moderate and vigorous exercise (Norton, Froelicher, Waters, & Carrieri-Kohlman, 2003; Tergerson & King, 2002; Winters, Petosa, & Charlton, 2003). Although current literature is inconclusive in determining if self-efficacy is a definite predictor of exercise during adolescence (Sallis, Prochaska, & Taylor, 2000), evidence exists to support a positive relationship between exercise self-efficacy and moderate-to-vigorous physical activity (Garcia, Pender, Antonakos, & Ronis, 1998; Winters et al., 2003). The novel exercise intervention was based on the current cardiovascular fitness level of each adolescent and personal exercise preferences that were identified with the adolescent and supported by a parent. The target for exercise adherence by the adolescents was at least 60 min of moderate to more vigorous physical activity (MVPA) on at least 5 days per week over the 16-week intervention.

Recruitment occurred in a pediatric diabetes clinic in the Southwestern region of the United States following institutional review board approval. Initially, 105 adolescents and their parents were approached and asked to participate in screening procedures. Informed consent and child assent were obtained from 62 parent-teen dyads; seven dyads were excluded after screening, and five additional dyads failed to keep pre-testing appointments. Out of the remaining 50 dyads, the final sample size of those adolescents completing all phases of the study (i.e., pre-testing, intervention, and post-testing) was 39: 20 with type 1 diabetes, 9 with type 2 diabetes, and 10 who were obese (age and gender-adjusted body mass index ≥ 95th percentile). Reasons for adolescents to not complete the study (n = 11) were as follows: (a) lost interest in the exercise program (n = 7), (b) problems with family support (n = 2), (c) disliked wearing the accelerometer (n = 1), and (d) diagnosed with aortic stenosis (n = 1).

The intervention study team consisted of the principal investigator (PI), a doctorally prepared exercise physiologist, and three graduate students who had degrees in exercise science, biology, or physiology and served as research assistants (RAs). Although the adolescents were required to speak English, one of the RAs was fluent in Spanish so that he could communicate with non-English-speaking parents. A detailed study protocol manual was developed and training sessions to orient the graduate students were developed and administered by the PI and exercise physiologist. Since the graduate students were not clinicians, training included not only learning the study protocol but also proper clinic etiquette for recruitment and professional communication with parents and adolescents. Role-playing sessions were established so that the RAs could become familiar with recruitment scripts, use of equipment (accelerometers for tracking exercise adherence), and conducting home visits and fidelity checks for the personalized exercise program. The fidelity checklist was completed at each home visit and included the following information: Has participant followed personalized exercise regimen? Has participant been taking his or her pulse following exercise? Has participant experienced any of the following: low blood sugar (if diagnosed with diabetes), muscle soreness, time restraints, compliance difficulty? What concerns does the participant have about his/her regimen? Any changes requested for the exercise regimen?

The study team met weekly to discuss home visits, fidelity checklists, and accelerometer downloads for frequency, duration, and intensity of the adolescents’ exercise programs. These weekly meetings were an essential aspect of treatment delivery for this feasibility trial so that participant concerns as well as any questions from RAs could be addressed. Any alteration in the personalized exercise program was made by the exercise physiologist and communicated with the adolescent and respective parent. The accelerometer downloads to laptop computers during home visits allowed the RAs to show the adolescent and parent the graphic representation of activity patterns since the previous visit, which averaged about every 2 weeks (treatment receipt). Raw accelerometer counts were used to determine age-specific energy expenditure, which was calculated using a prediction equation for youth developed by Freedson, Pober, and Janz (2005). Energy expenditure measures were used to determine the frequency, intensity, and duration of total activity and exercise bouts per day using SAS® (Cary, NC, USA) programming statements. Thus, accelerometer recordings of exercise over the 16-week intervention were used to determine treatment adherence or enactment. An average of 73.4 ± 27.2 days of accelerometry data were obtained from the final study sample of 39 adolescents. To our knowledge, this study is the first to follow longitudinal data over 16 weeks, with most studies reporting an average of 7 days (Trigona et al., 2010; Troiano et al., 2008).

As part of treatment receipt, we were also interested in obtaining feedback from the participants about refinement of the intervention to further enhance sustainability of exercise in adolescents with diabetes or those who may be at risk for developing diabetes due to obesity. Audio-recorded exit interviews were conducted separately with individual adolescents and parents to obtain information on their satisfaction with the personalized exercise program conducted in community settings of their choosing, the most and least helpful components, and suggestions for improvement. Any adolescents who did not complete the entire 16-week program and their parents were also offered an opportunity to be interviewed. Verbatim transcripts of the interviews were reviewed to identify common themes. Major positive themes included: gaining responsibility to exercise, improving glycemic control, having flexibility in the exercise routines, and wanting to include resistance and weight training. Less favorable themes were: preferring less accelerometer wear time and having difficulty in achieving the recommended goal of 60 min of MVPA per day.

How Do I Apply This Evidence to Nursing Practice?

Tremendous emphasis now exists on translating research evidence into practice. Nurses whose focus is on the delivery of high-quality health care for youth and their families must find innovative approaches for designing and ensuring studies that not only reveal compelling results but also can most easily be incorporated into real-world settings. By incorporating strategies for intervention fidelity measurement in study proposals that include considerations of factors that may affect the delivery, receipt, and enactment of behavioral change interventions, nurse researchers may begin to address the gap in implementing successful, well-controlled studies into applied settings (Glasgow, Lichtenstein, & Marcus, 2003). When reviewing research evidence for translation of credible interventions into practice, one should carefully examine the fidelity measures for study design, provider training, treatment delivery, treatment receipt, and skills in treatment enactment. All too often, studies are designed without forethought regarding how one might actually carry out the intervention in a practice setting. For health promotion interventions that would be done in diverse community settings, the resource availability and practicality of conducting the intervention and obtaining similar positive results are top priorities.

This paper describes the essential components of intervention fidelity, the reasons for including an intervention fidelity plan in study designs, and the potential complexity of intervening with youth and families. Strategies for examining and ensuring intervention fidelity and an example used in a recent feasibility trial are offered. While the inclusion of intervention fidelity plans requires more effort in conducting clinical trials, whether they are feasibility or randomized studies for improving behavioral health outcomes, the credibility of study results will likely be enhanced. Increased rigor in intervention integrity, as well as in overall study design, can be the benchmark for moving pediatric nursing research forward in improving the health of our youth.