Tailoring a Treatment Fidelity Framework for an Intensive Care Unit Clinical Trial

Abstract

Background

Treatment fidelity refers to methodological strategies and practices used to monitor and enhance the reliability and validity of behavioral interventions. Treatment fidelity monitoring enhances internal and external validity and is needed for study replication and generalizability.

Objectives

To describe the implementation, monitoring, and impact of treatment fidelity in an intensive care unit-based clinical trial testing music for anxiety self-management with mechanically ventilated patients.

Method

Development of the criteria was based on the five-component Treatment Fidelity Framework from the Treatment Fidelity Workgroup. Descriptive statistics were used to evaluate adherence rates to the key TF criteria and the reasons criteria were unmet. Descriptive and nonparametric statistics were used to evaluate the impact of TF on participants’ use of the assigned intervention.

Results

The Treatment Fidelity Framework was adapted easily to fit the study interventions. After the initial implementation phase of monitoring, adherence to key criteria was maintained at the targeted level of 80%. The majority of barriers to adherence affected the research nurses’ opportunity to interact with the participant and encourage use of the intervention. There was a trend toward increased use of equipment associated with the assigned condition after initiation of treatment fidelity; however this difference was not statistically significant.

Discussion

Treatment fidelity monitoring is an iterative process that requires ongoing vigilance. Identification of barriers and the implementation of methods to enhance protocol adherence are needed in order to enhance reliability, validity, and generalizability of clinical trials in the dynamic and challenging research environment of the ICU.

Conducting clinical trials in the dynamic, technologically complex setting of the intensive care unit (ICU) can present multiple methodological and logistical challenges. To enhance the reliability and validity of intervention protocols, the design and conduct of clinical trials should include the intentional monitoring of the research protocol integrity as articulated in the original study design. This can be achieved through the development and institution of a treatment fidelity monitoring plan.

Treatment fidelity (TF) refers to the methodological strategies used to monitor and enhance the reliability and validity of behavioral interventions, and the methodological practices used to ensure that a research study reliably and validly tests a clinical intervention (Bellg et al., 2004). The overall goal of enhancing TF is to increase scientific confidence that changes in the dependent variable are attributable to the independent variable (Borrelli et al., 2005). The monitoring not only enhances internal validity but also external validity, as a high degree of TF is needed for both study replication and for generalizability (Borrelli et al., 2005).

The purpose of this paper is to describe the adaptation of one treatment fidelity framework to an ICU-based clinical trial to test music for anxiety self-management in critically ill patients receiving mechanical ventilator support; to describe adherence rates to key TF criteria; and to explore the influence of adherence to key TF indicators on participants’ overall use of assigned condition (experimental or active control).

Method

Tailoring a Treatment Fidelity Framework

Description of the clinical trial

The primary aim of the randomized clinical trial was to determine the effect of individually preferred, relaxing music on anxiety self-management in critically ill patients receiving mechanical ventilatory support. This study was approved by the parent human subjects committee and all of the participating sites’ human subjects committees.

Mechanically ventilated patients were recruited from 12 ICUs in 5 medical centers in the Minneapolis urban area. Patients were approached who were consistently following commands, were participating in their daily care, and could provide informed consent. Participants were assigned randomly to one of three conditions: (a) experimental group of patient-directed music (PDM) listening whereby participants listen to music assembled by a music therapist based on their music preferences whenever they like for as long as they like; (b) active control condition of noise-canceling headphones whereby participants wear headphones to block out ICU noise whenever desired; and (c) control group of usual ICU care for the respective unit. Equipment (CDs, headphones) was kept at the participant’s bedside to enable self-initiation. Participants remained in the study as long as they were receiving mechanical ventilatory support in the ICU, for up to 30 days. Anxiety was assessed daily with two paper and pencil instruments: a 6-item shortened version of the Spielberger State Anxiety Inventory (Chlan, Savik, & Weinert, 2004) and a 100-mm visual analog scale-anxiety.

Development and implementation of the treatment fidelity monitoring plan

In the initial stages of study implementation, many challenges were encountered to consistent implementation of the experimental treatment (PDM) and the active control condition (headphones). These challenges included inconsistent presentation and explanation of the protocol by the research staff, inconsistent participant assistance with equipment by ICU nurses, and site-specific issues related to varying ICU unit cultures such as staffing patterns and staff familiarity with research. To address these challenges, a TF monitoring plan was developed to minimize random and unintended variability, with the goal of ensuring that the research protocol was implemented as originally designed and intended. While other existing fidelity frameworks were considered, the Five-Component Treatment Fidelity Framework was selected for tailoring; it was developed for health behavior change research and clinical practice by the Treatment Fidelity Workgroup, part of the NIH Behavioral Change Consortium (BCC; Borrelli et al., 2005). This model was the most suited for this clinical trial that tests a patient-driven intervention requiring behavior change.

Key considerations during development of the TF plan included monitoring adherence to the research protocol by the research nurses, the participants, and the ICU nursing staff. All research staff required competence in the delivery of the protocol. Competency includes knowledge of the PDM protocol, including suggestions for appropriate music listening times that considered the participant’s preferences and medical status. Further, strategies were needed to monitor compliance, behavior, and comprehension of the participants to verify their knowledge of how and when to use the assigned intervention. Finally, cooperation from the ICU nursing staff was necessary to ensure participants receive assistance with equipment. This is especially important for participants with limited peripheral muscle strength or other physical limitations due to weakness or paralysis.

Four of the five components of the BCC TF framework were tailored to meet the nuances of this clinical trial centered on a patient-initiated, self-directed intervention (a) training, (b) treatment delivery, (c) treatment receipt, and (d) treatment enactment. Issues related to the design component of the TF framework are not addressed in this paper. Guided by the BCC TF framework and previous research experience, the authors developed checklists to monitor and track the integral components of TF efficiently related to the clinical trial. Checklists (yes/no) are typically easy to complete and frequently used in ICUs.

Training

Selection of research nurses and their training were considered carefully by the principal investigator and project coordinator based on specific competencies required for the successful delivery of the protocol to participants. In addition to ICU nursing experience and skill communicating with mechanically ventilated patients, potential research nurses also needed to resonate with the conceptual foundation of the intervention and value the contributions of nonpharmacologic, adjunctive interventions in the ICU. These desired characteristics required careful screening of staff applicants and use of strategic questions during the interview process.

Checklists were used for the initial training of research nurses and quarterly verification of competencies. Training was conducted by the project coordinator and an experienced research nurse and was tailored to each new hire based on previous research and ICU nursing experience. A 2-week training period was set aside to ensure orientation and precepted experiences at all research sites.

Skill maintenance and verification of competencies was addressed through quarterly site visits with research nurses by the PI or project coordinator. A similar checklist format containing key indicators was utilized to ensure research nurses were engaging participants, delivering the protocol as intended, reviewing the study protocol manual, and reviewing principles of human research subjects’ protection yearly (Figure 1).

Figure 1.

Key Treatment Fidelity Criteria Embedded into Daily Checklists

Treatment delivery

The goal of treatment delivery is to increase the likelihood that the content and dose of the intervention are being delivered as conceptualized. In the protocol, this included participants freely self-initiating PDM or use of headphones whenever desired for as long as desired, with minimum listening twice daily for 30 minutes. Processes were instituted to ensure integrity of the treatment delivery component among research nurses, participants, and ICU nurses. Specific strategies were focused on monitoring and improving the delivery of the intervention. These strategies included working closely with the research and ICU nurses and to enhance the environment to allow full opportunity for participants to use the assigned intervention. A checklist was completed by a research nurse with each daily visit to ensure the equipment was working properly and within easy reach at the participant’s bedside. Additionally, research nurses carried the study manual of operations on daily site visits. It contained site-specific information and all protocol details to provide quick reference as issues arose to ensure delivery of the intervention as originally intended.

Treatment receipt

This component of TF ensures that participants understand the information provided about the intervention and includes assessing whether participants are able to use the skills learned. For the current study, this involved providing verbal information to participants on when to use music or headphones to manage their anxiety or provide some quiet time. Visual prompts reminding participants to listen to music or wear headphones and information for family and staff with specific suggestions on assisting the participant with music were posted in the room. Directions were provided also on the equipment operation. This TF component was crucial to the consistent implementation of the experimental treatment condition and the overall success of testing PDM. Strategies to improve participant comprehension and use of PDM were communicated to the research nurses as well as with nursing staff and family members.

Checklists were completed by research nurses during each daily visit to participants (Figure 1). Elements contained in the checklists surrounded the key indicators for TF monitoring to ensure participants had full opportunity to self-initiate PDM or wearing the headphones. These elements included engagement with each participant to review operation of the equipment, to provide prompts for using PDM, and to review music preferences. Research nurses made a point to interact with nursing staff to verify their understanding of equipment operation and when to offer or encourage use of PDM or headphones if a participant required assistance.

Treatment enactment

Processes to assess treatment enactment included strategies to monitor and improve the ability of participants to perform treatment-related skills (PDM use when feeling anxious or desiring rest or relaxation) while receiving ventilatory support. For this study, not only were cognitive and behavioral skills of participants being assessed, but also motor skills, given the prevalence of ICU-acquired weakness and the enrollment of participants with varying degrees of impairment (quadriplegia or paraplegia). Alternate plans for conveying desire for music listening or wearing headphones were required and necessitated verbal and written communication among the research nurses and ICU nurses for those participants with limited or no use of upper extremities. Appropriate communication strategies were determined jointly by staff, research nurses, and participants based on each participant’s physical ability and previous communication strategies such as eye movements or raising a specific number of fingers.

In addition to broad monitoring of the four components of the TF framework, Figure 1 summarizes the established key criteria essential to the integrity of the research protocol. The initial goal was set at 100% adherence to all of the key TF indicators and criteria.

Analysis

Descriptive statistics for barriers to meeting fidelity criteria are presented as percentages (Table 1). The description of the criteria met is presented as a stacked bar chart comparing percentages for criteria met over quarters of the study (Figure 2). The description of equipment time use by criteria met is presented as a bar chart of mean time by day (Figure 3). Comparison of equipment time use between those days when criteria were met and not met was performed using a Mann-Whitney U test as the time use data was highly skewed.

Table 1.

Specific Barriers to Meeting Key Treatment Fidelity Criteria

Specific Barrier	Identified (%)
Patient sedated1,3	27.6
Patient sleeping1	19.7
RN unavailable to discuss protocol (off unit or too busy)2	13.0
Equipment found out of patient reach3	8.4
Patient declined to discuss/review protocol1	5.3
Participant needs assistance with equipment3	4.2
Participant off unit/in procedure/in therapy1	3.6
Participant being transferred1	3.6
Patient weaning-request by RN not to disturb1	2.2
Participant using own CD/HP equipment3	1.5
Participant prefers to watch TV3	1.1
Other**1,2,3	9.9

Notes.

^**participant visiting with family/visitors; participant unstable; participant uncomfortable/in distress; participant chemically paralyzed; RN already familiar with study/equipment; participant being extubated.

¹Impacts research nurse’s ability to interact with research participant

²Impacts research nurse’s ability to interact with patient care nurse

³Impacts participant’s ability to enact the intervention

Figure 2.

Quarterly Adherence Rates to Key Treatment Fidelity Criteria

Note. Line on bar graphs provides reference to the target 80% adherence goal.

Figure 3.

Average Use (minutes) of Assigned Condition (Music or Headphones) by Research Participants Pre- and Posttreatment Fidelity Monitoring

Note. No indicates average equipment use/day prior to treatment fidelity monitoring (n = __). Yes indicates average equipment use/day after treatment fidelity monitoring was implemented (n = __). Five study days presented; the median length of time enrolled is 4.1 days.

Results

Results of adherence to key TF criteria and their influence on participants’ total use time of the assigned condition (PDM + headphones) are presented below.

Adherence to Key Criteria for Treatment Fidelity Monitoring

After the institution of daily checklists for assessing key indicators of TF, approximately 58% criteria met were met in Quarter 1 (Figure 2). Given that 42% of criteria were not met, information was collected on specific reasons why these criteria were unmet to identify barriers that would be amenable to new approaches (Table 1). Barriers identified by the PI and project coordinator were based on previous research experience, site visits, discussion with research nurses, and reasons recorded on the checklists for criteria unmet. Reasons that criteria were unmet fell into three general categories: impacting research nurses interactions with the participant (treatment delivery and receipt); impacting research nurses interactions with the staff nurse (treatment delivery, enactment, receipt); and impacting the participants ability to receive protocol information and enact the protocol. Most of the barriers impacted the research nurses’ opportunity to interact with the participant to encourage use of the intervention: participant sedated (27.6 %); sleeping (19.7%); declined to discuss or review protocol specifics (5.3%); unavailable due to a procedure or therapy in progress (4.2%); transferred from unit (3.6%); and weaning (2.2%). The primary barriers to participants’ enactment of the protocol included sedated/not arousable (27.6%); equipment out of reach (8.4%); required assistance with the equipment (4.2%); and preference for television (1.1%). The main barrier to the research nurses’ ability to educate ICU nurses about the protocol was nurses were either too busy or off the unit (19.7%). All reasons for unmet criteria were discussed in research staff meetings. Strategies to overcome these barriers were developed and implemented as a research team such as strategic timing of participant visits to avoid transfers or procedures and communication among the research nurses as to when individual participant’s would be most awake and available for interaction.

A steady improvement in adherence rates was realized after the second quarter, surpassing 80% consistently after postmonitoring implementation. In Quarters 5, 6, and 8, there was a slight dip in adherence rates; however, this slight drop was not as low as the first quarter postmonitoring initiation. The best overall TF criteria adherence was in Quarter 7 at 90% (Figure 2). Despite some initial improvement, the adherence to key TF criteria remained between 75 and 80%. It became clear that the initial goal of 100% adherence to all TF criteria was not realistic given the unpredictability of the environment and changes in participants’ medical status. Thus, the goal was revised to 80% adherence to the key criteria met on a daily basis with reasons provided for all unmet criteria. Since Quarter 5, all key TF criteria were met or explained if unmet (Figure 2).

Influence of Adherence to Key Criteria

Since the initiation of this clinical trial, daily PDM use and use of the noise-canceling headphones has been monitored closely. Mean equipment use time when the key TF criteria are met was 112.2 minutes (+/−163.6). When key criteria are not met, mean equipment use time was 80.3 minutes (+/−77.5). However, there was no statistically significant difference between these use times of the assigned condition (z = −.08; p = .94) given the wide overall variability among participants’ use times. Equipment use time by participants on day 1–5 of the protocol prior to and after implementation of TF monitoring is presented in Figure 3.

Discussion

Despite close monitoring of key TF criteria and regular discussion of adherence at research staff meetings, 100% adherence was not possible. After revising the adherence rate to 80% of key criteria met, this goal was achieved. When barriers to meeting TF criteria outside the control of the research staff are taken into account, 100% adherence was achieved. These findings highlight the need for TF monitoring to be an iterative process with vigilance to identify barriers and implement methods to enhance protocol adherence.

There are a few limitations to the information presented. When assessing adherence to TF criteria, data were combined from the experimental and active control groups in the belief that adherence to TF was equally important for both conditions. Difference in adherence between the two groups was not assessed. Additionally, given the dynamic nature of enrollment and staffing, adherence rates cannot be compared quarter by quarter, because each quarter contained different participants, nursing staff, research nurses, or other unanticipated occurrences at the research sites. Further, there may have been an unintended impact of the frequent TF monitoring on participants or nursing staff, such as research fatigue from the daily protocol reminders, which should be considered in future trials. Finally, one TF framework was adapted for this clinical trial. There are other models for TF monitoring that may be suitable for ICU studies, or a new framework may need to be developed for this setting.

Recommendations and Conclusions

Treatment fidelity is the extent to which a treatment is carried out as intended; the absence of which can obscure the conclusions about treatment effectiveness. The following are recommended when instituting a TF monitoring plan for behavioral interventions in the ICU or in other technologically complex settings: (a) careful consideration of the integral aspects of the research protocol, (b) identification of key criteria that can affect the integrity of the study specifically to the study site, participants, or nursing staff; (c) development of the TF plan and monitoring methods prior to protocol initiation to allow for any revisions as issues or barriers arise; (d) use of a concise TF checklist for each interaction with participants; (e) inclusion of specific reasons criteria are unmet to identify and address amendable barriers concerning the protocol, participants, or research site; (f) establishment of an evaluation plan for TF adherence (quarterly); and (g) discussion of TF monitoring and protocol adherence at monthly research team meetings to identify strategies to overcome barriers.

Intensive care unit researchers or those researchers engaged in technologically complex settings should develop a TF monitoring plan during the planning phase of a clinical trial and initiate the plan as soon as enrollment commences to enhance the quality and generalizability of findings for patients and the clinicians who care for them.