Establishing Treatment Fidelity in a Web-Based Behavioral Intervention Study

Abstract

Background

Treatment fidelity pertains to the methodological strategies used to monitor and enhance the reliability and validity of behavioral interventions. Approaches to establishing treatment fidelity in Web-based interventions differ from those used in interventions that are delivered in person.

Objective

To describe a methodology for ensuring treatment fidelity in a Web-based cognitive behavioral stress management intervention.

Methods

The intervention Coping with Cancer Workbook, adapted for Web-based delivery from an in-person intervention, was tested in a randomized controlled trial with 123 breast cancer survivors. Strategies for ensuring treatment fidelity were implemented and assessed.

Results

The National Institutes of Health Behavior Change Consortium Treatment Fidelity Guidelines were relevant to establishing treatment fidelity for the Web-based intervention.

Discussion

Web-based delivery of behavioral interventions is both a strength and a threat to treatment fidelity. Investigators must be cognizant of the elements of treatment fidelity and implement strategies to monitor and improve the reliability and validity of Web-based interventions.

The National Institutes of Health (NIH) Behavior Change Consortium (BCC) Treatment Fidelity Guidelines (Bellg et al., 2004) provides a framework for discussion of treatment fidelity strategies in the delivery of the CBSM intervention. Treatment fidelity requires methodological strategies for monitoring and enhancing the reliability and validity of behavioral interventions (Moncher & Prinz, 1991). It includes assessment of how closely a delivered treatment resembles the ideal model as described in treatment manuals. Researchers must measure treatment fidelity to ensure scientific confidence that the intervention has the direct effect on the dependent variable that was intended. If treatment fidelity is not monitored during evaluation of a new intervention, it will be unclear whether the outcomes are due to effective treatment or to unknown factors unintentionally added to or omitted from the treatment (Cook & Campbell, 1979). Inadequate treatment fidelity during research can lead to implementation of ineffective interventions or failure to implement effective interventions (Henggeler, Melton, Brondino, Scherer, & Hanley, 1997; Moncher & Prinz, 1991).

Web-Based Interventions

Web-based behavioral interventions introduce new research methodology issues that must be addressed. The Internet is increasingly being used to deliver empirically supported behavioral interventions, and is an important mode of health care delivery because it can reach a large number of people cost-effectively due to fewer personnel and infrastructure costs (Christensen & Griffiths, 2002). Specific methodological challenges have been identified in the areas of study design, recruitment, data collection, and data analysis (The National Cancer Institute and Robert Wood Johnson Foundation, 2001). Web-based behavioral interventions can be self-administered, self-paced, and private (Ybarra & Eaton, 2005). Online information can be implemented in a way that ensures the intervention will not change during implementation (Christensen & Griffiths, 2002).

Treatment Fidelity Guidelines

Treatment fidelity guidelines for health behavior research have been identified by the National Institutes of Health (NIH) Behavior Change Consortium (BCC; Bellg et al., 2004). These recommendations were developed to evaluate the integrity of projects designed to test innovative approaches to health behavior change. The five elements of treatment fidelity are (a) study design, (b) provider training, (c) treatment delivery, (d) treatment receipt, and (e) enactment of treatment skills.

To increase treatment fidelity, an intervention must be based on a theoretical model. The study design should minimize and monitor contamination between treatment and control groups, address predictable setbacks in the implementation of an intervention, and ensure that the intervention reflects its theoretical underpinnings. Fidelity also requires that provider training be standardized and that the interventionists obtain and maintain the skills necessary to deliver the intervention competently. Provider training should be reinforced to minimize skill loss and changes in technique that can occur over time. In addition, the research team should ensure standardization and monitoring of the intervention's delivery so that it is provided as planned, and enact processes to monitor and improve participants' ability to gain knowledge and skills from the intervention. Last, methods should be in place to monitor and improve the ability of participants to use these new skills effectively (Bellg et al., 2004).

Different strategies are used to establish treatment fidelity for a Web-based intervention than for an intervention delivered in person by an interventionist. Since a Web-based intervention is self-administered, it can be challenging to ensure that the appropriate dose is delivered and that the intervention is completed. Although different methodology issues pertain to delivering interventions via the Internet such as usability, tailoring messages to the individual, and ensuring confidentiality of submitted information (The National Cancer Institute and Robert Wood Johnson Foundation, 2001), treatment fidelity of Web-based interventions can be maintained and monitored successfully. The purpose of this report is to describe a methodology for ensuring treatment fidelity in a randomized controlled trial (RCT) of a Web-based cognitive behavioral stress management (CBSM) intervention.

Method

This discussion of treatment fidelity will use as illustration an RCT that tested a Web-based CBSM intervention for improving coping skills in breast cancer survivors. As there existed no formalized treatment fidelity plan at the start of the RCT, this guideline is now used to guide future studies and can be used to assist others in doing so.

The Web-based CBSM intervention delivered in the RCT was based on an empirically supported in-person group treatment for psychosocial distress. It included training in cognitive behavioral therapy techniques, coping skills, social support, anger management and assertiveness, and a variety of relaxation techniques. In-person CBSM has been reported to be effective for early stage breast cancer patients, HIV-positive men and women, and prostate cancer patients (Andersen et al., 2004; Antoni et al., 2009; Antoni et al., 2006; McGregor et al., 2004). It can also be an appropriate Web-based intervention: coping skills and relaxation techniques can be taught and social support accessed, via the Internet (Cuijpers, van Straten, & Andersson, 2008).

Human subjects protection review and approval was obtained from the appropriate institutional review board. Study participants were women within 18 months of a stage I–III breast cancer diagnosis who reported a moderate to severe level of distress as indicated by one of three distress measures. Participants had to have a telephone, an e-mail account, and access to a computer with an Internet connection for 2 hours a week. Study participants were recruited and self-selected primarily through the Army of Women's Website (www.armyofwomen.org). Women volunteered to be notified about breast cancer research studies and were sent an e-mail describing the study. Interested women provided their contact information to the Army of Women, who forwarded the information to the research team. The research team called the women to provide further information about the study, answer questions, and to screen for eligibility. The women were informed that their participation was purely voluntary and they could withdraw from the study at any time without penalty. An online consent process was used.

During the 10-week experimental phase, participants randomized to the intervention group (n = 62) used the Web-based CBSM intervention Coping with Cancer Workbook, completing one chapter each week (see Table, Supplemental Digital Content 1, which contains workbook content, http://links.lww.com/NRES/A62). Besides presenting didactic information in an intuitive, easy-to-access manner, the workbook provided self-assessments with tailored feedback, interactive activities and problem-solving exercises that encouraged self-reflection, and relaxation exercises. Participants were provided with a MP3 player preloaded with the relaxation exercises so they could use them at any time. Participants were able to print a hard copy of the workbook to review or reinforce what was learned. The last chapter of the workbook provided an opportunity for the participant to set future goals related to coping with cancer. Communication among intervention participants was made possible through the workbook's discussion board feature, a social support tool that allowed participants to discuss questions related to diagnosis, treatment, and other issues relevant to women with breast cancer.

The control group participants (n = 61) were on a wait list during the 10-week experimental phase, after which they received uncontrolled access to the online workbook. Both intervention and control participants had contact with the research team by phone for screening, biweekly check-in calls during the 10-week experimental phase, and a follow-up interview.

The study's primary outcome variables were self-efficacy for coping with cancer and benefit finding (i.e., perceived benefits from the breast cancer experience such as an enhanced sense of purpose, changes in life priorities). Secondary outcome variables included impact on health-related quality of life, cancer-related intrusive thoughts, and positive mood states. Assessments of outcome variables were completed online at baseline, midstudy (week 10), and follow-up (week 20). Preliminary analysis of the first 74 women (30 intervention, 44 control) who completed the baseline and 10-week follow-up indicated that women in the intervention group reported significantly decreased depression (Center for Epidemiological Studies Depression Scale, F(2,74) = 6.3, p = .01), confusion (Profile of Mood States confusion, F(2,74) = 6.3, p = .01), negative mood regulation (NMR; F(2,70) = 9.01, p = .004), and avoidance of stressful thoughts or stimuli (Impact of Event Scale, avoidance subscale, F(2,74) = 4.62, p = .035). Intervention participants also reported significant increases in finding benefit in having had breast cancer (Benefit Finding Scale, F(2,74) = 5.6, p = .021; Carpenter, McGregor, Schmitz & Doorenbos, 2011).

Results

The five elements of the NIH BCC Treatment Fidelity Guidelines were all relevant to the Web-based CBSM intervention. The elements are mutually exclusive, and treatment fidelity is threatened if any element is not monitored (Borrelli et al., 2005). The treatment fidelity features of the Coping with Cancer Workbook intervention are described in the following text and in Table 1.

Table 1. Treatment Fidelity Elements and Strategies Used in the Web-Based CBSM Intervention.

Treatment Fidelity Element	Description	Strategies
Study Design	Study design minimizes and monitors for contamination between treatment and control groups and addresses predictable setbacks in the implementation of an intervention so that the study can adequately test its hypotheses in relation to the underlying theory and clinical processes.	Ensured intervention was consistent with theory.
		Conducted usability and feasibility testing of the Web-based intervention.
		Developed protocol for intervention Website maintenance and resolution of technical problems during study.
		Sent weekly email check-ins to motivate participants to use the workbook regularly.
		Biweekly phone calls were made to assess use of the workbook and to motivate intervention participants to complete the workbook.
		Ensured control group participants were not given access to the intervention Website during the experimental phase of the study.
Provider Training	Training is standardized so that interventionists acquire the necessary skills to competently deliver the intervention; training is reinforced to minimize “decay” or interventionist “drift” over time.	Intervention was delivered via the Internet, not via providers.
		Training of all individuals who conducted telephone screening, biweekly check-ins, and follow-up interviews was delivered verbally by one person using standardized written materials. All phone assessments were scripted, with computer-based entry.
Delivery of Treatment	Dissemination of the intervention is standardized and monitored so that treatment is delivered as intended.	Intervention was delivered via Internet in exactly the same way to all participants.
		Preliminary usability and feasibility testing provided improvements to the delivery of treatment.
		The research coordinator regularly monitored the workbook throughout the study to ensure audios, videos, links, and workbook flow were working correctly.
		Technical difficulties with the delivery of the intervention were resolved in a timely manner.
Receipt of Treatment	Procedures are in place to monitor and ensure that the treatment was both received and understood.	Intervention dose was monitored by tools on the intervention Website.
		Intervention group received biweekly calls during the experimental phase to check in and motivate as needed to complete workbook.
Enactment of Treatment Skills	Strategies are in place to effectively monitor and improve participant application of treatment skills in daily life.	Intervention group received biweekly calls during the experimental phase to evaluate whether the participant was applying the coping skills learned from the workbook in daily life.

Study Design

The CBSM intervention was founded on the cognitive theory of stress and coping developed by Lazarus and Folkman (1984). The content of the Web-based intervention was translated from an in-person intervention and then reviewed by a multidisciplinary team of experts, including a clinical psychologist who was trained specifically in the CBSM intervention and had implemented previous research studies using the in-person version of the intervention. Other clinical psychologists reviewed the workbook in addition to oncology social workers, nurses, a psychiatrist, and an expert in mindfulness-based stress reduction. The Web-based intervention was determined to have a high degree of fidelity to the theoretical model.

Prior to the RCT, usability (Nielsen, 1994) and feasibility testing for the CBSM intervention were completed. Ten breast cancer survivors participated in the first usability study. In response to participant feedback, changes were made to the workbook (e.g., bullet points and graphics were added to highlight key content; length of chapters was reduced to no more than 24 pages; navigation was simplified, making each chapter accessible via the home page rather than on each standard page; confidentiality of the journal entries was specified). The second usability study included 9 breast cancer survivors, who reported no critical recommendations. An overview was added to help the user understand how each chapter is structured and the discussion board interface was updated so it more closely mimicked the interface of the workbook. Feasibility testing involved 6 participants with breast cancer. Participants were able to access and complete the workbook without problems. In addition, they reported that the skills presented in the workbook were helpful.

The RCT study design included planning for potential setbacks in implementation of the intervention. A protocol was created by the Website developers for maintaining the intervention Website and promptly resolving any technical difficulties that might arise. Weekly email check-ins were sent to participants to reinforce regular use of the Coping with Cancer Workbook. Biweekly phone calls were made by the research team to assess women's distress levels, and to determine intervention participants' use of the workbook and motivate them to complete the workbook.

Since participants resided throughout the country and were assigned randomly to the study groups, it was highly unlikely contact occurred between the intervention and control group participants during the 10-week experimental phase of the study.

A number of steps were taken to ensure confidentiality of submitted data. The online workbook required users to login with each visit. Unique logins were provided to study participants and kept secured in a password-protected database. The workbook required users to re-login after 60 minutes with no activity. A secure socket layer (SSL) certificate from VeriSign (www.verisign.com) was installed to ensure the security of the website itself; SSL is the industry standard for securing websites, intranets, and extranets. All subject-identifiable information was kept in a separate database that was not linked to the database used to gather study data. Last, a 128-Bit or 256-Bit Encryption (256-bit encryption when supported by compatible servers and clients) was used to make the data unreadable to others.

Provider Training

The Web-based CBSM intervention was delivered via Internet, and thus did not require an interventionist. Training was provided to the research team members who conducted the telephone screenings, biweekly check-ins, and follow-up interviews. Interviewers were instructed about the purpose of the phone calls, which was to collect information rather than to be therapeutic. Role-playing was used to ensure that the interviewers understood the protocol for interacting with the participants. All telephone screenings were scripted with simultaneous online data entry and occurred within 1 month of training to ensure fidelity to the interview protocol. The research coordinator randomly and regularly reviewed the biweekly call and follow-up interview data to ensure collection and accuracy of the data.

Treatment Delivery

The preliminary usability and feasibility testing led to improvements in the delivery of treatment. Because the Internet presented the Web-based intervention in the same way at all times, fidelity to intended delivery was monitored easily. The research coordinator regularly monitored the workbook throughout the study to ensure audios, videos, links, and workbook flow were working correctly. When technical difficulties prevented treatment delivery during the experimental phase, the participant called, sent an email, or notified the research team during a biweekly telephone check-in. Problems were then resolved.

The workbook included audio, video, and interactive exercises accessible with the Adobe Flash Player. The free Flash plugin is installed on over 90% of personal computers and a link to download it was provided (http://get.adobe.com/flashplayer/). Interactive exercises included assessments with personalized feedback. All study participants were sent a matchbook-sized MP3 player pre-loaded with relaxation and calming exercises to make it easier for participants to practice relaxation on the go. Relaxation exercises were also available in the workbook in three additional ways: (a) via an embedded Flash audio player, (b) by downloading the MP3 files directly to the user's computer, or (c) as episodes in an iTunes podcast.

Receipt of Treatment

Computer skills were assessed during the initial telephone screen by asking potential participants, “How comfortable are you with computers?” Response options included “not at all,” “somewhat,” or “very” comfortable. All women screened for participation indicated they were “somewhat” or “very” comfortable with computers. In addition, the women had to complete an online consent process successfully prior to study participation. The workbook required minimal computer skills (i.e., the ability to navigate using a mouse).

The research team made biweekly phone calls to assess workbook use and to motivate intervention participants to complete the workbook. Since the intention-to-treat analysis was used, once a participant was randomized to the intervention they were kept in the intervention group regardless of minimally participating. The phone calls were used to encourage or recover participants to the study. Participants were asked if the workbook had been used in the past 2 weeks and, if not, what prevented them from using it. Participants were asked also if they had been able to complete two chapters since the last check-in and what helped or prevented them from using it regularly.

Each chapter was designed to take between 90 and 120 minutes to complete, plus any time the participant chose to practice any of the skills. The time commitment for the Web-based intervention was similar to that of in-person CBSM group sessions, which usually last 2 or more hours (Antoni et al., 2006). The dose of the intervention for each participant was assessed through Website monitoring of chapter completion (see Table 2 for chapters completed). The individual chapters were completed by 58% to 87% of the intervention group (n = 62) with chapters the participants found most helpful being completed by more women than chapters with skills that were perceived as less useful to their lives. The participants' use of the workbook was not used as a measure of dose; a limitation to this strategy is that time spent does not guarantee receipt of treatment—that is, participants could have the workbook accessed on their computers, but not be using it.

Table 2. Chapters Completed by Intervention Group (n = 62).

Chapter	n (%)
1	54 (87%)
2	48 (77%)
3	51 (82%)
4	49 (79%)
5	47 (76%)
6	42 (68%)
7	41 (66%)
8	36 (58%)
9	36 (58%)
10	54 (87%)

Enactment of Treatment Skills

Monitoring of whether or not participants applied the coping skills learned from the workbook in daily life occurred during the biweekly phone calls. Participants were asked if they had used any of the techniques taught in the Coping with Cancer Workbook in the last 2 weeks and, if yes, which ones. If they had not used any of the techniques, they were asked why not. The entire intervention group reported using techniques from the workbook. The most common techniques used were the breathing and relaxation exercises, and meditation.

Discussion

The NIH BCC Treatment Fidelity Guidelines provided a comprehensive resource for identifying and developing strategies to ensure treatment fidelity of the Web-based CBSM intervention. In this study, the Web-based delivery method was both a strength and a threat to treatment fidelity. Since interventionists were not required, potential threats related to ensuring proper training and delivery of the intervention by an interventionist were eliminated. However, the potential for participants receiving an inadequate intervention dose posed a threat to treatment fidelity. Participants were encouraged to complete the workbook through weekly emails and biweekly phone interviews. Dose was measured by monitoring the number of chapters completed. A limitation to this strategy is that that the person completing the chapters may not be the study participant; however, requiring a user login and password to access the workbook increases the likelihood that the intervention dose was received by the participant.

With the Web-based intervention delivery, as with in-person delivery, it is a challenge to determine whether participants use the skills learned in daily life. Participants can be asked if skill use is occurring; however, an accurate account may not be given. Enactment of the skills may be increased by testing or reviewing the participants' understanding of the intervention in the biweekly phone interview, having participants set individual goals and use a diary to record skills applied in daily life. The Measure of Current Status has been utilized to assess whether skills taught by the in-person CBSM intervention were being used by participants (Antoni et al., 2006). The instrument measures the participant's current self-perceived status on several skills that are targeted by the intervention. These strategies were not used in the Coping with Cancer Workbook study, but will be considered in future studies. In all cases, investigators must be cognizant of the elements of treatment fidelity and implement strategies appropriate to the study's design.

Conclusion

Treatment fidelity must be established, using appropriate strategies, in order for a Web-based intervention to be moved into practice. If a Web-based intervention does not change outcomes, having treatment fidelity strategies in place will allow the research team to gain important knowledge to understand why and to develop and test future Web-based interventions. Hopefully, future investigators will build on the approach described here for using the NIH BCC Treatment Fidelity Guidelines to ensure treatment fidelity of Web-based behavioral interventions.

Supplementary Material

Table (Supplemental Digital Content): Contents of the Web-based Coping with Cancer Workbook