Abstract
Applying a comprehensive model of fidelity to interventions delivered by Information and Communication Technology (ICT) has multiple challenges. Fidelity must be considered in the design, implementation, evaluation, and reporting of the intervention. The fidelity strategies must address the unique aspects of the technology, including training providers to instruct participants to use the technology and to provide standardized feedback, rather than deliver the intervention in-person. Other challenges include the nonspecific effects resulting from participants accessing unintended content in interventions delivered by the Internet. ICT allows participant receipt and enactment of intervention skills to be assessed by electronic evidence, rather than in-person observation. Interventions using ICT, such as the Internet are unique, and there is less control of participant interaction with various electronic components. Monitoring participant use and providing standardized feedback for receipt and enactment of intervention skills is key to ensuring intervention fidelity. The final challenges involve evaluating and reporting fidelity.
Keywords: Intervention Fidelity, Internet Intervention, Health Behavior Change, Information and Communication Technologies
Fidelity is an integral part of health behavior intervention research. The concept of fidelity has evolved and expanded over the years from the initial focus on treatment integrity or delivery of the treatment as intended (Peterson, Homer, & Wonderlich, 1982; Yeaton & Sechrest, 1981) to a more comprehensive definition. Ensuring fidelity also includes monitoring and enhancing the reliability and validity of an intervention. The use of Information and Communication Technology (ICT), such as the Internet, to deliver behavioral interventions, presents challenges to ensure fidelity and requires careful consideration of strategies to be addressed in the design, implementation, evaluation and reporting of the intervention.
A comprehensive model of fidelity and strategies addressing five specific areas, including study design, training providers, delivery, receipt and enactment of intervention skills was developed by the Treatment Fidelity Workgroup of the National Institutes of Health Behavior Change Consortium (Bellg et al., 2004; Borrelli et al., 2005). This model includes such aspects as the intervention dose and the theory and/or guidelines on which the intervention was based as part of the study design. Other aspects include monitoring and improving performance of intervention skills during the intervention and in real-life settings as part of receipt and enactment of intervention skills.
Use of ICT to deliver interventions has resulted in unique considerations for ensuring fidelity. For example, an intervention delivered by the Internet provides advantages in relation to ensuring fidelity of various aspects of a study, such as the design, training providers and delivery of the intervention. Internet-based interventions offer consistent content and standard presentation of information that reduces unintended variability. There are aspects of an Internet intervention, however, that make it more difficult to ensure fidelity. Participants access information independently, and the technology allows for multiple interactive features (Kirsch & Lewis, 2001) that participants may access differently in the study. This results in the need to plan for methods and strategies to monitor and improve participant receipt of the intended intervention components, and enactment of intervention skills.
The purpose of this article is to discuss the challenges involved in applying the comprehensive fidelity model (Borrelli et al., 2005) to a physical activity intervention delivered by the Internet (Bosak, Yates, & Pozehl, 2010). This study involved an experimental design to investigate the effects of an Internet intervention compared to usual care on adherence to physical activity, cardiorespiratory fitness, lipid biomarkers (high-density lipoprotein [HDL] cholesterol, non-HDL cholesterol, low-density lipoprotein cholesterol, triglycerides, and total cholesterol) and high sensitivity C-reactive protein, and self-efficacy for physical activity in adult men and women with the metabolic syndrome. All participants were randomly assigned to the intervention plus usual care group or the usual care only group. Data was collected at baseline and six weeks. In addition to usual care, the intervention group received the multicomponent self-efficacy intervention strategies by the Internet. Specific examples of the fidelity strategies used are presented in Table 1. The challenges discussed apply to health behavior interventions delivered by technologies involving little face-to-face interaction with participants. Evaluating and reporting fidelity also requires special consideration, and specific examples are discussed and shown in Table 2.
Table 1.
Examples of Fidelity Strategies Applied to an Internet Physical Activity Intervention
| Fidelity Category | Strategies to Ensure Fidelity* |
| Intervention Design |
|
| Training Providers | Describe protocol for in-person interaction with participants: The intervention was delivered primarily by the Internet: The feedback provided to participants by the NP was standardized (to avoid nonspecific effects). |
| (Remainder of Table Applies to Internet Intervention Condition Only) | |
| Delivery of the Intervention |
|
| Receipt of the Intervention |
|
| Enactment of Intervention Skills |
|
Key
PA = Physical Activity
NP = Nurse Practitioner
RD/LD = Registered Dietician/Licensed Dietician
NCEP, ATP III = National Cholesterol Education Program, Adult Treatment Panel III (available at: http://www.ncep.org)
AHA = American Heart Association; Learn and Live (available at: http://www.americanheart.org)
NHLBI = National Heart, Lung and Blood Institute; Preventing Heart Disease (available at: http://www.nhlbisupport.com)
ACSM = American College of Sports Medicine; Perceived Exertion Scale (available at: http://www.acsm.org)
NIA = National Institute on Aging; Target Heart Rate Zone Calculator (available at: http://www.asaging.org/cdc)
MS SQL2000 = Microsoft Standard Query Language 2000 (software)
Intervention Design
The fidelity strategies in the Design category include addressing the aspects of the contact sessions in each condition, and other factors that must be defined in order to evaluate and replicate a study. For interventions using ICT, it is necessary to describe the aspects of the contact sessions in each condition. An objective method of tracking the intervention dose in each condition is also needed. In the example case, the number of contact sessions during the 6-week Internet intervention was tracked electronically. Participants accessed the Internet intervention on their own time, progressed at their own pace, and reviewed and repeated the content as necessary. Thus, the length of the contact sessions was determined by the participant, not specified by the provider. The number of contacts recommended was to log on to the study Website daily, Monday through Friday. New information was provided on the study Website each day of the week to encourage daily activity. The intervention allowed more contact between the intervention and individual participant than would be possible in person, so an objective and automated method of tracking the intervention dose delivered was essential. Other aspects of the intervention content are detailed in Table 1. The nurse practitioner (NP) provided standardized feedback about percent adherence to the National Cholesterol Education Program (NCEP, 2002) physical activity goals and positive reinforcement for goal setting, as well as individualized feedback about the appropriateness of the physical activity goals. Reminders to participate in the electronic discussion board were also included each week.
The credentials of the providers involved must be detailed to demonstrate the level of expertise needed to replicate the intervention. In this case, the usual care condition was administered in person to both the intervention and usual care control groups, and thus, a description of provider credentials was relevant (see Table 1). The Internet intervention was delivered electronically, and thus, credentials were not relevant for delivery of the intervention. Credentials were relevant for the content expert developing the intervention and providing electronic feedback by email and on the discussion board (although electronic communication was standardized). Replication of an intervention using ICT may not require special expertise as long as the intervention is standardized.
Describing the theoretical model and/or clinical guidelines on which an intervention is based is necessary to evaluate a study. The theoretical model and guidelines used for both the Internet intervention and the usual care condition are described in Table 1. A challenge for interventions using ICT is that theoretical constructs must be operationalized for electronic delivery, as direct transfer of an in-person intervention to the Internet may not be as effective. For example, in an Internet intervention using ICT, vicarious experience to build self-efficacy can best be provided by a video on the Website showing like-others participating in moderate intensity physical activity as opposed to observing others at a gym. Usability testing is recommended to determine if the theoretical constructs are operationalized optimally for electronic delivery.
An additional design strategy essential to the fidelity of Internet interventions includes, providing learning objectives and competencies specifying what participants are expected to accomplish during the intervention. Because participants access Internet Interventions independently, attention to the objectives and competencies is essential and must be visible to participants each time they access the study website. Standardized feedback must also be provided in a timely manner if objectives or competencies are not met.
Training Providers
This category involves describing the standardized protocol for interaction with participants. Provider training is essential in interventions involving participant and provider interaction to reduce the random and unintended variability that can threaten intervention fidelity. In contrast, the Internet intervention was delivered electronically, thus eliminating the need to train providers to deliver the intervention. However, a protocol was developed to ensure standardized feedback was provided by email messages and on the electronic discussion board. Furthermore, adherence to the protocol can be routinely assessed by review of the archives. Providers must also be trained in other aspects of the study protocol, such as how to use the technology, and how to access and navigate the study website.
Delivery of the Intervention
Intervention delivery focuses on the provider to ensure the content and the dose of the intervention are delivered as intended. In Internet interventions, it can be difficult to ensure participant access to the intervention content as intended. See Table 1 for details on how access was ensured in the physical activity intervention. Objective tracking of participant activity on the website is essential, including all linked pages and Websites explored, and all supplemental components used. For simplicity, the linked websites explored by participants in the Internet intervention were tracked by asking participants if the site was helpful by selecting “yes” or “no” from a dropdown menu. Data recorded on the website was downloaded into Microsoft Access 2000 file format. The data in the tables contained participant login time, dates, physical activity goals, self-reported minutes of moderate or greater physical activity as well as participants’ yes or no responses about the helpfulness of the linked sites visited. The BlackBoard Academic Suite™ provided the platform for the electronic discussion board. All discussion threads posted by the NP, participant’s questions and comments, and NP feedback were retrievable in the BlackBoard archives for assessment. Provider adherence to the intervention plan and competence in delivery of an in-person intervention is essential, but this is not a major challenge in Internet interventions that are automated and use standardized feedback. The comprehensive fidelity model includes addressing the nonspecific intervention effects resulting from provider interaction with participants. To address non-specific effects in the Internet intervention, participants did not receive interpersonal contact with the NP during the intervention period. All standardized messages were received electronically by participants. Participants were provided the credentials of the NP provider with whom they were interacting by email.
Finally, an intervention manual is typically developed for use by providers. The main purpose of the manual for the Internet intervention was a guide for development of the intervention website. In this case, the manual was provided to the website developer, who developed the intervention algorithms and published the program to the web. The intervention manual included the essential intervention content and skills needed by participants to meet the physical activity guidelines along with all of the hyperlinks. Thus, the website development was based on the intervention manual developed by the researcher.
Receipt of the Intervention
Receipt of the intervention focuses on subject behavior during the intervention period to ensure that the intervention was received and understood. Actual receipt of the intervention is difficult to assess and requires careful consideration. Receipt involves assessing participant comprehension of the intervention and providing strategies to improve comprehension and skill performance. When developing a plan to ensure receipt, the distinction must be made between strategies to ensure understanding and comprehension. In the Internet intervention, knowledge or understanding of the intervention content was easily assessed through administration of a quiz on the website at Weeks 3 and Week 5 of the 6-week intervention. Understanding was improved by providing the correct answers with explanations electronically in real-time. Tests of comprehension relate to the participant’s ability to apply the intervention skills, along with standardized electronic feedback to improve the performance of those skills. With ICT, there may be limited opportunity to directly observe participant-provider interactions or to observe participants performing a specific behavior during the intervention period. Indirect evidence of the participant’s comprehension by the ability to perform skills is necessary, such as a record of the expected behavior documented on the website (see Table 1) or remote real-time monitoring of vital signs, as described by Nangalia, Prytherch, and Smith (2010). Use of a webcam is another strategy that can be used to provide objective evidence of comprehension. Receipt of the intervention also includes strategies to improve performance beyond that included in the intervention. Receipt can be promoted by providing feedback using standardized electronic messages (to avoid nonspecific effects that can result from aspects of an intervention and it’s delivery that are unintended).
Enactment of Intervention Skills
The strategies to enhance enactment of intervention skills focus on participant performance of these skills in daily life. Ensuring enactment of ICT interventions may be more difficult when the intervention is delivered entirely in settings in which the skills may be applied, such as the home or work environment. Enactment also involves addressing performance of intervention skills in new settings or circumstances in which intervention might be applied. Monitoring of indirect electronic evidence of skill performance and providing feedback using standardized electronic messages is required to ensure enactment, and to improve performance of intervention skills delivered by ICT. In the Internet intervention, the electronic discussion board allowed participants to post questions for the NP about implementing physical activity skills in various settings, such as when traveling, when the weather was bad, when experiencing stress or depression. The NP anticipated participants’ questions and comments, and provided standardized feedback to overcome barriers to physical activity participation and to recover from relapse.
Evaluation of Fidelity
The final challenge of implementing a comprehensive fidelity plan involves evaluating fidelity for reporting. In the literature, behavioral interventions generally include a discussion of implementation of fidelity strategies, involving evaluation of provider’s adherence to the protocol and use of a treatment manual (Borrelli et al., 2005; Moncher & Prinz, 1991). The participant behaviors of receipt and enactment of intervention skills, however, are not always discussed. A strategy to assess if participants received an intervention as intended with particular relevance for interventions delivered using ICT is the qualitative manipulation check described by Donovan, Kwekkeboom, Rosenzweig, and Ward (2009). This involves evaluating transcripts from electronic discussions for themes reflecting study participation. Not only the finding of nonspecific factors, but the absence of specific factors in the postings identified by manipulation check indicates areas for further investigation. For example, the Internet physical activity intervention was based on Social Cognitive Theory (Bandura, 1997), with online activities to promote self-efficacy by mastery experience, vicarious experience, persuasion and reinterpretation of physiological signs and symptoms. Specific factors indicating receipt of the intervention as intended might include participant postings on the electronic discussion board detailing how they could relate to the online testimonials of how others with metabolic syndrome overcame difficulties, problem solved relapses, and achieved physical activity goals (indicating receipt of vicarious experience); and/or relate to the discussion about dealing with minor aches and pains and reducing stress and anxiety levels related to initiating or increasing physical activity (indicating reinterpretation of physiologic signs and symptoms). Nonspecific factors might include participant postings of how the discussion board provided social support, which was not intended to be part of the Internet physical activity intervention content but indicates a theme for assessment in a future study.
Another challenge in the evaluation of fidelity is the lack of specific evaluation criteria. High levels of fidelity were defined by Borrelli et al. (2005) as those studies that had .80 or greater proportion adherence across all strategies. In the Internet physical activity intervention, the proportion of adherence to the fidelity strategies was calculated for each component of the comprehensive model by summing the number of strategies coded as present and dividing by the total number of strategies (see Table 2). Thus, the Internet physical activity intervention addressed most of all the strategies considered relevant to the study design. Using these criteria, fidelity was high. It may be most informative for researchers to identify the components of the comprehensive fidelity plan related to intervention outcomes to provide evidence for the levels of fidelity needed to observe intervention effects.
Evaluating enactment of intervention skills can be challenging. Enactment itself is not considered a study outcome but focuses on the skills required to achieve study outcomes (Resnick et al., 2005). Enactment can be quantified by random sampling of records and evaluation using descriptive statistics. For example, the levels of physical activity over the course of the Internet intervention can be compared with the level participants were expected to have completed based on the NCEP guidelines. Enactment of the intervention can be calculated to determine percentage achievement of the study goals.
Fidelity related to receipt of the intervention has been analyzed by the input of the dose of the intervention that was actually received as an independent variable in regression analyses (Santacroce, Macarelli, & Gray, 2004). This method, however, may be misleading, as the intervention dose delivered and the dose received by participants may be quite different. The qualitative manipulation check (Donovan et al., 2009) can be used to assess participant understanding, along with objective evidence of performance of intervention skills to assess comprehension. Researchers may encounter a final challenge in reporting intervention fidelity. Editorial issues or space limitations beyond the researcher’s control may limit reporting intervention fidelity along with the effects or feasibility. A separate paper may be needed to report fidelity of a complex behavioral intervention, including all aspects of the comprehensive fidelity model.
Future Research
Fidelity is an essential part of conducting health behavior intervention research. A comprehensive model is necessary for incorporating essential aspects of fidelity in the design, implementation, evaluation, and reporting of complex health behavior interventions. The comprehensive plan described in this article is particularly relevant to ensure the fidelity of interventions delivered using ICT, such as the Internet. Interventions using technology are unique and offer some advantages compared with traditional in-person interventions, primarily consistency of implementation and ease of replication. These interventions also present unique challenges for ensuring fidelity. The greatest threat to the fidelity of Internet interventions shifts from implementing the intervention to participant receipt and enactment of intervention skills. Monitoring participant use of the intervention and providing timely, standardized feedback is necessary to ensure participant receipt and enactment. The strategies for evaluating fidelity are evolving, and more health behavior interventions are being conducted using technology. Future research is needed that focuses on the application and testing of novel methods for assessing intervention fidelity. Identifying the components of a comprehensive fidelity plan related to intervention outcomes can provide evidence for the levels of fidelity needed to observe intervention effects. Ultimately, reporting fidelity information is critical for translating the intervention to various clinical practice settings and populations.
Acknowledgments
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: American Heart Association Predoctoral Fellowship; University of Nebraska Medical Center Clinical Research Center, Sigma Theta Tau International, Gamma Pi Chapter.
Footnotes
Declaration of Conflicting Interests
The author(s) declared that they had not conflicts of interests with respect to their authorship or the publication of this article.
References
- Bandura A. Self-efficacy: The exercise of control. New York: W. H. Freeman and Company; 1997. [Google Scholar]
- Bellg AJ, Borrelli B, Resnick B, Hecht J, Minicucci DS, Ory M, et al. Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH behavior change consortium. Health Psychology. 2004;23(5):443–451. doi: 10.1037/0278-6133.23.5.443. [DOI] [PubMed] [Google Scholar]
- Borrelli B, Sepinwall D, Bellg AJ, Breger R, DeFrancesco C, Sharp DL, et al. A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research. Journal of Consulting and Clinical Psychology. 2005;73(5):852–860. doi: 10.1037/0022-006X.73.5.852. [DOI] [PubMed] [Google Scholar]
- Bosak KA, Yates B, Pozehl B. Effects of an internet physical activity intervention in adults with metabolic syndrome. Western Journal of Nursing Research. 2010;32(1):5–22. doi: 10.1177/0193945909333889. [DOI] [PubMed] [Google Scholar]
- Donovan HS, Kwekkeboom KL, Rosenzweig MQ, Ward SE. Nonspecific effects in psychoeducational intervention research. Western Journal of Nursing Research. 2009;31:983–998. doi: 10.1177/0193945909338488. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kirsch SD, Lewis FL. Using the world wide web in health-related intervention research. A review of controlled trials. Computers and Informatics in Nursing. 2001;22(1):8–18. doi: 10.1097/00024665-200401000-00005. [DOI] [PubMed] [Google Scholar]
- Moncher FJ, Prinz FJ. Treatment fidelity in outcome studies. Clinical Psychology Review. 1991;11:247–266. [Google Scholar]
- Nangalia V, Prytherch DR, Smith GB. Health technology assessment review: remote monitoring of vital signs-current status and future challenges. Critical Care. 2010;14(5):233. doi: 10.1186/cc9208. [DOI] [PMC free article] [PubMed] [Google Scholar]
- National Cholesterol Education Program (NCEP) [Accessed January 7, 2011];Expert Panel on Detection, Evaluation, and Intervention of High Blood Cholesterol in Adults (Adult Treatment Panel III) full report. Available at: URL www.nhlbi.nih.gov/guidelines/cholesterol/atp3_rpt.htm.
- Peterson L, Homer AL, Wunderlich SA. The integrity of independent variables in behavior analysis. Journal of Applied Behavior Analysis. 1982;15:477–492. doi: 10.1901/jaba.1982.15-477. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Resnick B, Inguito P, Orwig D, Yahiro JY, Hawkes W, Werner M, et al. Treatment fidelity in behavior change research: a case example. Nursing Research. 2009;54(2):139–143. doi: 10.1097/00006199-200503000-00010. [DOI] [PubMed] [Google Scholar]
- Santacroce SJ, Macarelli LM, Gray M. Intervention fidelity. Nursing Research. 2004;53(1):63–66. doi: 10.1097/00006199-200401000-00010. [DOI] [PubMed] [Google Scholar]
- Yeaton WH, Sechrest L. Critical dimensions in the choice and maintenance of successful treatments: strength, integrity, and effectiveness. Journal of Consulting and Clinical Psychology. 1981;49(2):156–167. doi: 10.1037//0022-006x.49.2.156. [DOI] [PubMed] [Google Scholar]