Lessons learned from a practice-based, multi-site intervention study with nurse participants

Abstract

Purpose

To identify challenges and solutions to the efficient conduct of a multi-site, practice-based randomized controlled trial to improve nurses’ adherence to personal protective equipment use in ambulatory oncology settings.

Design

The Drug Exposure Feedback and Education for Nurses’ Safety (DEFENS) study is a clustered, randomized, controlled trial. Participating sites are randomized to web-based feedback on hazardous drug exposures in the sites plus tailored messages to address barriers versus a control intervention of a web-based continuing education video.

Approach

The study principal investigator, the study coordinator, and two site leaders identified challenges to study implementation and potential solutions, plus potential methods to prevent logistical challenges in future studies.

Findings

Noteworthy challenges included variation in human subjects protection policies, grants and contracts budgeting, infrastructure for nursing-led research, and information technology variation. Successful strategies included scheduled web conferences, site-based study champions, site visits by the principal investigator, and centrally-based document preparation. Strategies to improve efficiency in future studies include early and continued engagement with contract personnel in sites, and proposed changes to the common rule concerning human subjects. The DEFENS study successfully recruited 393 nurses across 12 sites. To date, 369 have completed surveys and 174 nurses have viewed educational materials.

Conclusions

Multi-site studies of nursing personnel are rare and challenging to existing infrastructure. These barriers can be overcome with strong engagement and planning.

Clinical Relevance

Leadership engagement, onsite staff support, and continuous communication can facilitate successful recruitment to a workplace-based randomized, controlled behavioral trial.

Multi-site research is an important strategy to strengthen the external validity of nursing science (O’Mara, Bauer-Wu, Berry, & Lillington, 2007). In contrast to single-site studies, research projects conducted with multiple sites offer potentially larger, more diverse participants samples and reduce the likelihood of idiosyncratic research findings. Conversely, multi-site studies are more complicated to conduct and administer. New complexities also arise when research participants are staff, as opposed to patients or clients.

Workplace intervention studies are increasing, due in part to growing awareness that improved worker health and safety has downstream societal benefits (Anger et al., 2015). Specifically, the National Institute for Occupational Safety and Health (NIOSH) has launched the Total Worker Health initiative to respond to this challenge (National Institute for Occupational Safety and Health (NIOSH), 2016; Weisfeld, Lustig, & Board of Health Sciences Policy, 2014). Healthy workers are associated with lower turnover, improved economic productivity, and enhanced personal wellbeing. Due to labor shortages, high acuity, long shifts, and physical demands, NIOSH has identified healthcare workers as a vulnerable labor sector for intervention (National Institute for Occupational Safety and Health (NIOSH), 2013). For the past ten years, our interdisciplinary team has documented the specific concerns of oncology nurses employed in ambulatory oncology settings. These nurses face an unusual occupational threat of hazardous drug exposure given the high patient volume, the explicit emphasis on chemotherapy treatment and associated continuous risks of exposure.

Our team has documented that 18 percent of surveyed ambulatory oncology nurses experienced an unplanned hazardous drug spill in the past six months (Friese et al., 2014). Hazardous drug exposure is correlated with substantial short- and long-term health effects, such as nausea, vomiting, airway irritation, reproductive problems, and rare cancers (National Institute for Occupational Safety and Health, 2004). Despite 30 years of data to support the need for increased vigilance when handling hazardous drugs, surprisingly few nurses wear personal protective equipment as recommended (Connor & McDiarmid, 2006; Polovich & Clark, 2012). Except for the current project, only one published study examined an educational intervention for nurses, conducted in one Malaysian hospital (Keat, Sooaid, Yun, & Sriraman, 2013). Thus, we lack sufficient evidence on how to improve nurses’ use of personal protective equipment when handling hazardous drugs.

The Drug Exposure Feedback and Education for Nurses’ Safety (DEFENS) study is a four-year, multi-site cluster randomized controlled trial (Friese, Mendelsohn-Victor, et al., 2015). The study compares one-time static educational information about hazardous drug exposure prevention to quarterly feedback on study results, coupled with tailored messages designed to reduce barriers to protective equipment use. In planning for the project, we reviewed the sparse literature that describes multisite research project management with registered nurse employees as participants. In the current paper, we review successful study implementation strategies and identify important considerations for future research projects that plan to incorporate nurses as participants.

Approach

The DEFENS study is a cluster randomized controlled trial. Nurses who work 16 hours a week or more in ambulatory infusion within 12 large cancer centers in the United States were invited to participate. Full details may be found in the published protocol paper (Friese, Mendelsohn-Victor, et al., 2015) or in the ClinicalTrials.gov registry (National Institutes of Health, 2016a). Guided by extant models of health behavior and risk reduction, we hypothesized that one-time educational content is insufficient to improve nurses’ use of personal protective equipment when handling hazardous drugs (McCullagh, Ronis, & Lusk, 2010). Rather, we compared static educational content (control intervention) to quarterly feedback about data gleaned from study, coupled with video messages tailored to participants’ reported barriers to protective equipment use (experimental intervention). To avoid within clinic contamination, randomization occurred at the site level, stratified for clinic size and baseline use of personal protective equipment. The primary endpoint is nurse-reported use of personal protective equipment following one year of education or feedback plus tailored messages, using a validated self-report instrument (Polovich & Martin, 2011). To assess intervention fidelity, our team monitored participants’ frequency of accessing web-based materials and the duration of time they viewed website content.

Nurses also provided prospective reports of hazardous drug spills for quarterly analyses (delivered to the sites assigned to experimental intervention). Secondary analyses included measuring hazardous drug exposures in nurses’ plasma, as well as correlative analyses of immune and reproductive function. Informed consent, study questionnaires, educational content, and feedback content were housed on an encrypted, user-authenticated website.

For the present inquiry, study team members identified key challenges to study operations and strategies to assure study success. Team members also identified persistent and emerging issues for future investigators and participating site personnel to consider when embarking on a multi-site research study involving nursing personnel. To evaluate our study procedures, we constructed a flow diagram for participant and recruitment, as recommended by the Consolidated Standards of Reporting Trials (CONSORT) (Campbell, Piaggio, Elbourne, Altman, & CONSORT Group, 2012).

Findings

Leadership Engagement

Congruent with the implementation science literature (Yevchak et al., 2014), as well as organizational change theory (Tropman & Wooten, 2010), our team identified that endorsement and ongoing support by the senior nursing executive was crucial for success. Senior nursing leadership engagement facilitated timely protocol activation and encouraged clinical nurses to participate. Engagement began before the proposal was submitted and continues on period intervals throughout the project.

Before the original grant proposal was submitted, the principal investigator (PI) contacted senior nurse executives from National Cancer Institute-designated comprehensive cancer centers. He presented an overview of the proposed project at their annual meeting. He led one-hour informational webinars that reviewed the study team’s preliminary data and outlined the proposed research project. He prepared 5-page executive summaries for these leaders to share with their institution’s senior leadership. On several occasions, feedback from these executives led to important study protocol changes. For example, one leader recommended reviewing the policies of all participating institutions for differences in hazardous drug handling policy. Another identified strategies for nurses in satellite locations to participate.

After a favorable peer-review process by NIOSH’s study section, the PI re-engaged with interested leaders to plan for study activation. Re-engagement enabled leaders to identify key contacts, budgetary considerations, and information technology needs for participation. After re-engagement, several supportive leaders declined participation, principally due to major organizational changes in cancer care services and/or electronic health record implementation. The PI was able to replace these sites by contacting chief nursing officers from other cancer centers.

To demonstrate leadership support of the project to potential participants, we drafted study letter endorsements that would be sent to eligible staff nurses on behalf of the nurse leaders. The study team and the nursing leaders agreed that study participants would remain anonymous to the nurse leaders in the institution to promote trust in the study and ensure confidentiality of responses as well as of personal health information from employers.

Human Subjects Protections

Institutional review boards (IRBs) have extensive experience in protecting human subjects who are patients in a health care facility. They have less experience when employees are participants, and the interventions are not of a clinical nature. Timely, thorough, and efficient human subjects review was a critical priority for the study team. In partnership with leaders of our institution’s IRB, we carefully reviewed the criteria for “not-engaged” status for participating sites. An institution can be considered “not engaged” if the involvement of their employees or their agents is limited, among other things, to the following criteria:

1. the services performed do not merit professional recognition or publication privileges;

2. the services performed are typically performed by those institutions for non-research purposes; and

3. the institution’s employees or agents do not administer any study intervention being tested or evaluated under the protocol (US Department of Health and Human Services Office of Human Research Protections, 2016).

The advantage of not-engaged status meant that our protocol would be reviewed, critiqued, and approved centrally, that informed consent documents would be standardized, and administrative workloads would be reduced for participating sites. Another option to retain centralized control was to have site IRB cede control to the University by completing an IRB Authorization Agreement (IAA) form (National Academies of Science, Engineering, and Medicine, 2016).

Our initial approach was to review our IRB’s determination of not-engaged status with each site, provide requisite documentation, and ask them to confer with their IRB. We offered to speak with IRB staff, and highlight that participants were employees, not patients, the intervention was behavioral in nature, and a data safety monitoring board was in place at the primary institution in the event of an adverse event. In six cases, the participating sites’ IRBs agreed with our interpretation. In three cases, participant sites’ IRBs ceded authority to our institution’s IRB. In three cases, the participating institution required full review by their IRB. In the three latter cases, the study team provided as much assistance in preparing documents for review as possible. The time between initial IRB approval and final IRB approval at the last research site was 11 months.

The shift to not-engaged status required the team to modify several study procedures from our original plan. The coordinators at each site were no longer responsible for direct participant recruitment. Their role shifted to study facilitation, as they provided information, resources, and assisted participants with website navigation. Informed consent took place on the study website. Questions regarding consent and the study protocol were directed to the study personnel at the primary site. The downside of this approach is study coordinators did not know which nurses were enrolled in the study and could not provide personal reminders to complete study activities. A full-time project manager at the primary site was essential to manage participant inquiries.

Benefits of On-Site Study Coordinators

We asked each site to name at least one registered nurse to serve as a study coordinator. In most cases, the grant provided financial resources to the institution to partially subsidize the hours coordinators spent. These individuals provided information about the study to participants and clinic leaders, coordinated logistics of site visits, identified where and how blood would be drawn at each site, and directed participants to complete surveys and have blood drawn, when applicable.

To support these study coordinators, the project manager prepared a binder with all study materials, including the full protocol, a clean copy of the consent form, and a document of frequently-asked questions. These materials were updated as necessary, based on feedback from the study coordinators. The primary site also held 4 recorded webinars to review study procedures, answer questions, and address concerns. The primary site has held webinars approximately quarterly to keep study coordinators informed on study progress, address any ongoing challenges, and maintain enthusiasm for the project. We took steps to reduce the potential for cross-site contamination. After sites were randomized to intervention or control arms, separate telephone calls were held with study coordinators based on their randomization status. We also stressed to participants and study coordinators that all participants will learn the results of the study before the project concludes.

Finally, the PI and/or project manager conducted visits to all 12 sites at the time of study activation; another site visit occurs close to the primary endpoint collection time point. This visit enabled the PI and project manager to educate staff and engage nurses at each site in the study. It was also an opportunity to connect with study coordinators, thank them for their support, and outline logistics of study accrual and intervention procedures. Study coordinators were instrumental in arranging these visits and encouraging staff to attend information sessions with the study personnel.

Study coordinators assisted the project by troubleshooting reasons for low participation rates in educational video viewing. Study coordinators identified technology challenges and time constraints as barriers to timely completion. Coordinators also challenged our assumption that staff members would complete study activities after hours at home. They suggested communal “viewing parties” during scheduled work breaks with refreshments to facilitate completion. We also modified delivery of the materials to facilitate easy viewing based on their feedback. These suggestions were associated with improving our participation rate from 17.4% to 60.8% at the time of this publication.

Internet Access and Browser Compatibility

Advantages of Web-based study platforms include the capacity to standardize delivery, monitor access, and adjust content as needed. Our team experienced substantial challenges with the variation in informational technology and security restrictions across twelve participating sites. Despite substantial user testing before the project website launched, several institutions continued to use outdated and unsupported web browsers during the study period. This required unplanned modifications to the website design and scaled-down versions of materials for participants in affected sites. In addition to website browser incompatibility, several sites restricted the kinds of files staff members could access on clinic computers. Although we provided each site’s informational technology departments with web addresses in advance, several sites blocked viewing of video materials, regardless of source. For participants unable to access the videos, our team created one-page handouts that summarized the video content. To reduce the burden of using the website, we used Qualtrics™ (Provo, UT) software to deliver videos and handouts directly to participants’ email accounts.

Site Budgeting Challenges

Financial management of federally-supported multi-site projects intersect federal policy, primary site institutional policy, in addition the policies of participating sites. These policies are not always congruent. Moreover, grants and contract personnel occasionally do not understand the scope of work planned for the sites. In addition, policy changes that occur during the awards process require planning, attention, and flexibility by the primary research team.

In the case of the DEFENS study, the Department of Health and Human Services modified their policy in 2014, between the time of our original proposal and budget submission (Office of Management and Budget, 2014). The PI requested budgets from each site in the pre-award phase, with the expectation of no indirect costs included. However, after the policy change, participating institutions now expected full indirect costs in addition to their originally submitted budget. Yet the funds provided by the Centers for Disease Control and Prevention did not provide funds to support the work, plus full indirect costs at the participating sites.

The PI, in partnership with senior nurse executives at each site, engaged in discussions with respective grants and contracts departments to request waivers for full indirect costs for the project. These waiver requests highlighted the unique study focus on employees, not patients, the not-engaged human subjects determination for most sites, and the institutional benefits to participation. Whenever possible, the PI pledged non-financial resources to support sites with study activities, including primary site preparation of any requisite documents and on-site assistance with participant enrollment. In addition, the project manager assumed primary responsibility for several functions we anticipated study coordinators to assume. Fortunately, we prevailed in all twelve site negotiations. However in the future, closer consultation with grants and contracts offices in the pre-award phase should help clarify roles and expectations.

Enrollment and Participation Rates

Figure 1 shows the CONSORT diagram of study participants. The number of participants is slightly uneven in arms because sites, not participants, were randomized. Of 440 registered nurses identified by sites that met eligibility criteria, 393 completed the informed consent process and 369 (93.9%) of those completed baseline surveys. To date, 174 (47.2%) of the participants who completed baseline surveys have also viewed the control education video. To date, 32 participants have withdrawn from the study because of a change in employment or employment duties.

Figure 1.

Participant Flow Diagram

Discussion

Increasingly, nursing scientists turn to multi-site research designs to recruit larger samples of participants efficiently, boost statistical power to detect meaningful effect sizes, strengthen external validity, and promote implementation of efficacious interventions (Donovan, Nolte, Edwards, & Wenzel, 2014). Emerging interest in promoting a culture of health has shifted the lens of health promotion and risk reduction research to population-level interventions embedded in workplaces (Lavizzo-Mourey, 2015). These converging interests pose challenges and opportunities for nursing scientists. In our team’s two year experience conducting a multi-site randomized controlled trial with registered nurse participants, we identified five important considerations for PIs and study team members who plan to conduct similar studies.

Our project benefited from strong support from senior nursing leaders during study planning and execution. This approach was used successfully in a prior project that involved chief nursing officer participation in the research project, but required the trust and candor of registered nurse participants (Friese, Siefert, Thomas-Frost, Walker, & Ponte, 2015). While leaders should pledge and demonstrate support for research projects, they must also take care to avoid direct involvement in the project when employees are participants. In our case, the leaders understood that direct knowledge of which employees were participating could threaten the candor of responses.

We were fortunate to have thoughtful input from the IRB to pursue strategies for rigorous and efficient human subjects review. Not all projects will qualify for not-engaged designation. Recent policy changes regarding single IRB review of studies funded by the National Institutes of Health (NIH) may benefit researchers conducting multi-site research (National Institutes of Health, 2016b). Careful delineation of responsibilities, including clear roles and responsibilities of study site key contacts and primary site study team members, will be essential for smooth implementation as regulations and IRB policies change (O’Rourke et al., 2015).

While our change to our human subjects protections plan offered efficiency, we also had to adjust the planned roles of on-site study coordinators. They became less involved in participant recruitment and instead served as study facilitators. Yet we found their feedback about their organization and the experiences of their colleagues as study participants crucial for study success. They provided essential recommendations to amend study procedures and try alternate approaches, particularly when considering viewing educational materials. Implementation scientists have cited absence of local support as a key contributor to failed implementation (Scott et al., 2009). Our experience would support this observation. Another argument for on-site study staff is to meet the New Knowledge component for the American Nurses Credentialing Center Magnet Recognition Program (American Nurses Credentialing Center, 2013). One portion of the evaluation criteria assesses whether clinical nurses participate in nursing research within the organization. Our project assisted study sites pursuing Magnet recognition show evidence of ongoing nursing research.

To date, few investigators have documented internet access and browser compatibility issues across research sites. The study team’s experiences with these challenges are novel, and pose important implications for future researchers. Internet-based educational interventions are ubiquitous given the high rates of access and increased use of smartphones. Despite technological advances, healthcare facilities lag behind other employment sectors due to privacy and cost concerns (HIMSS Analytics, 2015). Information technology resources and policies vary substantially across healthcare settings, which makes intervention website design more complicated. Despite an upfront capabilities survey, careful planning, and pilot testing, several of our sites had difficulties with the website as initially designed. We encourage investigators to plan for additional programming costs after initial design for such a contingency. Our measurement of intervention fidelity is limited to data capture from the website; additional procedures to include direct observation of participants would strengthen the validity of our findings.

Several aspects of the current inquiry merit comment. First, the DEFENS study sites are primarily elite cancer cancers with robust research capacity. PIs conducting research in sites with less research capacity and experience may encounter different challenges. Multi-site projects consume substantial fiscal and human resource costs. Close collaboration with grants management professionals, coupled with frequent engagement with research sites, will minimize the impact of subsequent surprises. In our experience, senior leadership engagement coupled with pledging non-financial resources were key to overcoming obstacles. Yet we realize there are underappreciated costs to sites for research participation. Assuring that the project aligns with the organizational mission is an important consideration in recruiting sites.

While our investigation focused on a project that included employee participants, many of the findings are generalizable to sites where patients are participants. It is unclear how current revisions to NIH policy will impact future human subjects protection plans in projects not funded by NIH. Yet our findings, which include perspectives of the primary research team and leaders at participating sites, have notable relevance to the nursing scientists as they plan and conduct complex multi-site intervention studies.

Conclusions

As the third year of the study began, the DEFENS study team has successfully recruited 393 participants from 12 cancer centers across the country to understand factors that predict nurses’ use of personal protective equipment when handling hazardous drugs. Our team identified senior leadership engagement, on-site study coordinator participation, and partnership with IRB staff as key factors in the project’s success. PIs planning future web-based, multi-site intervention studies should pay careful attention to each site’s internet capabilities and policies, anticipate information technology challenges, and work closely with their team to overcome financial challenges. Anticipation and proactive actions to address these issues will improve the likelihood of successful study activation and participation.