Reducing Events of Noncompliance in Neurology Human Subjects Research: the Effect of Human Subjects Research Protection Training and Site Initiation Visits

Abstract

In an effort to minimize protocol noncompliance in neurological research studies that can potentially compromise patient safety, delay completion of the study, and result in premature termination and added costs, we determined the effect of investigator trainings and site initiation visits (SIVs) on the occurrence of noncompliance events. Results of protocol audits conducted at the National Institute of Neurological Disorders and Stroke from 2003 to 2019 on 97 research protocols were retrospectively analyzed. Based on the depth of auditing and provision of investigator research training, audit data were separated into four arms: 1) Early Period, 2003 to 2012; 2) Middle Period, 2013 to 2016; and Late Period, 2017 to 2019, further divided into 3) Late Period without SIVs; and 4) Late Period with SIVs. Events of noncompliance were classified by the type of protocol deviation, the category, and the cause. In total, 952 events occurred across 1080 participants. Protocols audited during the Middle Period, compared to the Early Period, showed a decrease in the percentage of protocols with at least 1 noncompliance event. Protocols with SIVs had a further decrease in major, minor, procedural, eligibility, and policy events. Additionally, protocols audited during the Early Period had on average 0.46 major deviations per participant, compared to 0.26 events in protocols audited during the Middle Period, and 0.08 events in protocols audited during the Late Period with SIVs. Protocol deviations and noncompliance events in neurological clinical trials can be reduced by targeted investigator trainings and SIVs. These measures have major impacts on the integrity, safety, and effectiveness of human subjects research in neurology.

Supplementary Information

The online version contains supplementary material available at 10.1007/s13311-020-01003-4.

Introduction

Over the last decade, there has been tremendous progress in developing treatments for neurological disorders [1, 2] and there has been an increase in clinical research opportunities to study these diseases [3]. Of paramount concern in the conduct of clinical research is the protection of human subjects participating in these studies. To address these concerns, regulations focused on protecting the safety and rights of human subjects in clinical research have grown exponentially [4], resulting in increasingly complex protocols [5]. As protocols increase in complexity, there is the potential for an increase in deviations from the institutional review board (IRB)-approved protocol [6], which is common in research [7].

The International Council for Harmonisation (ICH) was founded in 2015 on the back of the International Conference on Harmonisation, a 25-year veteran international agency dedicated to the harmonization of world-wide regulatory practices pertaining to human subjects research and regulatory requirements. The ICH E3 [8], a Guideline for Industry Structure and Content of Clinical Study Reports, references “important protocol deviations”. Precisely, important deviations are determined “in part by study design, the critical procedures, study data, subject protections described in the protocol, and the planned analyses of study data.” Although the ICH assigns importance to particular classes of deviations, it does not use the terminology “major” or “minor”—this is a National Institutes of Health (NIH) classification. The U.S. Department of Health and Human Services regulations 45 Code of Federal Regulations (CFR) 46.103(a) and (b)(5) require that institutions have written procedures to ensure that incidents, related to regulatory requirements pertaining to research conducted under an Office for Human Research Protections-approved assurance, are promptly reported. Adhering to this requirement, NIH has written procedures and policies which include definitions of protocol deviations as major/minor. This paper will adhere to the NIH terminology for research deviations and noncompliance.

While protocols are becoming more complex with an increased likelihood of protocol noncompliance, there is no required formal training in the methods of conducting research during medical school or residency. This lack of training and “learn as you go” approach can compromise the integrity of the study, expose patients to harm, and increase the vulnerability of the researchers with the possibility of disciplinary action from regulatory bodies. Therefore, procedures need to be established to mitigate potential harm.

The Intramural Research Program at the National Institute of Neurological Disorders and Stroke (NINDS) has conducted clinical research since its inception. Most patients are seen at the NIH Clinical Center, the world’s largest research infrastructure for clinical research. In 2014, NINDS established the Clinical Trials Unit (CTU) to oversee aspects of protocol development, regulatory monitoring, auditing, and education for researchers. Research auditing was the focus of this study—defined by ICH as, “A systematic and independent examination of trial-related activities and documents to determine whether the evaluated trial-related activities were conducted, and the data were recorded, analyzed, and accurately reported according to the protocol, sponsor’s standard operating procedures, Good Clinical Practice, and the applicable regulatory requirement(s).” In this study, auditing refers to an examination of trial-related activities and source documents at a moment in time during the course of the protocol, rather than ongoing monitoring of trial-related activities from beginning to closure of the protocol.

Studies in the literature have shown a lack of noncompliance reporting in surgical clinical trials [9] and the need for additional safety monitoring in the biopharmaceutical industry [10]. A site initiation visit (SIV) is comparable to a “time out” prior to a surgical procedure. It is a “pause” prior to the start of a study to ensure that all investigators are aware of their role and responsibilities in the conduct of the study and that all study-related procedures are reviewed and understood by the entire research team. While SIV research has shown a positive effect on patient recruitment, quality of data, and patients’ follow-up time [11], isolating the effect of SIVs on the occurrence of noncompliance has been rare in the literature.

The aim of this study was to investigate a database of noncompliance findings to determine the effects of clinical research training and SIVs on the incidence of noncompliance and protocol deviations identified during audits of non-Food and Drug Administration (FDA) regulated protocols conducted in the Intramural Research Program at NIH/NINDS.

Methods

The findings from protocol audits conducted at NIH/NINDS between January 2003 and December 2019 were included and analyzed in this study. All instances of protocol noncompliance (including protocol deviations) that were not previously reported by the study team, but were identified by auditors during routine audits, were tabulated in NINDS study data reports. Definitions and examples for noncompliance classification are provided in Table 1. Events were categorized by 1) Type; 2) Primary category; and 3) Cause. The type of each noncompliance event was classified as either: 1) Major or 2) Minor. The primary category of each noncompliance event was classified as either 1) Procedural; 2) Informed consent; 3) Eligibility; 4) Failure to follow NIH policy; or 5) Breach of personally identifiable information (PII). Causality was assigned to either of the following categories: 1) The study team; 2) A non-study team staff member; 3) The participant; or 4) Weather and/or travel issues.

Table 1.

Definitions and examples of noncompliance event categorizations

	Deviation	Definition	Example
Type	Major	Potential to negatively influence the rights, welfare, or safety of the patient or to substantially negatively influence the scientific integrity and validity of the study	Failure to perform safety labs at the protocol-specified timepoint, following administration of an investigational drug
	Minor	No potential to negatively influence the rights, safety, or welfare of patients or others, or the scientific integrity or validity of the study	Consenting a patient with an expired consent form, in which the information is not substantively different than the currently approved consent
Primary category	Procedural	Deviates from the IRB-approved protocol with respect to its procedures	Participant is administered a research procedure which is not described in the IRB-approved protocol
	Informed consent	Deviates from the protocol on any aspect of informed consent	Failing to sign and date the consent form during a screening visit, as specified in the protocol
	Eligibility	Failure to comply with the inclusion and exclusion criteria stated in the protocol	Participant does not meet all inclusion/exclusion criteria, but is enrolled in the study
	Failure to follow NIH policy	Failure to abide by National Institutes of Health policies	Failing to obtain a reliance agreement for a non-NIH investigator analyzing identifiable data at a non-NIH facility
	Breach of PII	An event that fails to protect the identity of the patient	Investigator downloads patient records with personally identifiable information (PII) onto an unencrypted laptop that is later stolen during a theft
Cause	Study team	A principal investigator, research coordinator, or similar Staff members	Principal Investigator fails to obtain informed consent during the screening visit prior to research procedures being performed
	Non-study team staff member	Support service members in pharmacy, phlebotomy, radiology, or nursing	Nursing documentation form indicates an incorrect dosage of a study drug was administered during a visit
	Participant	A research participant enrolled in the clinical trial	Participant fails to return to NIH to complete a required procedure within a 4-week window due to personal scheduling conflicts
	Weather and/or travel issues	Weather events (i.e., winter weather) that cause travel delays	A snowstorm delays bloodwork from being delivered to NIH and analyzed by the study team within the required 2-week window

Noncompliance classifications, used in this study, separated by type, primary category, and cause. Each deviation is defined and is explained with an example applicable to clinical research

Procedures in place for auditing protocols from 2003 to 2012 provided a groundwork for the institute (Table 2). Beginning in 2013, the breadth of the audit review broadened, and Human Subjects Training was introduced per Office of Human Subjects Research Protections (OHSRP). Four years later in 2017, NINDS implemented Events Reporting Training and started recommending SIVs for all newly approved protocols. Prior to 2017, SIVs were only conducted in intramural NINDS for FDA-regulated trials in compliance with the regulation for 21 CFR 312.53. Most natural history and screening trials do not fall under the FDA regulations and, as a result, did not undergo SIVs. Under the oversight of the CTU, an increase in protocol noncompliance was observed for non-FDA-regulated trials and due to this, the CTU offered SIVs for all protocols in 2017 in an attempt to mitigate the occurrence of noncompliance events, which may be deviations from the protocol or deviations from policy (either of which can result in harm to participants or data), across all protocols conducted in the NINDS Intramural Research Program. Events Reporting Training is offered both in-person and online and reviews OHSRP policies governing the reporting of protocol-related events. During a SIV, the auditor and study team review the protocol and discuss: 1) Protocol design; 2) Recruitment plan and methods for obtaining informed consent and/or assent; 3) Protocol procedures (screening/baseline/follow-up); 4) Oversight; 5) Investigator qualifications; 6) Data management and documentation; and 7) Regulatory records review. If any concerns are identified during the SIV, the auditors confirm that all issues have been resolved prior to the start of participant enrollment. The goal of these trainings is to protect patients’ safety by educating investigators on the importance of adhering to the IRB-approved protocol and reducing the occurrence of noncompliance events.

Table 2.

Items on audit report and policies from 2003 to 2019

Area of focus during audit	Early	Middle	Late
	2003–2012	2013–2016	2017–2019
Protocol files—IRB-approved protocol review (initial, amendments, continuing review)	✓	✓	✓
Informed consent—consent document review	✓	✓	✓
Enrollment log—complete/accurate	✓	✓	✓
Inclusion/exclusion documentation	✓	✓	✓
Case report form—present/complete	✓	✓	✓
Source documentation—present	✓	✓	✓
Data repository security	✓	✓	✓
Correspondence—FDA and radiation safety	✓	✓	✓
Adverse events—documented and reported	✓	✓	✓
Protocol deviations—identified and reported	✓	✓	✓
Retention of records	✓	✓	✓
Agreements (e.g., tech transfer) approved	✓	✓	✓
Study procedures: H&P and neuro exam	✓	✓	✓
Progress notes for each visit in medical records	✓	✓	✓
Consent forms—participant/investigator signed	✓	✓	✓
Consent forms—witness signed and dated	✓	✓	✓
Consent forms—correct version	✓	✓	✓
Consent forms—documentation of consent		✓	✓
Consent forms—personally identifiable information		✓	✓
Consent forms—present in medical record		✓	✓
Consent forms—short form consent process and assent utilized		✓	✓
Study procedures: screening procedures		✓	✓
Study procedures: procedures addressing primary and secondary outcome measures		✓	✓
Study procedures: safety procedures		✓	✓
Investigator files (CVs, licenses) and training certificates			✓
Inclusion/exclusion documentation via source documentation			✓
Institute-specific policies
Human subjects training		✓	✓
Events reporting training			✓
Site initiation visits*			✓

The evolution of procedures and areas of focus for protocol audits in the NINDS Intramural Research Program from 2003 to 2019. The addition of NINDS-specific policies is included to demonstrate increased oversight since 2003

^*Site Initiation Visits were offered for all protocols, but not required

As a result of implementing SIVs, this study contained four arms: 1) Early Auditing Period (2003 to 2012); 2) Middle Auditing Period (2013 to 2016); and Late Auditing (2017 to 2019), further divided into 3) Late Auditing Period without SIVs; and 4) Late Auditing Period with SIVs. These categories were delineated to best isolate the effects of new institute-wide trainings as well as the effect of SIV implementation. This study analyzed data from all protocols with complete audit reports conducted during each time period (e.g., 2003 to 2012, 2013 to 2016, and 2017 to 2019). As the data for this analysis were extracted from pre-existing audit reports, incorporation of additional bias in this aspect was minimized.

This study was deemed non-human subjects research per the NIH IRB.

Statistical Analysis

Within each arm, the number of protocols, participants, and events were described. The percentage of all protocols in a specific arm which experienced at least 1 event of noncompliance within each type (e.g., major or minor deviations) or primary category (e.g., procedural, informed consent) was described using percentages. This categorical data was analyzed using a two-sided Fisher’s exact test both for global effects across group, and Bonferroni-corrected for pairwise group comparisons. Similarly, the rate of deviations per participant was calculated for each protocol and summarized across type and category of noncompliance using mean and standard error. Deviation rate is a protocol-level variable which is the number of deviations per person in the protocol.

A one-way analysis of variance (ANOVA) was performed to assess for difference in mean event ratio between arms, while a post hoc Tukey honestly significant difference (HSD) test was performed to detect statistical significance between pairwise comparisons. GraphPad Prism 8 was used to create all artwork. All analyses were conducted using RStudio and p < 0.05 was considered a statistically significant result.

Results

Overall, a total of 97 protocols were reviewed from the 2003 to 2019 timeframe (Table 3). There were 49 protocols audited during the Early Auditing Period (2003 to 2012), 17 during the Middle Auditing Period (2013 to 2016), and 31 during the Late Auditing Period (17 without SIVs and 14 with SIVs). This included a total of 1080 participants reviewed, resulting in 952 noncompliance events. Three hundred and seventy events (from 606 participants; 0.61 events per participant) occurred during the Early Auditing Period, 279 events occurred (220 participants; 1.27 events per participant) during the Middle Auditing Period, 274 events occurred (187 participants; 1.47 events per participant) during the Late Auditing Period without SIVs, and 29 events occurred (67 participants; 0.43 events per participant) during the Late Auditing Period with SIVs. As expected, as more charts were reviewed, the number of observed events increased (R² = 0.80; p < 0.001).

Table 3.

Descriptive statistics across groups

	Early	Middle	Late		Total
	2003–2012	2013–2016	2017–2019
			No SIV	SIV
Protocols	49	17	17	14	97
Events	370	279	274	29	952
Participants	606	220	187	67	1080
Events per participant	0.61	1.27	1.47	0.43	0.88

Study characteristics for audits completed during 2003–2012, 2013–2016, and 2017–2019 (further divided into audits of protocols that did not have a site initiation visit and protocols that did have a site initiation visit)

Type of Noncompliance

Overall, the percentage of protocols with at least 1 major event, minor event, and the percentage of protocols with no deviations was related to group (Fisher’s exact p-values: p < 0.001, p < 0.001, p = 0.006, respectively). Of note, the percentage of protocols that experienced major deviations decreased by 52% with the addition of Human Subjects Training (86% during the Early Auditing Period, 41% during the Middle Auditing Period, adjusted p = 0.004), and decreased again by 70% in protocols with a SIV (47% during the Late Auditing Period without SIVs, 14% during the Late Auditing Period with SIVs, adjusted p = 0.41) during the Late Auditing Period. Similarly, the percentage of protocols that experienced minor deviations decreased by 50% with the addition of Human Subjects Training (94% during the Early Auditing Period, 47% during the Middle Auditing Period, adjusted p < 0.001), and decreased by 74% in protocols with a SIV (53% during the Late Auditing Period without SIVs, 14% during the Late Auditing Period with SIVs, adjusted p = 0.34) during the Late Auditing Period. The percentage of protocols that did not experience any deviations increased by 500% with the addition of Human Subjects Training (2% during the Early Auditing Period, 12% during the Middle Auditing Period, adjusted p = 0.96), and increased by 317% in protocols with a SIV (12% during the Late Auditing Period without SIVs, 50% during the Late Auditing Period with SIVs, adjusted p = 0.26) during the Late Auditing Period.

As with the percentage data, the ratio data for type of noncompliance event showed consistent decreases with additional trainings as well as with SIVs (Fig. 1). Regarding the number of major deviations per participant, there was a marginal effect of group on the ratio of noncompliance events (F(3,93) = 2.76; p = 0.047). There was a steady improvement across all groups, and in particular a drop of 0.15 deviations per participant from the Late Auditing Period without SIVs to the Late Auditing Period with SIVs (0.23 vs. 0.08). Regarding the number of minor deviations per participant, there was a clear overall effect of group (F(3,93) = 3.35; p = 0.02), with a decrease of 0.56 events per participant emerging between the Early Auditing Period and the Late Auditing Period with SIVs groups (0.62 ± 0.66 vs. 0.06 ± 0.15, respectively, adjusted p = 0.03).

Fig. 1.

Types of noncompliance by ratio. Mean deviation rates across protocols for each type of noncompliance (major or minor) are shown by time period and are analyzed by pairwise comparisons using post hoc Tukey HSD. Error bars represent standard error. *p = 0.03

Primary Category of Noncompliance

Due to the number of procedures in a given protocol, procedural-related deviations occurred with the greatest frequency. Protocols with at least one procedural-related deviation decreased over time (Fisher’s exact p value = 0.01). While a noteworthy decrease occurred between the Early Auditing Period and the Late Auditing Period with SIVs (92% vs. 57%, adjusted p = 0.03), a decrease in the percentage of protocols with deviations was also seen between the Late Auditing Period without SIVs and the Late Auditing Period with SIVs groups (82% vs. 57%, adjusted p = 1.0). Protocols with at least one informed consent deviation (Fisher’s exact p value = 0.01) showed a substantial drop in percentage of protocols with informed consent deviations from 59% in the Late Auditing Period without SIVs group to 14% in the Late Auditing Period with SIVs group. The percentage of protocols with eligibility deviations and with noncompliance related to failing to follow NIH policy were also related to group (Fisher’s exact p value: p = 0.03 and p < 0.001, respectively). In both cases, the Late Auditing Period with SIVs group showed lower rates (21%) in deviations related to eligibility and noncompliance related to failing to follow NIH policy than all earlier timepoints, or than protocols during the Late Auditing Period without SIVs. The percentage of protocols with a breach of PII decreased by 100% with the implementation of SIVs, compared to protocols audited during the Late Auditing Period without SIVs (0% vs. 12%, respectively, adjusted p = 1.0).

For the mean number of events per participant, the main primary categories that demonstrated reductions across group were Eligibility (F(3,93) = 3.45; p = 0.02) and Failure to Follow Policy (F(3,93) = 9.07; p < 0.001), although the former did not have pairwise comparisons that survived multiple correction (Fig. 2).

Fig. 2.

Primary categories of noncompliance by ratio. Mean deviation rates across protocols for each primary category of noncompliance are shown by time period and are analyzed by pairwise comparisons using post hoc Tukey HSD. Error bars represent standard error. **p = 0.001. PII = personally identifiable information

Policy-related noncompliance generally decreased over time, with a further reduction in the SIV arm, as 0.16 ± 0.18 events occurred per participant during the Early Auditing Period, 0.09 ± 0.19 events per participant during the Middle Auditing Period, 0.39 ± 0.32 events per participant during the Late Auditing Period without SIVs, and 0.05 ± 0.10 events per participant during the Late Auditing Period with SIVs.

Cause of Noncompliance

Focusing on the study teams, no overall effect of group was seen on the percentage of protocols with at least one event caused by the study team (p = 0.07), and there was no effect of group on the deviations per participant caused by the study team (F(3,93) = 1.08; p = 0.36).

Discussion

The primary goal of the present study was to observe differences in the occurrence of noncompliance events in protocols conducted at NIH/NINDS, where investigators have had Human Subjects Research Training, Events Reporting Training, and a SIV prior to beginning the protocol. Monitoring of study progress and compliance is an integral component in ensuring that the study team implements study procedures consistent with the IRB-approved protocol. Results indicated a reduction in the incidence of noncompliance events, as shown by fewer major and minor deviations, as well as fewer study team-caused events, across all protocols. Even with a lack of significant differences between SIV and No SIV groups, the data suggest that implementing SIVs generally decreased the incidence of noncompliance events. With the goal of reducing noncompliance events, any reduction in deviations is potentially worth achieving and shows the utility of SIVs in preventing protocol noncompliance.

A 2013 study showed that in 5 amyotrophic lateral sclerosis clinical trials, the most common forms of noncompliance, which accounted for 84% of all deviations, were events related to out of timeframe visits and outcomes and safety procedures [12]. Furthermore, 6% of deviations were related to eligibility, 3% to IRB and regulatory concerns, and 2% to informed consent. With an increasing number of study participants needed to strengthen the statistical power of clinical trials, large-scale studies are likely to place their participants at an increased risk for encountering safety events.

Another study found that of 100 publications of randomized controlled trials published in BMJ, New England Journal of Medicine, the Journal of the American Medical Association, and The Lancet, 98 experienced noncompliance related to the treatment protocol [13]. Of the 98 controlled trials, 51 implemented a statistical method, most commonly an intention to treat analysis, to address the potential loss of data from noncompliance. Our methods proposed in this study would serve to improve the quality and completeness of data from clinical trials studying neurological diseases.

In the NINDS Intramural Research Program, auditors are available to conduct a SIV according to the guidelines outlined in ICH E6 (R2) and Good Clinical Practice. An important aspect of a SIV is to ensure all required processes are in place prior to participant enrollment. This includes reviewing methods of obtaining informed consent and/or assent, discussing protocol procedures and the necessary documentation to support the study, and reviewing good data management practices. The aim is to identify protocol-specific areas of vulnerability that may result in noncompliance. This is particularly important for neurological studies where a wide variety of tools for patient assessment are applied that may range from genetic testing to neuropsychological assessment, neurophysiological testing, magnetic resonance imaging, positron emission tomography, and cerebrospinal fluid analysis. Patients with neurological diseases often have cognitive impairment or physical disabilities that require special considerations in obtaining consent [14, 15], which may be further complicated in non-English speaking populations. Further studies are needed to clarify what aspects of the SIV are most influential in decreasing events of protocol noncompliance.

SIVs have other advantages. They safeguard the principles of autonomy, beneficence, and justice throughout phase I clinical trials [16] and can help patient enrollment [17]. SIVs are also a model opportunity for the site staff to discuss the protocol and its procedures in depth with the monitor [18]. Additionally, it has been recommended that, if applicable, the monitor should remain on-site during initial patient recruitment or revisit the study site shortly after recruitment is initiated to troubleshoot any problems that may occur. While these studies are crucial to understand the importance of SIVs during the early onset of a clinical trial, our results provide data on the effect of SIVs for the entire life of a clinical trial.

A potential limitation of our analysis is that protocols audited during 2003 through 2012 had fewer criteria to meet to determine if a protocol was compliant or not, per the evolution in policy over time at NIH and NINDS. It is likely that noncompliance events were not identified in some earlier instances, compared to events identified from 2013 to 2019 when audit criteria were increasingly rigorous and analyzed the protocol in its entirety. However, while comparing the effect of SIVs within group (Late Auditing Period), it was found that the incidence of noncompliance events decreased in the group with SIVs, supporting the premise that SIVs also helped study teams adhere to required research policies to ensure protocol compliance.

This study shows the importance of clinical research training and SIVs in human subjects research and should be implemented for all levels of neurology clinical trial protocols, from minimal-risk, natural history studies to high-risk, FDA-regulated clinical trials. Conducting randomized controlled trials for neurological diseases is the gold standard for advancing our field [19], and protocol compliance, as evidenced in our results, should be of primary importance while performing clinical trials both for the protection of human subjects and for the integrity of the research data. Future studies are warranted to investigate if SIVs have a comparable benefit between smaller, single-site studies and larger, multi-site studies. Additional studies are also needed to quantify an improvement in patient safety (e.g., correlate a reduction in adverse events with additional research training and SIVs), and to analyze the specific qualities within a SIV that most greatly impact protocol compliance.