A Randomized Evaluation of On-site Monitoring Nested in a Multinational Randomized Trial Short title: Randomized Trial of On-site Monitoring

Abstract

Background:

Evidence from prospectively designed studies to guide on-site monitoring practices for randomized trials is limited. A cluster randomized study, nested within the Strategic Timing of AntiRetroviral Treatment (START) trial, was conducted to evaluate on-site monitoring.

Methods:

Sites were randomized to either annual on-site monitoring or no on-site monitoring. All sites were centrally monitored, and local monitoring was carried out twice each year. Randomization was stratified by country and projected enrollment in START. The primary outcome was a participant level composite outcome including components for eligibility errors, consent violations, use of antiretroviral treatment not recommended by protocol, late reporting of START primary and secondary clinical endpoints (defined as the event being reported more than 6 months from occurrence), and data alteration and fraud. Logistic regression fixed effect hierarchical models were used to compare on-site versus no on-site monitoring for the primary composite outcome and its components. Odds ratios (ORs) and 95% confidence intervals (CIs) comparing on-site monitoring vs no on-site monitoring are cited.

Results:

Ninety-nine sites (2,107 participants) were randomized to receive annual on-site monitoring and 97 sites (2,264 participants) were randomized to be monitored only centrally and locally. The two monitoring groups were well-balanced at entry. In the on-site monitoring group, 469 annual on-site monitoring visits were conducted, and 134 participants (6.4%) in 56 of 99 sites (57%) had a primary monitoring outcome. In the no on-site monitoring group, 85 participants (3.8%) in 34 of 97 sites (35%) had a primary monitoring outcome (OR=1.7; 95% CI: 1.1-2.7; p=0.03). Informed consent violations accounted for most outcomes in each group (56 versus 41 participants). The largest OR was for eligibility violations (OR=12.2; 95% CI: 1.8-85.2; p=0.01). The number of participants with a late START primary endpoint was similar for each monitoring group (23 versus 16 participants). Late START grade 4 and unscheduled hospitalization events were found for 34 participants in the on-site monitoring group and 19 participants in the no on-site monitoring group (OR=2.0; 95% CI: 1.1-3.7; p=0.02). There were no cases of data alteration or fraud. Based on the travel budget for on-site monitoring and the hours spent conducting on-site monitoring, the estimated cost of on-site monitoring was over $2 million.

Conclusions:

On-site monitoring led to the identification of more eligibility and consent violations and START clinical events being reported more than 6 months from occurrence as compared to no on-site monitoring. Considering the nature of the excess monitoring outcomes identified at sites receiving on-site monitoring, as well as the cost of on-site monitoring, the value to the START study was limited.

Background

The International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use guideline for Good Clinical Practice (GCP) E6¹ recommends on-site monitoring to safeguard the well-being of participants and to ensure trial data are accurate and complete and to ensure trial conduct is in compliance with currently approved protocol/amendment(s), GCP, and applicable regulatory requirement(s). Reference regulatory bodies such as the US Food and Drug Administration and the European Medicines Agency have adopted the International Conference on Harmonization guideline into their regulations. ^2,3 However, these regulations leave the nature, frequency and extent of monitoring to the discretion of the study sponsor. On-site monitoring provides an opportunity to train staff and perform source data verification. However, it is expensive, time-consuming, and as often practiced (100% source document verification) can be inefficient at assuring participant safety and data quality.^4,5,6

To improve efficiency, exploration of risk-based models of on-site monitoring has been endorsed by the Food and Drug Administration, European Medicines Agency,^7,8 the Clinical Trials Transformation Initiative,⁹ and TransCelerate.¹⁰ Central data monitoring has been used to guide the frequency and content of on-site visits,^{11,12,13,14,15} and some statistical approaches to identify problems in its use have been evaluated.^16,17,18,19 However, evidence to guide decisions about the frequency and intensity of on-site monitoring remains limited.^10,14,20,21

To guide future monitoring practices of large, international HIV strategy trials, a site monitoring substudy was nested within the Strategic Timing of AntiRetroviral Treatment (START) trial conducted by the International Network for Strategic Initiatives in Global HIV Trials (INSIGHT). Here we give substudy results and discuss difficulties defining monitoring outcomes and study conduct.

Methods

START study

The design of START has been described.²² START was an international, open-label, publicly funded randomized trial designed to assess whether immediate antiretroviral therapy was superior to deferral. The primary endpoint of the START study was a composite outcome of pre-defined clinical events. Each reported START primary endpoint was reviewed by an Endpoint Review Committee to determine if the event met pre-specified diagnostic criteria.^23,24 Grade 4 clinical events and unscheduled hospitalizations were also collected.²⁵ START required that the first antiretroviral therapy regimen prescribed in each treatment group be based on US Department of Health and Human Services guidelines.

Participants were enrolled in START between 2009 and 2013 at 215 sites overseen by one of four INSIGHT coordinating centers located in Copenhagen, London, Sydney and Washington DC.^22,26

START monitoring substudy

The design of the monitoring substudy has been described.²⁷ Sites were randomized to receive annual on-site monitoring in addition to central and local monitoring (“on-site group”) or to local and central monitoring only (“no on-site group”). The institutional review board (IRB) for the sponsor of START reviewed the substudy, but the sponsor did not require site IRB or ethics committee approval or additional participant consent since there was no direct participant contact related to the substudy. At any time, the START protocol team or a coordinating center could request a for-cause on-site monitoring visit at any site thought to not be meeting protocol standards.

Randomization

Site randomization was stratified by country and projected START enrollment (<15, 15-30, >30 participants), and was carried out by the statistical center using block randomization prior to the beginning of the substudy. Coordinating centers were informed of the assignments.

Central, local, and on-site monitoring

Central, local, and on-site monitoring was described briefly in the design paper.²⁷ Central monitoring by the statistical center was done continuously using central databases; reports provided to sites were updated regularly. Reports focused on tracking losses to follow-up, study visit attendance, and timely data reporting and query resolution, including START clinical endpoints. Summary data of central monitoring findings were also shared with site investigators at least twice a year at investigator meetings. Trained nurses at the statistical center also reviewed all reported grade 4 and unscheduled hospitalizations for possible START primary clinical events. If a possible START primary clinical event was identified, the site was asked to submit the necessary documentation.

A committee comprised of staff from the statistical center and coordinating centers also reviewed central monitoring findings for each site. Quantitative feedback focusing on participant retention, data quality, timeliness and completeness of START clinical event documentation, and adherence to local monitoring requirements was provided to sites.

Local monitoring of regulatory files, study specimens labeling and storage, study drug management and accountability, and source document verification was performed by site staff semi-annually. Source documents for eligibility, informed consent, changes in antiretroviral therapy, follow-up visits, and reportable START clinical events were reviewed for 15% of participants. All participant charts were reviewed at sites with less than 5 participants. Standardized case report forms were used to report specific findings to the statistical center. Coordinating centers were provided with reports listing each finding, and when available, the description of the finding and any corrective actions taken by the site. Coordinating centers were responsible for the review and follow-up of each finding to ensure proper resolution.

Annual on-site monitoring was performed by a coordinating center staff or coordinating center-designated staff located in the country of the sites being monitored. Each on-site monitoring visit included a review of the site’s study regulatory files, study specimen labeling and storage, study drug management and accountability, and source document verification (eligibility, informed consent, changes in antiretroviral therapy, reportable START clinical events). On-site monitors submitted standardized reports, which described each finding and any corrective actions required of the site, to their coordinating center for review and forwarding to the statistical center. A Visit Cost Summary case report form, documenting the number of person-hours spent conducting on-site monitoring-related activities, accompanied the reports.

The list of participants for both local and on-site monitoring source document verification was prepared by the statistical center. Participants were prioritized for review by: (1) START clinical event reported since the last review; (2) newly lost to follow-up since the last review; (3) transferred to another site since the last review; (4) previously identified as lost to follow-up and was still lost; (5) time from last review. While local monitors were provided a list of 15% of their participants, on-site monitors were provided a list of all participants at the site and were asked to review charts as many as possible.

Additional information is provided in the online supplemental appendix.

START monitoring substudy primary and secondary outcomes

The primary monitoring substudy endpoint was the subject of discussion amongst the study team. One important consideration was identifying issues that could impact the validity of START results. Consent and eligibility violations were also included as outcomes since such findings are important to identify possible improper participant consent or safety issues and are typically monitored in trials that conduct on-site monitoring.²⁰

With that background the primary outcome for the monitoring substudy was a participant-level composite outcome consisting of six major components:

1. Eligibility violations (HIV negative, lack of 2 CD4+ cell counts > 500 cells/mm³ within 60 days before randomization, prior antiretroviral therapy or interleukin-2 use, or pregnancy).

2. Informed consent violations were initially defined as:

   • Study-specific procedures performed or participant randomized prior to signing the appropriate IRB/ethics committee -approved consent.
   • Study-specific procedures performed prior to signing new IRB/ethics committee - approved consent (e.g., amendment)
   • Most recently signed consent not on file.
   • Signature or date on consent not made by participant or legal representative.

3. Use of antiretroviral therapy for initial therapy that was not permitted by START

4. START primary clinical event not reported within 6 months from occurrence (all potential primary endpoints were counted irrespective of later Endpoint Review Committee review).

5. START serious clinical event (grade 4 event or unscheduled hospitalization) not reported within 6 months from occurrence.

6. Data alteration or fraud.

The consent violations chosen were thought to be consistently documented and easily ascertained. However, during review of the reported consent violations, it became clear that documentation and ascertainment were not consistent across sites. This led to discussions about the varying requirements by sites for initial consent and reconsent. The primary outcome component for consent violations was modified in February 2016:

• For consent prior to randomization: 1) participant signed unapproved or incorrect consent; or 2) specimens for storage for future research collected prior to obtaining consent.

• For later consents due to amendments required locally or by the sponsor: 1) participant’s signature page is not on file; or 2) consent form not signed by participant or legal representative.

A procedure was implemented for statistical center staff to centrally review consent violations found by on-site monitors to determine if the violation met the revised criteria. Due to reporting methods (supplemental appendix), consent violations found through local monitoring were not centrally reviewed. Therefore, local consent violations included in analysis meet the original definition of a consent violation, but may not meet the revised definition.

All components of the primary monitoring outcome were counted in each randomized group regardless of how the event was identified. The study team could ascertain whether site staff or on-site monitors found the eligibility and consent violations. On-site and local monitors were required to review source documents related to the monitoring outcomes. Important errors related to tertiary data, e.g., laboratory values, could be detected with the central monitoring provided for both groups; those not detected would likely be random and have little effect on START results.^5,28

Henceforth, we refer to the primary endpoint of the monitoring substudy as the primary monitoring outcome. We hypothesized that on-site monitoring plus central and local monitoring would lead to more participants with primary monitoring outcomes identified than central and local monitoring alone.

Pre-specified secondary outcomes included:

• Percentage of participants lost to follow-up

• Percentage of missed follow-up data collection visits; and

• Data submission timeliness.

These secondary outcomes were all determined centrally. They were summarized in regularly updated web reports. We hypothesized that sites receiving annual on-site monitoring would have less missing data and more timely submission of data.

The monetary cost of annual on-site monitoring visits, including the person-hours spent conducting on-site monitoring-related activities, was also collected.

Follow-up

The monitoring substudy collected data between September 2011 and December 2016.

Statistical analyses

For the primary monitoring outcome and its components, generalized estimating equation models with fixed effects to account for clustering (PROC GENMOD) were used to compare the on-site group to the no on-site group. Sensitivity analyses for the primary monitoring outcome were performed using a generalized linear mixed model with random effects to account for clustering (PROC GLIMMIX) and logistic regression without accounting for clustering (PROC LOGISTIC). All of these models included a single indicator variable for randomized monitoring group. Odds ratios (OR) and 95% confidence intervals (Cl) are reported. All analyses were intention to treat. The intraclass correlation coefficient for discrete outcomes was estimated for the primary monitoring outcome using the method reported by Snijders and Bosker.²⁹ Subgroup analyses for the composite primary monitoring outcome, the components of the primary monitoring outcome, and secondary analyses were carried out using similar methods.

Subgroup analyses for the composite primary monitoring outcome were carried out according for: prior experience with an INSIGHT HIV study, whether any START-related on-site monitoring occurred before the substudy opened, coordinating center, geographic region, projected and actual enrollment by site, and START randomization group.

All analyses were performed with SAS version 9.3 (SAS Institute, Cary, North Carolina, US).

Results

Baseline characteristics

The START study enrolled 4,688 participants at 215 sites in 35 countries. Data collection for the monitoring substudy includes 4,371 participants from 196 sites in 34 countries. Sites in Germany were not allowed to participate due to national monitoring regulations (Figure 1). Ninety-nine sites (2,107 participants) were randomized to the on-site group, and 97 sites (2,264 participants) were randomized to the no on-site group. The groups were well balanced at baseline (Table 1). The percentage of sites with prior experience with an INSIGHT HIV trial was similar between the groups with 67% and 69% of sites in the on-site group and the no on-site group, respectively. Prior to the beginning of the monitoring substudy, 35% of sites in the on-site group and 39% of sites in the no on-site group had been on-site monitored.

Figure 1. Site monitoring substudy design and flow diagram.

^a 17 sites in Germany were excluded due to German national law required the clinical sites to be on-site monitored semi-annually. Two sites in the US closed to START before the monitoring substudy opened.

Table 1.

Baseline characteristics by random on-site monitoring assignment

Site Characteristic	On-site	No on-site
No. sites	99	97
Prior experience with INSIGHT (%)a	66.7	69.1
On-site monitored before start of substudy (%)b	35.4	39.2
Study enrollment (%)
< 15 participants	57.6	56.7
15-30 participants	22.2	27.8
> 30 participants	20.2	15.5
Geographic region
Africa	6.1	4.1
Asia	5.1	7.2
Australia	6.1	7.2
Europe and Israel	43.4	42.3
North America	28.3	25.8
South America and Mexico	11.1	13.4
Participant Characteristic	On-site	No on-site
No. participants	2107	2264
Age (years); median (IQR)	35 (28, 43)	36 (29, 44)
Gender (% female)	31.0	25.8
Race (%)
Asian	7.8	9.8
Black	30.1	33.7
Latino/Hispanic	15.4	13.6
White	44.7	37.5
Other	1.9	5.3
Geographic region
Africa	23.6	22.3
Asia	7.0	9.2
Australia	2.0	3.0
Europe and Israel	29.0	27.3
North America	13.1	10.0
South America and Mexico	25.3	28.3
Current smoker (%)	29.4	31.5
CD4+ (cells/mm3), median (IQR)	660 (588, 777)	648 (581, 759)
HIV-RNA (log10 copies/mL); median (IQR)	4.1 (3.5, 4.6)	4.1 (3.5, 4.6)
Hepatitis B (%)	2.4	3.4
Hepatitis C (%)	3.9	3.4
Framingham CHD 10-yr risk; median (IQR)	1.7 (0.4, 4.7)	1.9 (0.5, 5.1)

^aParticipated in either INSIGHT’s SMART or ESPRIT study

^bSTART opened on 09 April 2009 and the monitoring substudy opened on 01 September 2011

CHD = coronary heart disease

Description of randomized groups

Central monitoring was identical for sites in each group. Web reports were updated regularly for all sites and did not differ by randomized group.

A description of individual chart review frequency by both local and on-site monitors by randomized group is in Table 2. Percentage of participants with at least one review by local staff was similar for the two monitoring groups (89% for both groups).

Table 2.

Description of randomized groups

	On-site	No on-site
No. sites	99	97
No. of participants	2107	2264
Local monitoring
No. participants with ≥ 1 review (%)	1867 (88.6)	2026 (89.5)
Total participant chart reviews	5113	5133
On-site monitoring
No. monitoring visits	469
No. participants with ≥ 1 review (%)	1930 (91.6)
Total participant chart reviews	5345
Visit Cost Summary form received (%)	98.0
- Person-hours preparing for visit	2,328
- Person-hours conducting on-site visit	9,515
- Person-hours writing report	2,670
- Person-hours resolving issues post-visit	2,086

In total, 469 annual on-site monitoring visits were conducted at 99 sites (Table 2). The Visit Cost Summary case report form was received for 98% of on-site monitoring visits. A total of 16,599 person-hours by on-site monitors and coordinating centers were spent performing on-site monitoring related activities. Total travel costs allocated for START monitoring were $790,467. Salary allocation for on-site monitors was $75/hour. The cost for time of on-site monitors and travel is estimated as 16,599 hours x $75 = $1,244,925 + $790,467 = $2,035,392.

In the on-site and no on-site groups, there were 4 and 6 monitoring visits for-cause, respectively.

Primary monitoring outcome and components

In the on-site group, 134 participants (6.4%) at 56 sites (57%) had a primary monitoring outcome; in the no on-site group, 85 participants (3.8%) at 34 sites (35%) had a primary monitoring outcome (Table 3). The intraclass correlation coefficient was 0.08. The OR comparing the on-site group versus the no on-site group using the generalized estimating equation model with fixed effects to account for clustering was 1.7 (95% CI: 1.1-2.7; p=0.03). The ORs for the generalized linear mixed model using random effects to account for clustering and the logistic model without any account for clustering were 1.9 (95% CI: 1.2-3.0; p<0.01) and 1.7 (95% CI: 1.3-2.3; p<0.01), respectively. The log odds standard error for the generalized estimating equation, generalized linear mixed model, and logistic model with no accounting for clustering were 0.24, 0.23, and 0.14, respectively.

Table 3.

Primary endpoint results

	On-site		No on-site
	Pts	Pts with event (%)	Pts	Pts with event (%)	ICCb	OR (95% CI)c	p-valuec
Primary outcomea	2107 (99 sites)	134 (6.4) (56 sites)	2264 (97 sites)	85 (3.8) (34 sites)	0.08	1.7 (1.1, 2.7)	0.03
Components of primary endpoint
Eligibility violation		12 (0.6)		1 (0.04)		12.2 (1.8, 85.2)	0.01
- HIV negative		3 (0.1)		0 (0.0)
- Did not have 2 consecutive CD4+ cell counts >500 within 60 days prior to randomization		1 (0.04)		1 (0.04)
- Prior use of antiretroviral therapy		8 (0.4)		0 (0.0)
- Prior use interleukin-2 (IL-2)		0 (0.0)		0 (0.0)
- Pregnant		0 (0.0)		0 (0.0)
Informed consent violationd		56 (2.7)		41 (1.8)		1.3 (0.6, 2.7)	0.46
- Prior to randomization, participant signed an incorrect or unapproved main study consent form		13 (0.6)		0 (0.0)
- Baseline stored specimens were collected prior the participant signing the IRB/EC-approved main study consent form		8 (0.4)		17 (0.8)		0.7 (0.2, 1.9)	0.44
- The signature page of the most recently signed main study consent form is not on file		38 (1.8)		29 (1.3)		1.0 (0.4, 2.3)	>0.99
- The signature on the most recently signed main study consent form not signed by the participant or legal representative		2 (0.1)		3 (0.1)		0.7 (0.2, 3.3)	0.68
Use of ART drug for initial therapy not permitted by protocol		17 (0.8)		13 (0.6)		1.4 (0.6, 3.4)	0.47
START primary endpoint reported > 6 months from occurence		23 (1.1)		16 (0.7)		1.5 (0.7, 3.0)	0.27
- Confirmed START primary endpoints		10 (0.5)		11 (0.5)		0.9 (0.4, 2.3)	0.88
START serious event reported > 6 months from occurrenced		34 (1.6)		19 (0.8)		2.0 (1.1, 3.7)	0.02
- Grade 4 events		12 (0.6)		6 (0.3)		2.1 (0.9, 5.3)	0.10
- Unscheduled hospitalizations		25 (1.2)		17 (0.8)		1.7 (0.8, 3.3)	0.14
Data alteration/fraud		0 (0.0)		0 (0.0)		--	--

^aParticipants could have multiple outcomes identified. In total, there were 158 primary outcomes for 134 participants in the on-site group and 102 primary outcomes for 85 participants in the no on-site group

^bIntraclass correlation coefficient for discrete outcomes using the method reported by Snijders and Bosker

^cLogistic regression fixed effects hierarchical model comparing the on-site group to the no on-site group

^dParticipants could have more than one type of violation for this monitoring outcome

IRB = Institutional review board

EC = Ethics committee

ART = Antiretroviral therapy

Informed consent violations accounted for most of the primary outcomes in each group (56 participants in the on-site group and 41 participants in the no on-site group) with an OR of 1.3 (95% CI: 0.6-2.7; p=0.46). Sites receiving annual on-site monitoring had an excess of 15 participants with an informed consent violation; most were instances of a missing signature page for the most recently signed consent. Thirteen participants signed an incorrect version of the consent, all of which were identified by on-site monitors. In one coordinating center, the number of participants with a consent violation found was higher in the on-group compared to the no on-site group (20 vs 3 participants). This was balanced between groups for the other 3 coordinating centers (36 and 38 participants for the on-site and no on-site groups respectively) (data not shown). Consent violations found by local monitors were balanced between the groups (Table 4).

Table 4.

Informed consent violations

	On-site			No on-site
	Identified through			Identified through
	Local review, Pts (%)	On-site visit, Pts (%)	Either	Local review, Pts (%)	On-site visit, Pts (%)	Either
Any informed consent violation	33 (1.6)	24 (1.1)	56 (2.7)	40 (1.8)	1 (0.04)	41 (1.8)
Type of consent violation founda
- Prior to randomization, participant signed an incorrect or unapproved main study consent form	0 (0.0)	13 (0.6)	13 (0.6)	0 (0.0)	0 (0.0)	0 (0.0)
- Baseline stored specimens were collected prior the participant signing the IRB/EC-approved main study consent form	5 (0.2)	3 (0.1)	8 (0.4)	17 (0.8)	0 (0.0)	17 (0.8)
- The signature page of the most recently signed mains study consent form is not on file	28 (1.3)	10 (0.5)	38 (1.8)	28 (1.2)	1 (0.04)	29 (1.3)
- The signature on the most recently signed main study consent form was not made by the participant or legal representative	2 (0.1)	0 (0.0)	2 (0.1)	3 (0.1)	0 (0.0)	3 (0.1)

^aParticipants could have more than one type of violation for this monitoring outcome

IRB = Institutional review board

EC = Ethics committee

The largest relative difference between the groups was for eligibility violations (12 vs 1 participants; OR=12.2; 95% CI: 1.8-85.2; p=0.01). In the on-site group, 3 participants were found to be HIV-negative and 8 participants used antiretroviral therapy prior to enrollment. One participant in each group did not have 2 consecutive CD4+ cell counts > 500 cells/mm³ within 60 days prior to randomization. Some of these violations were identified within a few months following randomization; most were identified a year or more following randomization. Site staff found 38% of eligibility violations (supplemental Table S1).

In the on-site and no on-site groups, 17 and 13 participants respectively, were prescribed initial antiretroviral therapy regimen not permitted by START (OR=1.4; 95% CI: 0.6-3.4; p=0.47). While there was no overall difference between the monitoring groups, a greater number occurred in the deferred-antiretroviral therapy group of START.

A START primary endpoint was reported more than 6 months after occurrence for 23 participants in the on-site group and 16 participants in the no on-site group (OR=1.5; 95% CI: 0.7-3.0; p=0.27). Excluding START primary endpoints eventually ruled out by the Endpoint Review Committee, a START primary endpoint reported > 6 months from occurrence was found for 10 participants in the on-site group and 11 in the no on-site group (OR=0.9; 95% CI: 0.4-2.3; p=0.88). The majority of START primary endpoints reported > 6 months from occurrence ruled out by the Endpoint Review Committee were bacterial pneumonia (n=9) and tuberculosis (pulmonary or extrapulmonary) (n=7).

In the on-site and no on-site groups there were 34 and 19 participants, respectively, with a START serious event reported > 6 months from occurrence (OR=2.0; 95% CI: 1.1-3.7; p=0.02). Most of these were unscheduled hospitalization (25 vs 17 participants for the on-site and no on-site groups respectively). For both monitoring groups combined, 754 participants (17.3%) developed a START serious event; 6% of all events were reported late.

There were no findings of data alteration or fraud.

Primary monitoring outcome by baseline subgroups

There was no evidence of heterogeneity among the ORs for 6 of the 7 subgroups considered (supplemental Table S2). The estimated ORs varied according to prior experience with an INSIGHT HIV study (p=0.04 for interaction). The OR for prior experience with an INSIGHT HIV study was 1.4 (95% CI: 0.8, 2.4; p=0.26) and was 3.2 (95% CI: 1.6, 6.1; p<0.001) for no prior experience with an INSIGHT HIV study.

Secondary monitoring outcomes

There was a trend for fewer losses to follow-up in the on-site group compared to the no on-site group. This was largely due to fewer withdrawals due to site closures in the no on-site group. For other pre-specified secondary outcomes, sites in the on-site group had higher percentages of missing data and delayed submission of case report forms (Table 5).

Table 5.

Secondary endpoints of the Monitoring Substudy

	On-site	No on-site	OR (95% CI)a	p-valuea
No. sites	99	97
No. participants	2107	2264
Pct. lost to follow-upb	7.1	8.6	0.8 (0.5, 1.1)	0.19
- withdrawal due to site closure	0.4	1.1
- withdrawal of consent	2.3	2.4
- no data for 10 months	4.4	5.0
No. follow-up visits expectedc	30237	32585
Pct. of visits missed	8.6	7.8
No. follow-up visits attended	27629	30035
Pct. forms submitted > 2 weeks after visit	29.2	23.0
Pct. forms with query upon submission	38.7	31.2
Pct. forms clean > 2 weeks	20.9	15.7
Pct. forms with missing required data	8.4	5.9

^aLogistic regression fixed effects hierarchical model comparing the on-site group to the no on-site group

^bNo data in 10 months or withdrawal from START after the opening of the monitoring substudy

^cBaseline and START follow-up visits that took place or whose visit window closed between 1 Sep 2011 and 31 Dec 2016

Conclusions

In this cluster-randomized trial in which 196 clinical sites in a large, HIV trial were randomized to receive annual on-site monitoring in addition to local and central monitoring or to local and central monitoring only, a greater number of participants in the on-site group (n=134) were found to have a composite outcome consisting of eligibility violations, consent violations, and delayed reporting of major START clinical events than the group that only had local and central monitoring (n=85). While the major components of the composite primary outcome for this substudy varied in the frequency with which they were identified, for each component with an event, more outcomes were identified for participants in the on-site group than in the no on-site group. In total, with 469 on-site visits carried out in the on-site group, 49 more participants had a monitoring outcome identified than with local and central monitoring alone.

With a composite outcome like that used for this monitoring substudy, each component of the composite has to be evaluated to interpret the relevance of the overall composite result.³⁰ START primary and serious events reported > 6 months from occurrence were considered important because the results of START could be impacted. Considering START primary and serious events together, a START clinical event was reported > 6 months from occurrence for 22 excess participants in the on-site group (14 excess participants if only confirmed START primary events are counted). With consideration of these differences and the cost of on-site monitoring, the addition of annual on-site monitoring likely had minimal impact on START results.

Confirmed START primary endpoints were reported > 6 months from occurrence by only 10 participants in the on-site group and 11 in the no on-site group; losses to follow-up were similar (particularly after excluding site closures for which participants in both START treatment groups were withdrawn); and a similar percentage (8%) of study visits were missed by participants in each monitoring group. Taken together, these data indicate that it is unlikely on-site monitoring had a major impact on identifying START primary events that would have led to a biased treatment difference given that the percentage reduction in risk of the START primary endpoint for the immediate versus deferred antiretroviral therapy group was estimated as 57%.³¹

The difference between monitoring groups was greater for START serious events reported > 6 months after occurring (34 participants versus 19 participants in the on-site versus no on-site group). These 53 participants represent 6% of all participants with a START serious event. As previously reported, the number of START serious events reported for the immediate and deferred antiretroviral therapy groups were similar.³¹ Thus, if the excess number of participants with a START serious event reported > 6 months from occurrence identified by on-site monitoring (n=15) were missed in the no on-site group, it would have little impact on this safety outcome.

Numerically, the largest number of monitoring outcomes identified in each group were consent violations, and most of these were found by local monitors. The most common consent violation found was the most recently signed consent signature page being missing. The variation across coordinating centers suggests that surveillance for these consent violations by on-site monitors varied and instructions for monitoring consents need to be more specific.

Based on the descriptions and corrective action summaries provided by on-site monitors, there are no indications that any participant had been improperly consented or not consented. While there was no central review of consent violations found through local monitoring, the follow-up by the coordinating centers did not identify any individual that had not been properly consented. The statistical center did not find any systematic consent issues at a particular site or sites. Collectively, the consent violations identified gaps in local consent procedures, such as ensuring the correct version of the consent was signed, documented, or properly filed.

The most common eligibility violation was prior antiretroviral therapy use (8 participants). More than half of these were identified many months after randomization. Many participants were first seen by START sites (i.e., this was entry into care). Three participants were found to be HIV-negative (2 found by site staff, 1 found by an on-site monitor), of which one received antiretroviral therapy. There were no serious events reported for the HIV-negative participant who received antiretroviral therapy. It is not clear why eligibility violations were more commonly found in the on-site group, as 38% of eligibility violations were first identified by site staff. This may be a chance finding.

The number of violations due to use of antiretroviral therapy drugs not permitted by protocol was similar in the two monitoring groups. It is not surprising that a greater number occurred in the deferred-antiretroviral therapy group of START. Several of which resulted from drugs prescribed by a clinic other than where the participant was randomized. Some resulted from enrollment in another clinical trial. In other cases, START sites prescribed an antiretroviral therapy drug shortly before it was permitted by the protocol.

We hypothesized that sites randomized to receive on-site monitoring would have fewer missed visits and more timely data submission and query resolution. However, our results showed that these secondary outcomes generally favored the no on-site group. All of the secondary outcomes were monitored centrally, and sites in both monitoring groups were provided with the same reports on a regular basis throughout the study. Anecdotally, we learned that some sites relied on on-site monitoring visits for assistance in completing data collection forms and resolving queries. Clinical staff at sites randomized to the no on-site group may have taken more initiative. For whatever reason, the no on-site group was clearly not inferior in the secondary outcomes considered.

This substudy has several limitations. The primary monitoring outcome included a component for START primary and serious events reported > 6 months from occurrence. This may be too long, since sites were encouraged to report clinical events after occurrence. However, follow-up visits in START were scheduled every 4 months; a single missed visit could result in a clinical event being identified more than 6 months after occurrence. Another limitation was that the substudy opened 29 months after the first participant was randomized into START. Thus, nearly 40% of sites randomized to the no on-site group had already had an on-site monitoring visit. A third limitation was that informed consent violations were not consistently interpreted by local and on-site monitors. This led to modification of that outcome component during the substudy, and our results suggest there was variation by coordinating center. Finally, we did not collect data to evaluate the cost to the statistical center associated with on-site monitoring, nor did we collect data to evaluate the cost of local monitoring to the sites or costs for the support of local and central monitoring by the statistical center.

Other studies have been designed to evaluate on-site monitoring strategies for clinical trials. The OPTIMON non-inferiority study compared two monitoring strategies – an intensive approach in which all data were on-site monitored vs. a risk-adapted approach in which the intensity of monitoring was relative to a study’s risk to research participants.³² In 22 studies involving 759 participants and 68 sites, OPTIMON found that the risk-adapted strategy was inferior as shown by a significantly higher proportion of participant charts with major nonconformities (deviations). However study accrual was lower than expected, differences between the two strategies were small, and neither of the monitoring strategies reached their expected quality objectives.

The ADAMON cluster randomized non-inferiority trial compared risk-adapted monitoring with extensive on-site monitoring at 156 sites participating in 11 clinical trials, and concluded that the benefit of extensive on-site monitoring compared to risk-based monitoring was small.³³

The TEMPER study evaluated targeted (“triggered”) monitoring using a matched-pair design. Sites in 3 trials were matched on the number of participants and time since opening. Based on central data review, a site with a high “trigger” score (indicating concern for data quality) was compared to a site with a low “trigger” score. Triggered monitoring did not significantly distinguish sites with monitoring findings.³⁴

In our study, each of the aforementioned prospective studies, and in a retrospective review of risk-based monitoring,³⁵ informed consent violations were one of the more frequently identified issues found through on-site monitoring. Based on our experience, careful thought is needed to define informed consent violations and describing how to identify them. Additional research on more cost-efficient ways to identify consent violations without on-site monitoring is necessary. A plan similar to one described by Journot et al³⁶ could be implemented to centrally review consents prior to enrollment, or immediately after enrollment.

In summary, this trial of on-site monitoring was nested in a publicly funded strategic trial, and conducted at sites with experience in HIV research by a well-established clinical trials group with strong local and central quality management programs. While central data monitoring is a typical function performed by statistical centers, the frequency and intensity of the central monitoring and the robust local monitoring program described were developed specifically for START. Therefore, our findings may not be generalizable in the absence of rigorous central monitoring or the ability to implement a strong local monitoring management program. Because there were relatively few monitoring outcomes that could have impacted START results, no findings of participants who were inadequately consented, and no findings of data alteration or fraud, the added value of on-site monitoring in START was minimal considering the cost.