Protecting the Confidentiality of Interim Data: Addressing Current Challenges

Abstract

There is compelling evidence supporting the importance of maintaining confidentiality of interim data in clinical trials designed to reliably address the benefit-to-risk profile of interventions. While this is widely recognized, creative approaches are needed to achieve this in challenging settings where interim data are released for regulatory review and action, even though the trial would be continued to address its primary hypothesis. An illustration is the recently emerging setting of cardiovascular safety trials in type 2 diabetes mellitus. At the first stage of such trials, if large relative increases in cardiovascular major morbidity/mortality can be ruled out, data can be released solely for the purpose of allowing regulatory decision making about marketing approval. The trial then is continued in the post-marketing setting to address the primary hypothesis regarding whether smaller relative increases can be ruled out. Active rather than passive approaches are needed to protect the integrity of cardiovascular safety trials. Given the importance to trial integrity of maintaining confidentiality of interim data such as the estimated relative effect on cardiovascular risk, a Data Access Plan should be in place in these trials to ensure such data are not revealed to study participants and their caregivers, investigators involved in trial conduct, the sponsor's management team, and the public, until trial completion. A Performance Standards Document also should be developed to pre-specify targeted and minimally acceptable levels for recruitment rate, best real world achievable adherence, avoidance of cross-ins, and retention rate. This document should specify creative approaches for achieving these targets, oversight procedures during trial conduct to monitor performance levels, and actions to be taken if emerging data indicate minimally acceptable levels are not being reached. In settings where meaningful breaches in confidentiality have occurred, such oversight allows adverse effects on trial integrity to be detected earlier and more effectively addressed.

Introduction: Challenges to trial integrity when data from ongoing trials are used in regulatory decision-making

Extensive research and experience in the conduct of clinical trials provide compelling evidence in support of the importance of maintaining confidentiality of interim data in trials designed to reliably address the benefit-to-risk profile of interventions. For this reason, it is standard practice in the conduct of such trials to ensure that emerging data on efficacy and safety are not revealed to the trial sponsor, study investigators, caregivers, trial participants or others involved in trial conduct, or to the broader public. In many instances, a Data Monitoring Committee (DMC) is established to safeguard the interests of study subjects and to enhance the integrity and credibility of these trials. These goals are accomplished in part by ensuring that the DMC and the statistical center's ‘presenting’ or ‘independent’ statistician have sole access to emerging efficacy and safety data.^1-3 This reduces the risk of prejudgment that can adversely impact factors such as recruitment, treatment adherence, and retention.

While the importance of maintaining confidentiality of emerging data is widely recognized, creative approaches are needed to achieve this in some particularly challenging settings. One such setting arises when interim data are released for regulatory review and action, even though the trial would be continued to address its primary hypothesis. A specific illustration is the setting of regulatory ‘conditional’ or ‘accelerated’ approval, when early trial results provide definitive evidence about treatment effect on a shorter term outcome measure, such as a biomarker that is considered an acceptable basis for such early approval decisions, while the regulatory authority wishes to base ‘full’ regulatory approval decisions on later trial results providing definitive evidence about treatment effect on direct measures of major morbidity or mortality. A challenge is identifying and successfully implementing approaches that enable this early regulatory decision making while properly maintaining the blinding of ongoing trial data regarding the effects on the longer term outcome measures.

The illustration on which we will focus in this article is the recently emerging setting in type 2 diabetes mellitus (T2DM). For new products, cardiovascular (CV) safety trials are designed in essentially two stages. At the completion of the first stage, if large relative increases in morbidity and mortality can be ruled out, the product can be approved for marketing. However, the sponsor is required to continue the trial, using pre- and post-marketing study results to address the trial's primary hypothesis regarding whether smaller relative increases can be ruled out.

Our purpose in this article is to consider in greater depth the challenging issues arising in this setting of cardiovascular safety trials in T2DM. We discuss the importance of identifying and implementing approaches that reduce the risk of breaches in confidentiality when interim data informative about the trial's primary hypothesis are released for regulatory action even though the trial continues to collect data to address that hypothesis and others of interest. We discuss the information that could be publicly released during the conduct of a clinical trial without compromising the essence of confidentiality, the importance of establishing performance standards for quality of trial conduct, and the value of defining plans for ensuring proper restrictions for data access. Finally, we consider some alternative clinical development strategies that have been considered in this setting of cardiovascular safety trials.

The importance of maintaining confidentiality of interim data

The mission of a DMC is to safeguard the interests of study participants and to preserve the integrity and credibility of the clinical trial, enabling timely and reliable insights to the broader clinical community. A principle of fundamental importance in achieving this mission is that the DMC should have sole access to interim results on safety and efficacy of interventions.^1-5 This principle was emphasized by the DAMOCLES (‘Data Monitoring Committees: Lessons, Ethics, Statistics Study Group’) project, commissioned by the United Kingdom NHS Health Technology Assessment Program to investigate existing processes of monitoring accumulating data and to identify ways of improving the DMC process. DAMOCLES concluded, “There is near unanimity that the interim data and the deliberations of the DMC should be absolutely confidential...Breaches of confidentiality are to be treated extremely seriously”.⁴

Important evidence considered by DAMOCLES included a published review of 20 major clinical trials, 10 in each of two US cancer cooperative groups that were discordant in whether interim data were regularly released to investigators, but were concordant in key features such as calendar time, disease type and disease stage.⁵ In 10 trials conducted by the group that had DMCs in place and maintained confidentiality of interim results, the authors of the published review reported, “No studies were inappropriately terminated early. No study had early results published that were not confirmed at the time of final analysis. No final results were equivocal.” In contrast, in the 10 trials conducted by the group that did not have DMCs in place and that reported its results to investigators and others every 6 months, the rate of accrual declined in five trials, two trials closed early with inadequate data, and two others reported positive interim results that were not convincingly positive by the time of study closure. The authors concluded, “it is a reasonable supposition that routine reporting of results contributed to the decline in accrual in several studies (as investigators drew their own conclusions and treated accordingly) and to the early termination of others. No restrictions in reporting must also be part of the explanation of the publication of early (and overly optimistic) positive results in some studies.”

The experience from the Didanosine (ddI) versus Zalcitabine (ddC) trial conducted by the Community Programs for Clinical Research on AIDS⁶ also provides compelling evidence regarding the importance of maintaining confidentiality of interim data, (Figure 1). This non-inferiority trial was designed to assess whether it could be ruled out that ddC had a 1.25 relative increase or greater in the rate of death or symptomatic AIDS-defining events, relative to the standard-of-care ddI regimen. The evidence at the final analysis in August, 1992 was favorable for ddC regarding non-inferiority on the primary composite endpoint of death or symptomatic AIDS events and even suggested that survival results favored ddC. In stark contrast, the data at the interim analysis in August, 1991 were quite negative for ddC; ddC had twice the ddI rate of symptomatic AIDS events or deaths, (nominal p=0.009), and twice as many deaths.⁷ Further, these inferior clinical effects of ddC appeared to be biologically explained by nominally significant inferior effects of ddC on the biomarker endpoint, CD4 cell count.⁷ At that time, the HIV/AIDS community was strongly advocating for more reliance on surrogate endpoints such as CD4, advocacy that considerably influenced the establishment of the ‘accelerated approval’ regulatory pathway by FDA; CD4 was the biomarker used in the initial accelerated approvals granted in the early 1990's. Given the strength of the early unfavorable evidence about ddC's effects on clinical and biomarker endpoints, it is apparent that if the interim results had been disseminated, the trial would not have been able to be completed, and we would not have obtained the contradictory but more reliable final results on the relative efficacy of the two agents.⁷

Figure #1.

The Commuity Programs for Clinical Research on AIDS compared zalcitabine (ddC) against the control, didanosine (ddI). The number of events on ddC and ddI, respectively are provided in parentheses under the calendar date of each of the 4 analyses. At each analysis, the estimated ddC to ddI hazard ratio for the rate of progression to symptomatic AIDS or death is represented by asterisks, the square brackets represent nominal 95% confidence intervals, and the orange parentheses represent the O'Brien-Fleming repeated confidence intervals properly adjusting for the group sequential analyses.

Many authors have presented additional evidence that maintaining confidentiality of interim data enhances the integrity of definitive clinical trials,^1-5,7 reducing the risks for pre-judgment that could lead to declining enrollment rates, altered adherence, early release of misleading results or inability to complete trials. Another important benefit achieved by preventing breaches in confidentiality is that sponsors maintain some flexibility in making changes to trial design that may be suggested by external data. In an ongoing trial, design changes such as revising the targeted number of primary endpoints can only be made by those not having direct or indirect knowledge of the trial's emerging evidence about treatment effect. There is broad consensus about the importance of maintaining confidentiality of interim data.^8-11

Interim evidence that can be released while in essence preserving confidentiality

We begin the consideration of the type of information that could be released during ongoing CV Safety Trials by looking at this issue in the more familiar setting of superiority trials. Figure 2 provides the O’Brien-Fleming monitoring boundary¹² guiding early termination for benefit (in blue) in the Normal Hematocrit Trial designed to assess whether an erythropoietin stimulating agent (ESA) dosed to achieve ‘normal’ hematocrit levels provides a 25% reduction (i.e., an experimental to control hazard ratio 0.75) in the rate of death or myocardial infarction (death/MI) relative to regimen based on using standard lower dose levels.¹³ The figure also provides the symmetric monitoring boundary that could justify early termination if initial results are sufficiently unfavorable to rule out that level of benefit (in black). While these boundaries allow analyses at any point in time,¹⁴ Figure 2 illustrates the setting in which interim analyses occur after each quarter of the trial's total 742 events have occurred. At trial completion when 742 death/MI events have occurred, statistically significant evidence of benefit is achieved if the estimated hazard ratio is at least as favorable as 0.866.

Figure #2.

The Normal Hematocrit Trial was designed to evaluate the rates of death/MI events on a high dose relative to a standard dose regimen of an erythropoietin stimulating agent. The figure presents a monitoring boundary for superiority (in blue), ruling out the null hypothesis of a 1.00 hazard ratio, and a monitoring boundary to rule out a 0.75 hazard ratio (in black), when analyses are conducted after equal increments in the occurrence of the trial's planned 742 death/MI events. L denotes the number of events available at each analysis. The orange asterisks illustrate estimates for the hazard ratio that could be obtained at the mid-point of the trial.

Consider, for illustration, an interim analysis conducted at the midpoint of the trial when data with 371 death/MI events are available. With the monitoring boundaries displayed in Figure 2, trial continuation would be suggested by an estimated hazard ratio anywhere in the range from the null hypothesis of 1.0 to the highly powered alternative hypothesis of 0.75. Hence, little information about the likelihood the trial will yield positive results is provided by knowledge that the DMC recommended trial continuation after an interim analysis guided by these boundaries. On the other hand, if the actual point estimate of the hazard ratio were revealed, there would be considerable risk to trial integrity. For example, if the point estimate for the hazard ratio was a favorable 0.80, many would draw the conclusion that the high dose was superior. Such preliminary (but unreliable) estimates could create difficulties in continuing the trial participants on their assigned regimens, and/or continuing to recruit to the trial.

We now consider CV safety trials conducted in the T2DM setting. In 2008, the FDA released a guidance document indicating that large scale CV safety trials should be conducted for new agents in T2DM.¹⁵ In this guidance, the threshold for an unacceptable risk is specified to be a relative 30% increase in the rate of CV events. However, marketing approval could be granted if interim data rule out a ‘less stringent’ 80% increase.

A frequently applied approach to address these regulatory requirements has been to perform a single CV safety trial. A pre-specified analysis after 122 events is conducted to determine whether a 1.8 hazard ratio can be ruled out for the composite endpoint of ‘CV death, stroke, or MI’. The trial would then be continued until approximately 610 events had been observed, in order to determine whether a 1.3 hazard ratio for the composite endpoint could be ruled out. These issues recently have been discussed in detail.¹⁶

As displayed in Figure 3, at the pre-specified final analysis, the 1.3 hazard ratio would successfully be ruled out if the point estimate of the hazard ratio is more favorable than 1.10, and superiority would be concluded if that estimate is more favorable than 0.85. Early termination for benefit or for harm would occur if the monitoring boundary for superiority or for proven harm is crossed.

Figure #3.

In the type 2 diabetes mellitus setting, a 610-event cardiovascular safety trial may be conducted to determine whether one can rule out an experimental to control hazard ratio of 1.3 for the composite endpoint, ‘CV death, stroke, myocardial infarction’. At that final analysis, if the estimated experimental to control hazard ratio is less than 1.10, then the ‘1.3 margin’ is ruled out, if it is less than 0.85, then superiority is established, and if it is greater than 1.18, then harm is established. Early termination for superiority occurs if the blue boundary is crossed and for harm occurs if the black boundary is crossed. L denotes the number of events available at each interim analysis. Data from the 122-event analysis might be used to support early regulatory decisions about marketing approval if an experimental to control hazard ratio of 1.8 for the composite endpoint can be ruled out, requiring an estimate less than 1.26. Orange asterisks illustrate hazard ratio estimates that could be obtained at that analysis. Boundaries that do not trigger decisions for trial termination are denoted by dashed lines.

At the initial analysis, when 122 events are available, the 1.8 CV safety margin would be successfully ruled out if the point estimate of the hazard ratio is more favorable than 1.26. If that is achieved, the interim data could be released, but solely for the purpose of informing regulatory authorities in decision making regarding marketing approval. Suppose the analysis regarding the 1.8 margin is conducted at only this one point in time. Those who know only whether the 1.8 margin was ruled out would remain in equipoise. On the other hand, as in the superiority setting, there is considerable risk for pre-judgment that could meaningfully compromise trial integrity if the point estimate of the HR were revealed. For example, revealing that the estimated hazard ratio is 0.80 could lead to pre-judgments favoring the experimental intervention in a manner that poses the risk that the control patients already randomized would cross-in to the experimental drug, and that the rate of enrollment into the CV safety trial would be meaningfully reduced, leading to an unacceptable delay in obtaining important safety information for an intervention now being widely marketed. Furthermore, revealing that the estimated hazard ratio is 1.20 could lead to prejudgments favoring the control intervention, posing the risk that the experimental patients already randomized would become non-adherent, and again potentially adversely affecting the rate of enrollment into the CV safety trial. All of these issues—slow enrollment, lack of adherence by the patients on the experimental arm and cross-ins by the patients on the control arm—seriously adversely impact the integrity of the CV safety trial regarding its primary mission of addressing the 1.3 hazard ratio margin, (and addressing the 1.0 margin since the potential to establish superiority also is relevant in many of these settings.)

The performance standards document and the data access plan

Active rather than passive approaches are needed to ensure the integrity and credibility of clinical trials. This is especially important in settings such as long term CV safety trials designed to determine whether important differences between regimens can be ruled out, since noise in trial conduct meaningfully diminishes sensitivity to detecting true differences. Hence, in addition to the study protocol, the Statistical Analysis Plan and the DMC Charter, it is important to develop a Performance Standards Document that pre-specifies targeted and minimally acceptable levels of performance, creative approaches for achieving these targets, oversight procedures during trial conduct to monitor performance levels, and actions to be taken if emerging data should indicate minimally acceptable levels are not being reached.^1,16-18 Performance standards should be specified regarding achieving timely enrollment, having populations at sufficiently high risk or where it is most important to rule out safety concerns, achieving an adequately high event rate, having ‘best real world achievable’ adherence in the experimental arm, avoiding cross-ins in the control arm, maximizing retention, and achieving timely data capture and adjudication of potential events during the trial.

Figure 4 illustrates how releasing the actual point estimate of the hazard ratio from interim data in a CV safety trial would be detrimental to meeting performance standards. Hence, in ongoing trials, such as CV safety trials, where the DMC might agree to the release of interim data solely to allow consideration of marketing approval by regulatory authorities, a Data Access Plan should be in place. This document, to be developed jointly by the sponsor, principal investigator and DMC prior to data release, should emphasize the principle that access to these interim data should be restricted to those on an ‘unblinded team’ who must interact with regulators regarding marketing approval. Access should be solely for that purpose and not to address separate business considerations such as fund raising, securing partnerships or building manufacturing facilities. These restrictions in access should be honored until after the trial has been completed and the data base locked. Unblinded team members should be drawn from those within the sponsor whose roles are inherently internal, such as regulatory and science-based personnel, programmers, and personnel in clinical safety oversight and quality control, and should not include those within the sponsor whose roles are inherently external, such as corporate leadership, the board of directors, and those involved in marketing, partnering discussions, or relationships with investors.

Figure #4.

Some of the important performance standards measures in CV safety trials are provided. Illustrations are given for potential adverse consequences when there is direct or indirect public disclosure of the estimate from interim data for the experimental to control hazard ratio for the CV safety endpoint, such as when interim data yield a favorable 0.8 or an unfavorable 1.2 estimate.

In addition to defining criteria for membership on the unblinded team, the Data Access Plan should provide procedures for training unblinded team members about the need to maintain confidentiality. It should describe firewalls between unblinded and blinded personnel, including facilities and technologies that will be in place to ensure confidentiality, and should indicate that each member of the unblinded team should sign a formal confidentiality agreement before being provided access to interim data.

Once members of the ‘unblinded team’ have received access to confidential information, such as the point estimate and confidence interval for the hazard ratio estimate, they should not participate in the subsequent conduct and management of the study. Furthermore, the regulatory authorities and members of the ‘unblinded’ team should not be provided access to primary endpoint data that emerge after the pre-specified cutoff for the data upon which regulatory action is to be based, (e.g., data that emerge after the122 event analysis in Figure 3).

It is important for regulatory authorities who receive access to interim data from ongoing CV trials ensure that confidentiality is maintained. If external advisors are engaged by regulators during their review, such as in the formal advisory committee process, these advisors should be required to maintain confidentiality and their interactions with regulatory authorities regarding interim data from the CV safety data trial should be held in non-public sessions. Regulatory communications of their summary basis for approval decisions should confirm only that the data from the CV safety trial rule out the pre-specified ‘less stringent’ threshold for excess risk, (e.g., an 80% increase in CV risk in the T2DM setting), thereby retaining equipoise in the public regarding the likelihood of ruling out the threshold for a clinically unacceptable increase, (e.g., a 30% increase).

Alternative clinical development strategies to address CV safety in T2DM

Separate CV safety trials could be conducted to address the 1.8 and 1.3 hazard ratio margins in the T2DM setting. However, relative to the single trial approach, this could be less efficient, would delay important insights about CV safety during the early post marketing phase when there already could be widespread use of the intervention, and could result in the latter ‘1.3 trial’ having compromised integrity if confidentiality of results from the initial ‘1.8 trial’ is not maintained until both trials are completed.

A special case of the scenario of separate trials arises when the ‘1.8 trial’ is based on a meta-analysis of data from Phase 2 and 3 efficacy trials. Even though these trials provide important insights about effects on efficacy outcomes, such an approach might be particularly problematic regarding assessment of CV safety. Efficacy trials have uncertain generalizability about CV safety since they usually enroll low risk patients who often are followed for only 6-12 months post randomization. Furthermore, they often are not conducted according to adequately rigorous performance standards regarding achieving high levels of adherence and retention, low levels of cross-in, and avoidance of excess exposure of control group patients to cardio-toxic agents. They may have uneven quality of capture and adjudication of cardiovascular events. Furthermore, there usually is inadequate pre-specification, prior to conduct of these Phase 2 and 3 trials, about whether and how much data from each of these trials will be included in the data set used to address the 1.8 margin.

Given these concerns with having separate ‘1.8’ and ‘1.3’ trials, consider again the strategy of conducting a single CV safety trial that addresses both the 1.8 margin at an interim analysis and the 1.3 margin at later analyses. An approach that deserves some consideration would have the DMC release the interim data from the CV safety trial exclusively to an entity external to the sponsor. If that entity assumes the responsibility of submission of these interim data to regulatory authorities, this could nearly if not fully eliminate risks for sponsor-induced breaches in confidentiality.

Conclusion

The FDA's 2008 guidance document for development of new agents in T2DM requires the conduct of definitive CV safety trials. The document empowers sponsors to provide the public more timely access to promising interventions by indicating regulatory approval could be granted when only interim rather than final data are available from those definitive safety trials. These interim data need to reliably address whether large increases in CV morbidity and mortality events, such as a 1.8 hazard ratio, can be ruled out.

These CV safety trials then are continued in the post-marketing setting to address the guidance document's requirement for timely and reliable evidence regarding whether smaller but clinically relevant increases in CV risk, such as a 1.3 hazard ratio, can be ruled out. Active rather than passive approaches are needed to protect the integrity of this longer term evidence. Given the importance to trial integrity of maintaining confidentiality of interim data such as the estimated relative effect on CV risk, a Data Access Plan should be in place in these trials to ensure that access to interim data is reliably restricted to those who are essential for interactions with regulators regarding marketing approval, where such access is solely for that purpose. A Performance Standards Document also should be developed, pre-specifying targeted and minimally acceptable levels for key measures such as recruitment rate, best real world achievable adherence, avoidance of cross-ins, and retention rate. It also should specify oversight procedures during trial conduct to monitor performance levels, and actions to be taken if emerging data should indicate minimally acceptable levels are not being reached. If meaningful breaches in confidentiality are occurring, these oversight procedures enable earlier detection of their effects on measures of quality of trial conduct, and the potential to take timely actions that could reduce their negative impact on trial integrity.

Leadership from regulatory authorities is of considerable importance, in part through the guidance they provide about procedures for maintaining confidentiality of interim data from an ongoing CV safety trial, and in part by ensuring they themselves are implementing procedures that properly balance the importance of achieving transparency in the regulatory process with the need to protect the integrity of the ongoing clinical trial.

In CV safety trials and in other settings where, by trial design, interim data may be released for regulatory review and action even though the trial would be continued to address its primary hypothesis, it is imperative to maximize confidentiality of such data that are directly informative about that hypothesis that remains to be assessed at trial completion.