Trust and transparency in clinical trials of medical devices

Abstract

Regulatory approval of high-risk cardiovascular devices is on the basis of clinical studies submitted with a premarket approval application. Failure to publish many of these studies in peer-reviewed literature, and major discrepancies between premarket approval submissions and those studies that are published, raise important questions for clinicians and other stakeholders.

What should the public expect from new medical devices approved for clinical use? The answer depends greatly on geography, because evidentiary requirements for marketing new medical technology varies substantially between the USA and the European Union.¹ In the USA, the FDA is charged with adjudicating whether sponsors of new high-risk devices have provided “reasonable assurance of safety and effectiveness”, typically on the basis of one or more clinical studies submitted as part of a premarket approval application.² A new study published in the British Medical Journal raises questions about the quality of the clinical science supporting new high-risk cardiovascular devices.³

Chang et al. reviewed 177 studies (112 of them ‘pivotal studies’) supporting 106 new cardiac device approvals by the FDA from 2000 to 2010, focusing on the highest-risk category of devices, such as coronary stents, and drawing data from the ‘summary of safety and effectiveness’ documents available on the FDA website.⁴ The investigators aimed to identify and compare subsequent peer-reviewed publications of the same clinical studies, and found that only 49% of all studies and 59% of pivotal studies were eventually published, corresponding to 60 devices with data available in peer-reviewed journals.³ In addition, important study features varied between the summary results provided in FDA documents and the eventual published science. For example, more than one-quarter of studies compared had discrepancies in enrolment numbers and precise definitions of end points, and small, but important, differences in demographic data were also seen. Taken together, Chang and colleagues argue for greater transparency in making clinical data supporting new devices available to the public through peer-reviewed sources.³ In addition, their analysis extends previous work emphasizing the need for clear and consistent end-point definitions and diligent reporting of study outcomes.⁵

Few would quibble with the need for transparency and rigorous science supporting new, high-risk technology. Trial registries such as ClinicalTrials.gov are intended to hold investigators and sponsors accountable in a public forum for adherence to scientifically sound methodology with explicit, a priori selection of safety and effectiveness end points, enrolment targets, and a prespecified statistical analysis plan.⁶ Although valid reasons might exist for adjusting study design features mid-study, these changes and their justification should also be made public and clear.

The process for regulatory review of a clinical study for approval of a high-risk cardiovascular device by the FDA begins with approval of the protocol for the study, which indeed is often designed in conjunction with FDA reviewers and statisticians. Adherence to this protocol is monitored throughout the study and amendments require FDA approval. Major discrepancies between the final approved protocol analysis plan and data submitted in the premarket approval application should be apparent before approval, and documented with justification in the review process.

Pivotal studies in the USA, particularly those sponsored by the manufacturers of new devices premarket, are explicitly designed to meet the regulatory requirements to demonstrate safety and effectiveness. Without question, these studies should include clear and consistent definitions of study end points, enrolment, and treatment protocols, as well as detailed plans for analysis of results. In addition, the study participant flow, including the initial power and sample size assumptions and detailed accounting of subject screening, enrolment, and follow-up, should be available for public review whenever possible. However, Chang and colleagues’ insistence on peer-review as the arbiter of scientific rigour and transparency merits consideration. This approach conflates the purpose of regulatory review with those of peer-review in several ways. First, sponsors and investigators, having completed the studies and earned marketing approval, lack any clear motivation to dedicate the time and effort towards moving through the peer-review process. The cycle of submission, rejection, revision, and eventual publication typically takes many months even for highly-motivated investigators. Second, however valuable a manuscript-length explication of new device data might be to the public, peer-reviewers and journal editorial boards themselves have their own biases. These might include unfairly viewing studies sponsored by industry as lower in quality,⁷ and of limited scientific value to the readership. Lastly, many of the nonpivotal studies, which might include simple case series or first-in-man feasibility work with very small numbers of patients, face particular challenges in meeting sufficient priority for publication.

Nevertheless, we agree with Chang et al. that the results of pivotal studies leading to regulatory approval of high-risk cardiovascular devices should generally be made available in the peer-reviewed literature. In many cases, these devices constitute practice-changing therapies, and the study results might provide higher-level evidence for guidelines committees, where such evidence is frequently lacking.⁸ The peer-reviewed study results are also necessary for practitioners to make informed clinical decisions for individual patients, rather than relying on meeting presentations and marketing data. Although these objectives are clearly separate from the role of regulatory review, it is essential that the trial methodology and results are concordant between the scientific publication and the FDA submission. To this end, the publication process must also verify that submitted results conform to a published protocol. In many cases, this objective can be facilitated by the earlier publication of a design and methodology paper (Figure 1).

Figure 1.

Reviewing clinical trial data. Regulatory review and peer-review of clinical trial results have different objectives, but final published results for either process should conform to the approved clinical trial protocol and specify major deviations and amendments. The algorithm proposed here allows for the FDA and journal editors to verify that published results from either pathway are transparent and concordant.

Not mentioned by Chang and colleagues, but of equal importance to public health when considering the duties of investigators, are the study participants themselves. On average, each of the studies evaluated involved >300 participants—individuals who provided their consent to contribute to scientific advancement. Alongside investigators’ many obligations to regulators, sponsors, and a critical clinical community lies their central responsibility to take the commitment of clinical trial participants seriously. Failure to conduct pivotal studies rigorously—including consistently defining and reporting end points and tracking patient outcomes—is not only a scientific error, but also a moral one. The trust and consent provided by research participants demands the highest possible standards to ensure that studies meet their aims.

Ultimately, whether or not final study results are peer-reviewed might be an incomplete metric for Chang and colleagues’ analytical target: the quality of the science supporting life-saving devices. What clinicians and the public can and must demand, however, is that study design is consistent, transparent, rigorous, and worthy of the risks borne by study participants. Chang et al. rightly shine a light on the need for regulators, sponsors, investigators, and journal editors to work together to ensure scientific integrity.