Ethics and Methods in Surgical Trials

Abstract

This paper focuses on invasive therapeutic procedures, defined as procedures requiring the introduction of hands, instruments, or devices into the body via incisions or punctures of the skin or mucous membranes performed with the intent of changing the natural history of a human disease or condition for the better. Ethical and methodological concerns have been expressed about studies designed to evaluate the effects of invasive therapeutic procedures. Can such studies meet the same standards demanded of those, for example, evaluating pharmaceutical agents? In this paper, we describe a research project aimed at examining the interplay and sometimes apparent conflict between ethical standards for human research and standards for methodological rigor in trials of invasive procedures. We discuss how we plan to develop a set of consensus standards that, if met, would result in substantial and much-needed improvements in the methodological and ethical quality of such trials.

Introduction

Methodological standards for trials evaluating the efficacy of therapeutics in humans have become increasingly stringent over the past 60 years. Key developments have included the introduction of random allocation to treatment arm,[1] greater understanding of the need for blinding or masking,[2] and awareness of how sample size influences confidence that a treatment effect is present or absent. [3] In addition, uniform standards now exist for elements of reports of clinical trials,[4] and mandatory registration of clinical trials has become a prerequisite for the publication of their results.[5] Taken together, these methodological improvements reduce the influence of factors that bias the estimate of a treatment's effect, for example, selection bias (random allocation); expectancy bias (blinding), type I or type II errors (sample size), various threats to internal validity (standards for reporting and conduct); and publication bias, which can inflate estimates of treatment benefit to the extent that trials with negative results are under-reported (trial registration).

At the same time, ethical standards for the design and conduct of clinical trials have also become more developed.[6] The 1949 Nuremberg Code was followed by the 1964 Declaration of Helsinki, the 1979 Belmont Report, guidelines issued thereafter by the Council for International Organizations of Medical Sciences[7] , national laws such as the U.S. Common Rule,[8] and multi-national treaties such as guidelines issued by the Council of Europe.[9]

As these advances have occurred, a curious double standard has emerged for prescription drugs and invasive therapeutic procedures. Before their release for use in the public, new prescription drugs must be of proven efficacy and safety, demonstrated in randomized controlled trials, under regulations enforced by entities such as the US Food and Drug Administration or the EU European Medicines Agency or directives issued by bodies such as the International Conference on Harmonisation.[10] Conversely, new invasive therapeutic procedures are often launched and widely disseminated on the basis of clinical theories emerging from laboratory research, clinicopathological correlations, and weak human-studies designs from which no causal inferences should be made, with no regulatory body in charge of pre-dissemination oversight. (Medical devices are regulated but not the procedures in which they are used.) When randomized trials of an invasive procedure are conducted, it is often after the procedure has been widely used—in some cases in hundreds of thousands of patients—and doubts have emerged about its utility.

The relative dearth of randomized trials in surgery can be partly traced to a concern that the very nature of invasive therapeutic procedures precludes their evaluation using rigorous designs. This argument is based on the assumption that methods and ethics collide in invasive therapeutic interventions, such that it is impossible to design methodologically sound trials that adequately ensure the ethical treatment of human subjects.

In this paper, we make the case that there is a critical need for timely and rigorous randomized trials of invasive therapeutic procedures, mention briefly the choices of trial design that may challenge ethical standards, and describe current research in which we are developing a set of consensus standards that would have to be met to initiate, design, conduct, and report clinical trials of invasive procedures that are methodologically and ethically sound.

The Status Quo of Dissemination First, Evidence Later: the Example of Operative Interventions to Prevent Stroke

The story of carotid artery surgery for stroke prevention illustrates how procedures that look promising quickly disseminate into wide use, only later to be found, on the basis of randomized controlled trials, to be less effective or more harmful than originally believed. The first patients treated with carotid endarterectomy were described in the early 1950's.[11] Over a million Americans underwent the procedure during the subsequent decades, [12] before randomized trials established that the procedure's risks outweighed the likelihood of benefit in certain subgroups.[13-15] After the trial results were reported, the proportion of patients undergoing the procedure for inappropriate indications—no chance of benefit and/or excessive chance of harm—fell from 32% to less than 9%.[16]

Another operation for stroke prevention, extracranial-intracranial arterial bypass, was launched on the basis of a case report and performed widely for 15 years before the first randomized trial to test it showed that the operation increased, not decreased, the incidence of fatal and nonfatal strokes. [17] Few of these procedures are now performed in the US.[18]

Carotid artery stenting and angioplasty for stroke prevention were first described in the 1990's and diffused rapidly. Unlike endarterectomy, stenting does not require an incision in the neck to open the carotid artery but rather involves the percutaneous passage of a vascular catheter to implant a stent to prop open the blockage. Again, there was a long lag between the initial case reports and the first randomized trial. The results of the first stenting trial appeared in 2001,[19] 10−15 years after the procedure began to be performed in patients. The procedure's superiority to carotid endarterectomy has not yet been confirmed.[20-22]

The carotid artery surgery stories confirm concerns that the status quo for dissemination of invasive therapeutic procedures is, in itself, ethically objectionable, [23] because later trials so often show that the intervention is harmful or of no benefit, at least for some patients.

But what is also true is that the status quo of dissemination first, evidence later is responsible for most of the conflicts between ethics and methods that are encountered in designing, fielding, and conducting trials of the benefits and harms of an invasive procedure. Belief systems are allowed to develop among doctors and patients that make it difficult to rationalize the performance of a trial, or, even if a trial can be rationalized, that compromise key bias controls in trial design and conduct (e.g., choice of comparator, random allocation, and blinding). Surgeons’ beliefs that the benefits of an unvalidated procedure are self-evident or that random allocation is objectionable are rooted in the concept of equipoise, a state of uncertainty or disagreement within the clinical community about the efficacy of a particular intervention.[24-26] Some surgeons assert that they cannot ethically participate in or refer their patients to a randomized trial because they know what is best for their patients. Even if the trial is run, they object that random allocation will ensure that some of their patients will not receive it. For example, the Stenting and Angioplasty with Protection in Patients at High Risk for Endarterectomy (SAPPHIRE) trial had to be terminated early because of an abrupt drop in participant accrual rates coinciding with the opening of nonrandomized stent registries.[27] By choosing to participate in the registry, doctors and patients could avoid randomization and the possibility of being allocated to what they believed was inferior treatment. Accordingly, the benefits and risks of carotid stenting compared with endarterectomy are still uncertain,[20-22] a fact that may or may not be communicated to patients deciding whether or not to undergo the procedure.

“Knowing what's best” is a special problem when the comparator is a non-invasive therapy and blinding is impossible. When an individual with a strong preference about operative intervention is randomly allocated to medical treatment, or vice versa, that person may elect to forego the assigned treatment and obtain their preferred treatment by leaving the study and obtaining it outside the trial, or staying in the trial but “crossing over” to obtain it. Crossing over, in effect, reveals that many enrollees—and their surgeons—enter trials strongly preferring one arm over the other. In a recent trial of surgical vs. non-operative treatment for lumbar disk herniation,[28] within two years of being randomized, only 60% of those randomized to surgery had undergone surgery, while nearly half (45%) of those randomized to non-operative therapy had undergone surgery. From a statistical standpoint, when losses to follow-up or crossovers exceed a certain threshold it becomes impossible to draw definitive conclusions about the efficacy, or lack thereof, of the intervention being tested. Dropping out and crossing over represent deliberate selection of a treatment and thus undermine the value of random allocation in reducing bias in the estimate of treatment effect.

Is it Possible to Design and Conduct Ethically Sound as well as Methodologically Rigorous Randomized Trials of Invasive Therapeutic Procedures?

As the carotid artery surgery stories show, a major problem with invasive therapeutic procedures is that if and when they undergo rigorous tests of efficacy it is well after they have been widely used—wide use that fails to inform the scientific evidence base. This paradigm seems increasingly troublesome from an ethical standpoint and deserves scrutiny and change.

The question of the timing of trials aside, an urgent need exists to determine whether invasive therapeutic procedures can in fact be evaluated in trials that are ethically sound as well as methodologically rigorous. The need is greater than ever: the number and type of invasive procedures that can be, and are being, performed is exploding. Continued refinements in fiberoptics, the increasing use of robotics in procedures, advances in minimally invasive techniques, and rapid improvements in device technology and manufacturing are driving an unprecedented proliferation of new procedures. At the same time, more and more clinicians, not just classically trained surgeons, can perform invasive procedures, also contributing to the proliferation of interventions.

One explanation for the weak evidence base for most invasive therapeutic procedures is that many feel that it is impossible to test the benefits and harms of such procedures in randomized trials that are ethically sound and at the same time methodologically rigorous. The remainder of this paper will be devoted to describing a study during which we are examining the extent to which medical ethics and research methods actually do collide in evaluating invasive therapeutic procedures. The study is being funded by the US National Institutes of Health.

Brief Overview of the Study: Ethical and Methodological Standards for Trials of Invasive Therapeutic Procedures

In brief, the study is addressing the following questions: what are the ethical and methodological standards for clinical trials of therapeutic interventions (phase 1 of the study)? (Phase 1 covers standards for trials of any sort of therapeutic intervention in humans and is not restricted to invasive therapies.) Can trials of invasive therapeutic interventions meet these standards? If not, why not—at what junctures do ethics and methods appear to collide (phase 2)? Given what was learned from earlier phases of the study, what are reasonable, contemporary ethical and methodological standards for clinical trials of invasive therapeutic procedures (phase 3)? The assumption underlying the study is that an unavoidable interplay exists between the methodological quality and the ethical quality of a trial: “...scientifically unsound research on human subjects is unethical in that it exposes research subjects to risks without possible benefit...”[7] We are nearing completion of phase 1 of the study.

At the outset it is necessary to specify what is meant by the “methodological quality” and “ethical quality” of a clinical trial, because ethical and methodological standards can be said to operationalize these abstract concepts. We define the methodological quality of a trial as the extent to which its initiation, design, conduct, analysis and reporting minimizes or avoids biases in its estimates of the benefits and harms of the treatment it is evaluating.[29] The ethical quality of a trial is the extent to which trial initiation, design, conduct, analysis and reporting protects the moral commitments to respect for persons (treating individuals as autonomous agents and protecting persons with diminished capacity), beneficence (minimizing harms and maximizing benefits), and justice (fairness in the distribution of the benefits and burdens of research).[30] These definitions indicate that there are four key phases of a trial: initiation, design, conduct, and analysis and reporting, each with a particular set of ethical and methodological standards. Accordingly, building on other sources such as the Consolidated Standards for Reporting Trials [4] we developed and refined a four-part taxonomy with multiple subsections to use as the organizing framework for standards derived from source documents. Table 1 provides some example elements of the taxonomy. In the taxonomy ethical and methodological standards are fully integrated.

Table 1.

Example Elements of Taxonomy for Ethical & Methodological Standards

1.		Standards for Initiating a Clinical Trial (this section has 28 distinct subsections)
	1.1	Standards for research vs. innovative treatment
	1.2	Standards related to scientific background
	1.3	Standards related to social background (2 example standards) CIOM2345 Committees in the host country have a special responsibility to determine whether the objectives of the research are responsive to the health needs and priorities of that country. EMEA6870 There must be assurance that the scientific knowledge developed via research in a population or community with limited resources will be used for the benefit of the population.
	1.4	General standards related to protocol development and content
	1.5	Standards related to independent review and approval of protocol before trial commencement
	1.6	Standards related to registering the trial before it commences
	1.7	Standards related to choice of trial staff and centers
	Etc.
2.		Design Standards for a Clinical Trial (45 subsections)
	2.1	Standards related to hypothesis to be studied
	2.2	Standards related to choice of design and specification of intervention
	2.3	Standards related to specification of endpoints
	2.4	Standards related to choice of subjects
	2.5	Standards related to consent
	Etc.
3.		Standards for Conducting a Clinical Trial (15 subsections)
	3.1	Standards related to compliance with protocol and norms of conduct
	Etc.
	3.6	Standards related to data capture and processing (2 example standards) FDA0157 Access to the data at the clinical site should be restricted through the system's software with its required log-on, security procedures, and audit trail. FDA0135 Records should be backed up regularly in a way that would prevent a catastrophic loss and ensure the quality and integrity of the data.
	Etc.
4.		Standards for Analyzing and Reporting Trial Results (16 subsections)
	4.1	Standards for statistical analysis and reporting of data (3 example standards) ICH3940 Statisticians or other staff involved in unblinded interim analysis should not participate in the blind review. ICMJE2156 Researchers should not enter into agreements that interfere with their ability to analyze the data independently. ICH6798- Decisions concerning the analysis set should be guided by the avoidance of inflation of type I error.
	4.2	Standards for generalizing results and interpreting results in light of other data
	4.3	Standards for reporting results (2 example standards) CNSRT2367 Although P values may be provided in addition to confidence intervals, results should not be reported solely as P values. ICH8976 Any extreme or opposite results among centers should be noted, considering such possibilities as differences in study conduct, patient characteristics, or clinical settings.
	Etc.

Phase 1 of the Study: What are the Ethical and Methodological Standards for Clinical Trials of Therapeutic Interventions?

We reasoned that the most well-developed standards would come from countries and regions of the world that have a large number of active clinical trials, and that the best sources for consensus standards for trials of therapeutic interventions would be directives issued by governmental bodies funding and/or overseeing human research and medical product approval, and guidelines issued by internationally-recognized professional societies. We selected the source countries by rank-ordering nations according to the number of active clinical trials listed on clinicaltrials.gov as of July 24, 2008 and including nations with ≥700 registered trials. To this list of nations we added relevant transnational alliances and global organizations. We identified 31 nations, 26 transnational/regional alliances, and 29 global organizations as potential sources of ethical and methodological standards.

To locate the source documents for the standards, research assistants conducted internet-based searches for English-language documents from the target standard-setting bodies as well as other sources (e.g., the International Compilation of Human Subject Research Protections of the Office of Human Research Protections, U.S. Department of Health and Human Services [31]). As of the search closure date of November 14, 2008, the national searches yielded 887 directives (350 from the US alone) and the transnational and global searches yielded 157 additional unique source documents, for a total of 1,044 source documents.

We designated documents as “core” or “non-core” based on their international and/or regional influence and likelihood of extensive inclusion of number and type of standards, and then added documents to insure that the core set includes a balance of ethical and scientific topics and will populate all the subsections of the taxonomy of standards. Forty-eight documents were designated as “core” (see Table 2 for examples). Each core document was reviewed independently by two research assistants, who extracted the individual standards it contains and placed each standard into the taxonomy; the pair then met to identify differences in extraction and classification. The full project team met to resolve those differences. Over 9,000 non-compound standards were extracted from the 48 core documents.

Table 2.

Examples of core documents used as sources for ethical and methodological standards

	US Sources	International Sources
Predominantly ethical standards	• Code of Federal Regulations (CFR) Title 45 Part 46 – Protection of Human Subjects (US Federal Common Rule)	• International Ethical Guidelines for Biomedical Research Involving Human Subjects
	• Office of Human Research Protections Guidebook: Special Classes of Subjects	• International Council on Harmonisation (ICH) E6: Good Clinical Practice, Consolidated Guidelines
	• Office of Human Subjects Research: Continuing Review of Research Involving Human Subjects	• World Medical Association: Declaration of Helsinki
Predominantly methodological standards	• Guidance for Institutional Review Boards and Clinical Investigators: Cooperative Research	• ICH E9: Statistical Principles for Clinical Trials
	• 21 CFR 860: Device Trials	• ICH E10: Choice of Control Group and Related Issues in Clinical Trials
		• European Medicines Agency: Data Monitoring Committees

At present we are reviewing and extracting standards from all of the non-core documents pertaining to invasive therapeutic procedures or devices as well as 5% random sample of the remainder to ensure we have identified and included all ethical and methodological standards. We have also begun the process of reducing the number of standards in the set to a manageable number by identifying the key construct to which each standard pertains, eliminating redundant standards, merging similar ones, and partitioning out standards that pertain to largely administrative functions, e.g., standards that govern administrative aspects of ethics review boards.

Phase 2 of the Study: Can Trials of Invasive Therapeutic Interventions Meet These Standards?

With phase 2 we will turn our attention specifically to trials of invasive therapies (surgery and minimally invasive procedures). Phase 2 will be devoted to analyzing the extent to which it is possible for trials of invasive therapeutic procedures to meet normative ethical and methodological standards. To do this, we will apply the merged set of standards from phase 1 to 30−40 influential surgical trials whose findings were published in the last ten years. At least one trial will be selected for each of the 36 common clinical conditions (using the medical subject headings from the U.S. National Library of Medicine's cataloging classification [32]) for which one or more invasive therapeutic procedures is a treatment alternative. “Influence” will be measured by the number of literature citations adjusted for the number of years since the trial's main results were reported. Because the brevity of published papers precludes a thorough assessment of areas covered by the standards, we will apply the standards to dossiers for each trial that contain peer-reviewed publications emanating from the trial, study protocols, project final reports, review board materials, informed consent forms, and other materials obtained from trial investigators and sponsors.

When a trial failed to meet a standard, we will a) examine all records to determine why the investigators made the decision they did; b) consider alternative methods that might have been used; and c) analyze whether the conflict between ethics and methods was surmountable or insurmountable.

Phase 3: What Are Reasonable, Contemporary Ethical and Methodological Standards for Clinical Trials of Invasive Therapeutic Procedures?

In phase 3 we will convene a consensus conference to formulate a set of realistic contemporary standards for the initiation, design, conduct, and analysis and reporting of clinical trials evaluating the effects of invasive therapeutic interventions. Ethicists, methodologists, surgeon-clinical investigators, and members of the public will be represented among the conferees. As the basis for the final set of standards, conferees will use the findings of the ethical analyses of actual surgical trials reported in the past 10 years (phase 2 data). Conferees will vote privately and anonymously on the extent to which a given standard is realistic (practical, rather than imaginative or visionary) given the attributes of a particular surgical procedure. A consensus-building process will then be undertaken with standards identified in first-round voting as being nonconsensus standards, to determine if the standard should be refined or deleted. The final report of the panel will include consensus statements, statements about uncertain areas in need of further research, and minority opinions, and will be made available for release in the professional and lay literature.

Anticipated Contributions of the Project

The large number of nations with >700 active trials, the numerous standard-issuing bodies from those nations, and the multiplicity of standards from the source documents demonstrate the richness of our findings to date. But this same richness shows how insurmountable a task it is for a clinical researcher, research sponsor, or research oversight body engaged in multi-national trials to know about, find, and ensure compliance with standards such that therapeutics trials are of the highest ethical and methodological quality. Accordingly, as part of phase 1 we plan to produce a compendium in which ethical and methodological standards for clinical trials are fully integrated. This compendium will be web-based and the standards and other elements it contains will be searchable along multiple attributes. Such a compendium would be a tremendous boon to the research community and the members of the public who participate in clinical trials. Despite the increasing globalization of clinical research, [33] no such source exists. UNESCO's Global Ethics Observatory has recently launched a new database on ethics-related legislation and guidelines; however, it is not directed specifically at clinical trials, contains no methodological standards, and is not integrated.[34] If it is true that “scientifically unsound research [in humans] is unethical...” [7] then a source is critically needed in which ethical and methodological standards are found together in a format that is useful for those who plan trials, those who participate in them, and those who oversee them.

Phases 2 and 3 of this project will fill a serious gap in human research, namely standards for trials evaluating invasive procedures—standards that reflect consensus about ethics and methods and incorporate contemporary realities in surgical practice. Adherence to such standards should result in substantial and much-needed improvements in the scientific evidence base for invasive therapeutic procedures.