Randomized Controlled Trials in Health Technology Assessment: Overkill or Overdue?

Abstract

Evidence-based medicine has become a cornerstone in the development of radiation oncology and the randomized controlled phase III trial remains the gold standard for assessing differential benefits in clinical outcome between therapies. Health technologies aimed at improving treatment quality should primarily be tested using process measures or operational characteristics, the reason being that the sensitivity and specificity of clinical outcome is low for detecting quality improvements. The ongoing discussion of the relative merits of intensity modulated photon vs. proton radiotherapy is used to illustrate these concepts. Concerns over clinical and individual equipoise as well as the potential limitations of health economics considerations in this setting are also discussed. Working in a technology and science based medical discipline, radiation oncology researchers need to further develop methodology for critical assessment of health technologies as a complement to randomized controlled trials.

“Only two options exist. The first is that we accept that, under exceptional circumstances, common sense might be applied when considering the potential risks and benefits of interventions. The second is that we continue our quest for the holy grail of exclusively evidence based interventions and preclude parachute use outside the context of a properly conducted trial.”

From: “Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials”, GCS Smith & JP Pell [1]

1. Evidence-based medicine and randomized controlled trials

Evidence-based medicine is a compelling paradigm as a rational road-map, some would argue the only rational road-map, for improving the standard of health care [2,3]. The role of randomized controlled trials in this context cannot be overstated: while the allocation to different treatment options may be skewed even in a specific randomized trial, randomization is the only method that on average safe-guards against prognostic imbalances that could bias therapy effect estimates. History shows, that even when there has been overwhelming support for a novel therapeutic intervention in terms of a well-proven mechanism of action, promising pre-clinical and early clinical data as well as wide-spread consensus regarding an expected benefit among the experts, the cold-hearted scrutiny of a, preferably double-blind, randomized controlled trial may put a sudden end to its further clinical development.

One text-book case is the trials of erythropoietin (EPO) to increase the hemoglobin concentration in patients receiving radiation therapy. Low hemoglobin concentration is well-established as a prognostic factor in several solid cancers [4]; and in patients undergoing radiation therapy, this has traditionally been thought to be the result of hypoxia-related radioresistance. With the advent of EPO, there seemed therefore to be a strong case for trying to correct low hemoglobin concentrations before and during radiation therapy. Several placebo-controlled randomized controlled phase III trials were initiated to test this strategy, or to test EPO in combination with chemotherapy, and the outcome was a surprise. Despite a demonstrable improvement in hemoglobin concentration after EPO [5], the trials consistently found a worse tumor outcome in the EPO arm [6–10]. The biological mechanism behind this unexpected finding remains controversial [11] although it is likely the result of EPO acting as a survival/growth factor for cancer cells expressing erythropoietin surface receptors

Examples from other fields of oncology are abundant. For instance the recent demonstration in the large CALGB 89803 randomized controlled trial [12] that in patients with stage III colon cancer the addition of CPT-11, a drug with proven activity in metastatic disease, to fluorouracil and leucovorin, did not improve tumor outcome; or the INTACT-1 and INTACT-2 randomized controlled trials in patients with non-small cell lung cancer [13,14], each enrolling more than 1000 patients, showing that the addition of Gefitinib to two-drug chemotherapy regimens did not improve tumor outcome despite promising response data from randomized phase II trials. These and many other examples show why large randomized phase III trials are needed even to test the obvious, because experience shows that the obvious is often disproved [15].

Extrapolation of evidence-based medicine to treatment-related health technology assessment is straightforward: any new treatment technology can be seen as a therapeutic intervention, much like a new drug in the treatment of a disease [16]. So what, if anything, prevents a simple insistence that all new health technologies should go through the gold standard testing of an adequately powered randomized controlled trial before being implemented in the clinic? And what do we do as individuals and as a profession, in case Level I evidence – that is, evidence from randomized controlled trials – is lacking? The ongoing discussion of proton therapy as an alternative to photon therapy highlights many of the issues involved and will be used as a specific example in the following.

2. Lack-of-Evidence Based Medicine

After years of Evidence Based Everything in clinical research [3,17], it has become increasingly clear that this powerful paradigm has a dark companion: what we could call Lack-of-Evidence Based Medicine (LEBM). LEBM has been championed in the United Kingdom by the National Institute for Clinical Excellence (NICE) that has used it to restrict access to a number of therapies on the National Health Service. While evidence based medicine is concerned with weighing Level I evidence for or against a given intervention, LEBM is concerned with the systematic reviewing of the Lack-of-level-I evidence on a given topic. In the May 2007 issue of Radiotherapy & Oncology, two overviews and an invited editorial, running over a total of 28 pages, summarized the lack of Level I evidence for a benefit from proton therapy [18–20]. Add to this, that the same body of non-evidence underwent systematic review in the March 10, 2007 issue of Journal of Clinical Oncology [21] and in part by Widesott et al. in this issue of the Journal, and it is clear, that proton LEBM is a current hot topic.

Admittedly some Proton Protagonists may have been over-stating their case. Now, the Proton Skeptics have turned to LEBM, plausibly out of despair. Clearly, “the absence of evidence is not evidence of absence” of a benefit from protons. It just documents the widely-appreciated fact among anyone interested in the field, that there is no major body of randomized trials comparing protons with photons, and as suggested by Glimelius and Montelius [20], perhaps for good reasons. Still, it is concluded that “existing data do not suggest that the rapid expansion of hadron therapy as a major treatment modality would be appropriate” and “the introduction…of hadron therapy as a major treatment modality… into standard clinical patient care cannot be supported by the evidence base currently available” which is logically correct but can easily be mistaken as evidence against protons [18]. Again, there is an important difference between a lack of effect estimates vs. estimates showing a lack of effect! Like in all fields of medicine, absence of Level I evidence can also be a convenient excuse for doing nothing.

Unfortunately, these reviews let a more thoughtful reader down. It would have been so much more interesting if the authors had challenged the current paradigm that randomized controlled trials is the only valid source of clinical evidence.

That more philosophical speculation aside, the problem with these overviews is that the potential advantage of protons over photons is an ill-posed question and it is doubtful whether a systematic overview is meaningful at all in the absence of a clear question. As protons per se have no inherent biological advantage over photons, it is too imprecise to ask if they are better or not… better for what? The anticipated benefit from the improved dose distribution achievable with intensity modulated proton therapy [22] can only be teased out in study designs where both normal tissue toxicity and tumor control are carefully documented. One severe obstacle here is the well recognized problem of establishing high-quality late toxicity data [23,24]. A comparison of widely variable single-group studies using different radiation modalities will certainly not produce a valid argument for or against a benefit from protons. While Lodge et al. use the lack of level I evidence supporting proton therapy as basis for their conclusions it seems that they fall into their own trap by informally comparing outcomes of a number of uncontrolled single-arm studies using photons or protons.

Lack-of-Evidence Based Medicine is a rich source for filling future issues of Radiotherapy and Oncology: there is no evidence from randomized controlled trials demonstrating the clinical benefit from widely adopted port films, simulators, CT-based treatment planning, mega-voltage radiotherapy. Think of the profound health economic consequences if we got rid of all these unproven technologies! By the way, there are also no phase III trial data showing the benefit from sharp vs. dull scalpels for surgery.

3. Whose equipoise?

Trying to estimate improvements in outcome from comparison with historical controls is associated with well documented difficulties, see for example Hodge et al. [25]. So, large randomized phase III trials of proton versus photon intensity modulated radiotherapy would seem to be overdue… but are they scientifically and ethically justifiable? From a medical ethics perspective, the acceptability of a randomized controlled trial hinges on the principle of equipoise, a reasonable balance of risks and benefits to the human research subjects in both arms of the trial. The treatment options being compared must look as acceptable trade-offs between risks and benefits, perhaps for different reasons – and this is what will ultimately make a trial ethically acceptable to the participating investigators and eventually lead an informed patient to volunteer as a trial participant. Disturbingly, in this context, there are data suggesting that patients are less likely to consent to randomization if they get more detailed information on trial arms [26,27] and also that the willingness to volunteer seems to be inversely correlated with the level of education of the research subject or their medical legal proxy [28]. Would a patient information sheet for a proton vs. photon trial include an explained case of a plot of isodose contours on a CT scan for example?

As observed by Lilford and Jackson [29] it is important to distinguish between uncertainty and equipoise: even when a patient or a physician have no evidence for the superiority of one treatment over another (i.e. are uncertain regarding the relative merits of the two treatments) they may still have a preference for one treatment based on prior knowledge, such as preclinical data, the mechanisms of action or the specific trade-off between efficacy and toxicity. Freedman [30] introduced the distinction between theoretical and clinical equipoise, often referred to as individual and collective equipoise. The former applies to the individual clinician who must be of the opinion that the evidence is equally split or that the risks and benefits among the treatment options are balanced. Clinical equipoise refers to the profession as a whole, or at least to the segment of the professionals that can be regarded “experts” in the topic under consideration. Freedman’s proposition is that a clinician can ethically randomize patients in his or her care among various treatment options even when he or she would have a personal preference, provided that a collective equipoise exists. This proposition forms the widely accepted ethical basis for the conduct of clinical trials and does provide an ethical way around the fact that surrogate patient studies have shown that many clinicians are willing to accrue patients into randomized trials that they would not volunteer to participate in if they were patients themselves [31]. Logically, there will be some loosely defined limit to the lack of individual equipoise, before randomizing patients will cause ethical concerns for the individual treating physician.

So, how do the above ethical and philosophical considerations apply to the ongoing proton debate? The problem with many potential proton-versus-photon trials is that all the foreseeable benefits – and no specific risks to counter-balance these – are associated with the proton arm. This ethical dilemma is recognized in the review by Olsen et al. [19] but they conclude nonetheless that “randomized trials should in general be performed prior to introducing proton therapy as routine treatment”. While we all agree that there is a need for critical health technology assessment, we must also acknowledge the reality that the fundamental dose-volume-effects for tumors and normal tissues are established beyond reasonable doubt. Or to quote Stephen Jay Gould: ‘In science, "fact" can only mean "confirmed to such a degree that it would be perverse to withhold provisional assent”’. Are we already at that point with proton therapy?

Now, to the extent that there is uncertainty regarding the benefits of proton or light-ion therapy compared with photons, this relates to the clinical magnitude of the therapeutic benefit rather than to the principal dosimetric advantage of these radiation modalities. It is clear, just from the reviews published in this journal [18–20] and the papers by Widesott et al. and Suit et al. in this issue, that there is uncertainty regarding these questions, but does this really amount to collective equipoise? And are the proton skeptics “informed experts” or are they just…skeptics?

This is where individual equipoise enters the equation. A randomized trial allocating patients to proton vs. photon therapy can by necessity only be conducted in centers with proton therapy facilities and proton therapy expertise. It is doubtful whether physicians working in these centers will feel an individual equipoise between the two treatment options.

There is one additional player, often overlooked, in the discussion of equipoise: the patient. Even when outcomes are equivalent, say, in terms of overall survival, an individual may not assign the same utility to various toxicity:efficacy trade-offs. For instance, a person who still hopes to have children in the future may be more concerned about fertility than a person who has no such plans. Some patients may go for maximizing their life expectancy, other may be more concerned with health-related quality of life in their remaining life time; adding years to their life, or life to their years. Eric Hall has suggested in discussions at meetings, that a, say, 1% increase in the 10-year risk of a fatal second malignancy may not be balanced - in the eyes of a patient - by a 1% increase in 10-year survival from improved control of the primary malignancy. While the risk of dying from the primary cancer is a consequence of disease itself, treatment-related side-effects are more difficult to accept psychologically, even if they can be seen as the price for surviving the cancer, as they are actively inflicted by the therapy.

Only the individual patient can assign a utility to various outcomes and in many parts of the World patients are increasingly well informed about their disease and expect to be involved in therapeutic management decisions and to have a degree of self-determination in matters concerning their own health. It is likely that an increasing proportion of patients in the coming years will “vote with their feet” and for example seek proton or light-ion therapy rather than photon therapy. This is to some extent already happening, with a large number of self-referrals treated at the existing proton-therapy centers and would obviously undermine recruitment into a simple trial comparing the same tumor dose distribution with protons or photons. In a somewhat longer perspective it could potentially affect patient volumes in centers working in a competitive market.

4. Health technology assessment and quality of care

While new health technologies are ultimately aimed at improving treatment outcome, the immediate goal is to improve treatment quality. This is more than a play with words. Adding a novel agent to a multi-drug chemotherapy regimen is not in any simple sense aimed at improving treatment quality, but rather to improve outcome. It follows logically, that the benefit from adding this agent should be tested using the gold standard for comparing treatment outcomes: the randomized phase III trial. While improved quality will ultimately improve outcome, the principal problem with clinical outcome as an indicator of treatment quality is the low sensitivity and specificity [32]. Survival after cancer therapy is influenced by so many factors, some of which are not even known, that the outcome in a sample of patients varies widely.

As an alternative, process measures or operational characteristics are often used as a direct measure of quality. Image guidance for example is introduced to improve the geometrical precision of target localization and dose delivery. Documentation that this is worthwhile will typically consist of recording the positional adjustments performed as a result of the imaging. If these are minute compared to e.g. the estimated precision in target definition or treatment delivery, then image guidance is probably not worth the time and money. On the other hand, if major deviations are recorded in a proportion of all dose fractions, then common sense would suggest that the imaging is clinically useful. In some cases, this argument can be supported by “virtual trials” of the effect of performing the corrections or not, using treatment planning or individual patient dosimeters to estimate the dosimetric consequences and subsequently using mathematical dose-response models fitted to clinical data in assessing the likely clinical consequences [33]. An example is dosimetric beam-output quality assurance programs that can easily be shown to rectify clinically relevant deviations between intended and actual delivered dose, but can hardly be made the subject of a randomized phase III trial [33].

In summary, refraining from seeking direct evidence in terms of clinical outcome improvement does not mean that critical appraisal of health technologies should be taken lightly. One could argue that the requirements for well-conducted studies with operational endpoints and the subsequent estimation of the likely clinical consequences are just as demanding – and maybe even more intellectually challenging – than a hypothetical “cook-book-science” comparison of therapeutic outcome in patients treated with or without the new technology in a two-arm randomized trial. Evidently, it would be a big mistake to assume that all new technologies represent a worthwhile improvement in treatment quality. New technologies should be critically evaluated on a scale of their likely impact on outcome in the presence of other uncertainties in the overall therapy process [34]. The point is exactly this, that we can basically quantify quality improvements by using convincing quality indicators, such as the dosimetric descriptors for the clinical target volume or for relevant organs at risk.

5. Health economics and technology assessment

A flurry of new therapeutic agents and a seemingly endless stream of novel imaging, treatment technology and high-throughput assays are likely to be introduced in the future, also in the management of patients with radiation therapy [35–41]. This will increase the cost of managing a patient with radiation therapy, even without moving to proton or light ion therapy. Clearly, cost-utility will become an important element of evaluating these technologies.

The proton skeptics often bring health economics into the picture arguing that protons are unlikely to be cost-effective compared with photons in several sites [18,42,43]. However, the detailed assumptions of a cost-effectiveness estimation – in the absence of precise differential cost-effectiveness data from a randomized phase III trial with individual patient-level cost data – will critically affect the conclusions and a number of similar exercises have come out favorably for proton therapy [44–46]. Some debaters have argued that as long as there is an under-provision of (photon) radiation therapy in many countries [47], large investments in proton facilities should be put on hold. This is a flawed argument in so far as there is not a defined chest of money earmarked for radiation therapy in any environment, irrespective of the organization of the health care system. As pointed out by Lester Peters in 1995, radiation oncologists “…must acknowledge the need to prioritize the use of finite resources so that the maximal effort would be expended for those with most to gain from the treatment” [48]. Yet, Peters also documented that in fiscal year 1994, at the University of Texas M. D. Anderson Cancer Center, the entire budgets for the Departments of Radiotherapy and Radiation Physics were exceeded by the cost of purchasing the three most expensive drugs consumed at that time, G-CSF, Ondansetron, and Interferon [48]. It is a safe bet that the oncology drug budget at M. D. Anderson today exceeds the radiation therapy budget with an even larger margin, despite the new radiation technologies introduced after 1994. Obviously, inflating the cost of radiation therapy can never be a goal in itself. But radiation therapy is rather good value for money: The direct cost of delivering radiation therapy has been estimated at only 5.6% of the total cancer therapy cost in Sweden [49]. To put a brake on the development and implementation of new technologies seems ultimately to be self-destructive for the future of the specialty. By analogy, it is hard to imagine that investigators involved in pre-clinical and clinical development of molecular targeted agents in 4^th or 5^th line therapy for metastatic colon cancer will slow down their research efforts out of concern for the rising costs of therapy in this indication.

6. Where next?

The discussion of protons versus photons is just one example of the broader discussion of what evidence is needed to support the introduction of new technologies. With a number of such technologies becoming available over the coming years [36,40,50,51], we need to establish a broader health research paradigm – and more importantly, get this accepted in society – that will not force us to randomize between alternatives that are very asymmetric in terms of their operational characteristics. Establishing this broader paradigm is a scientific and political challenge we cannot turn away from as a science and technology based medical specialty. Arguably, much of the progress in radiation oncology, and one could make a case for extending this to medicine in general, has been down to improved technology, and most often this has been introduced based on its operational characteristics rather than its direct effect on treatment outcome.

The reduced incidence of secondary malignancies resulting from the lower integral dose to non-target tissues for intensity modulated proton versus photon therapy will be virtually impossible to test in a randomized trial due to the relatively low absolute incidence of these cancers. However, it is quite possible that such a reduction can be shown if large populations of patients were followed over extended periods of time, analogous to the post-marketing pharmacovigilance (phase IV studies) used with new pharmaceuticals. It seems to be about time that a system of radiovigilance is established.

Randomizing between two radiation treatment delivery technologies that yield the same tumor dose distribution but with a left-shifted dose-volume histogram in critical organs at risk is – in my view – not consistent with the principle of equipoise. However, trial questions and trial designs that address a clinico-biological question while indirectly testing the two technologies can be envisioned. It all depends on how the trial is designed and the way risks and benefits are balanced. A randomized phase I/II trial finding the maximum tolerable dose (MTD) of a drug with overlapping or interacting toxicity with radiation, in patients receiving proton or photon intensity modulated therapy would represent a situation with equipoise as the drug dose would be escalated to tolerance in both treatment groups. Likewise, a study escalating the dose of intensity modulated proton and photon therapy on the basis of an estimated normal-tissue complication probability would also seem acceptable, in principle this would not be different from an altered fractionation or a dose-escalation study.

Blind adherence to the principles of evidence-based medicine will eventually break down logically – as shown in the provocative and entertaining paper by Smith and Pell [1] on the lack of randomized trials proving the benefit from parachutes when jumping out of airplanes. Clearly, we rely increasingly on evidence from randomized controlled trials in many clinical situations, and there is no discussion that patient care is better for it. On the other hand, we have to accept the ethical and practical limitations to this methodology. The point here is that non-randomized, or “observational”, studies should be seen as a complement to, rather than a substitute for, randomized controlled trials of treatment outcome [52]. And finally: we need to provide the best possible care for the patient in front of us, even in the absence of Level I evidence. This necessitates clinical decision making on an imperfect basis over and over again. Paradoxically, this may be the real reason why clinical science is the most challenging and often most rewarding of all the sciences.