1 Introduction
Dr. Lauer discusses the wide range of studies that can fall under the umbrella of CER, ”including observational studies, experiments (meaning clinical trials), and syntheses”, urging us to focus on randomization and to utilize clinical trial designs. We by no means disagree with the scientific benefits that randomization provides. We caution, however, that the prescribed method should start with the careful formulation of the question to be answered. By only examining situations where randomization is feasible, we will be unable to answer many questions important to people, patients, and policy makers. To illustrate our point, we pose several clinical questions related to the management of carotid atherosclerosis.
1.1 Clinical Setting: Management of Carotid Atherosclerosis
Carotid atherosclerosis is a degenerative disease resulting in plaques in the carotid artery, the main blood vessels in the neck leading to the brain, which may lead to stroke. There are three treatment options available to patients: medical therapy (including utilization of anti-platelet drugs, statins, antihypertensives, smoking cessation, and life style modification), an open surgical procedure of carotid endarterectomy (CEA), and a catheter-based procedure of carotid angioplasty with stenting (CAS). CEA can diminish the risk of stroke in patients with atherosclerotic lesions in the cervical carotid artery. However, the benefit of CEA may differ according to patient sex, symptomatic status and degree of stenosis, as well as according to characteristics of the operators who perform the procedures. CAS, a relatively new procedure, may be as effective as CEA in treating carotid artery disease. The main peri-procedural outcomes of CAS include death, stroke, or myocardial infarction; long term outcomes include stroke, cranial nerve injury, patient life expectancy, or restenosis. The FDA labeling for carotid stents stipulates the use of embolic protection devices to catch debris that may become dislodged during the procedure. Given the current paucity of evidence to guide decisions among these three potential treatment options, several comparisons are required.
First, for elderly patients with asymptomatic severe carotid atherosclerosis, which treatment is the most efficacious (best medical therapy alone, CAS, or CEA) to improve outcomes of stroke and death? Second, for elderly patients with asymptomatic severe carotid atherosclerosis, which treatment is the most effective to improve outcomes of stroke and death when treated by typical physicians in the community? Finally, for subgroups of patients, including the extreme elderly, women, and high risk surgical candidates, which treatment is the most effective to improve outcomes of stroke and death? With the greying of America, we have posed these questions to target this increasingly large population.
What do we know? Several randomized trials have been conducted since 2000 comparing CEA and CAS (Table 1): SAPPHIRE (Stenting and Angioplasty with Protection in Patients at High Risk for Endarterectomy), EVA-3S (Endarterectomy versus Angioplasty in Patient with Symptomatic Severe Carotid Stenosis), CREST (Carotid Revascularization Endarterectomy versus Stenting Trial), SPACE (Stent-Protected Angioplasty versus Carotid Endarterectomy), and ICSS (International Carotid Stenting Study). Randomized studies comparing CEA to medical therapy were conducted in the 1990s. To date, no randomized studies have compared CAS or CEA to modern medical therapy of antiplatelet therapy, antihypertensive medications, and aggressive lipid lowering therapy with statins.
Table 1. CEA-CAS Randomized Trials.
Trials covering the period 2000 – 2008.
| Study | Asymptomatics Included |
% Embolic Protection Device |
Sample Size CAS/CEA |
Mean Age |
Time to Endpoint |
RR (95% CI) |
|---|---|---|---|---|---|---|
| SAPPHIRE1 | Yes | 96% | 167/167 | 73 | 1 year | 0.61 (0.37 to 1.05) |
| EVA-3S2 | No | 92% | 262/265 | 70 | 30 days | 2.5 (1.2 to 5.1) |
| SPACE3 | No | 27% | 599/584 | 69 | 30 days | 1.09 (0.69 to 1.72) |
| ICSS4 | No | 72% | 828/821 | 70 | 3 years | not available |
| CREST5 | Yes | 96% | 1262/1240 | 69 | 4 years | 1.11 (0.81 to 1.51) |
RR = relative risk of death or stroke. Embolic protection devices trap thromboemboli dislodged during carotid stenting. CAS = cartodi stenting; CEA = carotid endarterectomy.
2 Design and Ethical Considerations
2.1 For patients with asymptomatic carotid atherosclerosis, which treatment is the most efficacious (best medical therapy alone, CAS, or CEA) among the elderly to improve outcomes of stroke or death?
This is an ideal situation for a randomized clinical trial if there is equipoise or it has not been shown that CAS, CEA, or medical therapy alone will cause the greatest reduction in stroke or death among asymptomatic elderly patients with carotid atherosclerosis. Randomized trials are essential for testing efficacy of a new treatments in ideal settings. The benefit of randomization of patients to treatments is that we eliminate selection bias and balance the treatment arms with respect to all confounding variables (known and unknown). As illustrated in Table 1, several randomized trials were conducted to examine difference in outcomes of stroke or death for those receiving CAS versus CEA, as well as those receiving CEA versus medical therapy. However, these trials are of varying sizes and did not restrict to only the elderly with asymptomatic carotid atherosclerosis. SAPPHIRE and CREST both included asymptomatic patients, but used different stenosis severities for these patients to qualify for inclusion in the trial – SAPPHIRE indicating ≥ 80% stenosis severity and CREST only ≥ 60%. The American Heart Association and the American Stroke Association recently provided comments to the Medicare Evidence Development and Coverage Advisory Committee on the management of carotid atherosclerosis and called for ”well-designed, contemporary trials” as a necessity for comparing CEA, CAS, and medical therapy for asymptomatic patients.
While CREST provided some indication that CAS had higher four year stroke or death rates compared to CEA for asymptomatic patients, studies that are powered to detect a difference in rates of stroke or death in a population of asymptomatic elderly patients rather than looking at subgroups of patients are needed. Importantly, prior studies comparing medical therapy to CEA were conducted in an era in which medical therapy was significantly different than it is today. Indeed, stroke rates following the discovery of asymptomatic carotid disease treated medically have declined significantly over the previous two decades.6
2.2 For patients with asymptomatic severe carotid atherosclerosis, which treatment is the most effective among the elderly to improve outcomes of stroke and death when treated by typical physicians in the community?
Randomized clinical trials have long been criticized for conducting experiments that may not produce results that are generalizable to routine clinical practice settings. Dr. Lauer advocates for the conduct of practical or pragmatic clinical trials that more accurately recapitulate the patient profiles and clinical decisions truly encountered by physicians in the community. To date, trials comparing CAS to CEA conducted in the U.S. have included fairly stringent enrollment criteria, and, importantly, have typically involved highly skilled operators who were required to have demonstrated a high level of competence.1;5 A randomized trial comparing CAS, CEA and best medical therapy, with broad enrollment criteria and few limitations on physician participation may be able to successfully inform questions regarding the relative effectiveness, rather than efficacy, of various treatments.
Although practical clinical trials are useful from a CER perspective, they still have limitations in the actual interpretation of results from a causal perspective. For example, if efficacy is established in a narrow, but well-defined population what do we conclude if we are then unable to see effiectiveness in an expanded population of interest for generalizability of medical practice?7 Determining whether the conflicting finding is due to lack of efficacy in the broader patient population or if provider performance weakens the intervention may require statistical approaches from the realm of observational studies, such as adjusting for sample enrollment bias and confounding. If practical clinical trials necessitate statistical methods to handle biases and confounding, we may question why an observational study should not be performed in the first place.
Observational studies may be as, if not more, suitable for answering such a question. Physician and hospital participation in clinical trials is largely skewed toward academic centers which may have different outcomes8 than community hospitals and practices. In addition, patient participation in clinical studies, even when enrollment criteria are broad, is often confined to a healthier and more informed patient population. In the treatment of carotid artery disease, even post-marketing registries of CAS, which do not require randomization and have the explicit purpose of assessing real world outcomes, have enrolled healthier patients and improved outcomes compared to community practice.9 In such circumstances, investigators need to balance the benefits of randomization-safeguarding against bias due to unmeasured confounding-against the known compromises in generalizability which may diminish the applicability of study results, even in the most pragmatic of clinical trials.
2.3 For subgroups of patients, including the extreme elderly, women, and high risk surgical candidates, which treatment is the most effective to improve outcomes of stroke and death?
The primary findings of randomized clinical trials report average treatment effects across the entire population of enrolled individuals. Subgroup comparisons are typically limited to single variable stratifications which are grossly underpowered to find significant interactions. However, heterogeneity in treatment effectiveness is likely to be the rule, rather than the exception, particularly when treatment modalities are as diverse as those for carotid disease. Women represent one such subgroup in which revascularization for asymptomatic carotid artery disease has not been shown to be beneficial.10 The most valid method to compare treatment of asymptomatic carotid disease may involve enrollment of a large population of men and women, powered to examine the interaction between treatment modality and sex on outcomes. Such a trial is likely to lack feasibility because women have been historically underrepresented in clinical trials, and powering a three-armed clinical trial to detect a significant interaction would unlikely be able to successfully enroll a large enough population over a short enough time period to retain clinical relevance. In such circumstances in which feasibility of clinical trials is unlikely, observational studies represent the best approach to provide valuable information to help inform clinical practice..
Like any study, observational studies require rigorous design, data collection, and execution. A well-defined question addressed by an observational study informs the choice of covariates. Clinically rich data that include accurately collected confounders are crucial in order to understand the treatment assignment mechanism. Designs that exploit multiple comparison groups can help detect threats to residual confounding.11 Forward-looking designs would also include instrumental variables – those variables strongly related to treatment selection but not related to the outcome apart from its association with treatment selection. No observational study can guarantee elimination of ignorability of the treatment assignment. However, sensitivity analyses can provide a quantitative method to bound inferences. Finally, because of the lack of internal validity associated with observational studies, protocols and statistical code used to conduct all analyses should be made available so that findings can be replicated.
3 Conclusion
We have demonstrated that after defining the question of interest, a randomized clinical trial may not be the ideal way to inform clinical decision making – this observation is not new. When performed carefully, observational studies are able to answer questions of comparative effectiveness. Dr. Lauer is correct in stating” we have an opportunity to exploit the debate on CER to reinvigorate the clinical trial enterprise”, but the debate on CER should also reinvigorate methods for making causal inference from observational studies to improve clinical outcomes on both the patient and policy levels. Insistence on randomization as the only vehicle to CER is dangerous. Modern statistical thinking should embrace multiple strategies to create evidence.
Acknowledgments
Funding: Ms. Kunz was funded by the Research Participation Program, Center for Devices and Radiological Health, administered by the Oak Ridge Institute for Science and Education through an inter-agency agreement between the U.S. Department of Energy and the U.S. Food and Drug Administration (FDA). Dr. Yeh’s effort was supported by the Harvard Clinical Research Institute, Boston. Dr. Normand was supported, in part, by Contract No. 1074629 from the FDA. Dr. Normand serves as Vice Chair, Methodology Committee of the Patient Centered Outcomes Research Institute (PCORI). The views expressed by Dr. Normand do not necessarily reflect those of the PCORI.
References
- 1.Gurum HS, Yadav JS, Fayad P, et al. Long-term results of carotid stenting versus endarterectomy in high-risk patients. New England Journal of Medicine. 2008;358(15):1572–1579. doi: 10.1056/NEJMoa0708028. [DOI] [PubMed] [Google Scholar]
- 2.Mas JL, Trinquart L, Leys D, et al. Endarterectomy versus angioplasty in patients with symptomatic severe carotid stenosis (EVA-3S) trial: results up to 4 years from a randomised, multicentre trial. Lancet Neurology. 2008;7(10):885–892. doi: 10.1016/S1474-4422(08)70195-9. [DOI] [PubMed] [Google Scholar]
- 3.Eckstein HH, Ringleb P, Allenberg JR, et al. Results of the stent-protected angioplasty versus carotid endarterectomy (SPACE) study to treat symptomatic stenosis at 2 years: a multinational, prospective randomised trial. Lancet Neurology. 2008;7(10):893–902. doi: 10.1016/S1474-4422(08)70196-0. [DOI] [PubMed] [Google Scholar]
- 4.Ederle J, Dobston J, Featherstone RL, et al. Carotid artery stenting compared with endarterectomy in patient with symptomatic carotid stenosis (International Carotid Stenting Study): an interim analysis of a randomised controlled trial. Lancet. 2010;375(9719):985–987. doi: 10.1016/S0140-6736(10)60239-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Brott TG, Hobson RW, 2nd, Howard G, et al. Stenting versus endarterectomy for treatment of carotid artery stenosis. New England Journal of Medicine. 2010;363(1):11–23. doi: 10.1056/NEJMoa0912321. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Naylor AR. What is the current status of invasive treatment of extracranial carotid artery disease? Stroke. 2011;42:2080–2085. doi: 10.1161/STROKEAHA.110.597708. [DOI] [PubMed] [Google Scholar]
- 7.Luce BR, Kramer JS, Goodman SN, et al. Rethinking randomized clinical trials for comparative effectiveness research: The need for transformational change. Annals of Internal Medicine. 2009;151:206–209. doi: 10.7326/0003-4819-151-3-200908040-00126. [DOI] [PubMed] [Google Scholar]
- 8.Majumdar SR, Roe MT, Peterson ED, Chen AY, Gibler WB, Armstrong PW. Better outcomes for patients treated at hospitals that participate in clinical trials. Arch Intern Med. 2008;168(6):657–662. doi: 10.1001/archinternmed.2007.124. [DOI] [PubMed] [Google Scholar]
- 9.Yeh RW, Kennedy K, Spertus JA, Parikh SA, Sakhuja R, Anderson HV, White CJ, Rosenfield K. Do postmarketing surveillance studies represent real-world populations?. A comparison of patient characteristics and outcomes after carotid artery stenting. Circulation. 2011;123(13):1384–1390. doi: 10.1161/CIRCULATIONAHA.110.991075. [DOI] [PubMed] [Google Scholar]
- 10.ACAS Executive Committee. Endarterectomy for asymptomatic carotid artery stenosis. Journal of the American Medical Association. 1995;273(18):1421–1428. [PubMed] [Google Scholar]
- 11.Yoon F, Huskamp H, Busch A, Normand S-LT. Using multiple control groups and matching to address unobserved biases in comparative effectiveness research: An observational study of the effectiveness of mental health parity. Statistics in the Biosciences. 2011;3(1):63–78. doi: 10.1007/s12561-011-9035-4. [DOI] [PMC free article] [PubMed] [Google Scholar]