Observational research in the evidence based environment: eclipsed by the randomised controlled trial?

Patient based measures of outcome are assuming an increasing importance in the evaluation of therapeutic intervention. A classic, theoretical example of the importance of this approach involves the reporting of trials evaluating surgical procedures for benign prostatic obstruction. High quality studies might report significant improvements in appropriate outcome measures, such as the urinary flow rate or frequency of nocturia, and establish an apparent superiority over conservative treatment. When, however, patient satisfaction is considered, a side effect of erectile impotence in the surgical group may reverse this position.

A number of instruments have been developed for the quantitative assessment of what is now often referred to as “quality of life”. These may be generic, examining several aspects or dimensions of physical function and perceived well being. Others are disease specific with, for example, the Seattle Angina Questionnaire and the Cardiac Health Profile representing leading instruments in the assessment of angina pectoris.¹

CABG VERSUS PTCA TRIALS

The first generation of trials comparing revascularisation by coronary artery bypass grafting (CABG) or percutaneous transluminal coronary angioplasty (PTCA) did not demonstrate any significant differences in the subsequent incidence of death or non-fatal myocardial infarction. Initial treatment with coronary angioplasty was, however, associated with a less complete resolution of angina at short term follow up and with a 10 fold increase in the requirement for additional, repeat revascularisation.² An understanding of the impact of these findings on patient physical function and perceived heath state is essential if the relative merits of the two strategies are to be appreciated. A patient with stable angina contemplating elective revascularisation will be influenced by many factors and the reduced immediate procedural morbidity, shorter hospital stay, and option of subsequent bypass ensures that multivessel angioplasty enjoys an enduring immediate appeal. It is important to know if this translates into medium and long term satisfaction.

A number of the randomised PTCA versus CABG trials included an assessment of quality of life. The more recent trials—ARTS (arterial revascularisation therapy study) and SoS (stent or surgery)—re-examining this issue in the stent era both have designs with strong emphasis on the assessment of patient based outcomes, cost and cost benefit.³ The paper by Brorsson⁴ in this issue of Heart adds to this developing literature, by reporting on a prospective series of patients with one and two vessel coronary artery disease undergoing revascularisation with CABG or PTCA.

In the assessment of new developments in health care, the prospective randomised controlled trial (RCT) has assumed an appropriate, pre-eminent role. Even with rigorous, prospective design and meticulous conduct, the results of observational studies can be confounded by evident or unrecognised bias. As part of the establishment of the concept of a hierarchy of evidence quality, leading reviews have compared the results of observational and randomised studies designed to address the same clinical question. Recent work in this field has challenged the established finding that observational studies tend to report greater outcome differences than randomised trials.^5,6 It is possible that the nature, quality, and analysis of observational research has improved but fundamental limitations remain.^7,8 In its assessment of the clinical role of coronary stenting, the National Institute of Clinical Excellence (NICE) was disinclined to consider evidence from observational studies. The NICE approach may be too restricted and ignores the important complementary roles of the methodologies.

“Few studies have been able to overcome the practical (and funding) issues associated with prolonged follow up”

As the work of Darwin illustrates, RCT methodology may be inappropriate for some types of evaluation. A prospective RCT designed to examine different practice patterns or health service provision may be impossible to perform or involve unrepresentative and unethical manoeuvres. RCT initiatives are complex to mount and can be expensive to perform. As individual studies rarely address more than a few clinical questions only a very limited range of issues can be resolved with this approach. If the outcome of interest occurs very infrequently then it may be impossible to mount an appropriately powered, prospective RCT. (Postmarketing surveillance schemes for new drugs or medical devices are an example of an alternative approach to this problem). Sample size issues often restrict the ability to perform subgroup analyses that might help guide clinical practice. Other difficulties may be encountered if very long term observation is required to assess the outcome measure. Few studies have been able to overcome the practical (and funding) issues associated with prolonged follow up.

OBSERVATIONAL STUDIES

Observational studies can help address some of these issues. At the very least they help to identify the questions for future randomised controlled trials and to define the clinical conditions under which they will be addressed.

The RCT methodology was first developed for drug trials and it has proven difficult to translate this approach into procedure based interventions. A number of limitations have been described. Most operators and their supporting staff experience a “learning curve” effect as a new procedure is introduced. Early results may be less good than in subsequent cases. This can create difficulties in the timing of initiation of a study. If the evaluation is performed too early (or with less skilled operators) this may result in the condemnation of a potentially valuable procedure. If the research is initiated too late, an untested procedure may be in widespread use and have become so well established that some clinicians are unwilling to enrol patients in a randomised evaluation.

Rapid developments in a field can present additional problems. In coronary intervention, for example, advances in equipment, techniques, and adjunctive medication schedules meant that, on publication, the results of some studies had little relevance to prevailing practice patterns.

For procedure trials, the absence of a traditional placebo means that studies may compare very different treatments. This may involve the direct comparison of two procedures or a proportion of the trial population receiving alternative (usually medical) treatment. In the vast majority of cases it is impossible to blind either patients or observers to the treatment allocation and disparate treatments may present problems in the interpretation of outcome data. There may also be systematic differences in care that are not related to the intervention under consideration. In the soon to be reported SIMA trial, patients with single vessel coronary disease affecting the proximal left anterior coronary artery were randomised to management with coronary stenting or CABG. Lipid secondary prevention treatment was prescribed much more frequently in the non-surgical group (26% v 6% of cases) and may have influenced the medium term outcome.

When a trial seeks to compare two very different forms of treatment informed consent can be difficult, and patient and operator preferences can result in low recruitment rates and possible selection bias. This can exacerbate a more general problem; the majority of RCTs are performed in atypical settings and may recruit an unrepresentative patient population. In some studies only a small proportion of patients screened are randomised—as few as 2–5% in many of the first generation PTCA versus CABG trials. This may compromise the external validity of a study and hence limit its application in routine clinical practice.

APPLYING TRIAL FINDINGS TO THE INDIVIDUAL PATIENT

In the extension of trial findings to a management decision about an individual patient perhaps the most fruitful approach is to consider how well the individual under consideration matches the trial population. Although study inclusion and exclusion criteria frame a broad picture, more detailed data are usually available in the description of patient baseline characteristics. Steering committees concerned about the generalisation of results should provide a more complete description of trial subjects. This would be a better use of resources than, for example, the conduct of a screening registry. In this latter activity, investigators are asked to provide some information about all individuals screened for the study. This requires relentless hard work and the activity is difficult to fund and execute. In deference to this, the data set requested is often scant and is rarely recorded for all trial eligible patients. This undermines the supposed primary purpose of the registry—to describe the population norm from which the trial population was drawn. Furthermore the underlying concept is flawed, as population norms will vary between institutions, regions, nations, and health care systems. A description of the routine clinical caseload in a US teaching hospital may bear very little relation to the readers' activities and does nothing to refine understanding of the trial population. What matters is who was included in the trial and not who was excluded.

Observational research plays an important role in hypothesis generation, establishing questions for future randomised controlled trials and defining the clinical conditions under which they will be addressed. Beyond this, registry data provide the opportunity to bring research closer to practice, particularly in consideration of the external validity of trial results and their generalisation to local settings or for specific clinical presentations. When, as in the paper by Brorsson, the findings of experimental and observational ventures coincide, then this provides the strongest of evidence based mandates.

Abbreviations

• ARTS, arterial revascularisation therapy study

• CABG, coronary artery bypass graft

• NICE, National Institute of Clinical Evidence

• PTCA, percutaneous transluminal coronary angioplasty

• RCT, randomised controlled trial

• SoS, stent or surgery