Seeing what you want to see in randomised controlled trials: versions and perversions of UKPDS data

Randomised controlled trials are objective, free of bias, and produce robust conclusions about the benefits and risks of treatment, and clinicians should be trained to rely on them; so says the gospel of evidence based practice. In this article we argue, using the United Kingdom prospective diabetes study (UKPDS) as an example, that there is one stage in the conduct of a randomised controlled trial—the interpretation and dissemination of results—that is open to several biases that can seriously distort the conclusions. By bias, we mean the epidemiological definition: anything that systematically distorts the comparisons between groups. We will argue that certain biases arise when different stakeholders assign their individual values to the interpretation of the final results of randomised controlled trials.

Summary points

• Randomised trials are subject to interpretation bias as shown by the example of the UK prospective diabetes study

• The UK prospective diabetes study shows no benefit on macrovascular end points in patients with type 2 diabetes treated with sulphonylureas or insulin over 10 years

• The study shows a clinically important benefit on macrovascular end points from metformin in patients with type 2 diabetes that seems somewhat independent of the drug's ability to lower blood glucose concentrations

• Nevertheless, many authors, journal editors, and the wider scientific community interpreted the study as providing evidence of the benefit of intensive glucose control

• Journal editors should be aware of this important potential bias and encourage authors to present their results initially with a minimum of discussion so as to invite a range of comments and perspectives from readers

Marketing the UK prospective diabetes study results

Until 1998, type 2 diabetes had been treated for over 25 years with drugs such as the sulphonylureas, insulin, and metformin. Only one well designed, prospective clinical trial had evaluated the effect of these drugs on the development of microvascular and macrovascular disease. This was the university group diabetes program study, the results of which created considerable controversy because they showed an increased risk of death from cardiovascular disease in the group that received sulphonylureas.¹ Perhaps because of this controversy the results had little impact. The fact that the trial was never repeated, and that no further randomised controlled trials were published for another 25 years may surprise many clinicians. In September 1998, the long awaited results from the UK prospective diabetes study were presented in the BMJ, Lancet, and elsewhere.²,³ The 20 year study was conducted in 23 centres in the United Kingdom. More than 5000 patients with type 2 diabetes were recruited. The aim of the study was to determine the effect of intensive blood glucose control with sulphonylureas, insulin, or metformin on 21 predetermined clinical end points.

Despite some of the methodological limitations (the study was unblinded, the trial was continued when differences were not seen at the initial evaluation, and patients in the diet group received drug treatment if their fasting plasma glucose concentration was greater than 15 mmol/l), the papers have some important messages for clinicians and patients.⁴,⁵ Indeed, it is unlikely that any trials in the near future will provide us with more information about the effect of glucose lowering drug treatment on the microvascular and macrovascular complications of type 2 diabetes. In general, the reporting of the results of the trial was positive. Reviewers stated that:

• Clear and consistent evidence now exists that hyperglycaemia in diabetes is a continuous, modifiable risk factor for clinically important outcomes and that reduction in glucose is the key to improving outcomes⁶

• We now have convincing evidence that tight blood glucose control is an important goal for type 2 diabetes. Unless patients are seriously ill or have a short life expectancy, the long term benefits of intensive therapy clearly outweigh the few risks⁶

• The main translatable finding is that intensive treatment of type 2 diabetes is beneficial.⁷

Despite these widely disseminated conclusions, scrutiny of the published data seems to show that the sulphonylureas and insulin have no impact on clinically important outcomes.²,³ In this paper we present the raw data and invite readers to come to their own conclusions and recommendations.

What did the data show?

Table 1 summarises the 10 year results of the UK prospective diabetes study 33,² which evaluated drug treatment in 2505 non-obese and 1362 obese participants referred to hospital clinics with newly diagnosed type 2 diabetes. We have expressed the data as percentages rather than events per 1000 patient-years so that we can give absolute reductions and numbers needed to treat over a specific period. This allows comparison with the results of other trials that have been presented in this standardised way.⁸ We realise that there are advantages, disadvantages, and assumptions that have to be made when presenting the results either way.⁹

Table 1.

Effect of 10 years' treatment with chlorpropamide, glibenclamide, or insulin on patients with newly diagnosed type 2 diabetes

	Any diabetes related end points* (%)	Microvascular disease (%)	Individual macrovascular disease end points†	Median haemoglobin A1c (%)
Dietary advice plus chlorpropamide, glibenclamide, or insulin	35.3	8.2	No significant difference between the groups for any of the individual end points‡	Chlorpropamide 6.7; glibenclamide 7.2; insulin 7.1
Dietary advice only	38.5	10.6		7.9
Relative risk reduction	8.2	22.6		Significantly lower for all drugs compared with dietary advice
Absolute risk reduction	3.2§	2.4
No needed to treat for 10 years to prevent one event	31	42

^*Sudden death, death from hyperglycaemia or hypoglycaemia, fatal or non-fatal myocardial infarction, angina, heart failure, stroke, renal failure, amputation, vitreous haemorrhage, retinal photocoagulation, blindness in one eye, cataract extraction. 

^†Deaths related to diabetes, all cause mortality, myocardial infarction, stroke, blindness, renal failure, or neurological events. 

^‡P value for myocardial infarctions was 0.052 (dietary advice plus drug treatment 14.2% v dietary advice 16.3%). However, because the study was continued after the initial results showed no differences, a breakpoint for significance of 0.05 is debatable. 

^§2.7% of this 3.2% was due to a significant reduction in retinal photocoagulation. 

The primary outcome for these trials was a reduction in the number of patients with an aggregate of clinical end points (table 1) or diabetes related deaths. Over the 10 years of the study, there was a 3.2% absolute reduction in the occurence of one of the aggregated end points. Most of this benefit was due to a 2.7% absolute reduction in retinal photocoagulation, which was assessed by ophthalmologists independent of the study.

However, closer evaluation of the results showed that the use of glibenclamide, chlorpropamide, or insulin to lower blood glucose concentrations produced no significant benefit on any single macrovascular end point. A 2.4% absolute difference was seen for microvascular end points, and, again, most of the benefit was due to the reduction in retinal photocoagulation. Differences were detected in the surrogate end points of progression of retinopathy and albuminuria, but there were no differences in blindness, visual acuity, or renal failure.

Nevertheless, this trial has shown that the use of sulphonylureas probably does not increase the risk of death or serious disease events, which was a potential concern suggested by the results of the university group diabetes program study.¹ It seems, therefore, that clinicians can be confident in prescribing these drugs to control the symptoms of hyperglycaemia in patients whose glucose concentrations are not adequately controlled by diet, exercise, and other oral drugs.

The UK prospective diabetes study 33 suggests that the drugs used were well tolerated, although only hypoglycaemic events and weight gain were reported. Nevertheless, participants in the sulphonylurea and insulin groups gained a mean of 3.1 kg more weight than the diet alone group. Major hypoglycaemic episodes (those requiring third party help) occurred in 0.1%, 0.6%, 0.6%, and 2.3% of participants per year in the diet, chlorpropamide, glibenclamide, and insulin groups respectively (note that benefit was expressed over 10 years). The incidence of minor hypoglycaemic events was 1%, 11%, 18%, and 37% per year, respectively.

In contrast to the above results, the UK prospective diabetes study 34, which focused on 1704 obese newly diagnosed type 2 diabetic patients, found several clinically important differences in macrovascular disease end points with 10 years of treatment with metformin (table 2).³ In particular, the absolute risk reduction for the aggregate end points was more than 10%, and for overall mortality was 7%, giving numbers needed to treat of 10 and 14 respectively over 10 years. Furthermore, in these patients, metformin was not associated with increased weight gain or hypoglycaemic episodes compared with diet alone. Metformin reduced progression of retinopathy compared with dietary advice alone, but there were no differences in other surrogate markers between the treatment groups.

Table 2.

Effect of 10 years' treatment with metformin or chlorpropamide, glibenclamide, or insulin in overweight patients with newly diagnosed type 2 diabetes

	Any diabetes related end point (%)	Deaths related to diabetes (%)	All cause mortality (%)	Myocardial infarction (%)	Stroke (%)	Microvascular disease (%)	Median haemoglobin A1c (%)
Dietary advice plus metformin	28.7†	8.2*	14.6†	11.4*	3.5‡	7.0	7.4
Dietary advice plus chlorpropamide, glibenclamide, or insulin	36.8	10.8	20.0	14.6	6.3	7.8	All similar to metformin
Dietary advice only	38.9	13.4	21.7	17.8	5.6	9.2	8.0
Relative risk reduction (metformin v dietary advice)	26.2	38.8	32.7	36.0	44.4§	NS	Significantly lower for all drugs compared with dietary advice
Absolute risk reduction (metformin v dietary advice)	10.2	5.2	7.1	6.4	2.8§	NS
No needed to treat for 10 years to prevent one event (metformin v dietary advice)	10	19	14	16	36§	NS

^*Significant versus dietary advice. 

^†Significant versus both other groups. 

^‡Significant versus chlorpropamide, glibenclamide, or insulin group. 

^§These results are for the differences between the metformin and the chlorpropamide, glibenclamide, or insulin group. 

Contrary to expectations, treatment with sulphonylureas and insulin had no significant benefit on the occurrence of microvascular or macrovascular end points over 10 years in this obese population (table 2). Metformin also produced significant reductions in the aggregated diabetes related end points, all cause mortality, and stroke compared with the sulphonylureas and insulin.

With regard to the results of these two trials, one message seems to have been lost from many of the commentaries on the UK prospective diabetes study. That is, patients with type 2 diabetes seem to benefit not so much from the overall control of glucose but rather from taking metformin.

The study also raises an interesting point about haemoglobin A_1c, which to our knowledge has not been discussed in any detail. Haemoglobin A_1c concentration has been used for some years as a surrogate marker. Although it is a good marker of overall blood glucose control, it is not known whether reducing the haemoglobin A_1c concentration in patients with type 2 diabetes leads to an improved outcome. To establish a causal relation between a surrogate marker and a clinical outcome, it must be shown that a dose-response relation exists—that is, that a consistent, progressive clinical benefit is seen with progressive reductions in the surrogate marker.¹⁰ In the UK prospective diabetes study, changes in haemoglobin A_1c produced by drug treatment did not seem to correlate with treatment outcomes.

In study 33 an absolute reduction of 1% in haemoglobin A_1c concentration was observed with chlorpropamide, glibenclamide, or insulin over 10 years compared with diet alone; yet there was virtually no significant reduction in macrovascular outcomes.² In study 34, all the drugs given (metformin, chlorpropamide, glibenclamide, insulin) produced similar mean absolute differences in haemoglobin A_1c concentrations (about 0.6%) over the 10 years compared with diet alone, but only metformin produced significant reductions in clinically important macrovascular events.³ Not only did metformin reduce clinically important events compared with diet alone, it also produced reductions in some outcomes compared with other glucose lowering drugs. This shows that the studies in question were large enough, and of sufficient duration, to show macrovascular benefits. Clinicians and patients need to be aware of this and consider that either metformin may be conferring benefit independent of, or in addition to, blood glucose reduction, or that sulphonylureas and insulin may have an adverse effect on overall risk.¹¹ Further analysis of the study's findings may shed more light on this question.

Who inserts “spin” and why?

In summary, in contrast to the positive spin about overall glucose control applied by many editorialists, the data show that sulphonylureas and insulin produced only a small (3.2% absolute difference) reduction in an aggregate of clinical end points. In addition, these drugs produced no significant benefit on individual macrovascular end points in non-obese and obese patients with type 2 diabetes and no benefit at all in obese patients. However, metformin, which provided a similar level of glucose control to that of sulphonylureas and insulin in obese patients with type 2 diabetes, produced important (5-10%) absolute reductions or delays in clinically important end points (death, strokes, and myocardial infarctions).

Why were the results from these studies presented with such a positive spin on tight blood glucose control when the results seem to show a benefit of metformin over sulphonylureas and insulin? Are we so reluctant to give up old beliefs? A similar spin was found with the captopril prevention project, in which captopril was compared with diuretics and β blockers for the treatment of hypertension.¹² Although in general there were no differences in cardiovascular outcomes between the groups, patients taking diuretics and β blockers had a lower incidence of stroke (0.8% absolute difference) despite similar blood pressure reduction. If the reverse had been seen it is possible the researchers would have said something like “these results show that angiotensin converting enzyme inhibitors provide a unique benefit over other blood pressure lowering agents.” Instead, authors have tried to explain away the difference as being due to differences in baseline characteristics.

We believe that these cases illustrate the principle that interpretations of clinical trial results are often neither objective nor value-free. Rather, researchers, authors, and editors are highly susceptible to interpretive biases, including:

“We've shown something here” bias—that is, the researchers' enthusiasm for a positive result. It took 20 years to collect and analyse the UK prospective diabetes study data. To suggest that two of the three classes of drug used had little or no effect would have been a distinct anticlimax.

“The result we've all been waiting for” bias—that is, the clinical and scientific communities' prior expectations. It was widely believed in the 1980s and early '90s that the strict control of blood glucose concentrations was the raison d'être of the diabetologist and should be the principal objective of every well behaved patient.

“Just keep taking the tablets” bias—that is, the tendency of clinicians to overestimate the benefits and underestimate the harms of drug treatment. All the primary reports of the UK prospective diabetes study gave a relatively low emphasis to side effects (limited to hypoglycaemia and weight gain, with little discussion of the effect these had on patients and no mention of other adverse events). Side effects were presented as events per year, although the purported benefits were presented over 10 years.

“What the hell can we tell the public?” bias—that is, the political need for regular, high impact medical breakthroughs. Pressure from the press and patient support groups arguably drew staff from the British Diabetic Association, and perhaps even the trials' authors, into producing soundbites with a positive spin.

“If enough people say it, it becomes true” bias—that is, the subconscious tendency of reviewers and editorial committees to “back a winner.” The UK prospective diabetes study results were published in high quality, peer reviewed journals and were probably seen before publication by at least a dozen independent experts in either diabetes or research methodology. The writing—that the study was about to cause a sensation—was probably already on the wall, so it would have taken a brave and rebellious individual to be the first to jump off the bandwagon.

Looking back with the benefit of hindsight at how the UK prospective diabetes study results were presented and received at the time, we believe that this is a good example of the hidden biases inherent in the interpretation of randomised controlled trials. The relatively uncritical reception of the study by conference audiences, editorial committees, and the wider scientific community, could be an example of mass “groupthink”—a well described psychological phenomenon in which a group makes an overconfident and perhaps even irrational decision which it then defends fiercely against dissenting members, whose comments are subconsciously perceived as a threat to the group's own cohesion.¹³

We put it to the editors of medical journals that they should, in the interests of minimising interpretation bias, require investigators initially to present the results of clinical trials with a minimum of discussion so that individual clinicians and patients can decide if the results are clinically important. In addition, we suggest that editors should continue to provide space for readers to enter a discourse about the meaning and clinical importance of those results, and indeed they should actively stimulate discussion, perhaps by encouraging publication of dissenting views. Furthermore, when new evidence challenges old beliefs—let it.