Pitfalls of pharmacoepidemiology

Three months ago a paper in the BMJ analysed the incidence of venous thromboembolism before and after the warning from the UK Committee on Safety of Medicines about third generation oral contraceptives.¹ Using computer records of general practitioners, Farmer et al found that the incidence among pill users had not dropped, and they concluded that their findings were not compatible with a doubling of risk in women using third generation contraceptives (compared with older preparations). Their paper received wide publicity because it called into question an emerging consensus about this issue.²

This week's BMJ contains another analysis of computer records from British general practice, conducted by a group in Boston (p 1190).³ Jick et al found that, both before and after the warning in October 1995, the risk of venous thromboembolism in women using third generation oral contraceptives was about twice that in users of preparations containing levonorgestrel. Moreover, fewer cases occurred after the warning than would have been expected if the prescribing of oral contraceptives had not changed.

What is remarkable is that these two studies, reporting opposite conclusions, both used the same General Practice Research Database.⁴ How can we explain their discrepant findings? Part of the explanation must lie in the methods used. Farmer et al did not use all the information they held about the exposure and risk factors of individuals, presenting instead a time correlation study.¹ Users of any combined oral contraceptive were counted in the same way. As one correspondent observed, “Simple analyses have rhetorical power that exceeds their scientific merit.”⁵ Jick et al replicated this approach for the purpose of comparison but also presented cohort and nested case-control analyses.³ These uncovered important confounding factors: the reduction in use of third generation oral contraceptives mainly involved young women (who are at low risk of venous thromboembolism), and doctors also tended to avoid prescribing such contraceptives for obese women or smokers.

The first study involved little attempt to control for confounding.¹ There was adjustment for age, but even this may not have been fully adequate. In calculating the number of cases expected after October 1995, the authors stated that they standardised for age by using the data on overall use from the two periods. A subsequent sentence suggested that this referred to use of any combined oral contraceptive, rather than specific types. If so, there was no allowance for the fact that the switching from third generation oral contraceptives to other formulations was mainly by young women—who were at the lowest risk of venous thromboembolism.

Jick et al offer several other explanations for their different findings.³ Doctors may be tempted to discount these two studies, concluding that they cancel each other out in a “tit for tat” manner. This would be unwise, for only one of them can have the right answer. The elegant design and analysis of the new study mean that it could be the most important paper yet published on this vexed subject. As well as answering the previous report, it provides vital evidence on several controversial matters—including the increased risk in first time users of oral contraceptives, the role of risk factors such as obesity and smoking, and the irrelevance of prior switching of oral contraceptive preparations.³

The two groups have been producing conflicting results on this subject for several years,⁶,⁷ and Farmer et al have also sought to explain the differences.⁸ Surely it is time for the Medicines Control Agency, which now owns the General Practice Research Database, to conduct a thorough investigation. The whole stand-off is damaging to the credibility of pharmacoepidemiology in general and the General Practice Research Database in particular. The latter is a research tool of global importance.⁹ The Boston group and its collaborators have used it in over 100 publications, including studies on appetite suppressants and heart valve disorders,¹⁰ analgesics and gastrointestinal bleeding,¹¹ and antidepressants and suicide.¹² Such research can help lay to rest false alarms about drug safety.¹³ It can also disclose unexpected benefits of medicines, such as the possibility that statins may reduce the risk of fractures.¹⁴

The Medicines Control Agency is seeking to make the database more widely accessible for research and analysis. This seems desirable, but it also presents a challenge to researchers to be as rigorous as possible in the use they make of it.

There is a further, separate, problem raised by making the database more widely available, namely the risk of publication bias. It is notable that a third study on oral contraceptives and thromboembolism using the same database was conducted on behalf of a pharmaceutical company, but this has never seen the light of day. Pharmacoepidemiology is a powerful tool that can benefit patients and the public health, but only if it is used appropriately.

  Editorial footnote

We did poorly with our peer review of the study by Professor Farmer and others. We took six months to produce our initial decision and then, embarrassed by our slowness, accepted the revised paper without sending it back to the reviewer and our statistician. We should have done, particularly because the number of participants in the study was reduced by about a third. Although the authors explained clearly why the number of participants was reduced and we accepted the explanation, the paper should with hindsight have been treated as a new one. We have now sent the paper back to our statistician, and she is worried both about the adequacy of the age adjustment and the power of the study to detect an increase in risk as big as 50%.

We apologise to the authors, the reviewers, and readers for our performance in reviewing and publishing the study. A fuller explanation of our processes and our statistician's view on the published study is available on bmj.com.—Richard Smith, editor, BMJ

General practice p 1190