Convincing evidence from controlled and uncontrolled studies on the lipid‐lowering effect of a statin

John Bavosi/Science Photo Library

Lipitor (atorvastatin) is one of the best‐selling pharmaceutical drugs of all time. This month's issue of the Cochrane Database of Systematic Reviews includes a new review by Wright, Adams, and Tsang concerning the effects of this statin. The review is big, and it is not like most of the others in the database.

The authors focus their attention on lipid levels, without considering subsequent clinical outcomes such as cardiovascular morbidity or mortality. Specifically, they aim to characterise the dose‐response relationship between atorvastatin and lipid levels such as LDL cholesterol, total cholesterol and triglycerides. While clearly an important question, this is an unusual focus for a Cochrane Review. Reviews on the effects of drugs typically address clinical outcomes such as morbidity, mortality, and quality of life. Indeed, authors of Cochrane Reviews are often encouraged to relegate biochemical measures to secondary outcomes, and they may even be discouraged from including them at all. Many Cochrane Reviews have a narrow focus in terms of the interventions they address, so we would not expect many reviews to provide a complete picture sufficient for decision making. The current review has a narrow focus in its restriction to lipid outcomes (and the additional outcome of withdrawal from the study due to adverse effects), and the results would likewise need to be placed in the context of the wider evidence base in relation to subsequent clinical outcomes.

In order to address their question, Wright and colleagues draw heavily on uncontrolled (non‐randomised) studies. This is again unusual, since to evaluate the intended effects of drugs, Cochrane Reviews are typically limited to randomised trials, aiming to minimising bias by comparing similar groups of patients receiving the drug and a control. This review is a mammoth piece of work: 254 studies involving over 33 thousand participants are included, evaluated, and synthesised; while 835 studies are carefully annotated for reasons why they are not included. The authors must be congratulated for their substantial endeavour. I estimate that I would have to scroll down at least the height of the Statue of Liberty to read the whole review on my computer monitor. After doing that I would probably have an aching finger. But should I expect to achieve the lipid level reductions concluded by the authors after taking atorvastatin, such as a reduction of 36% to 53% in LDL cholesterol for doses of 10 to 80 mg/day?

Were I faced with the protocol for this review, I must admit that I would be sceptical about the validity of the proposed approach. The protocol drew on an earlier piece of work seemingly published only as a conference abstract, looking at studies of another statin, Lescol (fluvastatin). In this earlier work it was observed that patients in placebo groups of randomised trials did not experience changes in their lipid levels, and also that patients in the fluvastatin groups of the randomised trials had similar reductions to patients in uncontrolled studies of fluvastatin. This finding was used to argue that placebo groups are not necessarily required for assessment of lipid changes in patients taking atorvastatin. Thus the review was open to simple uncontrolled studies in which a group of patients was measured before and after taking atorvastatin.

Would the finding for fluvastatin justify the approach taken for atorvastatin? The vast majority of the included studies in the atorvastatin review (205 out of 254) were uncontrolled studies, with the remaining 49 being randomised trials as we would normally expect to see in a review of this kind. This enabled the authors to compare findings between randomised and uncontrolled studies, and this is of course a particular strength of the review. The results echoed the earlier findings for fluvastatin. What the authors end up with is a review with a large amount of consistent evidence of a substantial reduction in lipid levels, demonstrating a convincing dose‐response relationship. The evidence would fare well if evaluated against the commonly used set of considerations for causality proposed by Austin Bradford Hill. Because of this, and the dose‐response relationship in particular, I am left in little doubt about the validity of the findings.

The Cochrane Collaboration has long recognised the value of non‐randomised studies for evaluating the effects of interventions. They are particularly important for looking at adverse effects, and are routinely used in evaluation of organisational and public health interventions. Recent discussions have reconsidered the role of non‐randomised studies in Cochrane Reviews: a series of papers from the Cochrane Non‐Randomised Studies Methods Group is currently being published in the journal Research Synthesis Methods, and the Collaboration's annual strategic discussion meeting concluded earlier this year that they should be considered more often than they are currently. Nevertheless, inclusion of non‐randomised studies needs to be done with great care. This single example of an apparently successful analysis of lipid measures should not be used as justification to implement the same approach widely. As we are fond of saying in The Cochrane Collaboration, more research is needed on the issue.

Critical to the reliable inclusion of non‐randomised studies in systematic reviews is an awareness that such studies may be at a high risk of bias, so it is important that tools are available to evaluate them appropriately. To this end, the Collaboration has funded the extension of its ‘risk of bias’ evaluation tool to address a variety of types of non‐randomised studies. This should help both authors and readers of Cochrane Reviews to consider a wider body of evidence critically, enabling them to make more informed clinical, policy, and personal decisions.

Acknowledgements

Thanks to Barney Reeves, Jackie Chandler, Toby Lasserson, and Shah Ebrahim for comments on an earlier version of this editorial.

Feedback on this editorial and proposals for future editorials are welcome.

End Notes

Declarations of interest

The author has completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available upon request) and declares (1) no receipt of payment or support in kind for any aspect of the article; (2) no financial relationships with any entities that have an interest related to the submitted work; (3) royalties from Wiley‐Blackwell for the book Introduction to Meta‐Analysis; an honorarium from The Cochrane Collaboration for teaching on methods for assessing bias; reimbursement for travel and expenses for Cochrane Board of Trustees meetings and meetings to discuss use of non‐randomised studies and other methodological issues; but no other financial relationships with entities that have an interest in the content of the article; and (4) that there are no other relationships or activities that could be perceived as having influenced, or giving the appearance of potentially influencing, what was written in the submitted work.