The ‘Smallwood report’: method or madness?

In October 2005, a report was published entitled The Role of Complementary and Alternative Medicine in the NHS.¹ Its lead author is the economist Christopher Smallwood. The purpose of the report was:

‘… to examine evidence relating first to the effectiveness and then to the associated costs of mainstream complementary therapies.’

In the following, I will try to briefly summarise some of its findings and shortcomings based on my personal involvement with this project.

THE REMIT

In March 2005, I was interviewed by a team of ‘Freshminds’, a London-based research consultancy. They explained to me that The Prince of Wales' Foundation for Integrated Health had commissioned them to conduct a study of the cost-effectiveness of complementary/alternative medicine (CAM). Essentially this was to be based on a series of interviews with a range of experts, mostly outspoken proponents of CAM. At my second encounter, in July 2005, (this time in the presence of Mr Smallwood), the remit had changed fundamentally. The aim, I was told, was now to review the evidence related to the cost-effectiveness of five forms of CAM. I mentioned that we had just finished such a project for the World Health Organisation,² but they showed no interest. Their review was to be supplemented with ‘case studies’ and interviews. Mr Smallwood's plan was to submit it to UK health ministers in the hope to change health policy in Britain. He also explained that the report was no longer commissioned via the ‘Foundation’, but by the Prince of Wales personally. He informed me that the bulk of the funding was coming from Dame Porter. It appeared that this change of remit, about half way into the 9-month project, was detrimental to the credibility of the entire exercise.

INACCURACIES

There are too many inaccuracies to mention all. I will therefore only mention three obvious mistakes:

• ‘… a weekly course of St John's Wort cost just 82p’ and this expense, it was claimed, compares well with the £13.82 for conventional antidepressants. The truth is that, using a quality extract, which has been tested in clinical trials at a dose used in such studies, the weekly cost for St John's wort in the UK would range from £8.40 to £16.80.

• Chondroitin, glucosamine and melatonin are listed as herbal medicines which, of course, they are not.

• Phytodolor® is repeatedly recommended for the treatment of UK patients suffering from arthritis pain — yet this German preparation is not available in the UK.

MISLEADING INFORMATION

Much of the information in the report is grossly misleading. Again, a few examples must suffice.

• ‘… manipulative therapies offer advantages over conventional treatments for lower back pain, particularly acute pain’ (page 11). A Cochrane review, however, concludes that ‘there is no evidence that spinal manipulative therapy is superior to other standard treatments …’.³

• Viral infections such as the common cold are, according to Smallwood, treatable with Echinacea extract. The most conclusive series of trials, however, shows no benefit in terms of treatment or prevention.⁴

• Homoeopathy is recommended as a cost-saving treatment for asthma (page 16). A Cochrane review, however, fails to demonstrate efficacy.⁵

• The widespread use of homoeopathy, it is claimed, would save $4 billion on the national prescription drugs bill (page 16). This notion is based not on controlled data but on a ‘case study’. About a dozen systematic reviews/meta-analyses fail to confirm that homeopathic remedies are more than a placebo.^6,7

• ‘… economy-wide benefits running into the hundreds of millions of pounds could result if a significant reduction in the time off work associated with lower back pain alone could be achieved as a result of the wider application of CAM therapies’ (page 17). No conclusive data are provided to suggest that CAM could reduce absenteeism due to back problems.

• ‘… a number of CAM treatments offer the possibility of significant savings in direct health costs’ (page 17). The best evidence available to date does not support this statement.^2,8

• ‘There is no consensus on what constitutes the definite set of research methods for CAM’ (page 26). The majority of methodologists have long agreed that certain methods are best suited to answer certain CAM research questions.⁹

• ‘… there is a relatively large literature on the costs and benefits of homoeopathy’ (page 56). The articles cited in support of this statement are methodologically so weak that they do not lend themselves to meaningful conclusions. At present, there are no rigorous UK studies on this topic.⁸

• ‘… if 4% of GPs were to … [offer homeopathy] … a large saving (£190 million) would result’ (page 56). No conclusive data are provided to support this statement.

• Milk thistle is depicted as an effective treatment of liver problems. Yet a Cochrane review fails to demonstrate efficacy.¹⁰

• The section with interviews excludes at least one interview conducted by ‘Freshminds’ with one expert who happened to be critical about CAM (it also excludes my interview but this is because I withdrew my cooperation when I became convinced that this was no honest attempt at finding the truth).

EFFECTIVENESS GAPS

The report points out that the evidence presented indicates that ‘many of the most effective CAM therapies correspond to recognised ‘effectiveness gaps’ in NHS treatment, which suggests that they may have the potential to make an important contribution to the delivery of health in the UK’. It is true that CAM is used mostly for indications for which conventional medicine is not optimally effective. This foremost reflects patients' hope and CAM's claims of effective CAM therapies. It does not, however, demonstrate that CAM can be used cost-effectively in these areas. The report lacks a critical discussion of the concept and applicability of ‘effectiveness gaps’ in CAM.

REACTIONS TO THE REPORT

Only days after the publication of the ‘Smallwood report’, Prince Charles was quoted saying that he did not promote CAM ‘because of some self-indulgent pet projects, or because of some half-baked obsession with unsubstantiated quackery’.¹¹ Yet, this is precisely how many rational thinkers must view initiatives like the ‘Smallwood report’. Richard Horton, editor of the Lancet, for instance, did not mince his words: ‘Let's be clear: this report contains dangerous nonsense’.¹² Unsurprisingly perhaps, politicians would like to have it both ways. Peter Hain, Secretary of State for Wales and Northern Ireland and a staunch promoter of CAM; said, on 12 October 2005, in his speech to the Foundation for Integrated Health, that he ‘applauds’ its initiatives not least because ‘the growing popularity of such treatments is ultimately down to the fact that they deliver real benefits’. On the other hand, he also stressed that he ‘would certainly never advocate the squandering of public money on so-called treatments that have no proven benefits …’.¹³

CONCLUSION

The ‘Smallwood report’ is one of the strangest examples of an attempt to review CAM that I have ever seen. One gets the impression that its conclusions were written before the authors had searched for evidence that might match them. Both Mr Smallwood and the ‘Freshminds’ team told me that they understand neither health care nor CAM. Mr Smallwood stressed that this is positive as it prevents him from being ‘accused of bias’. My response was that ‘severely flawed research methodology almost inevitably leads to bias’.