Skip to main content
NCBI home page
The BMJ logoLink to The BMJ
. 2003 Feb 22;326(7386):445. doi: 10.1136/bmj.326.7386.445

New BMJ policy on economic evaluations

Response of NHS Economic Evaluation Database Research Team

Dawn Craig 1,2,3, John Nixon 1,2,3, Nigel Armstrong 1,2,3, Julie Glanville 1,2,3, Jos Kleijnen 1,2,3, Michael Drummond 1,2,3
PMCID: PMC1125321  PMID: 12595389

Editor—We, the NHS Economic Evaluation Database Research Team, agree with Smith that economic evaluations should contain comprehensive reporting of both clinical effectiveness and economic analysis and that the BMJ is right to implement this new policy.1 How the clinical trial results (which inform the economic evaluation) are obtained is often paramount to the understanding and quality of the economic analysis conducted.2

Research reports are included and abstracted in full on the NHS Economic Evaluation Database (www.york.ac.uk/inst/crd)—if they explicitly report costs and clinical outcomes for an intervention and at least one comparator.3 However, to critique the method adopted in the effectiveness study underpinning the economic evaluation appropriately, our template requires information that is often omitted in the report of the economic evaluation. When the parent clinical study has been previously published elsewhere, we obtain the study and use that alongside the economic research when writing the abstract. The abstract on the database then provides information on sample selection, study design, method of analysis, and so on, with the fact that the relevant information is cited from the parent study. graphic file with name bmjlett.f2r.jpg

Adhering to published guidelines, such as those provided by the BMJ,4 should produce publications of the highest quality, but authors are still likely to feel the need to be selective in their reporting, given word limits. If authors are required to report more effectiveness data other crucial aspects of the economic evaluation might receive less attention. The focus for BMJ editors should be to ensure that reporting of both important components of economic evaluations receives appropriate attention from the authors.

If the policy results in full reporting of both clinical and economic results in one place—for example, two papers in one issue of the journal—this will constitute an improvement. If, however, the new policy results in the combination of clinical and economic results in one short paper, this may be a step backwards.

References

  • 1.Smith R. New BMJ policy on economic evaluations. BMJ. 2002;325:1124. doi: 10.1136/bmj.325.7373.1124. . (16 November.) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Hoffmann C, Stoykova B, Nixon J, Glanville J, Misso K, Drummond M. Do health-care decision makers find economic evaluations useful? The findings of focus group research in UK health authorities. Value Health. 2002;5:71–78. doi: 10.1046/j.1524-4733.2002.52109.x. [DOI] [PubMed] [Google Scholar]
  • 3.NHS Centre for Reviews and Dissemination. Improving access to cost-effectiveness information for health care decision-making: the NHS economic evaluation database. 2nd ed. York: University of York; 2001. . (CRD report No 6.) [Google Scholar]
  • 4.Drummond MF, Jefferson TO. Guidelines for authors and peer reviewers of economic submissions to the BMJ. BMJ. 1996;313:275–283. doi: 10.1136/bmj.313.7052.275. [DOI] [PMC free article] [PubMed] [Google Scholar]
BMJ. 2003 Feb 22;326(7386):445.

Will the BMJ return clinical trials if submitted without any economic results?

Tracy Roberts 1

Editor—The implications of the BMJ's new policy for economic evaluations are unclear.1-1

Firstly, a lag often exists between the clinical and economic results, making simultaneous submission difficult. Typically, clinicians are eager to disseminate important clinical results immediately. For example, the extracorporeal membrane oxygenation (ECMO) trial was among the first research projects to incorporate economic evaluation in its design from the outset. But the preliminary clinical results were written up and fast tracked to the Lancet before I was even employed to continue the economic evaluation.1-2 The economic evaluation was published in the BMJ years later, having required the clinical evidence in its analysis and appropriate sensitivity analyses and having undergone delay to publication.1-3

Would it have benefited anyone to withhold dramatic clinical results until the economic results were ready? Clinical results are often more generalisable to an international audience than the concurrent economic results. The limitations of any clinical information in the absence of economic evidence should be made explicit. The pertinent concern is surely to ensure relevant policy makers exercise restraint until the full information is available.

Secondly, no incentive is given in the BMJ policy for clinicians to change their practice. Presumably clinicians send results to the Lancet for higher impact factors and wider dissemination. If economists cannot persuade colleagues to submit the clinical paper alongside the economic paper to the BMJ, they will resort to submitting results to economic journals for which a different style for different specialist audiences would be required, ensuring even poorer dissemination to clinical audiences and policy makers.

Finally, your editorial emphasised strong support for keeping clinical and economic results together, and Smith told us to send “somebody else your clinical results and us your economic results, and we will send them back, politely.” May I therefore ask, politely, is the converse also true? Will you return clinical trials if submitted without any economic results?

References

  • 1-1.Smith R. New BMJ policy on economic evaluations. BMJ. 2002;325:1124. doi: 10.1136/bmj.325.7373.1124. . (16 November.) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 1-2.UK Collaborative ECMO (Extracorporeal Membrane Oxygenation) Trial Group. UK collaborative randomised trial of neonatal extracorporeal membrane oxygenation. Lancet. 1996;348:75–82. [PubMed] [Google Scholar]
  • 1-3.Roberts TE. Extracorporeal Membrane Oxygenation Economics Working Group on behalf of the Extracorporeal Membrane Oxygenation Trial Steering Group. Economic evaluation and randomised trial. BMJ. 1998;317:911–916. doi: 10.1136/bmj.317.7163.911. [DOI] [PMC free article] [PubMed] [Google Scholar]
BMJ. 2003 Feb 22;326(7386):445.

Economic evaluations should be judged on scientific merit

Alastair M Gray 1,2, Andrew Briggs 1,2, Philip Clarke 1,2

Editor—Health economists have been grateful for the BMJ's hitherto supportive stance towards the publication of economic evaluations. The proposed new policy not to publish economic evaluations unless also offered the clinical results is disappointing and misjudged.2-1

Firstly, this policy denies the fact that, although clinical and economic results from a trial are both components of an overall evaluation, they also have many differences, often including the funding agencies supporting them, the researchers, and the timescale over which they are performed and published. Perhaps most importantly, important trials are often prepared for an international audience, but economic evaluations usually relate to specific healthcare systems; large trials may generate the need for several country specific economic evaluations.

These differences justify researchers in choosing to submit clinical and economic results to different journals, and entitle journals to use different criteria when deciding whether to publish or reject. Consequently, as in other disciplines, research findings that are closely related and possibly interdependent often appear in different journals. That poses no great problem to readers, especially in the era of electronic publication championed by the BMJ.

Secondly, what is the likely effect of this policy? Researchers aim to publish where they judge they make most impact. Surely no one will forgo an opportunity to publish trial results in the Lancet simply because the BMJ will not then consider publishing an economic evaluation?

Smith's editorial included no positive proposals to make the BMJ a more attractive outlet for trial results. Instead, this policy will inevitably mean turning away well conducted empirical research—such as the economic analysis of the multicentre aneurysm screening study that occasioned this announcement2-2—on strictly non-scientific grounds. Arguably these are precisely the more scientifically important papers, leaving the BMJ with a greater preponderance of non-trial based economic analyses and data-free “think pieces.” This is hardly the route to improving the journal's impact on the adoption of new treatments or technologies.

Smith admits this new policy owes something to petulance but nevertheless defends it as reasonable. We think it is unreasonable and ask him to reconsider.

Footnotes

We are currently involved in economic analyses of several large trials whose clinical results have recently been published in other journals, including the Lancet. If adopted, this policy will deny us the opportunity to have our scientific research results considered for publication by the BMJ .

References

  • 2-1.Smith R. New BMJ policy on economic evaluations. BMJ. 2002;325:1124. doi: 10.1136/bmj.325.7373.1124. . (16 November.) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2-2.Multicentre Aneurysm Screening Study Group. Multicentre aneurysm screening study (MASS): cost effectiveness analysis of screening for abdominal aortic aneurysms based on four year results from randomised controlled trial. BMJ. 2002;315:1135–1141. doi: 10.1136/bmj.325.7373.1135. . (16 November.) [DOI] [PMC free article] [PubMed] [Google Scholar]
BMJ. 2003 Feb 22;326(7386):445.

Economic evaluations are often based on many studies

Charles Normand 1

Editor—I understand the reasons for the new policy on publishing economic evaluation studies,3-1 but it is not clear how this will apply to many of the best evaluations that are based on reviews of many randomised controlled trials and other clinical studies, and use modelling to assess outcomes and cost effectiveness. There is no reason to exclude such studies.

If the new policy is to work it is important also for the BMJ to ensure that its processes of review and decision making are joined up in terms of the different components of studies. Too often in the past when pairs of papers were submitted or when a single paper reported the overall results of a study the reviewing of the economics has been weak. Smith's editorial raises the point that publishing clinical outcomes without economic ones is really incomplete evaluation. I look forward to results of high quality clinical trials being rejected for want of a proper economic evaluation.

References

BMJ. 2003 Feb 22;326(7386):445.

Will the Lancet play ball?

Christopher J McCabe 1,2, Jennifer Roberts 1,2

Editor—The debate ensuing from the BMJ's new policy on publication of economic evaluations is interesting.4-1,4-2 Smith made several important practical points, but we believe that some broader strategic issues still remain.

Thankfully, major research funders in the United Kingdom increasingly require that economic evaluation be an integral part of the design of a clinical trial. In many ways the BMJ's decision is the natural extension of this philosophy. The risk is, however, that unless other major journals follow suit the policy will damage the dissemination of cost effectiveness information.

The collaboration between clinicians and health economists is often delicate. The pressures that this new policy will place on this relationship will have implications for long term cooperation. Given the pressure to publish rapidly in high impact journals to secure long term funding, the interests of clinical and economic researchers will not always coincide.

An immediate effect of this policy for those of us participating in multidisciplinary research is the need to agree publication strategies at the outset of a project. It may be that a process similar to the Lancet's protocol pre-approval could facilitate these discussions by providing confidence that high quality clinical trials including an economic evaluation will be acceptable to the major journals.

Unless all researchers accept the need for the simultaneous publication of clinical and economic results, cost effectiveness information may be confined to more specialist journals, which are rarely seen by the clinical community. The best solution would be for the major journals to agree that clinical trials designed to inform policy decisions must include a high quality economic evaluation. We would be interested to know the Lancet's thoughts on this issue.

Footnotes

It is important for our research careers to be able to publish in high impact journals.

References

BMJ. 2003 Feb 22;326(7386):445.

Editor's clarifications

Richard Smith 1

Editor—We thank everybody who responded to our proposal to consider for publication economic evaluations that accompany clinical papers only if we are sent both the economic and the clinical papers. More respondents are in favour than against, but people have raised important questions that we must answer.

This is a further clarification of our policy.

(1) If both the clinical and the economic paper are submitted to us we might accept one and not the other.

(2) We will be willing to consider for publication economic papers that are submitted some time after the clinical paper if the clinical paper was also submitted to us. It will not matter if we decided not to publish the clinical paper. We might still be willing to publish the economic paper.

(3) We will not reject clinical papers if they are not accompanied by an economic evaluation. There would be a logic to such a policy, but we are first and foremost a clinical journal.

(4) We will be willing to consider either papers that combine clinical and economic results or pairs of papers. Pairs of papers will usually be better. Our ELPS (electronic long, paper short) policy means that we can publish long papers on bmj.com—longer than 2000 words.5-1 We prepare the shorter version for the paper edition of the BMJ. Authors approve it before publication. If we take two papers then we will publish them together.

(5) We will continue to consider for publication economic papers—perhaps modelling papers—that are not related to particular clinical papers.

References

  • 5-1.Müllner M, Groves T. Making research papers in the BMJ more accessible. BMJ. 2002;325:456. [Google Scholar]

Articles from BMJ : British Medical Journal are provided here courtesy of BMJ Publishing Group

RESOURCES