Reports of clinical trials should begin and end with up-to-date systematic reviews of other relevant evidence: a status report

Abstract

Objective Scientific and ethical justification for new clinical trials requires them to have been designed in the light of scientifically defensible assessments of relevant previous research. Reliable interpretation of the results of new clinical trials entails setting them in the context of updates of the reviews upon which they were deemed scientifically and ethically justifiable. We have shown previously that most reports of randomized trials published in five general medical journals in May 1997 and in May 2001 failed to set their results in the context of the findings from similar research. In the current study, we assess whether there had been progress in this respect in 2005 and also investigate the extent to which reports begin by referring to systematic reviews providing the justification for the new research reported.

Design Assessment of the Introduction and Discussion sections in all reports of randomized trials published during May 2005 in five general medical journals.

Setting Reports of randomized trials in five general medical journals.

Participants Annals of Internal Medicine, BMJ, JAMA, Lancet and New England Journal of Medicine.

Interventions None.

Main outcome measures The inclusion or mention of one or more systematic reviews in the Introduction or Discussion section of each report assessed.

Results We found 18 reports of randomized trials. The Introduction sections referred to systematic reviews in five (27%) of these reports. None of the Discussion sections of the 15 reports of trials that were not the first published trials to address the question studied placed the results of the new trial in the context of an updated systematic review of other research. Although reference was made to relevant systematic reviews in five of these 15 reports, there was no integration—quantitative or qualitative—of the results of the new trials in an update of these reviews. In the remaining ten reports there was no evidence that any systematic attempt had been made to set the new results in the context of previous trials.

Conclusions There is no evidence of progress between 1997 and 2005 in the proportion of reports of trials published in general medical journals which discussed new results within the context of up-to-date systematic reviews of relevant evidence from other controlled trials. Although the proportion of trials referring to systematic reviews in Discussion sections has increased, the majority of reports continued to fail even to do this. Similarly, most researchers appear not to have considered a systematic review when designing their trial. Researchers and journal editors do a disservice to the interests of the public and others involved in healthcare decision-making by acquiescing in this situation.

INTRODUCTION

Scientific and ethical justification for new clinical trials requires that they be designed in the light of scientifically defensible assessments of relevant previous research.¹ Then, reliable interpretation of the results of new trials entails setting them in the context of an updated systematic review of similar studies.²

In May 1997³ and again in May 2001⁴ we found and assessed 26 and 33 reports, respectively, of randomized trials in Annals of Internal Medicine, BMJ, JAMA, Lancet and the New England Journal of Medicine. We identified reports of apparently similar trials for 55 of these index trials, yet in only two were new results placed in the context of up-to-date systematic reviews of other relevant studies. In seven reports, systematic reviews were referred to in the Discussion sections of the reports, but in the other 46 reports no apparently systematic attempt had been made to set the results of the new trials in context. Thus, in both of our previous samples of trials, only a small proportion of the reports provided sufficient information to assess what contribution the new results had made to the totality of information, and thus permit reliable interpretation of their significance.

We repeated our studies in May 2005 to assess whether there had been any detectable improvement. In this study, as in the previous two, it was not our aim to assess the overall quality of the Discussion sections of trial reports, but simply to assess how well they had placed the results of the new trials in the context of other relevant research. Furthermore, we now also investigated the extent to which reports began by referring to systematic reviews to make clear the scientific and ethical justification for the design and conduct of the new research.

METHODS

Eligibility

A report was eligible for inclusion as a ‘trial’ if it met the following three criteria:

• It was published during May 2005 as a full report or paper (that is, not in the editorials, news, research letters, short reports or correspondence sections of the journals), in Annals of Internal Medicine, BMJ, JAMA, Lancet, or the New England Journal of Medicine.

• On the basis of information in the report, ‘the individuals (or other units) followed in the trial were assigned prospectively to one of two (or more) alternative forms of health-care using random allocation or some quasi-random method of allocation (such as alternation, date of birth, or case record number)’—that is, randomized and quasi-randomized trials, as defined by The Cochrane Collaboration.

• The report was principally concerned with the outcomes studied in the trial.

Identification of eligible reports and assessment of the Introduction and Discussion sections of the reports

We each searched the relevant issues of the journals in different random orders. Any report judged to be eligible by at least one of us was considered for inclusion.

We independently assessed the Introduction section of each eligible report to decide whether it referred to a systematic review (done by the authors or other people) that could have been used in planning the trial. We did not attempt to identify whether relevant systematic reviews were available at the time of planning those trials that did not refer to such a review in their Introduction. To do so would have required more knowledge than was available about when each trial was planned. Furthermore, the nonavailability of a review by others should not preclude the researchers from performing their own systematic review and, if they had done so, we might expect that they would have described this in their Introduction section.

We also independently assessed the Discussion section of each eligible report to decide whether an attempt had been made to integrate the results of the new trial within a systematic review, either qualitatively or quantitatively.

If a trial claimed to be the first or only trial of a topic, we searched for similar trials in the Cochrane Central Register of Controlled Trials (CENTRAL). We did not systematically search for all such trials or judge whether there was sufficient similarity between the new trial and other trials to combine them in a formal meta-analysis. We simply tried to find studies that might be considered for inclusion in a systematic review of the topic.

We resolved disagreements by discussion.

RESULTS

Eighteen reports of randomized trials (listed in Appendix A) were identified in the issues of these five journals published in May 2005. We did not include two economic analyses based on randomized trials.

Systematic reviews were referred to in the Introduction sections of five reports, and these could have been used in planning the newly reported trials. Reports of the remaining 13 trials did not provide any information on whether they had used systematic reviews done by themselves or other people to justify or inform the design of the index trial, or whether they had searched for systematic reviews.

In five reports, the authors claimed that their study was the only one to have addressed the question concerned, but we found apparently similar trials for two of these. None of the 15 reports for which there appear to be similar trials contained a discussion of the trial's results in the context of an up-to-date systematic review of earlier trials. Reference was made to relevant systematic reviews in the Discussion sections of five reports, but without integrating the results of the new trial—either qualitatively or quantitatively—into an update of the review. Four of these five had referred to a systematic review in their Introduction sections, while one had not done so. The fifth report in which a systematic review was referred to in the Introduction was categorized as a first and only trial when we assessed its Discussion section.⁵ It was a report of the 30-year follow-up of the first trial of a treatment, and none of the subsequent trials are old enough to evaluate this outcome. In ten reports, there was no evidence that any systematic attempt had been made to set the results of the new trials in context (Table 1).

Table 1.

Classification of Discussion sections in reports of RCTs published in May 1997, May 2001 and May 2005 in five general medical journals

Classification	May 1997 (n=26)	May 2001 (n=33)	May 2005 (n=18)
First trial addressing the question	1	3	3
Contained an updated systematic review integrating the new results	2	0	0
Discussed a previous review but did not attempt to integrate their results	4	3	5
No apparent systematic attempt to set the results in the context of other trials	19	27	10

DISCUSSION

There is no evidence of progress between 1997 and 2005 in the proportion of reports of trials published in general medical journals which discussed new results within the context of up-to-date systematic reviews of relevant evidence from other controlled trials. Although the proportion of trials referring to systematic reviews in Discussion sections has increased, the majority of reports continued to fail even to do this. Similarly, most researchers appear not to have considered a systematic review when designing their trial.

It is now a decade since the first CONSORT statement required that data from a new trial should be interpreted ‘in the light of the totality [emphasis added] of the available evidence’,⁶ and more than 40 years since Austin Bradford Hill suggested that the structure of a scientific paper could usefully be conceptualized in terms of four questions: ‘Why did you start? What did you do? What answer did you get? And what does it mean anyway?’.⁷ This is reflected in a common structure for scientific reports (sometimes referred to by the acronym ‘IMRaD’) comprising an ‘Introduction’; a description of the ‘Materials and Methods’; the ‘Results’; and a ‘Discussion’ of the findings. The Introduction should make clear why the study was worth starting and the Discussion should indicate the contribution of the new findings to the evidence available at the time of reporting.

The expectation that the results of a new randomized trial will be reported in the context of an up-to-date systematic review of earlier trials does not imply that the Discussion section of every report of a randomized trial should contain a full account of the material, methods and findings of such a review. The technology already exists to enable a brief review to be included in the Discussion section, with links to a relevant, up-to-date systematic review published elsewhere. And, now, with 3000 Cochrane reviews published in The Cochrane Library and protocols published for 1600 more, the availability and ready accessibility of systematic reviews has never been greater.

Randomized trials should be a key piece of the evidence for the public and others making decisions about health care, but to serve that purpose they need to be designed and reported in the light of other similar research. Systematic reviews provide the means to do this: they reduce unwanted duplication of research, help ensure that new research builds on lessons from earlier research, and place the findings of the new research in proper context.

In our three studies over the last eight years, we have repeatedly shown that the results of new randomized trials published in five general medical journals have not been presented within the context of updated systematic reviews of other relevant studies (Table 1). We have searched for but have not found any other reports of empirical research assessing the extent to which this issue has been addressed. We would welcome the conduct and reporting of such research.

In 2002, when we reported the second of our three studies, we wrote the following:

‘Because our expectations imply radical changes in the way that research is done and reported, we expect that not all researchers, journal editors, or publishers will agree with them. However, science is cumulative, and everyone, including the public, has a right to expect that this principle will be reflected more effectively in the way that science is conducted and reported. We feel that this imposes a duty on researchers to present their results in proper context and on journal editors to require them to do so.’⁴

A few years on, the findings we present here show that editors and authors—in these five high impact journals, at least—continue to fail to serve the needs of those who wish to use the results of randomized trials to make decisions about health care. Others have also shown how researchers designing new trials have not made proper use of systematic reviews, even where these are known to exist.⁸

Some years ago, the BMJ took the ground-breaking step of including a summary with each report of new research to show what is already known on a topic and what the new study adds. Unfortunately, this has not been followed by many other journals, nor extended to provide links to the evidence, such as systematic reviews, upon which these summaries are based. On a positive note, though, we are heartened to see that things might be changing in at least one of the journals we studied, with the announcement by the Lancet in mid 2005 that it will require all reports of clinical trials to include a clear summary of previous research findings and to explain how the new trial's findings affect this summary.⁹ Examples of this policy's implementation have appeared.¹⁰