Converting systematic reviews to Cochrane format: a cross-sectional survey of Australian authors of systematic reviews

Abstract

Background

Despite the growing reputation and subject coverage of the Cochrane Database of Systematic Reviews, many systematic reviews continue to be published solely in paper-based health care journals. This study was designed to determine why authors choose to publish their systematic reviews outside of the Cochrane Collaboration and if they might be interested in converting their reviews to Cochrane format for publication in the Cochrane Database of Systematic Reviews.

Methods

Cross-sectional survey of Australian primary authors of systematic reviews not published on the Cochrane Database of Systematic Reviews identified from the Database of Abstracts of Reviews of Effectiveness.

Results

We identified 88 systematic reviews from the Database of Abstracts of Reviews of Effectiveness with an Australian as the primary author. We surveyed 52 authors for whom valid contact information was available. The response rate was 88 per cent (46/52). Ten authors replied without completing the survey, leaving 36 valid surveys for analysis. The most frequently cited reasons for not undertaking a Cochrane review were: lack of time (78%), the need to undergo specific Cochrane training (46%), unwillingness to update reviews (36%), difficulties with the Cochrane process (26%) and the review topic already registered with the Cochrane Collaboration (21%). (Percentages based on completed responses to individual questions.) Nearly half the respondents would consider converting their review to Cochrane format. Dedicated time emerged as the most important factor in facilitating the potential conversion process. Other factors included navigating the Cochrane system, assistance with updating and financial support. Eighty-six per cent were willing to have their review converted to Cochrane format by another author.

Conclusion

Time required to complete a Cochrane review and the need for specific training are the primary reasons why some authors publish systematic reviews outside of the Cochrane Collaboration. Encouragingly, almost half of the authors would consider converting their review to Cochrane format. Based on the current number of reviews in the Database of Abstracts of Reviews of Effectiveness, this could result in more than 700 additional Cochrane reviews. Ways of supporting these authors and how to provide dedicated time to convert systematic reviews needs further consideration.

Background

The Cochrane Collaboration is an international organisation dedicated to preparing, maintaining and promoting the accessibility of systematic reviews of the effects of health care interventions [1]. These systematic reviews are published in the Cochrane Library, a regularly updated collection of evidence-based practice databases intended to provide high quality information to people providing and receiving care [2].

Systematic reviews prepared by members of the Cochrane Collaboration (referred to as Cochrane reviewers) are published in the Cochrane Database of Systematic Reviews (one of databases included in the Cochrane Library). The Cochrane Database of Systematic Reviews (CDSR) is a rapidly expanding collection of full-text summaries of the evidence. Since the Cochrane Library is still a relatively new publication, the 1500 Cochrane reviews only provide answers to about 10 to 15 per cent of all health care questions related to the effectiveness of interventions.

As a way of improving the usefulness of the Cochrane Library and complementing the coverage of topics in the Cochrane Database of Systematic Reviews, the Library also includes details of published systematic reviews prepared by people outside of the Cochrane Collaboration. The Database of Abstracts of Reviews of Effectiveness (DARE) includes structured abstracts of nearly 3000 such systematic reviews from around the world, which have been identified and quality assessed by researchers at the National Health Service (NHS) Centre for Reviews and Dissemination in the United Kingdom [3].

Several thousand health professionals, researchers and consumers from around 80 countries contribute to the work of the Cochrane Collaboration. Those involved in preparing Cochrane reviews are supported by one of about 50 collaborative review groups worldwide. These groups typically focus on particular conditions or group of conditions, such as tobacco addiction or acute respiratory infections. They provide specialist help, oversee the editorial process and ensure the reviews meet the high standards required for publication in the Cochrane Database of Systematic Reviews.

Cochrane centres, which have a role in promoting and co-ordinating the work of the Cochrane Collaboration at a regional level, are responsible for training Cochrane reviewers. As one of thirteen Cochrane centres worldwide, the Australasian Cochrane Centre provides training to Cochrane reviewers in Australasia and South East Asia [4]. The aim of the Centre's training program is to increase the number of high-quality systematic reviews published in the Cochrane Database of Systematic Reviews.

Despite the growing reputation and subject coverage of the Cochrane Database of Systematic Reviews, many systematic reviews continue to be published in paper-based health care journals. Some of these reviews follow the methodological guidelines recommended by the Cochrane Collaboration, but many do not. Recent studies have shown that Cochrane reviews have greater methodological rigour and are updated more frequently than reviews published solely in paper-based journals [5,6].

To identify why authors might choose to publish their reviews outside of the Cochrane Collaboration we undertook a survey of Australian authors of non-Cochrane systematic reviews. We also hoped to gauge whether authors were interested in converting their reviews to Cochrane format, and if so, how the Australasian Cochrane Centre could facilitate this process.

Methods

In September 2001, at the request of the Australasian Cochrane Centre (ACC), the NHS Centre for Reviews and Dissemination created a subset of the Database of Abstracts of Reviews of Effectiveness, listing all Australian primary authors. Eighty-eight reviews were identified out of the approximately 2200 reviews indexed in DARE. Contact information for 64 of the authors was obtained from the original publication. A further nine authors were identified by an earlier ACC survey, leaving 15 without contact information.

Each title from the DARE subset was then compared with the Cochrane Database of Systematic Reviews to see if the same or another author had published a systematic review on the same topic in CDSR [7].

A survey canvassing potential barriers to conducting a Cochrane review and seeking information about the potential conversion of the existing review to Cochrane format was developed by the staff of the ACC, with comment invited from all other Cochrane centres [See Additional File 1 for the complete survey]. The 73 available Australian primary authors were surveyed, 69 by post and four by email. A follow-up questionnaire was sent to non-responders three weeks after the original mailing.

Results

Survey response

Overall response rate after the two mailings was 63 per cent (46/73). Excluding the 29 per cent (21/73) of surveys returned as undeliverable, the response rate was 88 per cent (46/52) [Table 1].

Table 1.

Survey Response

	First Mailing 9 October 2001				Second Mailing 29 October 2001
Format	Number	Response	Un-deliverable	Response Rate of Deliverable	Number	Response	Un-deliverable	Response Rate of Deliverable	Total Response	Overall Response Rate of Deliverable
Post	69	26	10	44%	34	17	11	74%	43 (62%)	90%
Email	4	3	0	75%	0	0	0	n/a	3 (75%)	75%
Total	73	29	10	46%	34	17	11	74%	46 (63%)	88%

Ten authors replied, but did not complete the survey, citing the following reasons:

1. Two authors replied that their reviews had been published in the Cochrane Database of Systematic Reviews in addition to the original journal citation (publication in CDSR occurred after the date we conducted our searches).

2. One author's review was written with an aim of demonstrating that a published Cochrane review overstated treatment effect.

3. Two reviews concerned meta-analysis of studies of diagnostic tests, outside of the realm of Cochrane reviews.

4. One author felt that the review (from the early 1990s) was too old to consider conversion.

5. One respondent stated that the Cochrane editorial process took too long for students.

6. Three people left the survey blank.

The responses of these ten authors were not included in the analysis.

Survey analysis

Thirty-six valid surveys were available for analysis, with summary statistics generated using SPSS for Windows software, release 10.0.5.

All but one respondent had heard of the Cochrane Collaboration.

Reasons for undertaking systematic review outside of the Cochrane Collaboration

Seventy-eight per cent (18/23) did not undertake a Cochrane review due to lack of time available to complete a full Cochrane review. Almost half (10/21 and 11/24, respectively) identified a tight timeline and the need to undergo specific Cochrane training as barriers to completing Cochrane reviews. Thirty-six per cent of respondents (8/22) stated that they were unwilling to update their review. Twenty-six per cent (6/23) did not undertake a Cochrane review due to difficulty with the Cochrane system, and 21 per cent (4/19) found that the topic had already been registered [Table 2].

Table 2.

Results

	Yes	No	Total Responses	Potential Responses*
Why did you not undertake a Cochrane review?
Lack of time	18	5	23	36
Other issues	17	5	22	36
Need to undergo specific Cochrane training	11	13	24	36
Timeline too tight	10	11	21	36
Not willing to update review	8	14	22	36
Difficulty with Cochrane system	6	17	23	36
Topic already registered	4	15	19	36
Would you consider converting your review to Cochrane format?	17	19	36	36
Would you object to having someone else convert your review?	4	24	28	36
Would you like to be consulted on choice of author?	14	9	23	36
If you would convert your review, what resources would you need?
Dedicated time	14	1	15	17
Learning the Cochrane system	12	2	14	17
Assistance with updating	10	3	13	17
Statistical support	10	4	14	17
Data entry	9	4	13	17
Financial support	8	3	11	17
Institutional support	8	4	12	17
Obtaining articles	8	5	13	17
Obtaining translation of articles	7	5	12	17
Other support	6	4	10	17
Project management	6	6	12	17
If you are not interested in converting your review, what resources would lead you to reconsider?
Dedicated time	6	6	12	19
Assistance with updating	4	7	11	19
Financial support	3	7	10	19
Statistical support	3	8	11	19
Obtaining articles	2	8	10	19
Learning the Cochrane system	2	9	11	19
Institutional support	1	8	9	19
Obtaining translation of articles	1	8	9	19
Other support	1	8	9	19
Data entry	1	9	10	19
Project management	0	9	9	19

*Not all respondents chose to answer each question.

Qualitative comments echoed the quantitative replies. Four people stated that their review was too old to convert. Time constraints affected three respondents. One of these three respondents expressed frustration that in his case the Cochrane system entailed three separate instances of peer review, adding tremendously to the time until publication. Three authors researched topics with insufficient numbers of randomised trials, and believed that the Cochrane Collaboration would not allow other forms of evidence. Four respondents stated that they were unaware that their topic was suitable for a Cochrane review or that they could perform a Cochrane review without an invitation. Finally, one author (at the beginning of an academic career) was afraid of committing to the Cochrane policy of updating reviews before determining a long-term research focus.

Possible review conversion

Seventeen out of 36 respondents would consider converting the review to Cochrane format. Of these seventeen, helpful resources aiding conversion include: dedicated time (14 responses), learning the Cochrane system (12), assistance with updating (10), statistical support (10), data entry (9), financial support (8), institutional support (8), obtaining articles (8), obtaining translation of articles (7), and project management (6) [Table 2].

Of the nineteen respondents who would not consider conversion, only dedicated time emerged as a significant resource to lead them to reconsider (6 responses). Less helpful resources included assistance with updating (4), financial support (3), statistical support (3), obtaining articles (3), and learning the Cochrane system (2) [Table 2].

Only 14 per cent of respondents (4/24) would object to having someone else convert their review, but 61 per cent (14/23) would like to be consulted on the choice of author. Of the seventeen respondents who would consider converting their review to Cochrane format, only four had a review of either the same or a different topic published in the CDSR.

Discussion

The time taken to complete a Cochrane review, the need for specific training, difficulties navigating the publication process and the need to update the review, led some systematic reviewers to prepare and publish their reviews outside of the Cochrane Collaboration.

Although Cochrane reviews have been demonstrated, on average, to be of higher quality than those published elsewhere,[5,6] many of these outside reviews are of high quality. It would therefore benefit the Collaboration to address the perceived barriers to conducting a Cochrane review. It may be that the time to complete a Cochrane review is necessary to obtain validity; however, increasing the accessibility of training, addressing process difficulties, and developing strategies to assist with updating may encourage more authors to perform their reviews within the Collaboration.

Prestige, academic recognition and access to the reviews may be additional reasons for not undertaking Cochrane reviews. In countries where greater academic weight is attached to publications in traditional paper journals, the Cochrane Library has struggled to achieve equal recognition. This situation is changing however and several countries, including Australia and the United Kingdom, now bestow equal recognition on Cochrane and peer-reviewed paper publications. The situation with access is changing too. Cochrane reviews are now widely accessible by clinicians internationally and are indexed in MEDLINE. More recently, several countries have signed national provision agreements enabling all citizens to access Cochrane reviews.

Encouragingly, almost half of the authors responding to this survey would consider converting their review to Cochrane format. While four of the seventeen potential converters had a published Cochrane review, the remaining thirteen authors represent potential new (converted) Cochrane reviews. Extrapolating these findings, if half of all the authors of systematic reviews in DARE (approximately 2200 at the time of this survey) would consider converting their reviews to Cochrane format, and three-quarters of these represent new reviews, the Cochrane Collaboration could potentially achieve more than 700 additional reviews. This figure represents almost half the total number of current reviews in the Cochrane Database of Systematic Reviews. Issues of time, training, and support could facilitate this conversion, supporting both potential systematic review converters and encouraging those authors who at this stage are not considering converting their reviews.

Conclusions

Time required to complete a Cochrane review and the need for specific training are the primary reasons why some authors publish systematic reviews outside of the Cochrane Collaboration. Almost all of those primary authors who responded to the survey would be happy to have their review converted to Cochrane format by another author. Ways of supporting these authors and how to provide dedicated time to convert systematic reviews needs further consideration. Therefore, before further action in this area is undertaken, we must:

1. consider the role of the Cochrane Collaboration in facilitating these conversions, including provision for standardisation across Cochrane review groups;

2. conduct research comparing the time and resources necessary to convert an existing review with starting the review from scratch;

3. conduct research comparing the quality of converted systematic reviews with reviews begun from scratch; and

4. develop methodology to facilitate conversion of systematic reviews.

Based on the results of this survey, we propose development of a set of guidelines for the conversion of existing systematic reviews into Cochrane format, inviting some of this survey's respondents to pilot these guidelines. The resulting reviews will be compared for validity and resource requirements with reviews that have been published under current procedure. In addition, we believe that development of a methodology for assessing the need for updating a systematic review could potentially reduce the burden of updating on reviewers.

Competing interests

All authors were under the employ of the Australasian Cochrane Centre at the time of this survey.

Authors' contributions

The authors contributed equally to the design and analysis of this project.

Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1472-6963/3/2/prepub

Supplementary Material

Additional File 1

Australasian Cochrane Centre Conversion Needs Assessment Survey conversion survey.doc attached.